APIEvaluations

Create Run

Create an Evaluation Run.

Optionally specify the Dataset and version to be evaluated.

Humanloop will automatically start generating Logs and running Evaluators where orchestrated=true. If you are generating Logs yourself, you can set orchestrated=false and then generate and submit the required Logs via the API.

If dataset and version are provided, you can set use_existing_logs=True to reuse existing Logs, avoiding generating new Logs unnecessarily. Logs that are associated with the specified Version and have source_datapoint_id referencing a datapoint in the specified Dataset will be associated with the Run.

To keep updated on the progress of the Run, you can poll the Run using the GET /evaluations/{id}/runs endpoint and check its status.

Path parameters

idstringRequired
Unique identifier for Evaluation.

Headers

X-API-KEYstringRequired

Request

This endpoint expects an object.
datasetobjectOptional
Dataset to use in this Run.
versionobjectOptional
Version to use in this Run.
orchestratedbooleanOptionalDefaults to true

Whether the Run is orchestrated by Humanloop. If True, Humanloop will generate Logs for the Run; dataset and version must be provided. If False, a log for the Prompt/Tool should be submitted by the user via the API.

use_existing_logsbooleanOptionalDefaults to false

If True, the Run will be initialized with existing Logs associated with the Dataset and Version. If False, the Run will be initialized with no Logs. Can only be set to True when both dataset and version are provided.

Response

Successful Response
idstring
Unique identifier for the Run.
orchestratedboolean
Whether the Run is orchestrated by Humanloop.
added_atdatetime
When the Run was added to the Evaluation.
created_atdatetime
When the Run was created.
statusenum
The status of the Run.
Allowed values:
controlboolean
Stats for other Runs will be displayed in comparison to the control Run.
datasetobject or null
The Dataset used in the Run.
versionobject or null
The version used in the Run.
created_byany or null
The User who created the Run.

Errors