Create Run

1 from humanloop import Humanloop 2 client = Humanloop(api_key="YOUR_API_KEY", ) 3 client.evaluations.create_run(id='id', )

1 { 2 "id": "id", 3 "orchestrated": true, 4 "added_at": "2024-01-15T09:30:00Z", 5 "created_at": "2024-01-15T09:30:00Z", 6 "status": "pending", 7 "control": true, 8 "dataset": { 9 "path": "path", 10 "id": "id", 11 "name": "name", 12 "version_id": "version_id", 13 "created_at": "2024-01-15T09:30:00Z", 14 "updated_at": "2024-01-15T09:30:00Z", 15 "last_used_at": "2024-01-15T09:30:00Z", 16 "datapoints_count": 1, 17 "directory_id": "directory_id", 18 "description": "description", 19 "schema": { 20 "key": "value" 21 }, 22 "readme": "readme", 23 "tags": [ 24 "tags" 25 ], 26 "type": "dataset", 27 "environments": [ 28 { 29 "id": "id", 30 "created_at": "2024-01-15T09:30:00Z", 31 "name": "name", 32 "tag": "default" 33 } 34 ], 35 "created_by": { 36 "id": "id", 37 "email_address": "email_address", 38 "full_name": "full_name" 39 }, 40 "version_name": "version_name", 41 "version_description": "version_description", 42 "datapoints": [ 43 { 44 "id": "id" 45 } 46 ], 47 "attributes": { 48 "key": "value" 49 } 50 }, 51 "version": { 52 "path": "path", 53 "id": "id", 54 "directory_id": "directory_id", 55 "model": "model", 56 "endpoint": "complete", 57 "template": "template", 58 "template_language": "default", 59 "provider": "anthropic", 60 "max_tokens": 1, 61 "temperature": 1.1, 62 "top_p": 1.1, 63 "stop": "stop", 64 "presence_penalty": 1.1, 65 "frequency_penalty": 1.1, 66 "other": { 67 "key": "value" 68 }, 69 "seed": 1, 70 "response_format": { 71 "type": "json_object", 72 "json_schema": { 73 "key": "value" 74 } 75 }, 76 "reasoning_effort": "high", 77 "tools": [ 78 { 79 "name": "name", 80 "description": "description" 81 } 82 ], 83 "linked_tools": [ 84 { 85 "name": "name", 86 "description": "description", 87 "id": "id", 88 "version_id": "version_id" 89 } 90 ], 91 "attributes": { 92 "key": "value" 93 }, 94 "version_name": "version_name", 95 "version_description": "version_description", 96 "description": "description", 97 "tags": [ 98 "tags" 99 ], 100 "readme": "readme", 101 "name": "name", 102 "schema": { 103 "key": "value" 104 }, 105 "version_id": "version_id", 106 "type": "prompt", 107 "environments": [ 108 { 109 "id": "id", 110 "created_at": "2024-01-15T09:30:00Z", 111 "name": "name", 112 "tag": "default" 113 } 114 ], 115 "created_at": "2024-01-15T09:30:00Z", 116 "updated_at": "2024-01-15T09:30:00Z", 117 "created_by": { 118 "id": "id", 119 "email_address": "email_address", 120 "full_name": "full_name" 121 }, 122 "last_used_at": "2024-01-15T09:30:00Z", 123 "version_logs_count": 1, 124 "total_logs_count": 1, 125 "inputs": [ 126 { 127 "name": "name" 128 } 129 ], 130 "evaluator_aggregates": [ 131 { 132 "value": 1.1, 133 "evaluator_id": "evaluator_id", 134 "evaluator_version_id": "evaluator_version_id", 135 "created_at": "2024-01-15T09:30:00Z", 136 "updated_at": "2024-01-15T09:30:00Z" 137 } 138 ], 139 "raw_file_content": "raw_file_content" 140 }, 141 "created_by": { 142 "id": "id", 143 "email_address": "email_address", 144 "full_name": "full_name" 145 } 146 }

Create an Evaluation Run.

Optionally specify the Dataset and version to be evaluated.

Humanloop will automatically start generating Logs and running Evaluators where orchestrated=true. If you are generating Logs yourself, you can set orchestrated=false and then generate and submit the required Logs via the API.

If dataset and version are provided, you can set use_existing_logs=True to reuse existing Logs, avoiding generating new Logs unnecessarily. Logs that are associated with the specified Version and have source_datapoint_id referencing a datapoint in the specified Dataset will be associated with the Run.

To keep updated on the progress of the Run, you can poll the Run using the GET /evaluations/{id}/runs endpoint and check its status.

Create an Evaluation Run. Optionally specify the Dataset and version to be evaluated. Humanloop will automatically start generating Logs and running Evaluators where `orchestrated=true`. If you are generating Logs yourself, you can set `orchestrated=false` and then generate and submit the required Logs via the API. If `dataset` and `version` are provided, you can set `use_existing_logs=True` to reuse existing Logs, avoiding generating new Logs unnecessarily. Logs that are associated with the specified Version and have `source_datapoint_id` referencing a datapoint in the specified Dataset will be associated with the Run. To keep updated on the progress of the Run, you can poll the Run using the `GET /evaluations/{id}/runs` endpoint and check its status.

Path parameters

idstringRequired

Unique identifier for Evaluation.

Request

This endpoint expects an object.

datasetobjectOptional

Dataset to use in this Run.

versionobjectOptional

Version to use in this Run.

orchestratedbooleanOptionalDefaults to true

Whether the Run is orchestrated by Humanloop. If True, Humanloop will generate Logs for the Run; dataset and version must be provided. If False, a log for the Prompt/Tool should be submitted by the user via the API.

use_existing_logsbooleanOptionalDefaults to false

If True, the Run will be initialized with existing Logs associated with the Dataset and Version. If False, the Run will be initialized with no Logs. Can only be set to True when both dataset and version are provided.

Response

Successful Response

idstring

Unique identifier for the Run.

orchestratedboolean

Whether the Run is orchestrated by Humanloop.

added_atdatetime

When the Run was added to the Evaluation.

created_atdatetime

When the Run was created.

statusenum

The status of the Run.

Allowed values:

controlboolean

Stats for other Runs will be displayed in comparison to the control Run.

datasetobject or null

The Dataset used in the Run.

versionobject or null

The version used in the Run.

created_byany or null

The User who created the Run.

Path parameters

Headers

Request

Response

Errors