For AI agents: a documentation index is available at the root level at /llms.txt and /llms-full.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page.
Sign inBook a demo
DocsReferenceChangelog
DocsReferenceChangelog
  • Introduction
    • Overview
    • Errors
  • SDK
    • Overview
    • Decorators
    • Run Evaluation
  • API
      • GETList Evaluations
      • POSTCreate Evaluation
      • POSTAdd Evaluators
      • DELRemove Evaluator
      • GETGet Evaluation
      • DELDelete Evaluation
      • GETList Runs for Evaluation
      • POSTCreate Run
      • POSTAdd Existing Run
      • DELRemove Run
      • PATCHUpdate Evaluation Run
      • POSTAdd Logs to Run
      • GETGet Evaluation Stats
      • GETGet Logs for Evaluation
LogoLogo
Sign inBook a demo
APIEvaluations

Create Run

POST
https://api.humanloop.com/v5/evaluations/:id/runs
POST
/v5/evaluations/:id/runs
$curl -X POST https://api.humanloop.com/v5/evaluations/id/runs \
>     -H "X-API-KEY: <apiKey>" \
>     -H "Content-Type: application/json" \
>     -d '{}'
1{
2  "id": "id",
3  "orchestrated": true,
4  "added_at": "2024-01-15T09:30:00Z",
5  "created_at": "2024-01-15T09:30:00Z",
6  "status": "pending",
7  "control": true,
8  "dataset": {
9    "path": "path",
10    "id": "id",
11    "name": "name",
12    "version_id": "version_id",
13    "created_at": "2024-01-15T09:30:00Z",
14    "updated_at": "2024-01-15T09:30:00Z",
15    "last_used_at": "2024-01-15T09:30:00Z",
16    "datapoints_count": 1,
17    "directory_id": "directory_id",
18    "description": "description",
19    "schema": {
20      "key": "value"
21    },
22    "readme": "readme",
23    "tags": [
24      "tags"
25    ],
26    "type": "dataset",
27    "environments": [
28      {
29        "id": "id",
30        "created_at": "2024-01-15T09:30:00Z",
31        "name": "name",
32        "tag": "default"
33      }
34    ],
35    "created_by": {
36      "id": "id",
37      "email_address": "email_address",
38      "full_name": "full_name"
39    },
40    "version_name": "version_name",
41    "version_description": "version_description",
42    "datapoints": [
43      {
44        "id": "id"
45      }
46    ],
47    "attributes": {
48      "key": "value"
49    }
50  },
51  "version": {
52    "path": "path",
53    "id": "id",
54    "directory_id": "directory_id",
55    "model": "model",
56    "endpoint": "complete",
57    "template": "template",
58    "template_language": "default",
59    "provider": "anthropic",
60    "max_tokens": 1,
61    "temperature": 1.1,
62    "top_p": 1.1,
63    "stop": "stop",
64    "presence_penalty": 1.1,
65    "frequency_penalty": 1.1,
66    "other": {
67      "key": "value"
68    },
69    "seed": 1,
70    "response_format": {
71      "type": "json_object",
72      "json_schema": {
73        "key": "value"
74      }
75    },
76    "reasoning_effort": "high",
77    "tools": [
78      {
79        "name": "name",
80        "description": "description"
81      }
82    ],
83    "linked_tools": [
84      {
85        "name": "name",
86        "description": "description",
87        "id": "id",
88        "version_id": "version_id"
89      }
90    ],
91    "attributes": {
92      "key": "value"
93    },
94    "version_name": "version_name",
95    "version_description": "version_description",
96    "description": "description",
97    "tags": [
98      "tags"
99    ],
100    "readme": "readme",
101    "name": "name",
102    "schema": {
103      "key": "value"
104    },
105    "version_id": "version_id",
106    "type": "prompt",
107    "environments": [
108      {
109        "id": "id",
110        "created_at": "2024-01-15T09:30:00Z",
111        "name": "name",
112        "tag": "default"
113      }
114    ],
115    "created_at": "2024-01-15T09:30:00Z",
116    "updated_at": "2024-01-15T09:30:00Z",
117    "created_by": {
118      "id": "id",
119      "email_address": "email_address",
120      "full_name": "full_name"
121    },
122    "last_used_at": "2024-01-15T09:30:00Z",
123    "version_logs_count": 1,
124    "total_logs_count": 1,
125    "inputs": [
126      {
127        "name": "name"
128      }
129    ],
130    "evaluator_aggregates": [
131      {
132        "value": 1.1,
133        "evaluator_id": "evaluator_id",
134        "evaluator_version_id": "evaluator_version_id",
135        "created_at": "2024-01-15T09:30:00Z",
136        "updated_at": "2024-01-15T09:30:00Z"
137      }
138    ],
139    "raw_file_content": "raw_file_content"
140  },
141  "created_by": {
142    "id": "id",
143    "email_address": "email_address",
144    "full_name": "full_name"
145  }
146}
Create an Evaluation Run. Optionally specify the Dataset and version to be evaluated. Humanloop will automatically start generating Logs and running Evaluators where `orchestrated=true`. If you are generating Logs yourself, you can set `orchestrated=false` and then generate and submit the required Logs via the API. If `dataset` and `version` are provided, you can set `use_existing_logs=True` to reuse existing Logs, avoiding generating new Logs unnecessarily. Logs that are associated with the specified Version and have `source_datapoint_id` referencing a datapoint in the specified Dataset will be associated with the Run. To keep updated on the progress of the Run, you can poll the Run using the `GET /evaluations/{id}/runs` endpoint and check its status.
Was this page helpful?
Previous

Add Existing Run

Next
Built with

Create an Evaluation Run.

Optionally specify the Dataset and version to be evaluated.

Humanloop will automatically start generating Logs and running Evaluators where orchestrated=true. If you are generating Logs yourself, you can set orchestrated=false and then generate and submit the required Logs via the API.

If dataset and version are provided, you can set use_existing_logs=True to reuse existing Logs, avoiding generating new Logs unnecessarily. Logs that are associated with the specified Version and have source_datapoint_id referencing a datapoint in the specified Dataset will be associated with the Run.

To keep updated on the progress of the Run, you can poll the Run using the GET /evaluations/{id}/runs endpoint and check its status.

Authentication

X-API-KEYstring
API Key authentication via header

Path parameters

idstringRequired
Unique identifier for Evaluation.

Request

This endpoint expects an object.
datasetobjectOptional
Dataset to use in this Run.
versionobjectOptional
Version to use in this Run.
orchestratedbooleanOptionalDefaults to true

Whether the Run is orchestrated by Humanloop. If True, Humanloop will generate Logs for the Run; dataset and version must be provided. If False, a log for the Prompt/Tool should be submitted by the user via the API.

use_existing_logsbooleanOptionalDefaults to false

If True, the Run will be initialized with existing Logs associated with the Dataset and Version. If False, the Run will be initialized with no Logs. Can only be set to True when both dataset and version are provided.

Response

Successful Response
idstring
Unique identifier for the Run.
orchestratedboolean
Whether the Run is orchestrated by Humanloop.
added_atdatetime
When the Run was added to the Evaluation.
created_atdatetime
When the Run was created.
statusenum
The status of the Run.
Allowed values:
controlboolean
Stats for other Runs will be displayed in comparison to the control Run.
datasetobject
The Dataset used in the Run.
versionobject
The version used in the Run.
created_byany
The User who created the Run.

Errors

422
Create Run Evaluations ID Runs Post Request Unprocessable Entity Error