For AI agents: a documentation index is available at the root level at /llms.txt and /llms-full.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page.
Sign inBook a demo
DocsReferenceChangelog
DocsReferenceChangelog
  • Introduction
    • SDKs
    • Errors
  • Humanloop API
      • GETGet
      • GETList Datapoints
      • POSTCreate
      • POSTLog
      • POSTResult
      • PATCHUpdate Status
      • PATCHAdd Evaluators
      • GETGet Evaluations
LogoLogo
Sign inBook a demo
Humanloop APIEvaluations

Create

POST
https://api.humanloop.com/v4/projects/:project_id/evaluations
POST
/v4/projects/:project_id/evaluations
$curl -X POST https://api.humanloop.com/v4/projects/project_id/evaluations \
>     -H "X-API-KEY: <apiKey>" \
>     -H "Content-Type: application/json" \
>     -d '{
>  "config_id": "config_id",
>  "evaluator_ids": [
>    "evaluator_ids"
>  ],
>  "dataset_id": "dataset_id"
>}'
1{
2  "id": "id",
3  "status": "pending",
4  "config": {
5    "type": "model",
6    "id": "id",
7    "model": "model",
8    "chat_template": [
9      {
10        "role": "user"
11      }
12    ],
13    "description": "description",
14    "endpoint": "complete",
15    "frequency_penalty": 1.1,
16    "max_tokens": 1,
17    "name": "name",
18    "other": {
19      "key": "value"
20    },
21    "presence_penalty": 1.1,
22    "prompt_template": "prompt_template",
23    "provider": "anthropic",
24    "reasoning_effort": "high",
25    "response_format": {
26      "type": "json_object",
27      "json_schema": {
28        "key": "value"
29      }
30    },
31    "seed": 1,
32    "stop": "stop",
33    "temperature": 1.1,
34    "template_language": "default",
35    "tools": [
36      {
37        "id": "id",
38        "name": "name"
39      }
40    ],
41    "top_p": 1.1,
42    "tool_configs": [
43      {
44        "id": "id",
45        "status": "status",
46        "name": "name"
47      }
48    ]
49  },
50  "created_at": "2024-01-15T09:30:00Z",
51  "updated_at": "2024-01-15T09:30:00Z",
52  "evaluators": [
53    {
54      "name": "name",
55      "description": "description",
56      "arguments_type": "target_free",
57      "return_type": "boolean",
58      "type": "python",
59      "id": "id",
60      "created_at": "2024-01-15T09:30:00Z",
61      "updated_at": "2024-01-15T09:30:00Z",
62      "code": "code",
63      "model_config": {
64        "id": "id",
65        "model": "model"
66      },
67      "logging_project": {
68        "id": "id",
69        "name": "name",
70        "users": [
71          {
72            "id": "id",
73            "email_address": "email_address"
74          }
75        ],
76        "data_count": 1,
77        "feedback_types": [
78          {
79            "type": "rating"
80          }
81        ],
82        "team_id": "team_id",
83        "created_at": "2024-01-15T09:30:00Z",
84        "updated_at": "2024-01-15T09:30:00Z"
85      }
86    }
87  ],
88  "dataset": {
89    "id": "id",
90    "name": "name",
91    "datapoint_count": 1,
92    "created_at": "2024-01-15T09:30:00Z",
93    "updated_at": "2024-01-15T09:30:00Z",
94    "project_id": "project_id",
95    "description": "description"
96  },
97  "dataset_version_id": "dataset_version_id",
98  "dataset_snapshot": {
99    "id": "id",
100    "name": "name",
101    "datapoint_count": 1,
102    "created_at": "2024-01-15T09:30:00Z",
103    "updated_at": "2024-01-15T09:30:00Z",
104    "project_id": "project_id",
105    "description": "description"
106  },
107  "evaluator_aggregates": [
108    {
109      "model_config_id": "model_config_id",
110      "evaluator_id": "evaluator_id",
111      "evaluator_version_id": "evaluator_version_id",
112      "aggregate_value": 1.1
113    }
114  ]
115}
Create an evaluation.
Was this page helpful?
Previous

Log

Next
Built with

Authentication

X-API-KEYstring
API Key authentication via header

Path parameters

project_idstringRequired

String ID of project. Starts with pr_.

Request

This endpoint expects an object.
config_idstringRequired

ID of the config to evaluate. Starts with config_.

evaluator_idslist of stringsRequired

IDs of evaluators to run on the dataset. IDs start with evfn_

dataset_idstringRequired

ID of the dataset to use in this evaluation. Starts with evts_.

provider_api_keysobjectOptional
API keys required by each provider to make API calls. The API keys provided here are not stored by Humanloop. If not specified here, Humanloop will fall back to the key saved to your organization. Ensure you provide an API key for the provider for the model config you are evaluating, or have one saved to your organization.
hl_generatedbooleanOptionalDefaults to true

Whether the log generations for this evaluation should be performed by Humanloop. If False, the log generations should be submitted by the user via the API.

namestringOptional
Name of the Evaluation to help identify it.

Response

Successful Response
idstring

Unique ID for the evaluation. Starts with ev_.

statusenum
Status of an evaluation.
Allowed values:
configobject
created_atdatetime
updated_atdatetime
evaluatorslist of objects
datasetobject
dataset_version_idstring
dataset_snapshotobject
evaluator_aggregateslist of objects

Errors

422
Evaluations Create Request Unprocessable Entity Error