Humanloop Documentation and API Reference

Create an evaluator within your organization.

Request

This endpoint expects an object.

namestringRequired=1 character

The name of the evaluator.

descriptionstringRequired

The description of the evaluator.

arguments_type"target_free" or "target_required"Required

Allowed values: target_freetarget_required

Whether this evaluator is target-free or target-required.

return_typeenumRequired

Allowed values: booleannumberselectmulti_selecttext

The type of the return value of the evaluator.

typeenumRequired

Allowed values: pythonllmhumanexternal

The type of the evaluator.

codestringOptional

The code for the evaluator. This code will be executed in a sandboxed environment.

model_configobjectOptional

The model configuration used to generate.

Response

Successful Response

namestring=1 character

The name of the evaluator.

descriptionstring

The description of the evaluator.

arguments_type"target_free" or "target_required"

Allowed values: target_freetarget_required

Whether this evaluator is target-free or target-required.

return_typeenum

Allowed values: booleannumberselectmulti_selecttext

The type of the return value of the evaluator.

typeenum

Allowed values: pythonllmhumanexternal

The type of the evaluator.

idstring

Unique ID for the evaluator. Starts with evfn_.

created_atdatetime

updated_atdatetime

codestringOptional

The code for the evaluator. This code will be executed in a sandboxed environment.

model_configobjectOptional

The model config defining the LLM evaluator.

logging_projectobjectOptional

The project where the evaluator logs are stored.

Errors

1 curl -X POST https://api.humanloop.com/v4/evaluators \ 2 -H "X-API-KEY: <apiKey>" \ 3 -H "Content-Type: application/json" \ 4 -d '{ 5 "name": "name", 6 "description": "description", 7 "arguments_type": "target_free", 8 "return_type": "boolean", 9 "type": "python" 10 }'

1 { 2 "name": "name", 3 "description": "description", 4 "arguments_type": "target_free", 5 "return_type": "boolean", 6 "type": "python", 7 "id": "id", 8 "created_at": "2024-01-15T09:30:00Z", 9 "updated_at": "2024-01-15T09:30:00Z", 10 "code": "code", 11 "model_config": { 12 "id": "id", 13 "model": "model", 14 "other": { 15 "key": "value" 16 }, 17 "name": "name", 18 "description": "description", 19 "provider": "openai", 20 "max_tokens": 1, 21 "temperature": 1.1, 22 "top_p": 1.1, 23 "stop": "stop", 24 "presence_penalty": 1.1, 25 "frequency_penalty": 1.1, 26 "seed": 1, 27 "response_format": { 28 "type": "json_object", 29 "json_schema": { 30 "key": "value" 31 } 32 }, 33 "prompt_template": "prompt_template", 34 "chat_template": [ 35 { 36 "role": "user" 37 } 38 ], 39 "tools": [ 40 { 41 "id": "id", 42 "name": "name" 43 } 44 ], 45 "endpoint": "complete", 46 "tool_configs": [ 47 { 48 "id": "id", 49 "status": "status", 50 "name": "name" 51 } 52 ] 53 }, 54 "logging_project": { 55 "id": "id", 56 "name": "name", 57 "users": [ 58 { 59 "id": "id", 60 "email_address": "email_address" 61 } 62 ], 63 "data_count": 1, 64 "feedback_types": [ 65 { 66 "type": "rating" 67 } 68 ], 69 "team_id": "team_id", 70 "created_at": "2024-01-15T09:30:00Z", 71 "updated_at": "2024-01-15T09:30:00Z", 72 "active_config": { 73 "project_id": "project_id", 74 "project_name": "project_name", 75 "created_at": "2024-01-15T09:30:00Z", 76 "updated_at": "2024-01-15T09:30:00Z", 77 "last_used": "2024-01-15T09:30:00Z", 78 "config": { 79 "type": "generic", 80 "id": "config", 81 "name": "config", 82 "status": "config" 83 }, 84 "num_datapoints": 1, 85 "evaluation_aggregates": [ 86 { 87 "model_config_id": "model_config_id", 88 "evaluator_id": "evaluator_id", 89 "evaluator_version_id": "evaluator_version_id" 90 } 91 ] 92 }, 93 "config_type": "generic", 94 "directory_id": "directory_id" 95 } 96 }

1	curl -X POST https://api.humanloop.com/v4/evaluators \
2	-H "X-API-KEY: <apiKey>" \
3	-H "Content-Type: application/json" \
4	-d '{
5	"name": "name",
6	"description": "description",
7	"arguments_type": "target_free",
8	"return_type": "boolean",
9	"type": "python"
10	}'