Evaluators

Create

POST
Create an evaluator within your organization.

Request

This endpoint expects an object.
name
stringRequired
The name of the evaluator.
description
stringRequired
The description of the evaluator.
arguments_type
enumRequired
Whether this evaluator is target-free or target-required.
Allowed values: target_freetarget_required
return_type
enumRequired
The type of the return value of the evaluator.
Allowed values: booleannumber
type
enumRequired
The type of the evaluator.
Allowed values: pythonllmhumanexternal
code
stringOptional
The code for the evaluator. This code will be executed in a sandboxed environment.
model_config
objectOptional
The model configuration used to generate.

Response

This endpoint returns an object
name
string
The name of the evaluator.
description
string
The description of the evaluator.
arguments_type
enum
Whether this evaluator is target-free or target-required.
Allowed values: target_freetarget_required
return_type
enum
The type of the return value of the evaluator.
Allowed values: booleannumber
type
enum
The type of the evaluator.
Allowed values: pythonllmhumanexternal
id
string

Unique ID for the evaluator. Starts with evfn_.

created_at
datetime
updated_at
datetime
code
stringOptional
The code for the evaluator. This code will be executed in a sandboxed environment.
model_config
objectOptional
The model config defining the LLM evaluator.
logging_project
objectOptional
The project where the evaluator logs are stored.

Errors