Evaluating externally generated Logs

Install the latest version of the Humanloop Python SDK

$
pip install humanloop

In a new Python script, import the Humanloop SDK and create an instance of the client

1
humanloop = Humanloop(
2
    api_key=YOUR_API_KEY, # Replace with your Humanloop API key
3
)

Retrieve the ID of the Humanloop project you are working in

You can find this in the Humanloop app.

1
PROJECT_ID = ... # Replace with the project ID

Retrieve the dataset you’re going to use for evaluation from the project

1
# Retrieve a dataset
2
DATASET_ID = ... # Replace with the dataset ID you are using for evaluation.
3
								 # This must be a dataset in the project you are working in.
4
datapoints = humanloop.datasets.list_datapoints(DATASET_ID).records

Set up the model config you are evaluating

If you constructed this in Humanloop, retrieve by calling:

1
config = humanloop.model_configs.get(id=CONFIG_ID)

Alternatively, if your model config lives outside the Humanloop system, you can post it to Humanloop with the register model config endpoint.

Either way, you need the ID of the config.

1
CONFIG_ID = <YOUR_CONFIG_ID>

In the Humanloop app, create an evaluator

For this guide, we’ll simply create a Valid JSON checker.

1. Visit the Evaluations tab, and select Evaluators
2. Click + New Evaluator and choose Code from the options.
3. Select the Valid JSON preset on the left.
4. Choose the mode Offline in the setting panel on the left.
5. Click Create.
6. Copy your new evaluator’s ID from the address bar. It starts with evfn_.

1
EVALUATOR_ID = <YOUR_EVALUATOR_ID>

Create an evaluation run with hl_generated set to False

This tells the Humanloop runtime that it should not trigger evaluations itself, but wait for them to be posted via the API.

1
evaluation_run = humanloop.evaluations.create(
2
    project_id=PROJECT_ID,
3
    config_id=CONFIG_ID,
4
    dataset_id=DATASET_ID,
5
    evaluator_ids=[EVALUATOR_ID],
6
    hl_generated=False,
7
)

By default, the status of the evaluation after creation is pending. Before sending the generation logs, set the status to running.

1
humanloop.evaluations.update_status(id=evaluation_run.id, status="running")

Iterate through the datapoints in the dataset, produce a generation, and post it the evaluation

1
for datapoint in datapoints:
2
		# Use the datapoint to produce a log with the model config you are testing.
3
    # This will depend on whatever model calling setup you are using on your side.
4
    # For simplicity, we simply log a hardcoded
5
    log = {
6
        "project_id": PROJECT_ID,
7
        "config_id": CONFIG_ID,
8
        "messages":  [*config.chat_template, *datapoint.messages],
9
        "output": "Hello World!",
10
    }
11
12
    print(f"Logging generation for datapoint {datapoint.id}")
13
    humanloop.evaluations.log(
14
        evaluation_id=evaluation_run.id,
15
        log=log,
16
        datapoint_id=datapoint.id,
17
    )

Run the full script above.

If everything goes well, you should now have posted a new evaluation run to Humanloop, and logged all the generations derived from the underlying datapoints.

The Humanloop evaluation runtime will now iterate through those logs and run the Valid JSON evaluator on each of them. To check progress:

Visit your project in the Humanloop app and go to the Evaluations tab.

You should see the run you recently created; click through to it and you’ll see rows in the table showing the generations.

In this case, all the evaluations returned False because the string “Hello World!” wasn’t valid JSON. Try logging something which is valid JSON to check that everything works as expected.