January 19, 2024

Improved evaluation run launcher

We’ve made some usability enhancements to the launch experience when setting up batch generation & evaluation runs.

It’s now clearer which model configs, datasets and evaluators you’ve selected. It’s also now possible to specify whether you want the logs to be generated in the Humanloop runtime, or if you’re going to post the logs from your own infrastructure via the API.

Cancellable evaluation runs

Occasionally, you may launch an evaluation run and then realise that you didn’t configure it quite the way you wanted. Perhaps you want to use a different model config or dataset, or would like to halt its progress for some other reason.

We’ve now made evaluation runs cancellable from the UI - see the screenshot below. This is especially helpful if you’re running evaluations over large datasets, where you don’t want to unnecessarily consume provider credits.