January 12, 2024

Faster offline evaluations

We’ve introduced batching to our offline Evaluations to significantly speed up runtime performance and also improved the robustness to things going wrong mid-run.

In addition to our recent enhancements to the Evaluations API, we’ve also made some significant improvements to our underlying orchestration framework which should mean your evaluation runs are now faster and more reliable. In particular, we now batch generations across the run - by default in groups of five, being conscious of potential rate limit errors (though this will soon be configurable).

Each batch runs its generations concurrently, so you should see much faster completion times - especially in runs across larger datasets.