Changelog

Evaluation Names

You can now name your Evaluations in the UI and via the API. This is helpful for more easily identifying the purpose of your different Evaluations, especially when multiple teams are running different experiments.

In the API, pass in the name field when creating your Evaluation to set the name. Note that names must be unique for all Evaluations for a specific file. In the UI, navigate to your Evaluation and you will see an option to rename it in the header.

Introducing Flows

We’ve added a new key building block to our app with the first release of Flows. Flows allow you to collect multiple Logs into a Trace and evaluate them together, improving observability across your apps.

This release focuses on improving the code-first workflows for evaluating more complex AI applications like RAG and Agent-based apps.

You can start logging your app traces with the Flows log endpoint. (Check back later for a more in-depth tutorial we’ll be releasing shortly.)

What’s next

We’ll soon be extending support for evaluating Traces, allowing Evaluators to access all Logs inside a Trace. Additionally, we will build on this by adding UI-first visualisations and management of your Flows.

We’ll sunset Sessions in favour of Flows in the near future. Reach out to us for guidance on how to migrate your Session-based workflows to Flows.

Update Logs API

We’ve introduced the ability to patch Logs for Prompts and Tools. This can come in useful in scenarios where certain characteristics of your Log are delayed that you may want to add later, such as the output, or if you have a process of redacting inputs that takes time.

Note that not all fields support being patched, so start by referring to our V5 API References. From there, you can submit updates to your previously created logs.

Search files by path

We’ve extended our search interface to include file paths, allowing you to more easily find and navigate to related files that you’ve grouped under a directory.

Bring up this search dialog by clicking “Search” near the top of the left-hand sidebar, or by pressing Cmd+K.

Updated Gemini 1.5 models

Humanloop supports the three newly released Gemini 1.5 models.

Start using these improved models by specifying one of the following model names in your Prompts:

• gemini-1.5-pro-exp-0827 The improved Gemini 1.5 Pro model
• gemini-1.5-flash-exp-0827 The improved Gemini 1.5 Flash model
• gemini-1.5-flash-8b-exp-0827 The smaller Gemini 1.5 Flash variant

More details on these models can be viewed here.

Custom attributes for Files

You can now include custom attributes to determine the unique version of your file definitions on Humanloop.

This allows you to make the version depend on data custom to your application that Humanloop may not be aware of.

For example, if there are feature flags or identifiers that indicate a different configuration of your system that may impact the behaviour of your Prompt or Tool.

attributes can be submitted via the v5 API endpoints. When added, the attributes are visible on the Version Drawer and in the Editor.

Improved popover UI

We’ve expanded the information shown in the version popover so that it is easier to identify which version you are working with.

This is particularly useful in places like the Logs table and within Evaluation reports, where you may be working with multiple versions of a Prompt, Tool, or Evaluator and need to preview the contents.

Evaluate uncommitted versions

You can now evaluate versions without committing them first. This means you can draft a version of a Prompt in the editor and simultaneously evaluate it in the evaluations tab, speeding up your iteration cycle.

This is a global change that allows you to load and use uncommitted versions. Uncommitted versions are created automatically when a new version of a Prompt, Tool, or Evaluator is run in their respective editors or called via the API. These versions will now appear in the version pickers underneath all your committed versions.

To evaluate an uncommitted version, simply select it by using the hash (known as the “version id”) when setting up your evaluation.

Human Evaluator upgrades

We’ve made significant upgrades to Human Evaluators and related workflows to improve your ability to gather Human judgments (sometimes referred to as “feedback”) in assessing the quality of your AI applications.

Here are some of the key improvements:

• Instead of having to define a limited feedback schema tied to the settings of a specific Prompt, you can now define your schema with a Human Evaluator file and reuse it across multiple Prompts and Tools for both monitoring and offline evaluation purposes.
• You are no longer restricted to the default types of Rating, Actions and Issues when defining your feedback schemas from the UI. We’ve introduced a more flexible Editor interface supporting different return types and valence controls.
• We’ve extended the scope of Human Evaluators so that they can now also be used with Tools and other Evaluators (useful for validating AI judgments) in the same way as with Prompts.
• We’ve improved the Logs drawer UI for applying feedback to Logs. In particular, we’ve made the buttons more responsive.

To set up a Human Evaluator, create a new file. Within the file creation dialog, click on Evaluator, then click on Human. This will create a new Human Evaluator file and bring you to its Editor. Here, you can choose a Return type for the Evaluator and configure the feedback schema.

You can then reference this Human Evaluator within the Monitoring dropdown of Prompts, Tools, and other Evaluators, as well as when configuring reports in their Evaluations tab.

We’ve set up default Rating and Correction Evaluators that will be automatically attached to all Prompts new and existing. We’ve migrated all your existing Prompt specific feedback schemas to Human Evaluator files and these will continue to work as before with no disruption.

Check out our updated document for further details on how to use Human Evaluators:

• Create a Human Evaluator
• Capture End User Feedback
• Run a Human Evaluation

Evaluations improvements

We’ve made improvements to help you evaluate the components of your AI applications, quickly see issues and explore the full context of each evaluation.

A clearer Evaluation tab in Logs

We’ve given the Log drawer’s Evaluation tab a facelift. You can now clearly see what the results are for each of the connected Evaluators.

This means that it’s now easier to debug the judgments applied to a Log, and if necessary, re-run code/AI Evaluators in-line.

Ability to re-run Evaluators

We have introduced the ability to re-run your Evaluators against a specific Log. This feature allows you to more easily address and fix issues with previous Evaluator judgments for specific Logs.

You can request a re-run of that Evaluator by opening the menu next to that Evaluator and pressing the “Run Again” option.

Evaluation popover

If you hover over an evaluation result, you’ll now see a popover with more details about the evaluation including any intermediate results or console logs without context switching.

Updated Evaluator Logs table

The Logs table for Evaluators now supports the functionality as you would expect from our other Logs tables. This will make it easier to filter and sort your Evaluator judgments.