September 15, 2023

Evaluations improvements

We’ve released a couple of minor useability improvements in the evaluations workflow.

Summary statistics for evaluation runs

When reviewing past runs of evaluations, you can now see summary statistics for each evaluator before clicking into the detail view, allowing for easier comparison between runs.

Re-running evaluations

To enable easier re-running of past evaluations, you can now click the Re-run button in the top-right of the evaluation detail view.

Editor - copy tools

Our Editor environment let’s users incorporate OpenAI function calling into their prompt engineering workflows by defining tools. Tools are made available to the model as functions to call using the same universal JSON schema format.

As part of this process it can be helpful to copy the full JSON definition of the tool for quickly iterating on new versions, or copy and pasting it into code. You can now do this directly from the tool definition in Editor:

Selecting the Copy button adds the full JSON definition of the tool to your clipboard:

1
{
2
  "name": "get_current_weather",
3
  "description": "Get the current weather in a given location",
4
  "parameters": {
5
    "type": "object",
6
    "properties": {
7
      "location": {
8
        "type": "string",
9
        "name": "Location",
10
        "description": "The city and state, e.g. San Francisco, CA"
11
      },
12
      "unit": {
13
        "type": "string",
14
        "name": "Unit",
15
        "enum": [
16
          "celsius",
17
          "fahrenheit"
18
        ]
19
      }
20
    },
21
    "required": [
22
      "location"
23
    ]
24
  }
25
}

Single sign on (SSO)

We’ve added support for SOO to our signup, login and invite flows. By default users can now use their Gmail accounts to access Humanloop.

For our enterprise customers, this also unlocks the ability for us to more easily support their SAML-based single sign-on (SSO) set ups.