The most successful AI teams focus on two best practices:
**Evals-driven development**
## View the results
Navigate to the URL provided in your terminal to see the result of running your script on Humanloop.
This `Stats` view will show you the live progress of your local eval runs as well summary statistics of the final results.
Each new run will add a column to your `Stats` view, allowing you to compare the performance of your LLM app over time.
The `Logs` and `Review` tabs allow you to drill into individual datapoints and view the outputs of different runs side-by-side to understand how to improve your LLM app.
## Make a change and re-run
Your first run resulted in a `Semantic similarity` score of 3 (out of 5) and an `Exact match` score of 0. Try and make a change to your `callable` to improve
the output and re-run your script. A second run will be added to your `Stats` view and the difference in performance will be displayed.
## Next steps
Now that you've run your first eval on Humanloop, you can:
* Explore our detailed [tutorial](/docs/tutorials/rag-evaluation) on evaluating a real RAG app where you'll learn about versioning your app, customizing logging, adding Evaluator thresholds and more.
* Create your own [Dataset](/docs/explanation/datasets) of test cases to evaluate your LLM app against.
* Create your own [Evaluators](/docs/explanation/evaluators) to provide judgments on the output of your LLM app.
***
subtitle: Create and evaluate your Agent by running Evals in the Humanloop UI.
description: Getting up and running with Humanloop is quick and easy. This guide will explain how to set up evaluations through the Humanloop UI.
---------------------
For clean Markdown of any page, append .md to the page URL. For a complete documentation index, see https://humanloop.com/docs/llms.txt. For full documentation content, see https://humanloop.com/docs/llms-full.txt.
For this quickstart, we're going to evaluate **Outreach Agent**, which is designed to compose personalized outbound messages to potential customers. The Agent uses a [Tool](/docs/explanation/tools) to research information about the lead before writing a message.
We will assess the quality of the Agent using a [Dataset](https://humanloop.com/docs/evaluation/guides/create-dataset) and [Evaluators](/docs/explanation/evaluators).
This will create an **Outreach Agent** folder in your workspace. Inside the folder, you'll find:
* The **Outreach Agent**.
* **Hacker News Search** Tool used by the Agent to research potential customers.
* A **Customer Outreach** Dataset and three **Evaluators** that we will use to assess the Agent.
### Try out the Agent
The **Outreach Agent** looks up information about the lead on Hacker News and composes an outbound message to them.
Before we kick off the first evaluation, run the Agent in the Editor to get a feel for how it works:
1. Click on the **Outreach Agent** file.
2. On the left-hand side, you can configure your Agent's Parameters, Instructions, and add Tools.
3. On the right-hand side, you can specify Inputs and run the Agent on demand.
4. Enter *Coca-Cola* as the organization and *Humanloop* as the lead in the Inputs section in the top right-hand side.
5. Click the Run button.
### Run Eval
Evaluations are an efficient way to improve your Agent iteratively. You can test versions of the Agent against a [Dataset](https://humanloop.com/docs/evaluation/guides/create-dataset) and see how changing the Agent's configuration impacts the performance.
To test the **Outreach Agent**, navigate to the **Evals** tab and click on the **+ Evaluation** button.
Create a new Run by clicking on the **+ Run** button. Then, follow these steps:
1. Click on the **Dataset** button and select **Customer Outreach** Dateset.
2. Click on the **Agent** button and select the "v1" version.
3. Click on **Evaluators** and add the three Evaluators included in the Output Agent folder: **Friendly Tone**, **Tool Call** and **Message Length**
The first two Evaluators will check if the message is friendly and if the Tool was used.
The **Message Length** Evaluator will show the number of words in the output, providing a baseline value for all further evaluations.
Click **Save**. Humanloop will start generating Logs for the Evaluation.
### Review results
After the Run is completed, you can review the Logs produced and corresponding judgments in the **Review** tab.
The summary of all Evaluators is displayed in the **Stats** tab.
In the sidebar, rename this file to "Comedian Bot" now or later.
### Create the Prompt template in the Editor
The left hand side of the screen defines your Prompt – the parameters such as model, temperature and template. The right hand side is a single chat session with this Prompt.
In the editor, update the system message to:
```
You are a funny comedian. Write a joke about {{topic}}.
```
This message forms the chat template. It has an input slot called `topic` (surrounded by two curly brackets) for an input value that is provided each time you call this Prompt.
On the right hand side of the page, you’ll now see a box in the **Inputs** section for `topic`.
1. Add a value for `topic` e.g. music, jogging, whatever
2. Click **Run** in the bottom right of the page
This will call OpenAI’s model and return the assistant response. Feel free to try other values, the model is *very* funny.
You now have a first version of your prompt that you can use.
### Save your first version of this Prompt
1. Click the **Save** button at the top of the editor
2. Enter **Initial Comedian Setup** in the version name field
3. Enter **Prompt for making (not so) funny jokes!** in the description field
4. Click **Save**
### View the logs
Under the Prompt File, click ‘Logs’ to view all the generations from this Prompt
Click on a row to see the details of what version of the Prompt generated it. From here you can give feedback to that generation, see performance metrics, open up this example in the Editor, or add this log to a Dataset.
## Change the agent and rerun
Modify the `call_model` function to use a different model and temperature.
## Next steps
Logging is the first step to observing your AI product. Follow up with these guides on monitoring and evals:
* Add [monitoring Evaluators](/docs/v5/guides/observability/monitoring) to evaluate Logs as they're made against a File.
* Explore evals to improve the performance of your AI feature in our [guide on running an Evaluation](/docs/v5/guides/evals/run-evaluation).
* See logging in action on a complex example in our [tutorial on evaluating an agent](/docs/v5/tutorials/agent-evaluation).
***
subtitle: Add Humanloop observability to your Vercel AI SDK project.
description: Add logging to your Vercel AI SDK application with Humanloop.
---------------------
For clean Markdown of any page, append .md to the page URL. For a complete documentation index, see https://humanloop.com/docs/llms.txt. For full documentation content, see https://humanloop.com/docs/llms-full.txt.
The [Vercel AI SDK](https://sdk.vercel.ai/docs/introduction) is the TypeScript toolkit designed to help developers build AI-powered applications with React, Next.js, Vue, Svelte, Node.js, and more.
Humanloop supports receiving OpenTelemetry traces from the Vercel AI SDK, so you can see and evaluate your AI SDK application in your Humanloop Dashboard.
Take a look at the following guides to learn how to integrate Humanloop with your Vercel AI SDK application.
Files represent the core building blocks of your AI features on Humanloop. They exist within a flexible filesystem in an Organization.
### Function Files
[**Prompts**](./prompts)
A Prompt on Humanloop defines the instructions and configuration for guiding a Large Language Model (LLM) to perform a specific task.
Each change in any of the following properties creates a new Version of the Prompt:
* the **template** such as `Write a song about {{topic}}`.
Then, use the Agent Editor to configure and test the Agent. You can link existing Tools and Prompts from your workspace for the Agent to use. You can also define a tool inline by providing its JSON schema. You can also configure the `max_iterations` and `stopping_tools` for the Agent in the "Parameters" section.
### Agents via the SDK
To create an Agent programmatically, use our SDK.
```curl
curl -X POST https://api.humanloop.com/v5/agents \
-H "X-API-Key: $YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"path": "Banking/Teller Agent",
"provider": "anthropic",
"endpoint": "chat",
"model": "claude-3-7-sonnet-latest",
"reasoning_effort": 1024,
"template": [
{
"role": "system",
"content": "You are a helpful a digital bank teller, you help users navigate our digital banking platform."
},
],
"max_iterations": 3,
"tools": [
{
"type": "file",
"link": {
"file_id": "pr_1234567890",
"version_id": "prv_1234567890"
},
"on_agent_call": "continue"
},
{
"type": "json_schema",
"json_schema": {
"name": "stop",
"description": "Call this tool when you have finished your task.",
"parameters": {
"type": "object",
"properties": {
"output": {
"type": "string",
}
}
}
}
},
]
}'
```
Agents can be called through our UI for testing and development, or via API for production deployments.
```curl
curl -X POST https://api.humanloop.com/v5/agents/call \
-H "X-API-Key: $YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"path": "Banking/Teller Agent",
"messages": [
{
"role": "user",
"content": "I need to withdraw $1000"
}
]
}'
```
If your Agent calls a tool that Humanloop cannot run, the system will pause and wait until you provide the response required by the tool. You can then continue the Agent's execution using the `/continue` endpoint.
```curl
curl -X POST https://api.humanloop.com/v5/agents/continue \
-H "X-API-Key: $YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"log_id": "log_1234567890",
"messages": [
{
"role": "tool",
"tool_call_id": "tool_call_1234567890",
"content": "Not enough funds. Politely decline user request."
}
]
"stream": false
}'
```
You can also log Agent executions explicitly when running the Agent logic yourself:
```curl
curl -X POST https://api.humanloop.com/v5/agents/log \
-H "X-API-Key: $YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"path": "Banking/Teller Agent",
"messages": [
{
"role": "user",
"content": "I need to withdraw $1000"
},
]
}'
```
## Versioning
Versioning your Agents enables you to track how adjustments to template, tools, or model parameters influence the Agent's behavior. This is crucial for iterative development, as you can pinpoint which configuration produces the most effective results for your use cases. You can use the Editor or SDK to create and update Agent Versions and iterate on improving its performance.
The following components of an Agent contribute to its version:
* The Agent's **template** -- the system prompt and any other instructions
* The model call **parameters** -- such as `temperature`, `max_tokens`, and `reasoning_effort`
* The Agent's **workflow** configuration -- such as `max_iterations` and `stopping_tools`
* The Agent's **tools** -- including both inline JSON Schema tools and linked Tool or Prompt Versions
## Stopping Conditions
An Agent will stop executing in three circumstances:
1. The maximum number of iterations is reached
2. The Agent executes a tool that is configured to stop the Agent
3. The Agent has generated a response with no further tool calls
### Max Iterations
You can cap the maximum number of iterations an Agent can execute--the Agent will generate up to `max_iterations` times, and then stop.
This results in an Agent Log with a `finish_reason` of `"max_iterations_reached"`.
### Stopping Tools
You can designate that a tool should stop the Agent after it is called. If it can be run on Humanloop, it will be run and the result logged before the Agent stops.
This results in an Agent Log with a `finish_reason` of `"stopping_tool_called"` and `stopping_tool_names` populated.
### Agent Finished
Lastly, the Agent will stop if it has generated a response with no further tool calls. This represents a natural end to the Agent's execution, and results in an Agent Log with a `finish_reason` of `"agent_generation_finished"`.
## Evaluation
Agents can be evaluated both through the Humanloop UI and programmatically. After each Agent execution, configured evaluators will run automatically on Humanloop.
Get started with [evaluating your Agents in the UI](/docs/tutorials/evaluate-agent-in-ui) or [programmatically](/docs/tutorials/agent-evaluation-code).
## Tracing
A trace is the collection of Logs associated with an Agent execution, showing the complete path from input to output, including all intermediate steps.
Every Agent trace includes:
* The initial input and messages, and final output
* All intermediate LLM calls and their responses as nested Logs
* Tools called by the Agent and their inputs and outputs as nested Logs
* Any errors or unexpected behaviors
* Timing and performance metrics
### Metrics and Timing
Cost, latency, execution time, and token usage metrics are captured for each trace. They span from the earliest interaction to the final response.
## Serialization
The .agent file format is a serialized representation of an Agent Version, designed to be human-readable and suitable for integration into version control systems alongside code.
The format is heavily inspired by MDX, with model and parameters specified in a YAML header alongside JSX-style syntax for chat templates.
```jsx title=".agent" maxLines=75
---
model: claude-3-7-sonnet-latest
max_tokens: -1
provider: anthropic
endpoint: chat
reasoning_effort: 1024
tools: [
{
"type": "file",
"link": {
"file_id": "pr_1234567890",
"version_id": "prv_1234567890"
},
"on_agent_call": "continue"
},
{
"type": "file",
"link": {
"file_id": "tl_1234567890",
"version_id": "tlv_1234567890"
},
"on_agent_call": "stop"
},
{
"type": "inline",
"json_schema": {
"name": "stop",
"description": "Call this tool when you have finished your task.",
"parameters": {
"type": "object",
"properties": {
"output": {
"type": "string",
"description": "The final output to return to the user."
}
},
"additionalProperties": false,
"required": [
"output"
]
},
"strict": true
},
"on_agent_call": "stop"
},
{
"type": "file",
"link": {
"file_id": "tl_cBfvZ3Xre8PAfVAa5r9P6",
"version_id": "tlv_gzr3bxaVGu889O4lsOgtG"
},
"on_agent_call": "continue"
},
]
---
The core entity in the Humanloop evaluation framework is an **[Evaluator](/docs/explanation/evaluators)** - a function you define which takes an LLM-generated log as an argument and returns a **judgment**.
The judgment is typically either a boolean or a number, indicating how well the model performed according to criteria you determine based on your use case.
Evaluators can be leveraged for [Monitoring](/docs/guides/observability/monitoring) your live AI application, as well as for [Evaluations](/docs/guides/evals/run-evaluation-ui) to benchmark different version of your AI application against each other pre-deployment.
## Sources of Judgment
Currently, you can define three different Evaluator sources on Humanloop:
* **Code** - using simple deterministic rules based judgments against attributes like cost, token usage, latency, regex rules on the output, etc. These are generally fast and cheap to run at scale.
* **AI** - using other foundation models to provide judgments on the output. This allows for more qualitative and nuanced judgments for a fraction of the cost of human judgments.
* **Human** - getting gold standard judgments from either end users of your application, or internal domain experts. This can be the most expensive and slowest option, but also the most reliable.
## Online Monitoring versus Offline Evaluation
Evaluators can be deployed on Humanloop to support both testing new versions of your Prompts and Tools during development and for monitoring live apps that are already in production.
### Online Monitoring
Evaluators are run against the [Logs](./logs) generated by your AI applications. Typically, they are used to monitor deployed model performance over time and check for drift or degradation in performance.
The Evaluator in this case only takes a single argument - the `log` generated by the model. The Evaluator is expected to return a judgment based on the Log,
which can be used to trigger alerts or other actions in your monitoring system.
See our [Monitoring guides](/docs/guides/observability/monitoring) for more details.
### Offline Evaluations
Offline Evaluators are combined with predefined [**Datasets**](/docs/v4/datasets) in order to evaluate your application as you iterate in your prompt engineering workflow, or to test for regressions in a CI environment.
A test Dataset is a collection of **Datapoints**, which are roughly analogous to unit tests or test cases in traditional programming. Each datapoint specifies inputs to your model and (optionally) some target data.
When you run an offline evaluation, a Log needs to be generated using the inputs of each Datapoint and the version of the application being evaluated. Evaluators then need to be run against each Log to provide judgements,
which are then aggregated to provide an overall score for the application. Evaluators in this case take the generated `Log` and the `testcase` datapoint that gave rise to it as arguments.
See our guides on [creating Datasets](/docs/guides/evals/upload-dataset-csv) and [running Evaluations](/docs/guides/evals/run-evaluation-ui) for more details.
## Humanloop runtime versus your runtime
Evaluations require the following to be generated:
1. Logs for the datapoints.
2. Evaluator results for those generated logs.
Evaluators which are defined within the Humanloop UI can be executed in the Humanloop runtime, whereas Evaluators defined in your code can be executed in your runtime and the results posted back to Humanloop.
This provides flexibility for supporting more complex evaluation workflows.
## Return types
Evaluators apply judgment to Logs. This judgment can be of the following types:
* **Boolean** - A true/false judgment.
* **Number** - A numerical judgment, which can act as a rating or score.
* **Select** - One of a predefined set of options. One option must be selected.
* **Multi-select** - Any number of a predefined set of options. None, one, or many options can be selected.
* **Text** - A free-form text judgment.
Code and AI Evaluators can return either **Boolean** or **Number** judgments.
Human Evaluators can return **Number**, **Select**, **Multi-select**, or **Text** judgments.
***
subtitle: Tools give Agents and Prompts access to external data and enable them to take action.
description: Discover how Humanloop manages tools for use with large language models (LLMs) with version control and rigorous evaluation for better performance.
---------------------
For clean Markdown of any page, append .md to the page URL. For a complete documentation index, see https://humanloop.com/docs/llms.txt. For full documentation content, see https://humanloop.com/docs/llms-full.txt.
Tools on Humanloop are used to extend [Agents](/docs/explanation/agents) and [Prompts](/docs/explanation/prompts) with access to external data sources and enable them to take action.
## Types of Tools
### Runnable Tools
Runnable Tools can be executed by Humanloop, meaning they can form a part of an Agent's automatic execution.
Python code Tools, an example of these, are run in a [secure Python runtime](/docs/v5/reference/python-environment) and allow you to write custom code to be executed.
### Template Tools
Template Tools can be used within a Prompt's template to dynamically insert information for the model when called.
These include:
* **Snippet Tools**: Reusable text components that can be shared across multiple Prompts
* **Integration Tools**: Pre-built connectors to popular services like Google Search and Pinecone vector database
### Function calling Tools
Many models, such as those by [OpenAI](https://platform.openai.com/docs/guides/function-calling) and [Anthropic](https://docs.anthropic.com/en/docs/build-with-claude/tool-use), now support function calling.
You can provide these models with definitions of the available tools. The model then decides whether to call a tool, and which parameters to use.
On Humanloop, [JSON schema Tools](/docs/v5/guides/prompts/tool-calling-editor) record the schema definitions that are used for function calling,
allowing you to tweak and version control them.
```js
{
"name": "get_current_weather",
"description": "Get the current weather in a given location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"name": "Location",
"description": "The city and state, e.g. San Francisco, CA"
},
"unit": {
"type": "string",
"name": "Unit",
"enum": [
"celsius",
"fahrenheit"
]
}
},
"required": [
"location",
"unit"
]
}
}
```
In addition to JSON schema Tools, all Runnable Tools and Prompts can be used for function calling. Humanloop will automatically generate the schema definitions
for these when they're linked to a Agent or Prompt.
## Humanloop runtime versus your runtime
Humanloop supports running Python code Tools within the Humanloop runtime. If you have Tools that you want to run in your own environment,
you can use the Humanloop API to manage versions and record Logs.
The simplest way to do this is by decorating a function with the [Tool decorator](/docs/v5/sdk/decorators/tool) in the SDK.
Running the decorated function will automatically log the inputs and output of the function
to Humanloop. If you're using the Python SDK, the `@tool` decorator will also infer a JSON schema from the function's type hints.
***
subtitle: Instrument and monitor multi-step AI systems.
description: Humanloop Flows trace and evaluate complex AI workflows, from LLM agents to retrieval-augmented generation (RAG). By unifying all components, Flows provide the context needed to debug and iterate with confidence.
---------------------
For clean Markdown of any page, append .md to the page URL. For a complete documentation index, see https://humanloop.com/docs/llms.txt. For full documentation content, see https://humanloop.com/docs/llms-full.txt.
## Introduction
LLM-powered systems are multi-step processes, leveraging information sources, delegating computation to tools, or iterating with the LLM to generate the final answer.
Looking at the inputs and output of such a system is not enough to reason about its behavior. Flows address this by tracing all components of the feature, unifying Logs in a comprehensive view of the system.
## Basics
To integrate Flows in your project, add the SDK flow decorator on the entrypoint of your AI feature.
### Manual Tracing
If you don't want to use decorators, first create a Flow Log, then pass its `id` when creating Logs you want to add to the trace.
Datasets on Humanloop are collections of Datapoints used for evaluation and fine-tuning. You can think of a Datapoint as a test case for your AI application, which contains the following fields:
* **Inputs**: a collection of prompt variable values that replace the `{{variables}}` defined in your prompt template during generation.
* **Messages**: for chat models, you can have a history of chat messages that are appended to the prompt during generation.
* **Target**: a value that in its simplest form describes the desired output string for the given inputs and messages history. For more advanced use cases, you can define a JSON object containing whatever fields are necessary to evaluate the model's output.
## Versioning
A Dataset will have multiple Versions as you iterate on the test cases for your task. This tends to be an evolving process as you learn how your [Prompts](./prompts) behave and how users interact with your AI application in the wild.
Dataset Versions are immutable and are uniquely defined by the contents of the Datapoints. When you change, add, or remove Datapoints, this constitutes a new Version.
Each [Evaluation](/docs/guides/evals/run-evaluation-ui) is linked to a specific Dataset Version, ensuring that your evaluation results are always traceable to the exact set of test cases used.
## Creating a Dataset
A Dataset can be created in the following ways:
* [Upload a CSV in the UI](/docs/guides/evals/upload-dataset-csv)
* [Create a Datapoint from an existing Log](/docs/guides/evals/create-dataset-from-logs)
* [Create a Dataset via the API](/docs/guides/evals/create-dataset-api)
## Using Datasets for Evaluations
Datasets are foundational for Evaluations on Humanloop. Evaluations are run by iterating over the Datapoints in a Dataset, generating output from different versions of your AI application for each one. The Datapoints provide the specific test cases to evaluate, with each containing the input variables and optionally a target output that defines the desired behavior. When a target is specified, Evaluators can compare the generated outputs to the targets to assess how well each version performed.
***
subtitle: Directories can be used to group together related Files.
description: Directories can be used to group together related Files. This is useful for organizing your work as part of prompt engineering and collaboration.
---------------------
For clean Markdown of any page, append .md to the page URL. For a complete documentation index, see https://humanloop.com/docs/llms.txt. For full documentation content, see https://humanloop.com/docs/llms-full.txt.
Directories in Humanloop serve as an organizational tool, allowing users to group related [Files](/docs/explanation/files) and structure their work logically. They function similarly to folders in a traditional file system, providing a hierarchical structure for managing work.
### Paths
Every File and Directory has a unique `path` property which you can use to refer to Files or Directories. When you move a File or Directory, its path and the paths of all contained files and subdirectories will update accordingly.
Environments enable you to deploy different versions of your files to specific environments, allowing you to separately manage the deployment workflow between testing and production. With environments, you have the control required to manage the full LLM deployment lifecycle.
### Managing your environments
Every organisation automatically receives a default production environment. You can create additional environments with custom names by visiting your organisation's [environments page](https://app.humanloop.com/account/environments).
### Setup Google Search tool
### Add Tools to the Agent
Click on Outreach Agent File, then click on **+ Tools** button on left bottom corner and choose **Google Search** Tool and **Write Personalized Message** Prompt from the list. Remove HackerNews Search Tool as it's no longer needed.
Save the Agent and name it "v2".
### Run another Evaluation
We can now create a new Run with the new Agent version. Click on the **+ Run** button and select the newly created Agent version.
Navigate to the **Stats** tab to see how the two versions compare to each other.
To see the two versions side by side, click on the **Review** tab.
The second version of the Agent we evaluated used the Google Search Tool to extract more relevant information. It also searched for background information on each lead, something the initial version of the Agent lacked.
Adding a new Tools to the Agent resulted in a more personalized message and improved outreach.
## Add detailed logging
The decorators divide the code in logical components, allowing you to observe the steps taken to answer a question. Every step taken by the agent creates a Log.
## Next steps
We've built a complex agentic workflow and learned how to use Humanloop to add logging to it and evaluate its performance.
Take a look at these resources to learn more about evals on Humanloop:
* Learn how to [create a custom dataset](/docs/v5/guides/evals/create-dataset) for your project.
* Learn more about using [LLM Evaluators](/docs/v5/guides/evals/llm-as-a-judge) on Humanloop.
***
subtitle: Use Humanloop to evaluate more complex workflows like Retrieval Augmented Generation (RAG).
description: Evaluate a RAG application with Humanloop.
---------------------
For clean Markdown of any page, append .md to the page URL. For a complete documentation index, see https://humanloop.com/docs/llms.txt. For full documentation content, see https://humanloop.com/docs/llms-full.txt.
This tutorial demonstrates how to take an existing RAG pipeline and use Humanloop to evaluate it. At the end of the tutorial you'll understand how to:
1. Run an Eval on your RAG pipeline.
2. Set up detailed logging with SDK decorators.
3. Log to Humanloop manually
The full code for this tutorial is available in the [Humanloop Cookbook](https://github.com/humanloop/humanloop-cookbook/blob/main/tutorials/rag/evaluate-rag-flow.ipynb).
## Example RAG Pipeline
In this tutorial we'll first implement a simple RAG pipeline to do Q\&A over medical documents without Humanloop. Then we'll add Humanloop and use it for evals. Our RAG system will have three parts:
* **Dataset**: A version of the [MedQA dataset](https://huggingface.co/datasets/bigbio/med_qa) from Hugging Face.
* **Retriever**: [Chroma](https://docs.trychroma.com/getting-started) as a simple local vector DB.
* **Prompt**: Managed in code, populated with the user's question and retrieved context.
## Set up RAG pipeline
```bash
poetry install
```
Set up environment variables:
```python
from dotenv import load_dotenv
import os
from chromadb import chromadb
from openai import OpenAI
import pandas as pd
load_dotenv()
chroma = chromadb.Client()
openai = OpenAI(api_key=os.getenv("OPENAI_KEY"))
```
Set up the Vector DB:
```python
collection = chroma.get_or_create_collection(name="MedQA")
knowledge_base = pd.read_parquet("../../assets/sources/textbooks.parquet")
knowledge_base = knowledge_base.sample(10, random_state=42)
collection.add(
documents=knowledge_base["contents"].to_list(),
ids=knowledge_base["id"].to_list(),
)
```
Define the Prompt:
```python
model = "gpt-4o-mini"
temperature = 0
template = [
{
"role": "system",
"content": """Answer the following question factually.
Question: {{question}}
Options:
- {{option_A}}
- {{option_B}}
- {{option_C}}
- {{option_D}}
- {{option_E}}
---
Here is some retrieved information that might be helpful.
Retrieved data:
{{retrieved_data}}
---
Give you answer in 3 sections using the following format. Do not include the quotes or the brackets. Do include the "---" separators.
Every time the user presses the Send button, Humanloop receives the request and calls the AI model. The response from the model is then stored as a Log.
Let's check the `api/chat/route.ts` file to see how it works.
* The `path` parameter is the path to the Prompt in the Humanloop workspace. If the Prompt doesn't exist, it will be created.
* The `prompt` parameter is the configuration of the Prompt. In this case we manage our Prompt in code; if the configuration of the Prompt changes, a new version of the Prompt will automatically be created on Humanloop. Prompts can alternatively be managed [directly on Humanloop](/docs/development/guides/call-prompt).
* The `messages` parameter is the list of all messages exchanged between the Model and the User.
To learn more about calling Prompts with the Humanloop SDK, see the [Prompt Call](/v5/api-reference/prompts/call-stream) API reference.
```typescript api/chat/route.ts
const response = await humanloop.prompts.callStream({
// if Prompt doesn't exist, it will be created
path: "chatgpt-clone-tutorial/customer-support-agent",
prompt: {
model: "gpt-4",
template: [
{
role: "system",
content: "You are a helpful assistant.",
},
],
},
// messages is a list of objects: [{role: string, content: string}, ...].
// Role is either "user", "assistant", "system", or "tool".
messages,
providerApiKeys: {
// OpenAI API key, if you haven't already set it in Humanloop app
openai: process.env.OPENAI_API_KEY,
},
});
```
### Review the logs in Humanloop
After chatting with the AI model, go to the Humanloop app and review the logs.
Click on the `chatgpt-clone-tutorial/customer-support-agent` Prompt, then click on the Logs tab at the top of the page.
You see that all the interactions with the AI model are logged here.
### Modify the code to capture user feedback
Now, let's modify the code to start getting user feedback!
Go back to the code editor and uncomment lines 174-193 in the `page.tsx` file.
This snippet will add 👍 and 👎 buttons, that users can press to give feedback on the model's responses.
```typescript {13-31}
return (
Click on Log and then on **Editor ->** button in the top right corner to open the Prompt Editor. In the Prompt Editor, you can make changes to the instructions and the model's parameters to improve the model's performance.
Once you're happy with the changes, deploy the new version of the Prompt.
When users start interacting with the new version, compare the "good" to "bad" ratio to see if the changes have improved your users' experience.
## Next steps
Now that you've successfully captured user feedback, you can explore more ways to improve your AI product:
* If you found that your Prompt doesn't perform well, see our guide on [Comparing and Debugging Prompts](/docs/evaluation/guides/comparing-prompt-editor).
* Leverage [Code](/docs/evaluation/guides/code-based-evaluator), [AI](/docs/evaluation/guides/llm-as-a-judge) and [Human](/docs/evaluation/guides/human-evaluators) Evaluators to continuously monitor and improve your AI product.
***
title: Migrating from Humanloop
subtitle: How to export your data from Humanloop
description: Complete guide to exporting your Files, Versions, Logs, and Evaluations from Humanloop using our export tool.
---------------------
For clean Markdown of any page, append .md to the page URL. For a complete documentation index, see https://humanloop.com/docs/llms.txt. For full documentation content, see https://humanloop.com/docs/llms-full.txt.
Following our acquisition, the Humanloop platform will be sunset on **September 8th, 2025**.
This guide will walk you through exporting all your data from Humanloop, including Files, Versions, Logs, and Evaluations, so you can migrate to your chosen alternative platform.
## Frequently Asked Questions
### Set up an Evaluation Run
* Select a Dataset using **+Dataset**.
* Add the Prompt versions you want to compare using **+Prompt**.
* Add the Evaluators you want to use to judge the performance of the Prompts using **+Evaluator**.
* Below the spider plot, you can see the breakdown of performance per Evaluator.
* To drill into and debug the Logs that were generated, navigate to the **Review** tab at top left of the Run page.
The Review view allows you to better understand performance and replay logs in our Prompt Editor.
Alternatively you can see detailed stats in the Humanloop UI. Navigate to the Prompt, click on the **Evaluations** tab at the top of the page and select the Evaluation you just created. The stats are displayed in the **Stats** tab.
## Steps
To create a dataset from a CSV file, we'll first create a CSV in Google Sheets that contains values for our Prompt variable `{{query}}` and then upload it to a Dataset on Humanloop.
### Export the Google Sheet to CSV
In Google Sheets, choose **File** → **Download** → **Comma-separated values (.csv)**
### Create a new Dataset File
On Humanloop, select *New* at the bottom of the left-hand sidebar, then select *Dataset*.
### Click **Upload CSV**
First name your dataset when prompted in the sidebar, then select the **Upload CSV** button and drag and drop the CSV file you created above using the file explorer.
Press **Upload Dataset from CSV...**.
### Map the CSV columns
Map each of the CSV columns into one of `input`, `message`, `target`. To avoid uploading a column of your CSV you can map it to the `exclude` option.
To map in columns to Messages, they need to be in a specific format. An example of this can be seen in our example Dataset or below:
```
"[{""role"": ""user"", ""content"": ""Tell me about the weather""}]"
```
Once you have mapped your columns, press **Extend Current Dataset**
### Review your uploaded datapoints
You'll see the input-output pairs that were included in the CSV file and you can review the rows to inspect and edit the individual Datapoints.
Your dataset is now uploaded and ready for use.
## Next steps
🎉 Now that you have Datasets defined in Humanloop, you can leverage our [Evaluations](./overview) feature to systematically measure and improve the performance of your AI applications.
See our guides on [setting up Evaluators](./llm-as-a-judge) and [Running an Evaluation](/docs/guides/evals/run-evaluation-ui) to get started.
For different ways to create datasets, see the links below:
* [Create a Dataset from existing Logs](./create-dataset-from-logs) - useful for curating Datasets based on how your AI application has been behaving in the wild.
* [Upload via API](./create-dataset-api) - useful for uploading more complex Datasets that may have nested JSON structures, which are difficult to represent in tabular .CSV format, and for integrating with your existing data pipelines.
***
subtitle: In this guide, we will walk through creating a Dataset on Humanloop via the API
description: Learn how to create Datasets in Humanloop to define fixed examples for your projects, and build up a collection of input-output pairs for evaluation and fine-tuning.
---------------------
For clean Markdown of any page, append .md to the page URL. For a complete documentation index, see https://humanloop.com/docs/llms.txt. For full documentation content, see https://humanloop.com/docs/llms-full.txt.
[Datasets](/docs/explanation/datasets) are a collection of input-output pairs that can be used to evaluate your Prompts, Tools or even Evaluators.
### Prerequisites
If you are using the SDK, the only prerequisite is to have the SDK installed and configured. If you are using the API directly, you will need to have an API key.
### Select a subset of the Logs to add
Filter logs on a criteria of interest, such as the version of the Prompt used, then multi-select Logs.
In the menu in the top right of the page, select **Add to Dataset**.
### Add to a new Dataset
Provide a name for the new Dataset and click **Create** (or you can click **add to existing Dataset** to append the selection to an existing Dataset).
Then you can optionally add a version name. A description is pre-filled with details from the source Logs — you can edit this to add more context about
the new version if you’d like.
You will then see the new Dataset appear at the same level in the filesystem as your Prompt.
* Give the Evaluator a name when prompted in the sidebar, for example `Category Validator`.
### Define the Evaluator code
After creating the Evaluator, you will automatically be taken to the code editor.
For this example, our Evaluator will check that the feature category returned by the Prompt is from the list of allowed feature categories. We want to ensure our categoriser isn't hallucinating new features.
* Make sure the **Mode** of the Evaluator is set to **Online** in the options on the left.
* Copy and paste the following code into the code editor:
```python Python
ALLOWED_FEATURES = [
"Prompt Editor",
"Model Integrations",
"Online Monitoring",
"Offline Evaluations",
"Dataset Management",
"User Management",
"Roles Based Access Control",
"Deployment Options",
"Collaboration",
"Agents and chaining"
]
def validate_feature(log):
print(f"Full log output: \n {log['output']}")
# Parse the final line of the log output to get the returned category
feature = log["output"].split("\n")[-1]
return feature in ALLOWED_FEATURES
```
* Click the **Run** button at the far right of one of the loaded Logs to trigger a debug run. This causes the code to be executed with the selected Log as input and populates the **Result** column.
* Inspect the output of the executed code by selecting the arrow to the right of **Result**.
### Save the code
Now that you've validated the behaviour, save the code by selecting the **Save** button at the top right of the Editor and optionally provide a suitable version name and description.
### Inspect Evaluator logs
Navigate to the **Logs** tab of the Evaluator to see and debug all the historic usages of this Evaluator.
## Create an LLM Evaluator
* Click the **Run** button at the far right of one of the loaded Logs to trigger a debug run. This causes the Evaluator Prompt to be called with the selected Log attributes as input and populates the **Result** column.
* Inspect the output of the executed code by selecting the arrow to the right of **Result**.
### Save the code
Now that you've validated the behaviour, save the code by selecting the **Save** button at the top right of the Editor and optionally provide a suitable version name and description.
### Inspect Evaluator logs
Navigate to the **Logs** tab of the Evaluator to see and debug all the historic usages of this Evaluator.
### Create a new Run
To evaluate a version of your Prompt, click on the **+Run** button, then select the version of the Prompt you want to evaluate and the Dataset you want to use.
Click on **+Evaluator** to add a Human Evaluator to the Evaluation.
Click **Save** to create a new Run. Humanloop will start generating Logs for the Evaluation.
### Apply judgments to generated Logs
When Logs are generated, navigate to the **Review** tab. Turn on **Focus mode** and start providing judgments on the generated Logs.
When the last judgment is provided, the Run is marked as complete.
### Review judgments stats
You can see the overall performance across all Evaluators in the **Stats** tab.
Alternatively, you [upload Dataset via our SDK](/docs/v5/guides/evals/create-dataset-api)
### Run an Evaluation
Navigate to a Prompt you want to evaluate and create a new Evaluation Run.
### Split the workload between SMEs
To split the workload between your SMEs, navigate to the **Review** tab, turn on **Focus mode**, and click on the **Filters** button.
Filter the Dataset by identifiers, such as "chunk", to split the review work into smaller pieces.
### Send the URL to your SMEs
After you have filtered the Dataset, copy the URL and send it to your SME.
When they open the link, they will only see the relevant chunk of the Dataset.
### Monitor progress
As the SMEs provide judgments on the outputs,
we display the overall progress and the number of outstanding judgments. When the final judgment is given, the Evaluation is marked as complete.
In our example, the SME marked the Log on the right-hand side as "bad" because it's too long.
To take action, click on the Log ID above the Log output to open the Log drawer. In the drawer, click on the **Editor ->** button to load this Log in the Prompt Editor. Now, modify the instructions to explicitly state that the model should provide a concise answer.
## Next steps
We've successfully split the work among multiple SMEs to effectively evaluate the performance of our AI product.
Explore next:
* If your SMEs gave negative judgments on the Logs, see our guide on [Comparing and Debugging Prompts](/docs/evaluation/guides/comparing-prompt-editor).
* Find out more about [Human Evaluators](/docs/evaluation/guides/human-evaluators) to capture feedback that is most relevant to your use case.
***
subtitle: In this guide, we will walk through comparing the outputs from multiple Prompts side-by-side using the Humanloop Editor environment and using diffs to help debugging.
---------------------
For clean Markdown of any page, append .md to the page URL. For a complete documentation index, see https://humanloop.com/docs/llms.txt. For full documentation content, see https://humanloop.com/docs/llms-full.txt.
You can compare Prompt versions interactively side-by-side to get a sense for how their behaviour differs; before then triggering more systematic [Evaluations](/docs/guides/evals/run-evaluation-ui).
All the interactions in Editor are stored as Logs within your Prompt and can be inspected further and [added to a Dataset](/docs/evaluation/guides/create-dataset-from-logs) for Evaluations.
### Prerequisites
* You already have a Prompt — if not, please follow our [Prompt creation](/docs/development/guides/create-prompt) guide first.
## Compare Prompt versions
In this example we will use a simple support agent Prompt that answers user queries about Humanloop's product and docs.
Now save the new version of your Prompt by selecting the **Save** button in the top right and optionally provide a helpful version name (e.g. "Simple Support Agent v2") and/or description (e.g. "Changed model to gpt-4o-mini").
### Load up two versions of your Prompt in the Editor
To load up the previous version side-by-side, select the menu beside the Load button and select the **New panel** option (depending on your screen real-estate, you can add more than 2 panels).
Then press the *Load* button in the new panel and select another version of your Prompt to compare.
### Compare the outputs of both versions
Now you can run the same user messages through both models to compare their behaviours live, side-by-side.
### Select the versions to compare
In the table, select two rows you would like to see the changes between. Then select the **Show diff** button above the table.
### Call the Prompt by ID
Now you can use the SDK to generate completions and log the results to your Prompt:
This can be done by using the [Prompt Call](/docs/api/prompts/call) method in the SDK.
In the modal that appears, open the **Examples** dropdown and select the `get_current_weather` tool.
If you choose to edit or create your own tool, you'll need to use the [JSON Schema syntax](https://json-schema.org/).
When creating a custom tool, it should correspond to a function you have defined in your own code.
The JSON Schema you define here specifies the parameters and structure you want the AI model to use when interacting with your function.
### **Submit tool response**
You can also check how the Prompt would respond to a successful tool call.
Click on the **Add to Messages** button to move the assistant's response into the Messages section
for a subsequent call.
Then, you can paste in the exact response that the Tool would respond with into the **Output** section of the Tool call.
For prototyping purposes, you can also just simulate the response yourself. Provide in a mock response:
```json
{ "temperature": 65, "condition": "light rain", "unit": "fahrenheit" }
```
Remember, the goal is to simulate the tool's output as if it were actually fetching real-time weather data.
This allows you to test and refine your prompt and tool interaction without needing to implement the actual weather API.
After entering the simulated tool response, click on the **Run** button to send the Tool's response to the AI model.
### **Review assistant response**
The assistant should now respond using the information provided in your simulated tool response.
For example, if you input that the weather in Boston was 65°F and light rain, the assistant might say:
`The current weather in Boston is light rain with a temperature of 65°F.`
This response demonstrates how the AI model incorporates the tool's output into its reply, providing a more contextual and data-driven answer.
### **Iterate and refine**
Feel free to experiment with different queries. This iterative process helps you fine-tune your Prompt and understand how the AI model interacts with the tool,
ultimately leading to more effective and accurate responses in your application.
### **Save your Prompt**
By saving your Prompt, you will create a new version that includes the inline tool.
Congratulations! You've successfully learned how to use tool calling in the Humanloop Editor. This powerful feature allows you to test tool interactions, helping you create more dynamic and context-aware AI applications.
## Next steps
After you've created and tested your inline Tool, you might want to reuse it across multiple Prompts.
Humanloop allows you to link a Tool, making it easier to share and manage Tool definitions across multiple Prompts.
For more detailed instructions on how to link and manage Tools, check out our guide on [Linking a JSON Schema Tool](/docs/v5/guides/prompts/tool-calling-editor).
***
subtitle: How to re-use common text snippets in your Prompt templates with the Snippet Tool
description: Learn how to use the Snippet tool to manage common text snippets that you want to reuse across your different prompts.
---------------------
For clean Markdown of any page, append .md to the page URL. For a complete documentation index, see https://humanloop.com/docs/llms.txt. For full documentation content, see https://humanloop.com/docs/llms-full.txt.
The Snippet Tool supports managing common text 'snippets' that you want to reuse across your different prompts. A Snippet tool acts as a simple key/value store, where the key is the name of the common re-usable text snippet and the value is the corresponding text.
For example, you may have some common persona descriptions that you found to be effective across a range of your LLM features. Or maybe you have some specific formatting instructions that you find yourself re-using again and again in your prompts.
Instead of needing to copy and paste between your editor sessions and keep track of which projects you edited, you can instead inject the text into your prompt using the Snippet tool.
## Create and use a Snippet Tool
### Prerequisites
* You already have a Prompt — if not, please follow our [Prompt creation](/docs/development/guides/create-prompt) guide first.
To create and use a snippet tool, follow the following steps:
### Name the Tool
Name it `assistant-personalities` and give it a description `Useful assistant personalities`.
### Add a key called "helpful-assistant"
In the initial box add `helpful-assistant` and give it a value of `You are a helpful assistant. You like to tell jokes and if anyone asks your name is Sam.`
### Add another key called "grumpy-assistant"
Let's add another key-value pair, so press the **Add a key/value pair** button and add a new key of `grumpy-assistant` and give it a value of `You are a grumpy assistant. You rarely try to help people and if anyone asks your name is Freddy.`.
### Save and Deploy your Tool.
Press the **Save** button, and optionally enter a name and description for this new version. When presented with the deployment dialog, click the **Deploy** button and deploy to your *production* environment.
Now your Snippets are set up, you can use it to populate strings in your prompt templates across your projects.
### Navigate to the **Editor**
Go to the Editor of your previously created file.
### Add `{{ assistant-personalities(key) }}` to your prompt
Delete the existing prompt template and add `{{ assistant-personalities(key) }}` to your prompt.
### Change the key to `grumpy-assistant`.
The Snippet tool is particularly useful because you can define passages of text once in a Snippet tool and reuse them across multiple prompts, without needing to copy/paste them and manually keep them all in sync. Editing the values in your tool allows the changes to automatically propagate to the Prompts when you update them, as long as the key is the same.
### Click the **Change deployment** button
### Select a version
Choose the version you want to deploy from the list of available versions.
### Click the **Deploy** button.
### (Optional) Move a File into the Directory
1. In the File navigation sidebar, hover over the file in the sidebar to show the context menu. From the menu select "Move".
2. Choose the destination directory
After you create the Tool, you can edit the file name in the sidebar. Rename the file to "Weather tool".
### Define your tool's structure
Paste the following JSON into the provided dialog to define your tool's structure:
```json
{
"name": "get_current_weather",
"description": "Get the current weather in a given location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"name": "Location",
"description": "The city and state, e.g. San Francisco, CA"
},
"unit": {
"type": "string",
"name": "Unit",
"enum": [
"celsius",
"fahrenheit"
]
}
},
"required": [
"location",
"unit"
]
}
}
```
If you choose to edit or create your own Tool, you'll need to use the [JSON Schema syntax](https://json-schema.org/).
When creating a custom Tool, it should correspond to a function you have defined in your own code.
The JSON Schema you define here specifies the parameters and structure you want the AI model to use when interacting with your function.
### Save Tool version
Press the **Save** button to save this version of the Tool.
Click the **Deploy...** button to deploy the Tool to your production environment, following the steps in the dialog.
You should now see "Weather tool" in the list of Tools.
### Run Prompt with Tool
Now that your Tool is linked, you can start using it. In the panel on the right, add a User message with "What is the weather in London?".
Press the **Run** button.
You should see the **Assistant** respond with the tool call.
Press the **Add to Messages** button so you can make a subsequent call to the Prompt with the tool call result,
to see how the model uses the tool call.
This will move the Assistant's messages, containing the tool call, to the Messages section.
You should now also see an **Output** section within the tool call, allowing you to insert an answer.
Enter `22` into the Output section and press **Continue**.
The model will respond with "The current temperature in London is 22°C."
### Save the Prompt
Now that you've linked a Tool to your Prompt, let's save it. Press the **Save** button and name this Prompt version `weather-model`.
### Click **Send invite**.
An email will be sent to the entered email address, inviting them to the organization. If the entered email address is not already a Humanloop user, they will be prompted to create an account before being added to the organization.
### Copy the generated API key
Save it in a secure location. You will not be shown the full API key again.
## Rename an environment
You can rename an environment to re-arrange your development workflows. Since each new file is automatically deployed to the default environment, which is production unless altered, it may make more sense to create a separate production environment and rename your current environments.
To help you get started, we provide a library of example Evaluators, Prompts, Tools and Datasets. These examples have been designed for common AI use-cases like RAG, customer service, and agents. Each example can be duplicated to your workspace and customized for your use cases.
## Browse the Library
The Template Library includes various examples built for common AI application needs, including:
#### Evaluators
Pre-built evaluators for measuring semantic similarity, cost, latency, and other key metrics. Quickly assess your AI's performance with ready-made evaluators that are suited for RAG. Measure response accuracy, prompt quality, and user satisfaction with tools designed for reliable and consistent evaluation.
#### Prompts
Example prompts for common use cases including classification, question-answering, content generation, conversational agents, and RAG (Retrieval-Augmented Generation).
#### Tools
Ready-to-use tools for tasks like retrieval, data extraction, and calling external functions with JSON Schema function calling.
#### Datasets
Example datasets for common use cases including classification, question-answering, and content generation.
#### Example Agents and Flows
Example agent templates designed for common use cases like customer support, guided workflows, and RAG applications. These templates demonstrate best practices for building AI agents that can classify queries, answer questions, generate content, and provide context-aware responses by retrieving and incorporating relevant information.
## How to use templates
1. Navigate to the 'Library' page in the sidebar
2. Filter by type, category, or tags
3. Select a template you'd like to use
4. Click "Duplicate" to create a copy in your workspace that you can customize
Once you've added a template to your workspace, you can modify it to match your specific requirements. Learn more about customizing your files in our guides for [Evaluators](/docs/evaluation/guides/code-based-evaluator), [Prompts](/docs/guides/prompts/create-prompt), and [Tools](/docs/guides/prompts/tool-calling-editor).
***
subtitle: How to integrate Humanloop with the Vercel AI SDK
description: Learn about the ways you can use Humanloop with the Vercel AI SDK.
---------------------
For clean Markdown of any page, append .md to the page URL. For a complete documentation index, see https://humanloop.com/docs/llms.txt. For full documentation content, see https://humanloop.com/docs/llms-full.txt.
## Observability integration
The Vercel AI SDK supports [tracing via OpenTelemetry](https://sdk.vercel.ai/docs/ai-sdk-core/telemetry). You can export these traces to Humanloop by enabling telemetry and configuring the OpenTelemetry Exporter.
Humanloop provides serialized File formats (`.prompt`, `.agent`) for storing Prompts and Agents as human-readable, version-control-friendly files.
These formats enable you to integrate your Prompts and Agents into standard software development workflows.
## File Types
Humanloop currently supports the following File formats:
* `.prompt` — Serialized representation of a [Prompt](/docs/explanation/prompts)
* `.agent` — Serialized representation of an [Agent](/docs/explanation/agents)
## Usage
The Humanloop SDK supports working with local Files in your codebase, enabling a code-first development approach.