For AI agents: a documentation index is available at the root level at /llms.txt and /llms-full.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page.
Sign inBook a demo
DocsReferenceChangelog
DocsReferenceChangelog
  • Getting Started
    • Overview
    • Quickstart
  • Explanation
    • Integrating Humanloop
  • Tutorials
    • Evaluate an Agent in the UI
    • Evaluate an Agent in code
    • Evaluate a RAG app
    • Capture user feedback
  • How-To Guides
    • Migrating from Humanloop
      • Run an Evaluation via the UI
      • Run an Evaluation via the API
      • Upload a Dataset from CSV
      • Create a Dataset via the API
      • Create a Dataset from existing Logs
      • Set up a code Evaluator
      • Set up LLM as a Judge
      • Set up a Human Evaluator
      • Run a Human Evaluation
      • Manage multiple reviewers
      • Compare and Debug Prompts
      • Set up CI/CD Evaluations
      • Spot-check your Logs
      • Use external Evaluators
      • Evaluate external logs
  • Reference
    • Deployment Options
    • Supported Models
    • Template Library
    • Vercel AI SDK
    • .prompt and .agent Files
    • Humanloop Runtime Environment
    • Security and Compliance
    • Data Management
    • Access roles (RBACs)
    • SSO and Authentication
    • LLMs.txt
LogoLogo
Sign inBook a demo
On this page
  • Prerequisites
  • Steps
  • Next steps
How-To GuidesEvaluation

Create a Dataset via the API

In this guide, we will walk through creating a Dataset on Humanloop via the API
Was this page helpful?
Previous

Create a Dataset from existing Logs

In this guide, we will walk through creating a Dataset on Humanloop from your existing Logs.
Next
Built with

Datasets are a collection of input-output pairs that can be used to evaluate your Prompts, Tools or even Evaluators.

Prerequisites

If you are using the SDK, the only prerequisite is to have the SDK installed and configured. If you are using the API directly, you will need to have an API key.

Install and initialize the SDK

First you need to install and initialize the SDK. If you have already done this, skip to the next section.

Open up your terminal and follow these steps:

  1. Install the Humanloop SDK:
1pip install humanloop
  1. Initialize the SDK with your Humanloop API key (you can get it from the Organization Settings page).
1from humanloop import Humanloop
2humanloop = Humanloop(api_key="<YOUR HUMANLOOP KEY>")
3
4# Check that the authentication was successful
5print(humanloop.prompts.list())

Steps

Using the API is a great way to integrate Humanloop with your existing data pipeline or just to once-off upload a more complex Dataset that is hard to represent in a CSV file, such as one that contains an array of messages and JSON targets.

1

Post data to the Datasets API

We first define some sample data that contains user messages and desired responses from our Support Agent Prompt and call the POST /datasets endpoint to upload it as follows:

POST
/v5/datasets
1curl -X POST https://api.humanloop.com/v5/datasets \
2     -H "X-API-KEY: <apiKey>" \
3     -H "Content-Type: application/json" \
4     -d '{
5  "datapoints": [
6    {
7      "messages": [
8        {
9          "role": "user",
10          "content": "How do i manage my organizations API keys?\n"
11        }
12      ],
13      "target": {
14        "response": "Hey, thanks for your questions. Here are steps for how to achieve: 1. Log in to the Humanloop Dashboard \n\n2. Click on \"Organization Settings.\"\n If you do not see this option, you might need to contact your organization admin to gain the necessary permissions.\n\n3. Within the settings or organization settings, select the option labeled \"API Keys\" on the left. Here you will be able to view and manage your API keys.\n\n4. You will see a list of existing API keys. You can perform various actions, such as:\n     - **Generate New API Key:** Click on the \"Generate New Key\" button if you need a new API key.\n     - **Revoke an API Key:** If you need to disable an existing key, find the key in the list and click the \"Revoke\" or \"Delete\" button.\n     - **Copy an API Key:** If you need to use an existing key, you can copy it to your clipboard by clicking the \"Copy\" button next to the key.\n\n5. **Save and Secure API Keys:** Make sure to securely store any new or existing API keys you are using. Treat them like passwords and do not share them publicly.\n\nIf you encounter any issues or need further assistance, it might be helpful to engage with an engineer or your IT department to ensure you have the necessary permissions and support.\n\nWould you need help with anything else?"
15      }
16    },
17    {
18      "messages": [
19        {
20          "role": "user",
21          "content": "Hey, can do I use my code evaluator for monitoring my legal-copilot prompt?"
22        }
23      ],
24      "target": {
25        "response": "Hey, thanks for your questions. Here are steps for how to achieve: 1. Navigate to your Prompt dashboard. \n 2. Select the `Monitoring` button on the top right of the Prompt dashboard \n 3. Within the model select the Version of the Evaluator you want to turn on for monitoring. \n\nWould you need help with anything else?"
26      }
27    }
28  ],
29  "path": "datasets/support-queries",
30  "version_name": "Initial version",
31  "version_description": "Add two new questions and answers"
32}'
Try it
2

Inspect the uploaded Dataset

After running this code, in your Humanloop workspace you will now see a Dataset called Support Query Ground Truth (or whatever value was in path) with your sample data.

Inspect the Dataset uploaded via API.

Next steps

šŸŽ‰ Now that you have Datasets defined in Humanloop, you can leverage our Evaluations feature to systematically measure and improve the performance of your AI applications. See our guides on setting up Evaluators and Running an Evaluation to get started.

For different ways to create datasets, see the links below:

  • Create a Dataset from existing Logs - useful for curating Datasets based on how your AI application has been behaving in the wild.
  • Upload data from CSV - useful for quickly uploading existing tabular data youā€™ve collected outside of Humanloop.