> This page is for version v5.0 (default). > For other versions, use one of these documentation indexes: > - v5.0 (default): https://humanloop.com/docs/v5/llms.txt > - v4.0: https://humanloop.com/docs/v4/llms.txt > For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://humanloop.com/docs/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://humanloop.com/docs/_mcp/server. > Learn how to create Datasets in Humanloop to define fixed examples for your projects, and build up a collection of input-output pairs for evaluation and fine-tuning. [Datasets](/docs/explanation/datasets) are a collection of input-output pairs that can be used to evaluate your Prompts, Tools or even Evaluators. ### Prerequisites If you are using the SDK, the only prerequisite is to have the SDK installed and configured. If you are using the API directly, you will need to have an API key. #### Install and initialize the SDK First you need to install and initialize the SDK. If you have already done this, skip to the next section. Open up your terminal and follow these steps: 1. Install the Humanloop SDK: ```python pip install humanloop ``` ```typescript npm install humanloop ``` 2. Initialize the SDK with your Humanloop API key (you can get it from the [Organization Settings page](https://app.humanloop.com/account/api-keys)). ```python from humanloop import Humanloop humanloop = Humanloop(api_key="") # Check that the authentication was successful print(humanloop.prompts.list()) ``` ```typescript import { HumanloopClient, Humanloop } from "humanloop"; const humanloop = new HumanloopClient({ apiKey: "YOUR_API_KEY" }); // Check that the authentication was successful console.log(await humanloop.prompts.list()); ``` ## Steps Using the API is a great way to integrate Humanloop with your existing data pipeline or just to once-off upload a more complex Dataset that is hard to represent in a CSV file, such as one that contains an array of messages and JSON targets. ### Post data to the Datasets API We first define some sample data that contains user messages and desired responses from our [Support Agent Prompt](./create-dataset-from-logs) and call the `POST /datasets` endpoint to upload it as follows: ### Request POST [https://api.humanloop.com/v5/datasets](https://api.humanloop.com/v5/datasets) **`CreateSupportDataset`** ```curl CreateSupportDataset curl -X POST https://api.humanloop.com/v5/datasets \ -H "X-API-KEY: " \ -H "Content-Type: application/json" \ -d '{ "datapoints": [ { "messages": [ { "role": "user", "content": "How do i manage my organizations API keys?\n" } ], "target": { "response": "Hey, thanks for your questions. Here are steps for how to achieve: 1. Log in to the Humanloop Dashboard \n\n2. Click on \"Organization Settings.\"\n If you do not see this option, you might need to contact your organization admin to gain the necessary permissions.\n\n3. Within the settings or organization settings, select the option labeled \"API Keys\" on the left. Here you will be able to view and manage your API keys.\n\n4. You will see a list of existing API keys. You can perform various actions, such as:\n - **Generate New API Key:** Click on the \"Generate New Key\" button if you need a new API key.\n - **Revoke an API Key:** If you need to disable an existing key, find the key in the list and click the \"Revoke\" or \"Delete\" button.\n - **Copy an API Key:** If you need to use an existing key, you can copy it to your clipboard by clicking the \"Copy\" button next to the key.\n\n5. **Save and Secure API Keys:** Make sure to securely store any new or existing API keys you are using. Treat them like passwords and do not share them publicly.\n\nIf you encounter any issues or need further assistance, it might be helpful to engage with an engineer or your IT department to ensure you have the necessary permissions and support.\n\nWould you need help with anything else?" } }, { "messages": [ { "role": "user", "content": "Hey, can do I use my code evaluator for monitoring my legal-copilot prompt?" } ], "target": { "response": "Hey, thanks for your questions. Here are steps for how to achieve: 1. Navigate to your Prompt dashboard. \n 2. Select the `Monitoring` button on the top right of the Prompt dashboard \n 3. Within the model select the Version of the Evaluator you want to turn on for monitoring. \n\nWould you need help with anything else?" } } ], "path": "datasets/support-queries", "version_name": "Initial version", "version_description": "Add two new questions and answers" }' ``` **`CreateSupportDataset`** ```python CreateSupportDataset import requests url = "https://api.humanloop.com/v5/datasets" payload = { "datapoints": [ { "messages": [ { "role": "user", "content": "How do i manage my organizations API keys? " } ], "target": { "response": "Hey, thanks for your questions. Here are steps for how to achieve: 1. Log in to the Humanloop Dashboard 2. Click on \"Organization Settings.\" If you do not see this option, you might need to contact your organization admin to gain the necessary permissions. 3. Within the settings or organization settings, select the option labeled \"API Keys\" on the left. Here you will be able to view and manage your API keys. 4. You will see a list of existing API keys. You can perform various actions, such as: - **Generate New API Key:** Click on the \"Generate New Key\" button if you need a new API key. - **Revoke an API Key:** If you need to disable an existing key, find the key in the list and click the \"Revoke\" or \"Delete\" button. - **Copy an API Key:** If you need to use an existing key, you can copy it to your clipboard by clicking the \"Copy\" button next to the key. 5. **Save and Secure API Keys:** Make sure to securely store any new or existing API keys you are using. Treat them like passwords and do not share them publicly. If you encounter any issues or need further assistance, it might be helpful to engage with an engineer or your IT department to ensure you have the necessary permissions and support. Would you need help with anything else?" } }, { "messages": [ { "role": "user", "content": "Hey, can do I use my code evaluator for monitoring my legal-copilot prompt?" } ], "target": { "response": "Hey, thanks for your questions. Here are steps for how to achieve: 1. Navigate to your Prompt dashboard. 2. Select the `Monitoring` button on the top right of the Prompt dashboard 3. Within the model select the Version of the Evaluator you want to turn on for monitoring. Would you need help with anything else?" } } ], "path": "datasets/support-queries", "version_name": "Initial version", "version_description": "Add two new questions and answers" } headers = { "X-API-KEY": "", "Content-Type": "application/json" } response = requests.post(url, json=payload, headers=headers) print(response.json()) ``` **`CreateSupportDataset`** ```typescript CreateSupportDataset import { HumanloopClient } from "humanloop"; const client = new HumanloopClient({ apiKey: "YOUR_API_KEY" }); await client.datasets.upsert({ path: "datasets/support-queries", datapoints: [{ messages: [{ role: "user", content: "How do i manage my organizations API keys?\n" }], target: { "response": "Hey, thanks for your questions. Here are steps for how to achieve: 1. Log in to the Humanloop Dashboard \n\n2. Click on \"Organization Settings.\"\n If you do not see this option, you might need to contact your organization admin to gain the necessary permissions.\n\n3. Within the settings or organization settings, select the option labeled \"API Keys\" on the left. Here you will be able to view and manage your API keys.\n\n4. You will see a list of existing API keys. You can perform various actions, such as:\n - **Generate New API Key:** Click on the \"Generate New Key\" button if you need a new API key.\n - **Revoke an API Key:** If you need to disable an existing key, find the key in the list and click the \"Revoke\" or \"Delete\" button.\n - **Copy an API Key:** If you need to use an existing key, you can copy it to your clipboard by clicking the \"Copy\" button next to the key.\n\n5. **Save and Secure API Keys:** Make sure to securely store any new or existing API keys you are using. Treat them like passwords and do not share them publicly.\n\nIf you encounter any issues or need further assistance, it might be helpful to engage with an engineer or your IT department to ensure you have the necessary permissions and support.\n\nWould you need help with anything else?" } }, { messages: [{ role: "user", content: "Hey, can do I use my code evaluator for monitoring my legal-copilot prompt?" }], target: { "response": "Hey, thanks for your questions. Here are steps for how to achieve: 1. Navigate to your Prompt dashboard. \n 2. Select the `Monitoring` button on the top right of the Prompt dashboard \n 3. Within the model select the Version of the Evaluator you want to turn on for monitoring. \n\nWould you need help with anything else?" } }], versionName: "Initial version", versionDescription: "Add two new questions and answers" }); ``` **`CreateSupportDataset`** ```go CreateSupportDataset package main import ( "fmt" "strings" "net/http" "io" ) func main() { url := "https://api.humanloop.com/v5/datasets" payload := strings.NewReader("{\n \"datapoints\": [\n {\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"How do i manage my organizations API keys?\\n\"\n }\n ],\n \"target\": {\n \"response\": \"Hey, thanks for your questions. Here are steps for how to achieve: 1. Log in to the Humanloop Dashboard \\n\\n2. Click on \\\"Organization Settings.\\\"\\n If you do not see this option, you might need to contact your organization admin to gain the necessary permissions.\\n\\n3. Within the settings or organization settings, select the option labeled \\\"API Keys\\\" on the left. Here you will be able to view and manage your API keys.\\n\\n4. You will see a list of existing API keys. You can perform various actions, such as:\\n - **Generate New API Key:** Click on the \\\"Generate New Key\\\" button if you need a new API key.\\n - **Revoke an API Key:** If you need to disable an existing key, find the key in the list and click the \\\"Revoke\\\" or \\\"Delete\\\" button.\\n - **Copy an API Key:** If you need to use an existing key, you can copy it to your clipboard by clicking the \\\"Copy\\\" button next to the key.\\n\\n5. **Save and Secure API Keys:** Make sure to securely store any new or existing API keys you are using. Treat them like passwords and do not share them publicly.\\n\\nIf you encounter any issues or need further assistance, it might be helpful to engage with an engineer or your IT department to ensure you have the necessary permissions and support.\\n\\nWould you need help with anything else?\"\n }\n },\n {\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"Hey, can do I use my code evaluator for monitoring my legal-copilot prompt?\"\n }\n ],\n \"target\": {\n \"response\": \"Hey, thanks for your questions. Here are steps for how to achieve: 1. Navigate to your Prompt dashboard. \\n 2. Select the `Monitoring` button on the top right of the Prompt dashboard \\n 3. Within the model select the Version of the Evaluator you want to turn on for monitoring. \\n\\nWould you need help with anything else?\"\n }\n }\n ],\n \"path\": \"datasets/support-queries\",\n \"version_name\": \"Initial version\",\n \"version_description\": \"Add two new questions and answers\"\n}") req, _ := http.NewRequest("POST", url, payload) req.Header.Add("X-API-KEY", "") req.Header.Add("Content-Type", "application/json") res, _ := http.DefaultClient.Do(req) defer res.Body.Close() body, _ := io.ReadAll(res.Body) fmt.Println(res) fmt.Println(string(body)) } ``` **`CreateSupportDataset`** ```ruby CreateSupportDataset require 'uri' require 'net/http' url = URI("https://api.humanloop.com/v5/datasets") http = Net::HTTP.new(url.host, url.port) http.use_ssl = true request = Net::HTTP::Post.new(url) request["X-API-KEY"] = '' request["Content-Type"] = 'application/json' request.body = "{\n \"datapoints\": [\n {\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"How do i manage my organizations API keys?\\n\"\n }\n ],\n \"target\": {\n \"response\": \"Hey, thanks for your questions. Here are steps for how to achieve: 1. Log in to the Humanloop Dashboard \\n\\n2. Click on \\\"Organization Settings.\\\"\\n If you do not see this option, you might need to contact your organization admin to gain the necessary permissions.\\n\\n3. Within the settings or organization settings, select the option labeled \\\"API Keys\\\" on the left. Here you will be able to view and manage your API keys.\\n\\n4. You will see a list of existing API keys. You can perform various actions, such as:\\n - **Generate New API Key:** Click on the \\\"Generate New Key\\\" button if you need a new API key.\\n - **Revoke an API Key:** If you need to disable an existing key, find the key in the list and click the \\\"Revoke\\\" or \\\"Delete\\\" button.\\n - **Copy an API Key:** If you need to use an existing key, you can copy it to your clipboard by clicking the \\\"Copy\\\" button next to the key.\\n\\n5. **Save and Secure API Keys:** Make sure to securely store any new or existing API keys you are using. Treat them like passwords and do not share them publicly.\\n\\nIf you encounter any issues or need further assistance, it might be helpful to engage with an engineer or your IT department to ensure you have the necessary permissions and support.\\n\\nWould you need help with anything else?\"\n }\n },\n {\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"Hey, can do I use my code evaluator for monitoring my legal-copilot prompt?\"\n }\n ],\n \"target\": {\n \"response\": \"Hey, thanks for your questions. Here are steps for how to achieve: 1. Navigate to your Prompt dashboard. \\n 2. Select the `Monitoring` button on the top right of the Prompt dashboard \\n 3. Within the model select the Version of the Evaluator you want to turn on for monitoring. \\n\\nWould you need help with anything else?\"\n }\n }\n ],\n \"path\": \"datasets/support-queries\",\n \"version_name\": \"Initial version\",\n \"version_description\": \"Add two new questions and answers\"\n}" response = http.request(request) puts response.read_body ``` **`CreateSupportDataset`** ```java CreateSupportDataset import com.mashape.unirest.http.HttpResponse; import com.mashape.unirest.http.Unirest; HttpResponse response = Unirest.post("https://api.humanloop.com/v5/datasets") .header("X-API-KEY", "") .header("Content-Type", "application/json") .body("{\n \"datapoints\": [\n {\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"How do i manage my organizations API keys?\\n\"\n }\n ],\n \"target\": {\n \"response\": \"Hey, thanks for your questions. Here are steps for how to achieve: 1. Log in to the Humanloop Dashboard \\n\\n2. Click on \\\"Organization Settings.\\\"\\n If you do not see this option, you might need to contact your organization admin to gain the necessary permissions.\\n\\n3. Within the settings or organization settings, select the option labeled \\\"API Keys\\\" on the left. Here you will be able to view and manage your API keys.\\n\\n4. You will see a list of existing API keys. You can perform various actions, such as:\\n - **Generate New API Key:** Click on the \\\"Generate New Key\\\" button if you need a new API key.\\n - **Revoke an API Key:** If you need to disable an existing key, find the key in the list and click the \\\"Revoke\\\" or \\\"Delete\\\" button.\\n - **Copy an API Key:** If you need to use an existing key, you can copy it to your clipboard by clicking the \\\"Copy\\\" button next to the key.\\n\\n5. **Save and Secure API Keys:** Make sure to securely store any new or existing API keys you are using. Treat them like passwords and do not share them publicly.\\n\\nIf you encounter any issues or need further assistance, it might be helpful to engage with an engineer or your IT department to ensure you have the necessary permissions and support.\\n\\nWould you need help with anything else?\"\n }\n },\n {\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"Hey, can do I use my code evaluator for monitoring my legal-copilot prompt?\"\n }\n ],\n \"target\": {\n \"response\": \"Hey, thanks for your questions. Here are steps for how to achieve: 1. Navigate to your Prompt dashboard. \\n 2. Select the `Monitoring` button on the top right of the Prompt dashboard \\n 3. Within the model select the Version of the Evaluator you want to turn on for monitoring. \\n\\nWould you need help with anything else?\"\n }\n }\n ],\n \"path\": \"datasets/support-queries\",\n \"version_name\": \"Initial version\",\n \"version_description\": \"Add two new questions and answers\"\n}") .asString(); ``` **`CreateSupportDataset`** ```php CreateSupportDataset request('POST', 'https://api.humanloop.com/v5/datasets', [ 'body' => '{ "datapoints": [ { "messages": [ { "role": "user", "content": "How do i manage my organizations API keys?\\n" } ], "target": { "response": "Hey, thanks for your questions. Here are steps for how to achieve: 1. Log in to the Humanloop Dashboard \\n\\n2. Click on \\"Organization Settings.\\"\\n If you do not see this option, you might need to contact your organization admin to gain the necessary permissions.\\n\\n3. Within the settings or organization settings, select the option labeled \\"API Keys\\" on the left. Here you will be able to view and manage your API keys.\\n\\n4. You will see a list of existing API keys. You can perform various actions, such as:\\n - **Generate New API Key:** Click on the \\"Generate New Key\\" button if you need a new API key.\\n - **Revoke an API Key:** If you need to disable an existing key, find the key in the list and click the \\"Revoke\\" or \\"Delete\\" button.\\n - **Copy an API Key:** If you need to use an existing key, you can copy it to your clipboard by clicking the \\"Copy\\" button next to the key.\\n\\n5. **Save and Secure API Keys:** Make sure to securely store any new or existing API keys you are using. Treat them like passwords and do not share them publicly.\\n\\nIf you encounter any issues or need further assistance, it might be helpful to engage with an engineer or your IT department to ensure you have the necessary permissions and support.\\n\\nWould you need help with anything else?" } }, { "messages": [ { "role": "user", "content": "Hey, can do I use my code evaluator for monitoring my legal-copilot prompt?" } ], "target": { "response": "Hey, thanks for your questions. Here are steps for how to achieve: 1. Navigate to your Prompt dashboard. \\n 2. Select the `Monitoring` button on the top right of the Prompt dashboard \\n 3. Within the model select the Version of the Evaluator you want to turn on for monitoring. \\n\\nWould you need help with anything else?" } } ], "path": "datasets/support-queries", "version_name": "Initial version", "version_description": "Add two new questions and answers" }', 'headers' => [ 'Content-Type' => 'application/json', 'X-API-KEY' => '', ], ]); echo $response->getBody(); ``` **`CreateSupportDataset`** ```csharp CreateSupportDataset using RestSharp; var client = new RestClient("https://api.humanloop.com/v5/datasets"); var request = new RestRequest(Method.POST); request.AddHeader("X-API-KEY", ""); request.AddHeader("Content-Type", "application/json"); request.AddParameter("application/json", "{\n \"datapoints\": [\n {\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"How do i manage my organizations API keys?\\n\"\n }\n ],\n \"target\": {\n \"response\": \"Hey, thanks for your questions. Here are steps for how to achieve: 1. Log in to the Humanloop Dashboard \\n\\n2. Click on \\\"Organization Settings.\\\"\\n If you do not see this option, you might need to contact your organization admin to gain the necessary permissions.\\n\\n3. Within the settings or organization settings, select the option labeled \\\"API Keys\\\" on the left. Here you will be able to view and manage your API keys.\\n\\n4. You will see a list of existing API keys. You can perform various actions, such as:\\n - **Generate New API Key:** Click on the \\\"Generate New Key\\\" button if you need a new API key.\\n - **Revoke an API Key:** If you need to disable an existing key, find the key in the list and click the \\\"Revoke\\\" or \\\"Delete\\\" button.\\n - **Copy an API Key:** If you need to use an existing key, you can copy it to your clipboard by clicking the \\\"Copy\\\" button next to the key.\\n\\n5. **Save and Secure API Keys:** Make sure to securely store any new or existing API keys you are using. Treat them like passwords and do not share them publicly.\\n\\nIf you encounter any issues or need further assistance, it might be helpful to engage with an engineer or your IT department to ensure you have the necessary permissions and support.\\n\\nWould you need help with anything else?\"\n }\n },\n {\n \"messages\": [\n {\n \"role\": \"user\",\n \"content\": \"Hey, can do I use my code evaluator for monitoring my legal-copilot prompt?\"\n }\n ],\n \"target\": {\n \"response\": \"Hey, thanks for your questions. Here are steps for how to achieve: 1. Navigate to your Prompt dashboard. \\n 2. Select the `Monitoring` button on the top right of the Prompt dashboard \\n 3. Within the model select the Version of the Evaluator you want to turn on for monitoring. \\n\\nWould you need help with anything else?\"\n }\n }\n ],\n \"path\": \"datasets/support-queries\",\n \"version_name\": \"Initial version\",\n \"version_description\": \"Add two new questions and answers\"\n}", ParameterType.RequestBody); IRestResponse response = client.Execute(request); ``` **`CreateSupportDataset`** ```swift CreateSupportDataset import Foundation let headers = [ "X-API-KEY": "", "Content-Type": "application/json" ] let parameters = [ "datapoints": [ [ "messages": [ [ "role": "user", "content": "How do i manage my organizations API keys? " ] ], "target": ["response": "Hey, thanks for your questions. Here are steps for how to achieve: 1. Log in to the Humanloop Dashboard 2. Click on \"Organization Settings.\" If you do not see this option, you might need to contact your organization admin to gain the necessary permissions. 3. Within the settings or organization settings, select the option labeled \"API Keys\" on the left. Here you will be able to view and manage your API keys. 4. You will see a list of existing API keys. You can perform various actions, such as: - **Generate New API Key:** Click on the \"Generate New Key\" button if you need a new API key. - **Revoke an API Key:** If you need to disable an existing key, find the key in the list and click the \"Revoke\" or \"Delete\" button. - **Copy an API Key:** If you need to use an existing key, you can copy it to your clipboard by clicking the \"Copy\" button next to the key. 5. **Save and Secure API Keys:** Make sure to securely store any new or existing API keys you are using. Treat them like passwords and do not share them publicly. If you encounter any issues or need further assistance, it might be helpful to engage with an engineer or your IT department to ensure you have the necessary permissions and support. Would you need help with anything else?"] ], [ "messages": [ [ "role": "user", "content": "Hey, can do I use my code evaluator for monitoring my legal-copilot prompt?" ] ], "target": ["response": "Hey, thanks for your questions. Here are steps for how to achieve: 1. Navigate to your Prompt dashboard. 2. Select the `Monitoring` button on the top right of the Prompt dashboard 3. Within the model select the Version of the Evaluator you want to turn on for monitoring. Would you need help with anything else?"] ] ], "path": "datasets/support-queries", "version_name": "Initial version", "version_description": "Add two new questions and answers" ] as [String : Any] let postData = JSONSerialization.data(withJSONObject: parameters, options: []) let request = NSMutableURLRequest(url: NSURL(string: "https://api.humanloop.com/v5/datasets")! as URL, cachePolicy: .useProtocolCachePolicy, timeoutInterval: 10.0) request.httpMethod = "POST" request.allHTTPHeaderFields = headers request.httpBody = postData as Data let session = URLSession.shared let dataTask = session.dataTask(with: request as URLRequest, completionHandler: { (data, response, error) -> Void in if (error != nil) { print(error as Any) } else { let httpResponse = response as? HTTPURLResponse print(httpResponse) } }) dataTask.resume() ``` ### Inspect the uploaded Dataset After running this code, in your Humanloop workspace you will now see a Dataset called `Support Query Ground Truth` (or whatever value was in `path`) with your sample data. ![Inspect the Dataset uploaded via API.](/docs/_fern-img/8bc80016b97eca457bd3d7f469123d4b86e121d66a067772dec78c5060310c3d.webp) ## Next steps 🎉 Now that you have Datasets defined in Humanloop, you can leverage our [Evaluations](./overview) feature to systematically measure and improve the performance of your AI applications. See our guides on [setting up Evaluators](./llm-as-a-judge) and [Running an Evaluation](/docs/guides/evals/run-evaluation-ui) to get started. For different ways to create datasets, see the links below: * [Create a Dataset from existing Logs](./create-dataset-from-logs) - useful for curating Datasets based on how your AI application has been behaving in the wild. * [Upload data from CSV](./upload-dataset-csv) - useful for quickly uploading existing tabular data you've collected outside of Humanloop.