Log to a Prompt

How to log generations from any large language model (LLM) to Humanloop

This guide will show you how to capture the Logs of your LLM calls into Humanloop.

The easiest way to log LLM generations to Humanloop is to use the Prompt.call() method (see the guide on Calling a Prompt). You will only need to supply prompt ID and the inputs needed by the prompt template, and the endpoint will handle fetching the latest template, making the LLM call and logging the result.

However, there may be scenarios that you wish to manage the LLM provider calls directly in your own code instead of relying on Humanloop. For example, you may be using an LLM provider that is not directly supported by Humanloop such as a custom self-hosted model, or you may want to avoid adding Humanloop to the critical path of the LLM API calls.

Prerequisites

  • You already have a Prompt — if not, please follow our Prompt creation guide first.

Log data to your Prompt

To log LLM generations to Humanloop, you will need to make a call to the /prompts/log endpoint.

Note that you can either specify a version of the Prompt you are logging against - in which case you will need to take care that you are supplying the correct version ID and inputs. Or you can supply the full prompt and a new version will be created if it has not been seen before.

1

Get your Prompt

Fetch a Prompt from Humanloop by specifying the ID. You can ignore this step if your prompts are created dynamically in code.

GET
1curl https://api.humanloop.com/v5/prompts/pr_30gco7dx6JDq4200GVOHa \
2     -H "X-API-KEY: <apiKey>"
Response
1{
2  "path": "Personal Projects/Coding Assistant",
3  "id": "pr_30gco7dx6JDq4200GVOHa",
4  "name": "Coding Assistant",
5  "version_id": "prv_7ZlQREDScH0xkhUwtXruN",
6  "type": "prompt",
7  "environments": [
8    {
9      "id": "env_ffSVxEBzJcBZ1H5jcNMVj",
10      "created_at": "2023-06-27T23:16:07.992339",
11      "name": "development",
12      "tag": "default"
13    }
14  ],
15  "created_at": "2024-07-08T22:40:35.656915",
16  "updated_at": "2024-07-08T22:40:35.656915",
17  "created_by": {
18    "id": "usr_01RJO1k2spBVqNUt1ASef",
19    "email_address": "raza@humanloop.com",
20    "full_name": "Raza Habib"
21  },
22  "status": "committed",
23  "last_used_at": "2024-07-08T22:40:35.656915",
24  "model": "gpt-4o",
25  "endpoint": "chat",
26  "template": [
27    {
28      "content": "You are a helpful coding assistant specialising in {{language}}",
29      "role": "system"
30    }
31  ],
32  "provider": "openai",
33  "max_tokens": -1,
34  "temperature": 0.7,
35  "top_p": 1,
36  "presence_penalty": 0,
37  "frequency_penalty": 0,
38  "other": {},
39  "tools": [],
40  "linked_tools": [],
41  "commit_message": "Initial commit",
42  "version_logs_count": 0,
43  "total_logs_count": 0,
44  "inputs": [
45    {
46      "name": "messages"
47    }
48  ]
49}

Here’s how to do this in code:

1import re
2PROMPT_ID = "<YOUR PROMPT ID>"
3prompt = humanloop.prompt.get(id=PROMPT_ID)
4
5# This will fill the prompt template with the variables
6def fill_template(template, variables):
7    def replace_variable(match):
8        variable = match.group(1).strip()
9        if variable in variables:
10            return variables[variable]
11        else:
12            raise ValueError(f"Error: Variable '{variable}' is missing.")
13
14    filled_template = []
15    for message in template:
16        content = message['content']
17        filled_content = re.sub(r'\{\{\s*(.*?)\s*\}\}', replace_variable, content)
18        filled_template.append({**message, 'content': filled_content})
19
20    return filled_template
21
22template = fill_template(prompt.template, {"language": "Python"})
2

Call your Prompt

This can be your own model, or any other LLM provider. Here is an example of calling OpenAI:

1import openai
2
3client = openai.OpenAI(api_key="<YOUR OPENAI API KEY>")
4
5messages = template + [{ "role": "user", "content": "explain how async works" }]
6
7chat_completion = client.chat.completions.create(
8    messages=messages,
9    model=config.model,
10  	temperature=config.temperature
11)
12
13# Parse the output from the OpenAI response.
14output = chat_completion.choices[0].message.content
3

Log the result

Finally, log the result to your project:

POST
1curl -X POST https://api.humanloop.com/v5/prompts/log \
2     -H "X-API-KEY: <apiKey>" \
3     -H "Content-Type: application/json" \
4     -d '{
5  "path": "persona",
6  "output_message": {
7    "content": "Well, you know, there is so much secrecy involved in government, folks, it\'s unbelievable. They don\'t want to tell you everything. They don\'t tell me everything! But about Roswell, it’s a very popular question. I know, I just know, that something very, very peculiar happened there. Was it a weather balloon? Maybe. Was it something extraterrestrial? Could be. I\'d love to go down and open up all the classified documents, believe me, I would. But they don\'t let that happen. The Deep State, folks, the Deep State. They’re unbelievable. They want to keep everything a secret. But whatever the truth is, I can tell you this: it’s something big, very very big. Tremendous, in fact.",
8    "role": "assistant"
9  },
10  "prompt_tokens": 100,
11  "output_tokens": 220,
12  "prompt_cost": 0.00001,
13  "output_cost": 0.0002,
14  "finish_reason": "stop",
15  "messages": [
16    {
17      "role": "user",
18      "content": "What really happened at Roswell?"
19    }
20  ],
21  "prompt": {
22    "model": "gpt-4",
23    "template": [
24      {
25        "role": "system",
26        "content": "You are {{person}}. Answer questions as this person. Do not break character."
27      }
28    ]
29  },
30  "created_at": "2024-07-19T00:29:35.178992",
31  "provider_latency": 6.5931549072265625,
32  "inputs": {
33    "person": "Trump"
34  }
35}'
1# Get the output from the OpenAI response.
2output_message = chat_completion.choices[0].message
3
4# Log the inputs, outputs and config to your project.
5log = humanloop.prompts.log(
6    id=PROMPT_ID,
7    output_message=output_message,
8    messages=messages,
9)