For AI agents: a documentation index is available at the root level at /llms.txt and /llms-full.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page.
Sign inBook a demo
DocsReferenceChangelog
DocsReferenceChangelog
  • Getting Started
    • Overview
    • Quickstart
      • Evals in code
      • Agent Evals in the UI
      • Create Prompt in the UI
      • Set up logging
  • Explanation
    • Integrating Humanloop
  • Tutorials
    • Evaluate an Agent in the UI
    • Evaluate an Agent in code
    • Evaluate a RAG app
    • Capture user feedback
  • How-To Guides
    • Migrating from Humanloop
  • Reference
    • Deployment Options
    • Supported Models
    • Template Library
    • Vercel AI SDK
    • .prompt and .agent Files
    • Humanloop Runtime Environment
    • Security and Compliance
    • Data Management
    • Access roles (RBACs)
    • SSO and Authentication
    • LLMs.txt
LogoLogo
Sign inBook a demo
On this page
  • Prerequisites
  • Create the chat agent
  • Log to Humanloop
  • Run the code
  • Check your workspace
  • Change the agent and rerun
  • Next steps
Getting StartedQuickstart

Set up logging

Log an AI project to Humanloop.
Was this page helpful?
Previous

Vercel AI SDK

Add Humanloop observability to your Vercel AI SDK project.
Next
Built with

This quickstart takes a chat agent and adds Humanloop logging to it so you can observe and reason about its behavior.

Prerequisites

Account setup

Create a Humanloop Account

If you haven’t already, create an account or log in to Humanloop

Add an OpenAI API Key

If you’re the first person in your organization, you’ll need to add an API key to a model provider.

  1. Go to OpenAI and grab an API key.
  2. In Humanloop Organization Settings set up OpenAI as a model provider.

Using the Prompt Editor will use your OpenAI credits in the same way that the OpenAI playground does. Keep your API keys for Humanloop and the model providers private.

Install dependencies
1pip install humanloop openai

Create the chat agent

To demonstrate how to add logging, we will start with a chat agent that answers math and science questions.

Create a script and add the following:

1import json
2
3from humanloop import Humanloop
4from openai import OpenAI
5
6openai = OpenAI(api_key="<YOUR_OPENAI_KEY>")
7humanloop = Humanloop(api_key="<YOUR_HUMANLOOP_KEY>")
8
9
10def calculator(operation: str, num1: int, num2: int) -> str:
11 """Do arithmetic operations on two numbers."""
12 if operation == "add":
13 return num1 + num2
14 elif operation == "subtract":
15 return num1 - num2
16 elif operation == "multiply":
17 return num1 * num2
18 elif operation == "divide":
19 return num1 / num2
20 else:
21 return "Invalid operation"
22
23
24def call_model(messages: list[str]) -> str:
25 output = openai.chat.completions.create(
26 messages=messages,
27 model="gpt-4o-mini",
28 tools=[{
29 "type": "function",
30 "function": {
31 'name': 'calculator',
32 'description': 'Do arithmetic operations on two numbers.',
33 'parameters': {
34 'type': 'object',
35 'required': ['operation', 'num1', 'num2'],
36 'properties': {
37 'operation': {'type': 'string'},
38 'num1': {'type': 'integer'},
39 'num2': {'type': 'integer'}
40 },
41 'additionalProperties': False
42 },
43 },
44 }],
45 temperature=0.7,
46 )
47
48 # Check if model asked for a tool call
49 if output.choices[0].message.tool_calls:
50 for tool_call in output.choices[0].message.tool_calls:
51 arguments = json.loads(tool_call.function.arguments)
52 if tool_call.function.name == "calculator":
53 result = calculator(**arguments)
54 return f"[TOOL CALL] {result}"
55
56 # Otherwise, return the LLM response
57 return output.choices[0].message.content
58
59
60def conversation():
61 messages = [
62 {
63 "role": "system",
64 "content": "You are a a groovy 80s surfer dude "
65 "helping with math and science."
66 },
67 ]
68 while True:
69 user_input = input("You: ")
70 if user_input == "exit":
71 break
72 messages.append({"role": "user", "content": user_input})
73 response = call_model(messages=messages)
74 messages.append({"role": "assistant", "content": response})
75 print(f"Agent: {response}")
76
77
78if __name__ == "__main__":
79 conversation()

Log to Humanloop

If you use a programming language not supported by the SDK, or want more control, see our guide on logging through the API for an alternative to decorators.

Use the SDK decorators to enable logging. At runtime, every call to a decorated function will create a Log on Humanloop.

1@humanloop.tool(path="Logging Quickstart/Calculator")
2def calculator(operation: str, num1: int, num2: int) -> str:
3 ...
4
5@humanloop.prompt(path="Logging Quickstart/QA Prompt")
6def call_model(messages: list[str]) -> str:
7 ...
8
9
10@humanloop.flow(path="Logging Quickstart/QA Agent")
11def conversation():
12 ...
13
14if __name__ == "__main__":
15 conversation()

Run the code

Have a conversation with the agent. When you’re done, type exit to close the program.

1> python main.py
2You: Hi dude!
3Agent: Tubular! I am here to help with math and science, what is groovin?
4You: How does flying work?
5Agent: ...
6You: What is 5678 * 456?
7Agent: [TOOL CALL] 2587968
8You: exit

Check your workspace

Navigate to your workspace to see the logged conversation.

Inside the Logging Quickstart directory on the left, click the QA Agent Flow. Select the Logs tab from the top of the page and click the Log inside the table.

You will see the conversation’s trace, containing Logs corresponding to the Tool and the Prompt.

Change the agent and rerun

Modify the call_model function to use a different model and temperature.

1@humanloop.prompt(path="Logging Quickstart/QA Prompt")
2def call_model(messages: list[str]) -> str:
3 output = openai.chat.completions.create(
4 messages=messages,
5 model="gpt-4o-mini",
6 tools=[
7 # The @tool utility adds a .json_schema attribute
8 # to avoid redefining the schema
9 calculator.json_schema
10 ],
11 temperature=0.2,
12 )
13
14 # Check if model asked for a tool call
15 if output.choices[0].message.tool_calls:
16 for tool_call in output.choices[0].message.tool_calls:
17 arguments = json.loads(tool_call.function.arguments)
18 if tool_call.function.name == "calculator":
19 result = calculator(**arguments)
20 return f"[TOOL CALL] {result}"
21
22 # Otherwise, return the LLM response
23 return output.choices[0].message.content

Run the agent again, then head back to your workspace.

Click the QA Prompt Prompt, select the Dashboard tab from the top of the page and you should find a new version of the Prompt at the top of the list.

By changing the hyperparameters of the OpenAI call, you have tagged a new version of the Prompt.

Next steps

Logging is the first step to observing your AI product. Follow up with these guides on monitoring and evals:

  • Add monitoring Evaluators to evaluate Logs as they’re made against a File.

  • Explore evals to improve the performance of your AI feature in our guide on running an Evaluation.

  • See logging in action on a complex example in our tutorial on evaluating an agent.