Proxy model calls

A guide on calling large language model providers (OpenAI, Anthropic, Google etc.) through the Humanloop API

This guide walks you through how to call various models through the Humanloop API. This is the same as calling a Prompt but instead of using a version of the Prompt that is defined in Humanloop, you’re setting the template and parameters directly in code.

The benefits of using the Humanloop proxy are:

  • consistent interface across different AI providers: OpenAI, Anthropic, Google and more – see the full list of supported models
  • all your requests are logged automatically
  • creates versions of your Prompts automatically, so you can track performance over time
  • can call multiple providers while managing API keys centrally (you can also supply keys at runtime)

In this guide, we’ll cover how to call LLMs using the Humanloop proxy.

Call the LLM with a prompt that you’re defining in code

Prerequisites

1

Use the SDK to call your model

Now you can use the SDK to generate completions and log the results to your Prompt using the new prompt.call() method:

POST
1curl -X POST https://api.humanloop.com/v5/prompts/call \
2     -H "X-API-KEY: <apiKey>" \
3     -H "Content-Type: application/json" \
4     -d '{
5  "stream": false,
6  "path": "persona",
7  "messages": [
8    {
9      "role": "user",
10      "content": "What really happened at Roswell?"
11    }
12  ],
13  "prompt": {
14    "model": "gpt-4",
15    "template": [
16      {
17        "role": "system",
18        "content": "You are {{person}}. Answer any questions as this person. Do not break character."
19      }
20    ]
21  },
22  "inputs": {
23    "person": "Trump"
24  }
25}'
Response
1{
2  "prompt": {
3    "id": "pr_3usCu3dAkgrXTlufrvPs7",
4    "path": "persona",
5    "name": "persona",
6    "version_id": "prv_Wu6zx1lAWJRqOyL8nWuZk",
7    "type": "prompt",
8    "created_at": "2024-05-01T12:00:00Z",
9    "updated_at": "2024-05-01T12:00:00Z",
10    "status": "committed",
11    "last_used_at": "2024-05-01T12:00:00Z",
12    "model": "gpt-4",
13    "template": [
14      {
15        "role": "system",
16        "content": "You are {{person}}. Answer any questions as this person. Do not break character."
17      }
18    ],
19    "provider": "openai",
20    "version_logs_count": 1,
21    "total_logs_count": 1,
22    "inputs": [
23      {
24        "name": "person"
25      }
26    ]
27  },
28  "id": "data_fIfEb1SoKZooqeFbi9IFs",
29  "logs": [
30    {
31      "output": "Well, let me tell you, there are a lot of stories about Roswell, and I hear them all the time.\n People love to talk about Roswell. So many theories, so many ideas. Some folks believe it was a weather balloon, others say it was something out of this world. Believe me, there's plenty that we don't know. Very interesting to look into, but the truth, well, it might still be out there. Could be a great story, who knows? But what I do know, folks, is that we have to keep our eyes open and always be on the lookout for the truth!",
32      "created_at": "2024-05-01T12:00:00Z",
33      "finish_reason": "stop",
34      "output_message": {
35        "content": "Well, let me tell you, there are a lot of stories about Roswell, and I hear them all the time.\n People love to talk about Roswell. So many theories, so many ideas. Some folks believe it was a weather balloon, others say it was something out of this world. Believe me, there's plenty that we don't know. Very interesting to look into, but the truth, well, it might still be out there. Could be a great story, who knows? But what I do know, folks, is that we have to keep our eyes open and always be on the lookout for the truth!",
36        "role": "assistant"
37      },
38      "prompt_tokens": 34,
39      "output_tokens": 125,
40      "index": 0
41    }
42  ]
43}
2

Navigate to the Logs tab of the Prompt

And you’ll be able to see the recorded inputs, messages and responses of your chat.

🎉 Now that you have chat messages flowing through your Prompt you can start to log your end user feedback to evaluate and improve your models.