Set up Webhooks

In this guide, we will demonstrate how to set up webhooks via API for alerting on your monitoring evaluators.

Under Development

This content is currently under development. Please refer to our V4 documentation for the current docs.

Paid Feature

This feature is not available for the Free tier. Please contact us if you wish to learn more about our Enterprise plan

In this guide, we’ll walk you through the process of setting up webhooks using the Humanloop API to notify you in Slack when certain events occur with your monitoring evaluators.

Prerequisites

Before you begin, make sure you have:

  • A Humanloop account with API access
  • A Slack workspace where you have permissions to add webhooks
  • A Humanloop project with at least one LLM model and monitoring evaluator set up

Setting up a webhook

To set up a webhook, you’ll use the hl.webhook.create() method from the Humanloop Python SDK. Here’s a step-by-step guide:

1

Create a Slack incoming webhook

  1. Go to your Slack workspace and create a new Slack app (or use an existing one).
  2. Under “Add features and functionality”, choose “Incoming Webhooks” and activate them.
  3. Click “Add New Webhook to Workspace” and choose the channel where you want to receive notifications.
  4. Copy the webhook URL provided by Slack.
2

Import the Humanloop SDK and initialize the client

1import humanloop as hl
2
3hl.init(api_key="your-api-key")

Replace "your-api-key" with your actual Humanloop API key.

3

Create a webhook

1webhook = hl.webhook.create(
2    url="https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK",
3    description="Webhook for monitoring evaluator alerts",
4    events=["EVALUATION_COMPLETED", "DRIFT_DETECTED"],
5    model_name="your-model-name",
6    status="ACTIVE",
7    http_url_spec={
8        "secret": "your-shared-secret"
9    }
10)

Replace the following:

  • "https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK" with your Slack webhook URL
  • "your-model-name" with the name of the model you want to monitor
  • "your-shared-secret" with a secret string of your choice for added security
4

Test the webhook

To test if your webhook is working correctly, you can trigger an evaluation:

1evaluation_run = hl.evaluations.create(
2    project_id=PROJECT_ID,
3    config_id=CONFIG_ID,
4    dataset_id=DATASET_ID,
5    evaluator_ids=[EVALUATOR_ID],
6    hl_generated=False,
7)

Replace "your-project-id" and "your-model-name" with your actual project ID and model name.

Verifying the webhook

After setting up the webhook and triggering an evaluation, you should see a message in your specified Slack channel. The message will contain details about the evaluation event, such as:

New event: EVALUATION_COMPLETED
Model: your-model-name
Timestamp: 2023-07-29T12:34:56Z
Evaluation ID: eval_123456
Result: Pass/Fail

Managing webhooks

You can list, update, or delete webhooks using the following methods:

1# List all webhooks
2webhooks = hl.webhook.list()
3
4# Update a webhook
5updated_webhook = hl.webhook.update(
6    id="webhook-id",
7    description="Updated description",
8    status="DISABLED"
9)
10
11# Delete a webhook
12hl.webhook.delete(id="webhook-id")

Replace "webhook-id" with the ID of the webhook you want to manage.

Conclusion

You’ve now set up a webhook to receive notifications in Slack when your monitoring evaluators complete evaluations or detect drift. This will help you stay informed about the performance and behavior of your LLM models in real-time.