Capture user feedback

In this guide, we show how to record end-user feedback using the Humanloop Python SDK. This allows you to monitor how your generations perform with your users.

This guide shows how to use the Humanloop SDK to record end-user feedback on Logs.

Different use-cases and user interfaces may require different kinds of feedback that need to be mapped to the appropriate end user interaction. There are broadly 3 important kinds of feedback:

  1. Explicit feedback: these are purposeful actions to review the generations. For example, ‘thumbs up/down’ button presses.
  2. Implicit feedback: indirect actions taken by your users may signal whether the generation was good or bad, for example, whether the user ‘copied’ the generation, ‘saved it’ or ‘dismissed it’ (which is negative feedback).
  3. Free-form feedback: Corrections and explanations provided by the end-user on the generation.

You should create Human Evaluators structured to capture the feedback you need. For example, a Human Evaluator with return type “text” can be used to capture free-form feedback, while a Human Evaluator with return type “multi_select” can be used to capture user actions that provide implicit feedback.

If you have not done so, you can follow our guide to create a Human Evaluator to set up the appropriate feedback schema.

Prerequisites

  • You already have a Prompt — if not, please follow our Prompt creation guide first.
  • You have created a Human Evaluator. This can be done by following the steps in our guide to Human Evaluator creation.

Attach Human Evaluator to enable feedback

In this example, we’ll be attaching a “Tweet Issues” Human Evaluator to an “Impersonator” Prompt. The specifics of the “Tweet Issues” Evaluator are not important for this guide, but for completeness, it is a Human Evaluator with the return type “multi_select” and options like “Inappropriate”, “Too many emojis”, “Too long”, etc.

1

Go to the Prompt’s Dashboard

2

Click Monitoring in the top right to open the Monitoring Dialog

Prompt dashboard showing Monitoring dialog

3

Click Connect Evaluators and select the Human Evaluator you created.

Dialog connecting the "Tweet Issues" Evaluator as a Monitoring Evaluator

You should now see the selected Human Evaluator attached to the Prompt in the Monitoring dialog.

Monitoring dialog showing the "Tweet Issues" Evaluator attached to the Prompt

Record feedback against a Log by its ID

With the Human Evaluator attached to the Prompt, you can now record judgments against the Prompt’s Logs. To make API calls to record feedback, you will need the Log ID of the Log you want to record feedback against. The steps below illustrate a typical workflow for recording feedback against a Log generated in your code.

1

Retrieve the Log ID from the client.prompts.call() response.

1log = client.prompts.call(
2    version_id="prv_qNeXZp9P6T7kdnMIBHIOV",
3    path="persona",
4    messages=[{"role": "user", "content": "What really happened at Roswell?"}],
5    inputs={"person": "Trump"},
6)
7log_id = log.id
2

Call client.evaluators.log(...) referencing the above Log ID as parent_id to record user feedback.

1feedback_2 = client.evaluators.log(
2    # Pass the `log_id` from the previous step to indicate the Log to record feedback against
3    parent_id=log_id,
4    # Here, we're recording feedback against a "Tweet Issues" Human Evaluator,
5    # which is of type `multi_select` and has multiple options to choose from.
6    path="Feedback Demo/Tweet Issues",
7    judgment=["Inappropriate", "Too many emojis"],
8)

Viewing feedback

You can view the applied in two main ways: through the Logs that the feedback was applied to, and through the Human Evaluator itself.

Viewing Feedback applied to Logs

The feedback recorded for each Log can be viewed in the Logs table of your Prompt.

Logs table showing feedback applied to Logs

Your internal users can also apply feedback to the Logs directly through the Humanloop app.

Log drawer showing feedback section

Viewing Feedback through its Human Evaluator

Alternatively, you can view all feedback recorded for a specific Evaluator in the Logs tab of the Evaluator. This will display all feedback recorded for the Evaluator across all other Files.

Logs table for "Tweet Issues" Evaluator showing feedback