Capture user feedback

Record end-user feedback using Humanloop; monitor how your model generations perform with your users.

Prerequisites

  • You already have a Prompt — if not, please follow our Prompt creation guide first.
  • You have created a Human Evaluator. For this guide, we will use the “rating” example Evaluator automatically created for your organization.

First you need to install and initialize the SDK. If you have already done this, skip to the next section. Otherwise, open up your terminal and follow these steps:

  1. Install the Humanloop TypeScript SDK:

    $npm install humanloop

  2. Import and initialize the SDK:

    1import { HumanloopClient, Humanloop } from "humanloop";
    2
    3const humanloop = new HumanloopClient({ apiKey: "YOUR_API_KEY" });
    4
    5// Check that the authentication was successful
    6console.log(await humanloop.prompts.list());

Configure feedback

To collect user feedback, connect a Human Evaluator to your Prompt. The Evaluator specifies the type of the feedback you want to collect. See our guide on creating Human Evaluators for more information.

You can use the example “rating” Evaluator that is automatically for you. This Evaluator allows users to apply a label of “good” or “bad”, and is automatically connected to all new Prompts. If you choose to use this Evaluator, you can skip to the “Log feedback” section.

Different use-cases and user interfaces may require different kinds of feedback that need to be mapped to the appropriate end user interaction. There are broadly 3 important kinds of feedback:

  1. Explicit feedback: these are purposeful actions to review the generations. For example, ‘thumbs up/down’ button presses.
  2. Implicit feedback: indirect actions taken by your users may signal whether the generation was good or bad, for example, whether the user ‘copied’ the generation, ‘saved it’ or ‘dismissed it’ (which is negative feedback).
  3. Free-form feedback: Corrections and explanations provided by the end-user on the generation.

You should create Human Evaluators structured to capture the feedback you need. For example, a Human Evaluator with return type “text” can be used to capture free-form feedback, while a Human Evaluator with return type “multi_select” can be used to capture user actions that provide implicit feedback.

If you have not done so, you can follow our guide to create a Human Evaluator to set up the appropriate feedback schema.

1

Open the Prompt’s monitoring dialog

Go to your Prompt’s dashboard. Click Monitoring in the top right to open the monitoring dialog.

Prompt dashboard showing Monitoring dialog

2

Connect your Evaluator

Click Connect Evaluators and select the Human Evaluator you created.

Dialog connecting the "Tweet Issues" Evaluator as a Monitoring Evaluator

You should now see the selected Human Evaluator attached to the Prompt in the Monitoring dialog.

Monitoring dialog showing the "Tweet Issues" Evaluator attached to the Prompt

Log feedback

With the Human Evaluator attached to your Prompt, you can record feedback against the Prompt’s Logs.

1

Retrieve Log ID

The ID of the Prompt Log can be found in the response of the humanloop.prompts.call(...) method.

1log = humanloop.prompts.call(
2    version_id="prv_qNeXZp9P6T7kdnMIBHIOV",
3    path="persona",
4    messages=[{"role": "user", "content": "What really happened at Roswell?"}],
5    inputs={"person": "Trump"},
6)
7log_id = log.id
2

Log the feedback

Call humanloop.evaluators.log(...) referencing the above Log ID as parent_id to record user feedback.

1feedback = humanloop.evaluators.log(
2    # Pass the `log_id` from the previous step to indicate the Log to record feedback against
3    parent_id=log_id,
4    # Here, we're recording feedback against a "Tweet Issues" Human Evaluator,
5    # which is of type `multi_select` and has multiple options to choose from.
6    path="Feedback Demo/Tweet Issues",
7    judgment=["Inappropriate", "Too many emojis"],
8)

The “rating” and “correction” Evaluators are attached to all Prompts by default. You can record feedback using these Evaluators as well.

The “rating” Evaluator can be used to record explicit feedback (e.g. from a 👍/👎 button).

1rating_log = humanloop.evaluators.log(
2    parent_id=log_id,
3    # We're recording feedback using the "rating" Human Evaluator,
4    # which has 2 options: "good" and "bad".
5    path="rating",
6    judgment="good",
7
8    # You can also include the source of the feedback when recording it with the `user` parameter.
9    user="user_123",
10)

The “correction” Evaluator can be used to record user-provided corrections to the generations (e.g. If the user edits the generation before copying it).

1correction_log = humanloop.evaluators.log(
2    parent_id=log_id,
3    path="correction",
4    judgment="NOTHING happened at Roswell, folks! Fake News media pushing ALIEN conspiracy theories. SAD! "
5    + "I know Area 51, have the best aliens. Roswell? Total hoax! Believe me. 👽🚫 #Roswell #FakeNews",
6)

If the user removes their feedback (e.g. if the user deselects a previous 👎 feedback), you can record this by passing judgment=None.

1removed_rating_log = humanloop.evaluators.log(
2    parent_id=log_id,
3    path="rating",
4    judgment=None,
5)

View feedback

You can view the applied feedback in two main ways: through the Logs that the feedback was applied to, and through the Evaluator itself.

Feedback applied to Logs

The feedback recorded for each Log can be viewed in the Logs table of your Prompt.

Logs table showing feedback applied to Logs

Your internal users can also apply feedback to the Logs directly through the Humanloop app.

Log drawer showing feedback section

Feedback for an Evaluator

You can view all feedback recorded for a specific Human Evaluator in the Logs tab of the Evaluator. This will display all feedback recorded for the Evaluator across all other Files.

Logs table for "Tweet Issues" Evaluator showing feedback

Next steps

Built with