Capture user feedback

Prerequisites

• You already have a Prompt — if not, please follow our Prompt creation guide first.
• You have created a Human Evaluator. For this guide, we will use the “rating” example Evaluator automatically created for your organization.

1
pip install humanloop

1
from humanloop import Humanloop
2
humanloop = Humanloop(api_key="<YOUR HUMANLOOP KEY>")
3
4
# Check that the authentication was successful
5
print(humanloop.prompts.list())

Configure feedback

To collect user feedback, connect a Human Evaluator to your Prompt. The Evaluator specifies the type of the feedback you want to collect. See our guide on creating Human Evaluators for more information.

You should now see the selected Human Evaluator attached to the Prompt in the Monitoring dialog.

Log feedback

With the Human Evaluator attached to your Prompt, you can record feedback against the Prompt’s Logs.

1
log = humanloop.prompts.call(
2
    version_id="prv_qNeXZp9P6T7kdnMIBHIOV",
3
    path="persona",
4
    messages=[{"role": "user", "content": "What really happened at Roswell?"}],
5
    inputs={"person": "Trump"},
6
)
7
log_id = log.id

1
feedback = humanloop.evaluators.log(
2
    # Pass the `log_id` from the previous step to indicate the Log to record feedback against
3
    parent_id=log_id,
4
    # Here, we're recording feedback against a "Tweet Issues" Human Evaluator,
5
    # which is of type `multi_select` and has multiple options to choose from.
6
    path="Feedback Demo/Tweet Issues",
7
    judgment=["Inappropriate", "Too many emojis"],
8
)

1
rating_log = humanloop.evaluators.log(
2
    parent_id=log_id,
3
    # We're recording feedback using the "rating" Human Evaluator,
4
    # which has 2 options: "good" and "bad".
5
    path="rating",
6
    judgment="good",
7
8
    # You can also include the source of the feedback when recording it with the `user` parameter.
9
    user="user_123",
10
)

1
correction_log = humanloop.evaluators.log(
2
    parent_id=log_id,
3
    path="correction",
4
    judgment="NOTHING happened at Roswell, folks! Fake News media pushing ALIEN conspiracy theories. SAD! "
5
    + "I know Area 51, have the best aliens. Roswell? Total hoax! Believe me. 👽🚫 #Roswell #FakeNews",
6
)

1
removed_rating_log = humanloop.evaluators.log(
2
    parent_id=log_id,
3
    path="rating",
4
    judgment=None,
5
)

View feedback

You can view the applied feedback in two main ways: through the Logs that the feedback was applied to, and through the Evaluator itself.

Feedback applied to Logs

The feedback recorded for each Log can be viewed in the Logs table of your Prompt.

Your internal users can also apply feedback to the Logs directly through the Humanloop app.

Feedback for an Evaluator

You can view all feedback recorded for a specific Human Evaluator in the Logs tab of the Evaluator. This will display all feedback recorded for the Evaluator across all other Files.

Next steps

• Create and customize your own Human Evaluators to capture the feedback you need.
• Human Evaluators can also be used in Evaluations, allowing you to collect judgments from your subject-matter experts.