For AI agents: a documentation index is available at the root level at /llms.txt and /llms-full.txt. Append /llms.txt to any URL for a page-level index, or .md for the markdown version of any page.
Sign inBook a demo
DocsReferenceChangelog
DocsReferenceChangelog
  • Getting Started
    • Overview
    • Why Humanloop?
    • Quickstart Tutorial
  • Tutorials
    • Create your first GPT-4 App
    • ChatGPT clone with streaming
  • Guides
    • Create a Prompt
      • Overview
      • Run an evaluation
      • Set up evaluations using API
      • Use LLMs to evaluate logs
      • Self-hosted evaluations
      • Evaluating externally generated Logs
      • Evaluating with human feedback
      • Set up Monitoring
    • Fine-tune a model
    • Manage API keys
    • Invite collaborators
    • Deploy to environments
  • Core concepts
    • Prompts
    • Tools
    • Datasets
    • Evaluators
    • Logs
    • Environments
    • Key Concepts
  • Reference
    • Supported Models
    • Access Roles
    • .prompt files
    • Postman Workspace
LogoLogo
Sign inBook a demo
On this page
  • Prerequisites
  • Set up an evaluator to collect human feedback
  • Configuring the feedback schema
GuidesEvaluation and Monitoring

Evaluating with human feedback

This guide demonstrates how to run a batch generation and collect manual human feedback.
Was this page helpful?
Previous

Set up Monitoring

In this guide, we will demonstrate how to create and use online evaluators to observe the performance of your models.
Next
Built with

Prerequisites

  • You need to have access to evaluations.
  • You also need to have a Prompt – if not, please follow our Prompt creation guide.
  • Finally, you need at least a few logs in your project. Use the Editor to generate some logs if you don’t have any yet.

Set up an evaluator to collect human feedback

1

Create a ‘Human’ Evaluator

From the Evaluations page, click New Evaluator and select Human.

2

Give the evaluator a name and description and click Create in the top-right.

3

Return to the Evaluations page and select Run Evaluation.

4

Choose the model config you are evaluating, a dataset you would like to evaluate against and then select the new Human evaluator.

5

Click Batch generate and follow the link in the bottom-right corner to see the evaluation run.

6

View the details

As the rows populate with the generated output from the model, you can review those outputs and apply feedback in the rating column. Click a row to see the full details of the Log in a drawer.

7

Apply your feedback either directly in the table, or from the drawer.

8

Once you’ve finished providing feedback for all the Logs in the run, click Mark as complete in the top right of the page.

9

You can review the aggregated feedback results in the Stats section on this page.

Configuring the feedback schema

If you need a more complex feedback schema, visit the Settings page in your project and follow the link to Feedbacks. Here, you can add more categories to the default feedback types. If you need more control over feedback types, you can create new ones via the API.