Compare and Debug Prompts
In this guide, we will walk through comparing the outputs from multiple Prompts side-by-side using the Humanloop Editor environment and using diffs to help debugging.
You can compare Prompt versions interactively side-by-side to get a sense for how their behaviour differs; before then triggering more systematic Evaluations. All the interactions in Editor are stored as Logs within your Prompt and can be inspected further and added to a Dataset for Evaluations.
Prerequisites
- You already have a Prompt — if not, please follow our Prompt creation guide first.
Compare Prompt versions
In this example we will use a simple Support Agent Prompt that answers user queries about Humanloop’s product and docs.
Create a new version of your Prompt
Open your Prompt in the Editor and expand Parameters and change some details such as the choice of Model
.
In this example, we change from gpt-4o
to gpt-4o-mini
.
This will create a new uncommitted version of the Prompt.
Now commit the new version of your Prompt by selecting the blue Commit button over Parameters and providing a helpful commit message like:
Load up two versions of your Prompt in the Editor
To load up the previous version side-by-side, select the menu beside the Load button and select the New panel option (depending on your screen real-estate, you can add more than 2 panels).
Then select to Load button in the new panel and select another version of your Prompt to compare.
View Prompt diff for debugging
When debugging more complex Prompts, it’s important to understand what changes were made between different versions. Humanloop provides a diff view to support this.
- While in the Compare tab, look for the Diff section.
- This section will highlight the changes made between the selected versions, showing additions, deletions, and modifications.
- Use this diff view to understand how specific changes in your prompt configuration affect the output.
By following these steps, you can effectively compare different versions of your Prompts and iterate on your instructions to improve performance.