October 16, 2024
Evaluate Flow Log contents
Flow Logs allow users to represent complex mult-step apps on Humanloop. Each step can be a Prompt, Tool, or Evaluator Log; or even another Flow Log. Logs can also be nested so that you can represent your app’s execution trace.
Prior to now, Evaluators could only reference the inputs and outputs of Flow Logs when providing Judgements. We’ve now added the ability to access the entire contents of a Flow Log in an Evaluator. This allows you to write more complex Evaluators that can inspect the entire execution trace of your app.
How to use
The contents of the Flow Log are accessible via the new children
field. Logs within the trace can also have children depending on the level of nesting in your code.
For example, if your Flow Log represent a conversation between a user and a chatbot, you can now write an Evaluator that inspects the entire conversation to make a judgement. Below is a simple example of checking how many steps there were in the conversation:
Or maybe you want to count how many Logs in the trace returned an empty output, where there may have been nesting:
You can access children
within any of the Evaluator Editors.