October 16, 2024

Evaluate Flow Log contents

Flow Logs allow users to represent complex mult-step apps on Humanloop. Each step can be a Prompt, Tool, or Evaluator Log; or even another Flow Log. Logs can also be nested so that you can represent your app’s execution trace.

Prior to now, Evaluators could only reference the inputs and outputs of Flow Logs when providing Judgements. We’ve now added the ability to access the entire contents of a Flow Log in an Evaluator. This allows you to write more complex Evaluators that can inspect the entire execution trace of your app.

How to use

The contents of the Flow Log are accessible via the new children field. Logs within the trace can also have children depending on the level of nesting in your code.

For example, if your Flow Log represent a conversation between a user and a chatbot, you can now write an Evaluator that inspects the entire conversation to make a judgement. Below is a simple example of checking how many steps there were in the conversation:

1def count_number_steps(log):
2 """Counts the number of steps in the Flow Log."""
3 # This assumes there was no subsequent nesting
4 return len(log.get('children', []))

Or maybe you want to count how many Logs in the trace returned an empty output, where there may have been nesting:

1def count_null_output_logs(log):
2 """Count the number of logs in the trace where output is null."""
3 def count_null_output(log):
4 """Helper function for recursively counting."""
5 null_count = 1 if log.get('output') is None else 0
6 for child in log.get('children', []):
7 null_count += count_null_output(child)
8 return null_count
9 return count_null_output(log)

You can access children within any of the Evaluator Editors.

Evaluate Flow Log Contents