Introduction
A single run is a number. Two runs over the same dataset are evidence. Comparison is the only thing offline evaluation does that online evaluation cannot, because only here do you control what stays the same.
What makes two runs comparable
Must be identical | May differ |
The dataset | The prompt |
The metrics | The model |
The passing criteria | The tool set |
Retrieval configuration |
Note: Changing the dataset between runs invalidates the comparison, silently. Adding ten hard cases to a set will lower the score whether or not the agent got worse. If the dataset has to grow, the honest move is to treat the enlarged set as a new baseline and re-run the previous configuration against it — not to compare across the change.
What to read, in order
The verdict — Passed or failed. Coarse, and the fastest signal.
Per-metric scores — Where an aggregate hides the trade. A change that improves accuracy and degrades tone is one number up and one down, and only the per-metric view shows both.
Which specific cases changed verdict — The most useful comparison available. Cases that newly fail are a regression; cases that newly pass are the improvement you were aiming for.
Cost and latency — Each scored case carries them. A version that scores two points higher and costs twice as much is a decision, not an improvement.
Quality is not the only axis
Because runs carry cost and latency alongside scores, the comparison you can make here is richer than "which scored higher" — and the richer one is usually the decision you actually face:
Score | Cost | Reading |
Up | Flat | Ship it |
Up | Up sharply | A judgement call. Is the gain worth the per-run cost at your volume? |
Flat | Down | Often the best result available — same quality, cheaper |
Down slightly | Down sharply | Worth considering for non-critical paths |
That third row is the one teams under-pursue. Moving a step to a smaller model and confirming the score held is the most reliable saving in the product, and an experiment is how you confirm it rather than hoping.
"The best experiment result is usually not a higher score. It is the same score for less money, proven rather than assumed."
Small differences are usually noise
Agent outputs vary between runs even with identical inputs, and AI-graded metrics add their own variance because a model is doing the grading. A one- or two-point difference on a modest dataset is not a finding.
Three ways to tell signal from noise:
Run the same configuration twice. The gap between two identical runs is your noise floor, and it is worth measuring once.
Look at case-level changes. Five cases flipping in both directions is churn; five flipping one way is a change.
Prefer larger effects. If a change needs statistics to detect, it will not be visible to users either.
Treat a newly failing case as a regression
A case that passed before and fails now is the single most actionable output of this module. It is specific, reproducible and attributable to the one thing you changed.
Read the failing output rather than the score — the metric's own reasoning is recorded alongside the verdict, and it usually names the problem faster than you would find it. See Reviewing Failed Cases.