Introduction
Evaluation questions split cleanly in two: nothing is being scored, or the scores are not believable. The first is nearly always configuration; the second is nearly always the metric.
Nothing is being scored
Online — check in this order:
Is an evaluator bound to the agent? — None bound means an empty
Evaluatedtab however much traffic there is.Does the evaluator's filter match your traffic? — Records outside its scope are never candidates.
Is the sampling rate low? — Evaluators score a proportion. Little scored yet is expected at a low rate.
Is it a conversational evaluator? — Those run once the session is marked closed, so results lag live traffic.
Offline — the usual cause is parameters. A metric scores only cases carrying every parameter it declares, so a metric asking for ground-truth context skips every case without it — silently. See Input Parameters Reference.
The scores are not believable
Symptom | Usual cause |
Everything passes | The threshold is too low, or the dataset is all easy cases |
Everything fails | The threshold is too high, or the criteria describe something the agent was never asked to do |
Scores swing between identical runs | Vague criteria. An AI judge with a loose brief answers differently each time |
The metric disagrees with you | It is measuring something adjacent to what you meant. Rewrite the criteria; more data will not help |
A case fails and the output looks fine | Bad metric or bad expected output — not necessarily a bad agent |
Note: Doubt the metric before you doubt the agent. An AI-graded metric is a model forming a judgement and can be as wrong as the agent it grades. Teams that skip this step spend weeks fixing agents that were behaving correctly.
Evaluation is costing more than expected
Both halves cost money and for different reasons. Online: an AI judge is a model call per scored session — the product says it "adds spend every session". Offline: every test case is a real agent run, and every AI metric adds a call per case.
The levers, in order of effect: lower the sampling rate online; use fewer, better-chosen AI metrics; prefer Automation or Derived metrics wherever they express the check; and keep datasets focused rather than large. See Evaluation Cost.
A comparison between runs looks wrong
Did the dataset change? — Adding hard cases lowers the score whether or not the agent changed. Check the import history.
Did you change one thing or two? — A run that changed prompt and model tells you the score moved and nothing about why.
Is the difference bigger than your noise floor? — Run the same configuration twice to find out what that floor is. One or two points is usually variance.
Frequently asked
Online or offline? — Offline before shipping a change; online continuously afterwards. They answer different questions — see Online vs Offline Evaluation.
Why is my quality score based on so little traffic? — Evaluation is sampled. Quote coverage alongside quality or neither — see Sampling & Coverage.
Does Failed mean the run broke? — No. It means the agent did not meet the bar. A failed run is a successful experiment.
Can I evaluate past traffic? — Yes, with a backfill — but records older than thirty days have aged out of retention.
Why do evaluated sessions and evaluated traces show different things? — Conversational evaluators score sessions; the rest score traces. Two lists populated by different evaluators, not two views of one dataset.
Can evaluation block a bad response? — No. Evaluation scores; controls act. Different mechanisms — see Controls Overview.
How many metrics should I run? — Few, answering distinct questions. Every metric is a cost on every case, forever.