Introduction
A quality score is a number produced by a model applying a definition you chose to a sample you sized. Reading it well means holding all three of those in view.
The four tiles
Tile | Is |
| A blended score out of 100 across the agent's evaluators |
| Records scored, of records sampled |
| Points moved against the previous window |
| What the scoring cost |
Quality trend reads no prior window when there is nothing to compare against — expected for a new evaluator, and a reason not to react to the first week's number.
What "blended" hides
Overall quality combines every evaluator on the agent. That is convenient and it is where most misreadings start.
A stable blend can hide a moving component — One evaluator improving while another degrades produces a flat headline.
Mixed scopes blend incomparable things — A session score and a trace score answer different questions; averaging them produces a number that answers neither.
Adding an evaluator moves the blend — Without anything about the agent changing.
Read the evaluator table, not just the tile. Per-evaluator RESULT and COVERAGE are where the actual information is.
A perfect score usually means nothing was scored
An agent showing 100 blended quality is far more often an agent with zero coverage than an agent that is perfect.
Check Coverage first, every time. 0 of 0 alongside a perfect score means the evaluator has not run — wrong filter, no traffic, or sampling so low nothing was selected.
"The two tiles sit next to each other for a reason. A score without its coverage is an opinion with a number attached."
The evaluator table
Columns: EVALUATOR, TYPE, SCOPE, SAMPLING, RESULT, COVERAGE, LAST EVALUATED, COST, ADDED BY.
Two columns to read together: SAMPLING and COVERAGE. A high sampling rate with low coverage means evaluation is not keeping up. No data in range means nothing was scored in the selected window.
Reading an individual result
A score is only trustworthy once you have read the records behind it. Opening a scored record shows the metrics applied, each marked Passed or Failed, alongside the material the evaluator judged — ACTUAL OUTPUT, RETRIEVED CONTEXT and TOOLS CALLED.
Not evaluated means no evaluator has scored that record — normal under sampling.
Read ten badly-scored records before trusting any aggregate. If you disagree with the model, the metric is wrong for this agent, and that finding is worth more than the score.
Reading the trend
Traffic changes, not just agents — A score can move because users started asking different questions. Online evaluation cannot separate the two — only a fixed dataset can. See Online vs Offline Evaluation.
Small samples produce large swings — Before investigating a drop, check whether coverage is large enough for the movement to mean anything.
Configuration changes break comparability — Changing a filter, scope or metric restarts the series. Note when you do it.