Introduction
A single quality score answers almost nothing. Is 82 good? Against what?
A baseline is what turns a number into information: the score this agent produces when it is working normally, on this traffic, under this configuration. Everything after that is a comparison.
Establishing one
Step 1: Fix the configuration
A baseline is only valid for the setup that produced it. Settle the metric, the scope, the filter and the sampling rate before you start counting — changing any of them restarts the series.
Step 2: Let it run for two weeks
Long enough to cover the shape of your traffic. Most agents see different traffic on a Monday than a Saturday, and a baseline drawn from three days encodes whichever days those were.
Step 3: Read the records, not just the number
Open the low scores and decide whether you agree with the model. If you do not, you have calibrated a metric that measures the wrong thing, and the baseline is worthless.
Step 4: Write it down with its conditions
The number alone is not the baseline. Record the metric, the scope, the sampling rate, the coverage and the window — without those, a future comparison is not a comparison.
What moves a score, and what it means
Cause | Look like | Do |
The agent changed | A step, dated to a deployment | Run an offline experiment to confirm attribution |
Traffic changed | A drift with no deployment behind it | Read new low-scoring records — users are asking something new |
Small sample | Noise, week to week | Raise sampling or widen the window |
Configuration changed | A step dated to your own edit | Treat as a new baseline, not a regression |
An evaluator was added | The blend moves, components do not | Read per-evaluator results |
The first two are the pair that matters, and online evaluation cannot separate them. Both the agent and the traffic moved between windows, so a change in score has two candidate causes and no way to choose.
Which is why a baseline needs a dataset
The only way to attribute a change to something you did is to hold the input still. That is what a fixed dataset is for.
Online establishes the baseline and detects movement — It watches real traffic, which is the only place real problems appear first.
Offline explains it — Import the low-scoring sessions as test cases, change the agent, re-run against the same dataset. The difference is attributable.
"Online tells you something moved. Only a fixed dataset can tell you whether it was you."
See Online vs Offline Evaluation for the full loop.
Using a baseline day to day
Before a change — Record the current score and coverage. Without a before, there is no after.
After a change — Wait for coverage to rebuild before judging. A day of data at 5% sampling is not a verdict.
As a release gate — "Quality must not fall more than N points against baseline" is a usable rule, provided the baseline is written down with its conditions.
Re-baseline deliberately — When the agent's purpose changes, the old baseline stops being a fair comparison. Retire it rather than carrying it forward.