Introduction
Sampling decides how much you score. Coverage tells you how much you actually scored. Together they decide whether a quality number means anything.
They are also your only real cost dial: an AI evaluator charges per scored record, so sampling rate multiplied by traffic is your evaluation bill.
Sampling
"Score only a subset of matching records." Set as a rate, with an estimate of the resulting volume over seven days shown alongside it.
Read that estimate before saving. It is the difference between a considered decision and discovering the bill at month end.
Traffic | Reasonable start | Why |
Low — tens per day | 100% | Cheap, and a sample would be too small to read |
Moderate — hundreds | 10–25% | Enough to see a trend at a fraction of the cost |
High — thousands | 1–5% | Trends are stable well below full coverage |
Filtering first is better than sampling harder. Narrowing to the traffic the evaluator is meant to judge improves the signal and reduces the cost; a lower sampling rate only does the second.
Coverage
The tile reads records scored, of records sampled — and the distinction between the two halves matters.
Sampled — Records the filter matched and sampling selected.
Scored — Records the evaluator actually completed.
A gap between them means evaluation is not keeping up or is failing on some records. Coverage well below 100% of sampled is worth investigating; coverage below 100% of traffic is normal and intended.
An empty state reads no records scored yet, which is expected for a new evaluator and a problem for an old one.
Why the two tiles sit together
They are adjacent by design. A blended quality score of 94 over 2% of traffic and the same score over all of it are very different claims, and only one of them supports a decision.
"Quote quality and coverage in the same breath, or do not quote quality."
The practical rule: when reporting a score, state the coverage. When reading someone else's score, ask for it.
Sampling is random, not representative of everything you care about
A random sample tells you about typical traffic. It is a poor way to find rare failures — the interesting ones are, by definition, rare.
For the general picture — Sample broadly at a low rate.
For a specific worry — Add a second evaluator with a tight filter and a high sampling rate on just that slice. Narrow and thorough beats broad and thin.
Backfill
Also evaluate past records applies the same filter and sampling to history. Two limits:
30-day retention — Older records have aged out and cannot be scored.
Set a maximum — The cap limits how many records the run scores, oldest-first. A backfill with no cap on a busy agent is an unbounded one-off charge.