Introduction
Evaluation is model work, and model work costs money. An LLM-judged evaluator makes a call per scored record, continuously, for as long as it exists.
That spend is not hidden: it appears as Eval spend on the agent's Evaluation tab, and as a distinct Eval cost type in Cost Explorer.
What drives it
Driver | Effect |
Evaluator type |
|
Sampling rate | Directly proportional. Halving it halves the cost. |
Filter breadth | Narrowing the filter reduces what is eligible to be sampled at all. |
Scope | Trace scope scores every turn; session scope scores once per conversation. |
Judge model | A smaller model evaluating costs less than a large one. |
Number of evaluators | Each one scores independently. |
Backfill | A one-off charge proportional to the records selected. |
Controlling it
In the order worth trying.
Narrow the filter — The only lever that improves signal and cost together. An evaluator judging answer quality should not be scoring sessions that asked nothing.
Lower the sampling rate — Trends stay readable well below full coverage. Check the seven-day estimate as you change it.
Use session scope where the question is conversational — One score per conversation instead of one per turn.
Use a smaller judge model — It grades rather than generates.
Prefer a Derived metric where one fits — It costs nothing beyond the metrics it reads.
Where the cost shows up
Eval spend — On the agent's Evaluation tab — this agent's evaluators, this window.
COST per evaluator — In the evaluator table, so you can see which one is expensive.
Eval cost type — In Cost Explorer, grouped by
Cost Type, across the estate.
Eval is deliberately separate from Governance in Cost Explorer, because the two are tuned by different dials: sampling rate for evaluation, check design and applied-on scope for controls. See Where Spend Comes From.
Budgeting it deliberately
The useful question is not "how do we minimise this" but "what proportion of spend are we willing to spend on knowing what our agents did".
Decide that number, then set sampling rates to match it. Teams that skip this step either over-instrument and get a surprise, or under-instrument and cannot answer questions about quality when asked.
"Evaluation spend is not overhead. It is what visibility costs — and the alternative is not cheaper, it is unmeasured."
One case worth checking
An AI-judge outcome verification also charges per session, and it lands in the same place. Where an outcome is verified by a judge, compare Value per outcome against what judging a session costs.
An outcome whose measurement costs more than the outcome returns is a negative-value measurement of a possibly-positive agent. Rare, but it happens on low-value, high-volume outcomes.