Introduction

An evaluator scores an agent's live traffic. Adding one is four decisions — which metric, over what unit, on which records, and how many of them — and the last two are where the cost lives.

Where to find it

Agent Registry > select an agent > Evaluation > + Add evaluator

Before you begin

1. The agent must be producing traces — An evaluator on a silent agent scores nothing. Check LAST TRACE first.

2. Know what "good" means for this agent — Choosing a metric is choosing a definition of quality. Pick the one you would defend.

3. Start with one — Read its output before adding a second. Five evaluators on an agent nobody has calibrated is five costs and no insight.

Add an evaluator

Step 1: Open + Add evaluator

A drawer opens with a searchable list. Filter by Type — AI, Automation, Human in the loop, Derived — or by Test case type, LLM or Conversational.

The type matters for cost. An AI evaluator makes a model call per scored record; a Derived one does not.

Step 2: Set the scope

The configuration panel opens as Configure <type> · <name>. Scope decides what one score covers.

See Scope: Session vs Trace — it is the decision people get wrong most often.

Step 3: Filter what gets scored

"Only matching records are scored." Add conditions to narrow to the traffic this evaluator is meant to judge.

Filtering is the cheapest way to control both cost and noise. An evaluator judging answer quality has no business scoring sessions that never asked a question.

Step 4: Choose the model

For an AI evaluator, Model is "used by this evaluator to score" the records. It evaluates rather than generates, so a smaller model is often adequate.

Step 5: Set the sampling rate

"Score only a subset of matching records." The panel shows an estimate of the volume this produces over seven days, which is the number to look at before saving.

Step 6: Optionally backfill

Also evaluate past records scores history using the same filter and sampling. Two limits worth knowing:

1. Retention — Records older than 30 days have aged out and cannot be scored.

2. A cap — Maximum records limits how many the backfill scores, oldest-first. Leave it unset only if you have checked what that costs.

Step 7: Save

The evaluator appears in the table with its type, scope, sampling rate, result, coverage, last-evaluated time, cost and who added it.

Read it before adding another

Open the records it scored badly and decide whether you agree. If you do not, the metric is wrong for this agent — and that is a finding worth more than the score.

Important considerations

1. Evaluators cost money continuously — Visible as Eval spend on this tab and under Eval in Cost Explorer.

2. A backfill is a one-off charge proportional to what you select — Set a maximum.

3. Evaluation never intervenes — A terrible score raises no incident and blocks nothing.

4. Coverage is not quality — Both tiles matter, and the score means little without the coverage beside it.