Introduction

"Evaluation" means two different activities in AI Governance, and they answer different questions. Confusing them is the most common source of wasted effort in this section.

1. Online evaluation — Scores traffic your agents are handling now. Answers "how are we doing?"

2. Offline evaluation — Runs an agent against a fixed dataset. Answers "did this change make it better?"

You need both, and for different reasons. Neither substitutes for the other.

Where to find it

Online — on an agent: Agent Registry > select an agent > Evaluation, titled Online Evaluation.

Offline — the Evaluation workspace: Experiments, Datasets and Metrics.

Access to AI Governance is granted through platform roles. Ask your workspace administrator, or see Settings > Governance > Roles for how permissions are assigned.

The difference

The sharpest way to hold the distinction: online evaluation cannot tell you whether a change helped, because the traffic also changed. Only a fixed dataset holds the input still.

Online evaluation

You attach one or more evaluators to an agent, and they score its live traffic as it happens.

What you configure

1. Scope — What one score covers. An evaluator runs on traces — scoring one run at a time — on sessions, scoring the whole conversation, or on spans, scoring one model call.

2. Filter — Only matching records are scored. Narrow to the traffic the evaluator is meant to judge.

3. Sampling rate — Score a subset rather than everything. This is your main cost dial.

4. Backfill — Optionally score past records too, using the same filter and sampling. Records older than 30 days have aged out of retention, so a backfill reaches back only so far.

What you read

Overall quality as a blended score out of 100, Coverage as records scored out of records sampled, Quality trend against the previous window, and Eval spend.

Always read quality with coverage. A blended score over two per cent of traffic is a different claim from the same score over all of it, and the tile shows both for exactly that reason.

"A quality score without its coverage is an opinion with a number attached."

Offline evaluation

You build a dataset of test cases, choose metrics, and run an experiment. Because the input is fixed, two runs are comparable — which is what makes this the only way to answer "did that change help?"

Where datasets come from

You are not required to write test cases by hand. A dataset can be built from a CSV, generated with AI, imported from knowledge — or, most usefully, built from real sessions and OpenTelemetry traces.

That last route is the one worth designing around. It closes the loop between the two halves of this page.

Using them together

The two are not alternatives. They form a cycle, and each hands work to the other.

1. Online tells you something is wrong — Quality drops, or a class of interaction scores badly.

2. Turn those sessions into a dataset — Import the failing sessions as test cases. You now have a fixed, repeatable version of the real problem.

3. Change the agent and run the experiment — Re-run against the same dataset. The difference is attributable to your change, because nothing else moved.

4. Ship, and watch online — Confirm the improvement holds on traffic you did not curate.

Skipping step 2 is the common failure: a change is made, online quality moves, and nobody can say whether the change or the traffic caused it.

Which to reach for

Important considerations

1. Both cost money — Scoring is model work. Online spends continuously at your sampling rate; offline spends per run. Both appear as governance spend. See Where Spend Comes From.

2. Online is agent-scoped — Evaluators attach to an agent. There is no tenant-wide online evaluation.

3. Retention bounds backfill — Records older than 30 days cannot be scored retrospectively.

4. Evaluation is not governance — An evaluator scores quality; it does not enforce policy. A low score raises no incident. Controls do that.