Introduction

Controls tell you whether an agent did something it should not. Evaluation tells you whether it did its job well — which no policy can answer, because a correct-but-useless answer breaks no rule.

This is the quality half of governance, and it is measured rather than enforced.

Where to find it

Evaluation has two homes, because it does two things.

1. Per agent — Agent Registry > select an agent > Evaluation — scoring live traffic.

2. The Evaluation workspace — Experiments, Datasets and Metrics — testing against fixed inputs.

Access to AI Governance is granted through platform roles. Ask your workspace administrator, or see Settings > Governance > Roles for how permissions are assigned.

What are you trying to do?

That last row is the one worth reading twice. Evaluation never intervenes. A terrible score raises no incident and blocks nothing.

The objects

1. Evaluator — A scoring rule bound to an agent, with a type, a scope, a sampling rate and a cost. Scores live traffic.

2. Metric — A standardised measurement. Some are model-judged, some run as automations, some derived, some human-in-the-loop.

3. Dataset — A collection of test cases — input, expected output, and whatever context the metric needs.

4. Test case — One case inside a dataset.

5. Experiment — One offline run: a dataset, an agent, a set of metrics, a result you can compare against the last run.

Reading the numbers

On an agent, four figures summarise quality.

Never quote quality without coverage. The two tiles sit together deliberately: a score is only as meaningful as the share of traffic behind it.

"Evaluation is the only part of governance that tells you whether the agent is worth running at all."

Evaluation costs money

Scoring is model work. An LLM-judged evaluator makes a model call per scored record, and that lands in your spend as governance cost — the same figure surfaced as Eval spend.

Which makes sampling a design decision rather than a detail. Scoring everything on a busy agent buys precision you rarely need at a cost you will notice. See Where Spend Comes From.

Getting started

1. Pick one agent that matters — Preferably one with real traffic and a clear idea of what "good" looks like.

2. Add one evaluator — One, at a low sampling rate. Resist configuring five before you have read the output of one.

3. Read the scores against real traces — Open the records it scored badly and decide whether you agree. If you do not, the metric is wrong for this agent.

4. Build a dataset from the failures — Import those sessions as test cases. That dataset is now your regression suite.

5. Change something and run an experiment — Only now can you attribute an improvement to what you did.

Important considerations

1. Evaluation observes; controls enforce — Different systems, different questions. A quality problem is not a policy violation.

2. Online evaluation is agent-scoped — Evaluators bind to an agent.

3. Coverage is usually a sample — By design, and by cost.

4. A metric encodes a judgement — Choosing one is choosing a definition of good. Read what it measures before trusting its number.