Introduction

Some results are real and leave no trace. The agent gave a complete, correct answer and the user went away satisfied — no tool fired, no tag was set, nothing in the session says it worked.

An AI judge reads the output and grades it. The product is direct about the cost: it adds spend every session, and it captures what no signal records. That trade is the whole decision.

When to use it

Check the middle row before reaching for the judge. If an evaluator is already scoring the agent, a rule on its score gets you a quality-gated outcome at no additional cost.

Define the outcome

Step 1: Name the outcome

As for any outcome. Name the result, not the activity.

Step 2: Add an agent and choose AI judge

Under 2. Measure outcome, click Add agent, pick the agent, and set How it is verified to AI judge. Set What it is worth as for any agent.

Step 3: Choose the judge model

Judge model is the model that grades the output; the list offers text-generation models only. It evaluates rather than generates, so a smaller model is often adequate — and it is being called once per session, which makes the choice a cost decision as much as a quality one.

Step 4: Write the evaluation criterion

Describe what a passing output looks like. The product's own placeholder is a good model: "The output is complete and correct enough to use without changes."

Two things make a criterion work:

1. Describe the output, not the intent — "The reply answers the question asked" is judgeable. "The agent was helpful" is not.

2. Set the bar where you would — "Complete and correct enough to use without changes" is a real bar. "Correct" alone invites the judge to pass anything not actively wrong.

Step 5: Set the pass threshold

Pass score threshold is a slider from 0 to 1, default 0.85: "Sessions the judge scores below this do not count." The default is a reasonable starting point.

When in doubt, go higher. An ROI figure built on a generous threshold is the first thing a reviewer will attack, and undercounting is a much easier position than overcounting.

Step 6: Save

The source summarises as Graded by an AI judge, passes at ….

What it costs

One model call per session, continuously, for as long as the source exists. On a high-volume agent that is a standing line in governance spend — and it appears under Eval and Governance in Cost Explorer alongside everything else you configured.

Worth stating the awkward case: an AI judge on a low-value outcome can cost more to measure than the outcome is worth. Compare what a counted session is worth against what judging a session costs before committing to it.

"Measuring value is not free. An outcome whose measurement costs more than the outcome returns is a negative-value measurement of a positive-value agent."

Verify the judge agrees with you

Before reporting anything, read ten sessions the judge passed and ten it failed, and decide whether you agree.

If you disagree on the passes, the criterion is too loose or the threshold too low. If you disagree on the failures, the criterion is describing something narrower than you meant.

Known limitations

1. Non-deterministic — Two similar sessions can be graded differently. Borderline cases will be inconsistent.

2. Cost scales with traffic — Unlike a rule, which is free however busy the agent is.

3. It judges the output, not the truth — A confident, well-formed and wrong answer can pass.