Introduction

Write a custom metric when the library cannot express what your domain considers correct. That is a narrower set of cases than it first appears — and the ones that qualify are usually worth a lot, because they encode a judgement only your organisation can make.

Before you write one

Can a library metric do it? — Adopting and tuning beats writing. See Browse the Metrics Library.
Can a rule or a workflow decide it? — An Automation metric gives the same answer every time and never needs calibrating. Prefer it whenever the rule is expressible.
Is it a combination of metrics you already have? — Then it is a derived metric, not a new judge.

Criteria or steps

For an AI metric, the editor offers two ways to describe the judgement, and it is a toggle between them rather than two different metric types:

Mode	You write	Suits
`Evaluation Criteria`	A description of what good looks like	A holistic judgement
`Evaluation Steps`	An ordered procedure to follow	A judgement with a method — identify, then compare, then score

Steps produce more consistent scoring when the judgement has genuine structure, because they stop the model weighing things differently from case to case. The product's own example criterion shows the shape worth aiming for: identify relevant versus irrelevant segments, then assign a score representing the proportion, with a brief justification.

Note: Vague criteria produce inconsistent scores, and inconsistency looks like agent variance. "Is the response good?" gives a different answer on Tuesday. Name what you are looking for, say how to weigh it, and say what the score should represent — then the same output scores the same way twice.

Writing a criterion that holds up

Do	Avoid
Name the specific property being judged	"Good", "high quality", "appropriate"
Say what the score should represent	Leaving the scale's meaning implicit
Say what to do with edge cases	Letting the model decide silently
Ask for a justification	A bare number you cannot audit

That last row matters more than it sounds. Scored results carry the reasoning behind the verdict, and it is the fastest way to find out whether the metric is measuring what you meant — see Reviewing Failed Cases.

Building a derived metric

A derived metric is a formula over metrics you already have. The editor walks it in steps: basic details, select the metrics to combine, apply a formula, configure the evaluator, set passing criteria, review.

Quick suggestions cover the common shapes — sum, average, product — and you can reference a metric directly in a formula. Two constraints worth knowing before you start:

A metric used in the formula cannot be deselected until it is removed from the formula. The editor says so.
Selecting more metrics than the formula uses prompts a warning — the extra ones contribute nothing and cost their own scoring.

The reason to prefer this over another AI judge is that the weighting becomes explicit. A composite produced by a formula can be argued with in a review; one produced by a model asked to consider everything cannot.

Test it against cases you have already judged

The editor runs a metric against sample input and reports the result, the score and its reasoning. Use it before the metric influences any decision.

Pick cases where you know the answer — Some clearly good, some clearly bad, and — most importantly — some genuinely borderline.
Read the reasoning, not just the score — A metric can reach the right verdict for the wrong reason, and it will not keep doing so.
Check the borderline cases hardest — Clear cases score correctly almost regardless of wording. The borderline ones are where your criterion is actually tested.
Set the threshold from observed scores — Not from a round number chosen in advance.

"A metric that disagrees with your judgement is not measuring rigorously. It is measuring something else."

Create a Custom Metric