Introduction

Four kinds of metric, and the choice between them is a choice about cost, consistency and what you can actually measure. The instinct is to reach for the AI judge for everything. It is usually the wrong instinct.

The four types

Type	Scores by	Cost per case	Consistent?
`AI (LLM as a judge)`	A model reading the output	A model call	Approximately
`Automation`	A platform automation	Depends what it does	Yes
`Human in the loop`	A person	Someone's time	Varies by person
`Derived Metric`	A formula over other metrics	None beyond its inputs	Perfectly

Note: Reach for an AI judge only when the check needs one. Use it when the thing you are measuring genuinely requires reading and understanding. Where a rule or a workflow can decide the check, an Automation metric gives the same answer every time and never needs calibrating.

Choosing

Can a rule or a workflow decide it? — Format, length, presence of a required element, whether a specific tool was called, a value in range, checking a record was created, calling out to a service. Use Automation.
Does it need judgement about meaning? — Tone, faithfulness to context, whether an answer actually addressed the question. Use AI (LLM as a judge).
Does it need a person? — Domain judgement no model reliably has — clinical, legal, safety-critical. Use Human in the loop, and accept it will not run at volume.
Is it a combination of metrics you already have? — Use Derived Metric rather than a new judge that re-reads everything.

Derived metrics deserve more use than they get

A derived metric computes a formula over other metrics' scores — sum, average, product, or something you write. It costs nothing beyond the metrics it reads, because those already ran.

That makes it the right tool for a composite quality number: instead of a second AI judge asked to weigh accuracy, tone and completeness together — another model call, another calibration problem — combine the three metrics you already have. The weighting becomes explicit and arguable rather than buried in a prompt.

"A composite score computed from a formula can be argued with. One produced by a model asked to 'consider everything' cannot."

How a metric expresses a result

Evaluation type	Produces	Suits
`Binary`	Pass or fail	Requirements — it either did or it did not
`Scale`	A number in a range	Qualities that admit degree
`Star`	A rating	Human review

A scale needs a threshold to become a verdict, and that threshold is where your actual standard lives — see Create a Custom Metric.

The cost that surprises people

AI metrics are model calls. Three of them over a hundred-case dataset is three hundred extra calls per run, and the point of a dataset is to run it repeatedly.

Two ways to keep it proportionate: use Automation or Derived Metric metrics for everything that does not need reading, and keep the AI ones few and well-chosen. Five AI metrics scoring the same output is usually three metrics too many.

Metric Types