Evaluation Metrics are how you score an agent's answers — built-in and custom, judged by an AI, a workflow, a person, or a formula.

Overview

Metrics are the measurements you apply to an agent's answers in an experiment. A metric is "A standardized measurement tool for evaluating agent performance against predefined criteria." You can use the built-in Metrics Library or author your own. Each metric reads certain test-case fields, produces a score, and passes or fails the case against criteria you set.

The Metrics Library

Explore Metrics Library offers a large set of pre-built metrics so you can "effortlessly evaluate any AI Agent across standard performance parameters." It includes, among others: Answer Relevancy, Faithfulness, Hallucination, Bias, Toxicity, Prompt Alignment, Task Completion, Json Correctness, SQL Generation, Text to SQL, Code Generation, QA, Check Accuracy, Summarization, Tool Correctness, Citation Evals, Language Mismatch, User Frustration, and AI vs Human (Groundtruth) — plus the RAG-focused Contextual Precision, Contextual Relevancy, and Chunk Overlap. For multi-turn agents there are Conversation Relavancy, Conversation Completeness, Knowledge Retention, and Role Adherence. Similarity metrics include Cosine Similarity and Embedding Similarity (Automation metrics) and BERT Score (an AI metric).

Search the library to find one and use its Add button to copy it into your workspace metrics — Add only shows up if you have metric create access.

Metric Types You Can Create

Use New Metric to author your own. You choose a type:

Pick AI for nuanced quality judgments, Automation when you can score deterministically with a workflow, Human in the loop when people must review, and Derived to combine other metrics' scores with a formula.

Automation Metrics

An Automation metric runs an automation you build to compute the score. Build an automation that starts with AI Evaluator by UnifyApps > Triggers LLM metrics evaluation (or Triggers Conversational metrics evaluation), computes a score within your scale, and ends with Respond to metric evaluation. Then select it in the metric. Only automations with one of these triggers appear in the list.

Configuring a Custom Metric

The custom-metric editor walks through Basic Details → Configure Evaluator → Passing Criteria → Review. Key choices:

• Evaluation mode — define the metric as Criteria (what a good answer looks like) or Steps (an explicit checklist). Switch between them with "Switch to Evaluation Steps / Criteria."

• Evaluation type — a Scale (the default) or Binary rating; Star is offered only for Human in the loop metrics. Scale and Star carry a scale range (1–5 by default).

• Input Parameters — which test-case fields the metric reads.

• Passing Criteria — two rules: Pass test case, if and Pass criteria of metric, if.

Use Test Metric to run it on sample inputs before saving — it shows the Result, Score, and Reasoning so you can confirm it judges the way you intend.

The Two Passing-Criteria Rules

Passing Criteria — "Passing criteria for your query and metric" — sets one rule for an individual test case and one for the metric across a whole run. Both are read as field · operator · value, and the field is fixed for you.

Pass test case, if — "Define passing criteria for a particular test case." The field is Score, and the operators depend on the Evaluation type you chose:

• Binary — a single operator, Is, with a true/false value. It defaults to Is true.

• Scale and Star — Greater than or equal to or Less than or equal to, with a number. The description shows you the live range ("Score range: 1 – 5"), and a threshold outside it is rejected with "Value should be between 1 and 5." No default threshold is filled in — you must enter one, or you'll get a required-field error on Test Case Success Criteria.

Switching the Evaluation type resets this rule to that type's default, so set the type before the threshold.

Pass criteria of metric, if — "Define passing criteria for an experiment." The field is Test Cases passed, the operators are Greater than or equal to and Less than or equal to, and the value is a percentage between 0 and 100. This is what decides whether the metric as a whole passed a run, as distinct from whether one test case passed. It has no default either — leaving it empty raises a required-field error on Metric Success Criteria.

What You Can Still Change After a Metric Is Created

Editing a metric reopens the same fields, with one restriction: if the metric was created from a standard library metric, its Input Parameters become read-only and the Evaluation type and Evaluation scale are hidden — you can retune its name, description, criteria, and passing criteria, but not re-point it at different test-case fields or re-scale it. A metric you authored from scratch has no such restriction.

Creating a Derived Metric

A derived metric computes its score from other metrics' scores instead of judging the answer itself — useful for a blended measure, like an overall quality score that averages relevancy and faithfulness. Its editor skips Configure Evaluator and instead runs Basic Details → Select Metrics → Apply Formula → Passing Criteria:

• Select Metrics — choose one or more existing metrics to use as inputs for your formula. You can pick any metric type except another derived metric (formulas can't be nested), and for LLM datasets the inputs must match the dataset's test-case type.

• Apply Formula — use operators & functions on selected metrics to create your derived metric. Reference a metric by typing {{ (or click one from the list) and combine them with math operators. Three Quick Suggestions build the formula for you over the selected metrics: Average (a + b) / n, Sum a + b, and Product a * b. Formulas evaluate plain math only; an invalid one shows "Invalid formula. Please check the formula and try again."

• Passing Criteria — the same pass rules as any other metric.

In run details a derived metric shows "You have applied the formula …" with the formula and each input metric's individual metric score, so you can see how the combined number was reached.

How an AI Metric Actually Computes Its Score

AI metrics are LLM-as-judge: each evaluation is a model call that must return a JSON verdict with a score and a reason (the reason is what you see in run details). The mechanics differ by metric:

• Built-in library metrics each have their own judging prompt with an explicit method. For example, Faithfulness extracts every claim in the answer, checks each against the retrieval context, and scores truthful claims ÷ total claims (0 if no claims are found); the RAG metrics (Contextual Precision and Contextual Relevancy) issue per-chunk verdicts and aggregate them; conversational metrics grade the whole dialogue.

• Custom Criteria metrics work G-Eval-style: the platform first generates evaluation steps from your criteria, then judges the answer against those steps and your criteria on your scale range. (Defining the metric as Steps skips generation and uses your checklist directly.) The judge is told the exact score range — 0.0–1.0 unless you set a custom Scale — and mentions specifics from the evaluation parameters in its reason.

• The judge model is the one you pick on the metric, and every judgment records its own tokens and cost — that's the Metric Cost you see on runs, separate from what the agent itself spent.

• Automation metrics (for example Cosine Similarity and Embedding Similarity in the library) run a workflow and use its output as the score, with no judge call. A Derived metric kind also exists in the platform's metric model — a score computed from other metrics' results rather than judged directly.

Note: If a judge response can't be parsed, the case records an error rather than a fabricated score.