Evaluation Experiments run a chosen agent against a dataset and score every test case with the metrics you selected.

Overview

An experiment runs a chosen agent (or callable automation) version against a dataset and scores every test case with the metrics you selected — "Create and manage experiments to evaluate your Agents." Each execution of the experiment is a run, and a run produces a pass/fail per test case plus aggregate score, cost, and latency. Experiments are how you compare versions, prove a change improved quality, and catch regressions before deploying.

Creating and Running an Experiment

Use New Experiment and work through the steps — Configure Dataset → Metrics → Additional Setup:

• Pick the dataset(s).

• Choose where outputs come from: Agent ("Run agent live and use its outputs" — pick the agent), Dataset ("Use existing dataset outputs"), or Task ("Build custom workflow for outputs" — a callable automation; the experiment runs its latest deployed version).

• Select the metrics, and map dataset/task fields to each metric's inputs. For custom scoring, use an Automation metric.

• In Additional Setup, set Max Concurrency, toggle Skip Approval Before Tool Execution, assign a team to Annotate on report (for Human-in-the-loop metrics), and optionally Schedule Runs.

• Turn on Auto-start run to run immediately on create.

To use Task, first build an automation that starts with AI Evaluator by UnifyApps > Trigger task execution and ends with Respond to task execution. Only such automations appear in the Task list.

After it exists you can Start New Run, Stop Run, Restart Run, and Export Experiment Runs.

Permissions

The Experiments / Datasets / Metrics tabs each appear only with view access to that asset type; the New … buttons need create access. On an experiment, Start New Run and Stop Run both need execute access — the two are kept in sync, so anyone who can start a run can also stop it, and no one else sees either button. Config changes — the sidebar's edit pencils for Metrics/Schedule/Annotations, the schedule on/off toggle, and the runs table's Delete Runs bulk action — need edit access on the experiment; row selection and Export Experiment Runs stay available to viewers either way. A dataset's add/edit affordances need edit access on that dataset.

Note: If a button described here is missing, ask your admin for the matching permission.

How a Run Executes Under the Hood

When a run starts, the platform counts the test cases (a single run processes up to 20,000), stamps the run in-progress, and splits the cases into parallel workers according to Max Concurrency — the batch size is simply total ÷ concurrency, and 10 workers is the default when you don't set one. Each worker takes its slice of cases and, per case:

1. Produces the output — calls the agent live (or your Task automation), or reuses the dataset's stored output, depending on the output source you chose. Live calls capture latency, tokens, cost, and the trace.

2. Scores it — each selected metric judges the case (metric judging itself runs with a small parallelism of its own, a few evaluations at a time) and records a score, a reason, and the judging cost.

3. Applies your passing criteria to turn scores into the per-case pass/fail and the metric-level pass/fail.

The run waits for all workers to signal completion, then aggregates: average score per metric plus pass/fail counts (computed across the run's score records), and the cost/latency rollups. Errors on individual cases are recorded per case without sinking the whole run.

What the Run Results Show

A run shows aggregates — Agent Cost, Metric Cost, Agent Latency (with averages and totals) — and a table of every test case with its pass/fail result and per-metric scores. Open a row to see its detail: the inputs and outputs, the per-metric score and reason, and a Trace Timeline of how the agent produced the answer. (When the data came from a dataset rather than a live run, you'll see No Trace Timeline Found.) Failing cases sort to the top so you can triage quickly.

Reading One Metric's Result Inside a Test Case

Open a test case in a run and each metric gets its own panel, shaped by the metric's type:

• AI in the loop — the score summary, then the judge's explanation.

• Automation — the automation that scored it, with a link that opens that workflow run in a new tab, the run's latency, and a pass/fail pill; then a Score section reading the score against the metric's maximum (or a dash if it didn't produce one).

• Human in the loop — a Rated by card naming the person who scored it, plus their rating shown in the metric's own control (a Linear scale, a star Rating, or a like/dislike for a binary metric), read-only. Until someone rates it, there's nothing to show.

Every one of them ends with a Reasoning section — the judge's explanation of the score. Plain-text reasons are shown as written; structured ones are rendered as readable, browsable data rather than a wall of text; and if the judge returned no reason at all, you'll see a dash. This is the fastest way to understand why a case failed rather than just that it failed.

Run Statuses

A run is Running, Completed, Failed, or Cancelled. Inside a run, you can filter test cases by Passed, Failed, or Pending. While a run is still going the table reads Evaluation In Progress — "This can take a few minutes. Once finished, your run data will show up here." A per-test-case result is a green check (passed its criteria), a red cross (failed), or blank when no metrics scored it.

Adding Human Review (Annotation)

Human review happens inside an experiment run. Open a run — when test cases need human scores, a banner reads {n} test cases pending for annotation — Annotate Now. Click it to score each case by hand — a like/dislike, a star rating, or a linear scale per metric — and to Add Annotation and Add Comment. The team you set in Annotate on report receives this work. Use annotation for Human in the loop metrics and to spot-check what the automated metrics scored.