Introduction
A run has four statuses, and two of them describe progress while the other two describe a verdict. Conflating those two categories is the misreading this page exists to prevent.
The four statuses
Status | Kind | Means |
| Progress | Queued, not started |
| Progress | Executing test cases now |
| Verdict | Finished, and met its criteria |
| Verdict | Finished, and did not |
Note:
Failedmeans the agent did not meet the bar — not that the run broke. This is the misreading worth guarding against. A failed run is a successful experiment: it executed, it scored, and it told you the answer was no. That is the entire point of running it before you ship.
What decides Passed or Failed
The verdict comes from criteria defined on the metrics, and they are set at two levels:
Per test case — "Define passing criteria for a particular test case." Whether an individual case passed.
Per metric, across the run — "Define passing criteria for an experiment." How many cases must pass for the metric to be satisfied.
So a run can fail while most of its cases pass, if the threshold for the metric is higher than the proportion achieved. That is intended, and it is where the bar you set actually lives — see Create a Custom Metric.
A run that will not finish
Symptom | Check |
| Queued behind other work. Give it time before treating it as stuck |
| Large dataset, slow agent, or AI-graded metrics adding a model call per case |
Progress stalled entirely | The agent under test may be failing. Try one case by hand |
What to do with each verdict
Passed — check what it did not test — A pass is a statement about the cases in the dataset. Everything absent from it is untested, and a dataset of easy cases passes reliably.
Failed — go to the cases, not the score — The aggregate says no; the individual failures say why. See Reviewing Failed Cases.
"A run that fails has done its job. A run that passes has only told you about the questions you thought to ask."