Introduction
An experiment runs a dataset of test cases against an agent and scores the results with metrics. It is the offline half of evaluation — a controlled comparison you can repeat, as opposed to online evaluation's sampling of whatever real traffic happened to arrive.
The three things an experiment needs
Part | Is | Answers |
Dataset | The test cases | What are we asking it? |
Agent | The thing under test | Who is answering? |
Metrics | The scoring | What counts as good? |
All three have to exist before you can create an experiment, which makes the order of work dataset → metrics → experiment. Most first attempts stall because one of the first two was skipped.
Creating one
Have a dataset — Built from CSV, from live sessions, from OpenTelemetry traces, from knowledge, or generated with AI — see Create a Dataset.
Have the metrics you intend to score with — From the library or your own — see Browse the Metrics Library.
Create the experiment — Name it, pick the agent, pick the dataset, attach metrics.
Run it — Each test case is executed against the agent and scored. The run reports progress as cases complete.
Read the failures, not the score — The aggregate tells you whether to ship. The failed cases tell you what to fix — see Reviewing Failed Cases.
Note: An experiment is a comparison, so it needs something to compare against. A single run produces a number with no meaning — is 78% good? The first run of any experiment is a baseline, and its only job is to exist so the second run means something.
Naming an experiment
Experiments accumulate faster than anything else in this module, and a name that made sense when you created it rarely survives three months. Name for what varies rather than what is being tested:
Better | Weaker |
Support agent — refund policy prompt v3 | Test 3 |
Claims agent — smaller model, same prompt | Claims experiment |
Search agent — retrieval top-k 5 vs 10 | Retrieval test |
Change one thing at a time
The discipline that makes experiments worth running is unglamorous and constantly violated: vary a single thing between runs. A run that changed the prompt and the model tells you the score moved and nothing about why.
The variables usually worth isolating are the prompt, the model, the tool set, and the retrieval configuration. Each is a separate run.
"Two changes and one number is not an experiment. It is a coin flip you paid to run."
Experiments cost money
Every test case is a real agent run, and every AI-graded metric is another model call. A hundred-case dataset scored by three AI metrics is four hundred model calls per run.
Two consequences worth planning for:
Dataset size is a budget decision, not just a coverage one. A focused set of fifty well-chosen cases usually beats five hundred repetitive ones, and costs a tenth as much to iterate on.
Keep AI metrics to the checks that need them. Each one is a model call per case — see Metric Types.
When to use an experiment rather than online evaluation
Experiment (offline) | Online evaluation |
Before shipping a change | After shipping, continuously |
The same cases every time | Whatever real traffic arrives |
Comparable across runs | Representative of reality |
Answers "is this version better?" | Answers "is it still good?" |
They are complements, and the bridge between them matters: a dataset can be built from live sessions, so the cases that went wrong in production become the test set that stops them recurring. See Online vs Offline Evaluation.