Introduction
A dataset is a set of test cases — the questions you put to an agent every time you want to know whether a change helped. Building one well is the highest-leverage work in offline evaluation, because every experiment you ever run is only as good as what it asks.
Two dataset types
Type | A case is | Use for |
| One input and its expected output | Single-turn work — classification, extraction, a direct answer |
| A multi-turn exchange | Anything where the agent has to hold context across turns |
Conversational datasets come in two shapes: a scenario, which describes a situation to be played out, and a transcript, which is a recorded exchange. A scenario tests how the agent handles a situation; a transcript tests it against something that actually happened.
Note: The type has to match how the agent is used, not how it is built. An agent that is technically single-turn but is used conversationally will pass an LLM dataset and fail its users — because the failures live in turn three, and an LLM dataset never gets there.
Four ways to build one
Upload a CSV — Most control, most effort. Right when you know exactly what you need to test.
Import from live sessions — Real traffic becomes test cases. This is the most valuable path in the module — see below.
Import from Knowledge — Generates cases from indexed content. Useful for retrieval-based agents that answer from a corpus.
Create with an AI agent — Generates cases from a description. Fastest to volume, and needs review — a generated expected output is a guess until someone checks it.
Building from live sessions closes the loop
This is the bridge the evaluation category talks about and rarely ships: sessions that went wrong in production become the test set that stops them recurring.
It changes what a dataset is. Instead of the cases somebody imagined at design time, you get the cases your users actually produced — including the ones nobody anticipated, which is precisely where agents fail.
One constraint the product states plainly: only sessions with at least one user query are imported. A session with no user turn has nothing to ask the agent.
"The test cases you invent are the failures you already thought of. The ones you import are the failures you did not."
What makes a dataset worth having
Good | Weak |
Cases that have actually failed | Cases the agent obviously handles |
Edge cases and ambiguous inputs | Twenty variations of the same easy question |
Expected outputs a human checked | Generated expectations nobody read |
Fifty focused cases | Five hundred repetitive ones |
Stable, so runs stay comparable | Growing between every run |
The last row matters more than it looks. A dataset that changes between runs makes comparison meaningless — the score moved and you cannot say whether the agent or the questions changed. Grow it deliberately, and re-baseline when you do. See Compare Experiment Runs.
Size is a cost decision
Every case is a real agent run, and every AI-graded metric adds a model call per case. A hundred cases scored by three AI metrics is four hundred model calls each time you run it — and the whole point of a dataset is to run it often.
Start small enough to iterate cheaply. Grow it when a real failure shows you a gap, not to reach a round number.
Import history
Datasets retain a record of what was imported and when. When a score moves unexpectedly, "did the dataset change?" is the first question to eliminate, and this is where you answer it.