Introduction
Getting cases in is easy; keeping them worth running is the work. A dataset degrades quietly — expectations go stale, duplicates accumulate, and the set drifts toward whatever was easy to import rather than what actually breaks.
The import paths
Path | You provide | Best for |
CSV upload | A file | Cases you have already curated elsewhere |
From sessions | An agent, a sampling rate, how many cases | Real traffic — the highest-value path |
From Knowledge | A knowledge source | Retrieval agents answering from a corpus |
With an AI agent | A model and a count | Reaching volume fast, with review |
Importing from sessions
You choose the agent, a sampling rate, and how many cases to take. One rule the product states plainly: only sessions with at least one user query are imported — a session with no user turn has nothing to ask.
Two habits make this path much more valuable than sampling at random:
Import the sessions that went wrong, not a random slice — Sessions that scored badly in online evaluation, or that raised incidents, are known-interesting. A random sample is mostly cases the agent already handles.
Review the expected outputs afterwards — An imported session records what the agent did say. That is not the same as what it should have said, and if you leave it as the expectation you have encoded the bug as the target.
Note: This is the most consequential mistake in the module. Importing a failed session and keeping its actual output as the expected output turns a regression test into a guarantee that the failure keeps passing. Every imported case needs a human to say what the right answer was.
Generated cases need review
AI generation reaches volume faster than any other path and produces expectations nobody has checked. A generated expected output is a model's guess at the right answer, and a dataset of unreviewed guesses will fail correct behaviour and pass incorrect behaviour in ways that are hard to attribute later.
Use it to draft the inputs — where variety is genuinely useful — and review the expectations before the set is used to make a decision.
Keeping a dataset healthy
Correct expectations that turn out to be wrong — A stale expectation fails every future run and trains the team to ignore that case. One ignored case becomes several.
Remove duplicates — Repetitive cases cost money per run and weight the score toward whatever is over-represented.
Add cases when production surprises you — The best moment to grow a dataset is right after a real failure, while you still know what should have happened.
Re-baseline when you grow it — Adding hard cases lowers the score whether or not the agent changed. Re-run the previous configuration against the enlarged set so comparisons stay honest.
Check the import history when a score moves unexpectedly — "Did the dataset change?" is the first thing to eliminate, and it is recorded.
"A dataset nobody prunes becomes a measure of how much of it is easy."
Exporting
Datasets and results can be exported as CSV, XLS or XLSX. That is genuinely useful for two things: reviewing expectations in bulk with someone who does not use the platform, and taking evaluation results into a report.
It is worth noting the contrast — evaluation has export and incidents does not. If you need a regular evidence extract, this is not the surface that provides it. See Evidence & Export.