Evaluation Datasets are the test cases you measure an agent against — the starting point of every evaluation run.

Overview

Evaluation is where you measure AI-agent quality. It has three areas — Datasets, Experiments, and Metrics — each opened from the navigation (or from AI Governance > Overview > AI Evaluation). The flow is: build a dataset of test cases, choose or author metrics, then run an experiment that executes an agent (or automation) against the dataset and scores every test case. Human review happens inside an experiment run. Datasets is the starting point — "Organise your test cases to quickly evaluate your AI Agents."

Creating a Dataset

Use New Dataset and pick the Type of Test Cases:

Type	Description
LLM	Used to test single input/output set of responses (e.g. RAG Agents).
Conversational	Used to test multi-step conversations (e.g. customer care agents). Also asks for a Conversation Type: Transcript Based ("Create test cases with defined input/output" — the default) or Scenario Based ("Define different scenarios for evaluation").

Choose LLM for question-answer style checks and Conversational when you need to evaluate a back-and-forth dialogue.

Adding Test Cases

Open a dataset and Add Test Cases. You can build the set several ways:

Add from Sessions — turn real agent conversations into test cases.
Add from OTel Traces.
Upload CSV.
Generate with AI Prompt — generate realistic cases.
Generate using Knowledge Set — derive cases from a knowledge set.
Add Manually — available for LLM datasets.

Scenario-based conversational datasets don't use Add Test Cases — open the dataset and use Generate Scenarios or Create New Scenario instead. For other datasets, each import is tracked in an import history you can re-run, and you can export the dataset.

A note on how the generated sources work: Add from Sessions converts real conversation messages into test cases (capturing the input, the agent's actual output, and the tools it used); Generate with AI Prompt and Generate Scenarios use an LLM with a generation prompt to produce realistic cases from your description; and Generate using Knowledge Set samples chunks from a knowledge set (randomly or targeted) and has an LLM write question/answer cases grounded in those chunks — batch sizes and chunk counts are platform-capped so large knowledge sets don't explode into unbounded generation.

Test Case Fields

A test case carries standard fields, grouped as input, output, and additional:

Field	Description
Input	The input to your agent.
Expected Output	Expected agent output for a given input.
Context	Ground truth for a given input.
Expected Tools	Expected tools to be called by your agent.
Actual Output	Generated agent output for a given input.
Retrieval Context	Retrieved text chunks for a given input.
Tools Called	Tools called by your agent.
Turns, Agent Role, Latency, Cost	Additional recorded fields.

You also manage Metadata Fields on the dataset. Not every metric uses every field — fill the ones the metrics you plan to run actually read (for example, Expected Output for accuracy checks, Retrieval Context for RAG metrics).

Permissions

Creating datasets needs create access. Every affordance that adds or changes a dataset's contents — Add Test Cases, a test case's delete button, bulk-selecting rows, Manage fields, renaming or deleting a conversation, a scenario's edit/regenerate/delete menu, and retrying a failed import — needs edit access on that dataset and disappears without it.

Note: Viewing is not gated the same way. A test case still opens without edit access, just read-only (titled View Test Case / View Message, with no save/cancel controls), and the show/hide-columns picker stays usable — only saving your column choices back to the dataset needs edit access.