Introduction

A test case is a structured record, not just a question and an answer. Which fields you populate determines which metrics can score it — so the fields are worth understanding before you build a dataset rather than after a metric refuses to run.

Two families

`LLM`	One input, one output. Single-turn
`Conversational`	A multi-turn exchange, carrying turns and roles

The fields, and what each is for

Field	The product says	You supply it
`Input`	"The input to your agent"	Always
`Expected Output`	"Expected agent output for a given input"	When you know the right answer
`Context`	"Ground truth for a given input"	When correctness depends on facts
`Expected Tools`	"Expected tools to be called by your agent"	When the right behaviour is an action
`Actual Output`	"Generated agent output for a given input"	Never — the run fills it
`Retrieval Context`	"Retrieved text chunks for a given input"	Never — the run fills it
`Tools Called`	"Tools called by your agent"	Never — the run fills it
`Turns`	—	Conversational cases
`Agent Role`	—	Conversational cases
`Latency` · `Cost`	—	Never — measured

Note: Context and Retrieval Context are opposites and are constantly confused. Context is ground truth — what is actually true, supplied by you. Retrieval Context is what the agent fetched — filled in by the run. One is the answer key; the other is the agent's working. A metric comparing them is asking whether retrieval found the right material.

Two kinds of field

What you supply — the question and what should happen. This is the test.
What the run fills in — what did happen, and what it cost. This is the result.

Metrics score by comparing across that line. Which comparisons are available depends entirely on what you supplied, which is why an under-populated case silently limits what you can measure.

Populate the fields the question needs

What you want to know	Populate
Did it give the right answer?	`Input` + `Expected Output`
Did it stay faithful to the facts?	`Input` + `Context`
Did retrieval find the right material?	`Input` + `Context` — compared against retrieved chunks
Did it take the right action?	`Input` + `Expected Tools`
Did it handle the conversation?	A conversational case with `Turns`

"An under-populated test case does not fail. It quietly narrows what you are allowed to ask about the result."

Expected Tools is the field governance readers should care about

Most evaluation attention goes to whether the answer was right. For an agent that acts, whether it did the right thing matters more — and Expected Tools is how you test it.

It catches the two failures that matter: an agent that should have called a tool and did not, and an agent that called one it should not have. The second is the same risk the blocked-tools control guards at runtime — testing it offline is how you find it before it ships. See the Blocked Tools check.

Custom fields

Beyond the standard set, cases can carry fields of your own for domain-specific checks. A metric can reference them the same way it references the standard ones — see Input Parameters Reference.

Test Case Types