Introduction
Scope decides what one score covers, and it is the setting people get wrong most often — because both choices produce a plausible-looking number, and only one of them answers the question you asked.
The three scopes
Scope | Scores | Answers |
| One run at a time | Was that answer good? |
| The whole conversation | Did the user get what they came for? |
| One model span at a time | Was that model call good? |
Choosing between trace and session
The distinction is not granularity. It is what counts as success.
Trace scope judges each answer on its own — An agent that answers every turn correctly scores well. That is right for a question-answering agent where each turn stands alone.
Session scope judges the conversation — An agent that took six turns to get somewhere a good agent reaches in two scores badly at session scope and well at trace scope — because every individual answer was fine.
"Trace scope tells you the agent gives good answers. Session scope tells you the user got what they needed. Those are different claims, and only one of them is what you were asked."
Rule of thumb: if the agent holds a conversation, score sessions. If it processes items independently, score traces.
Session scope needs a definition of "finished"
A conversational evaluator cannot score a session that is still open, so sessions need a point at which they are considered closed.
Session Settings on the Evaluation tab controls that, and it is subtitled "how live sessions are timed and swept for scoring".
Setting | What it does |
| The idle period after which a session is treated as finished. "Conversational evaluators run once the session is marked closed." |
| How often sessions are swept for evaluation. |
Two consequences worth planning for:
Session scores lag — A session closed after 30 minutes and swept every hour means a score up to 90 minutes after the conversation ended. Trace scores do not have this delay.
Too short a close time splits conversations — A user who steps away and returns produces two sessions, each scored as an incomplete conversation. Both score badly, and neither reflects what happened.
Span scope
Scores one model call at a time. This is the scope for evaluating a model rather than an agent — a model's activity is spans, so its evaluators score spans.
Model evaluation carries its own settings, subtitled "how this model's traffic is swept for scoring", with sweep frequency described in terms of spans rather than sessions.
Running more than one scope
An agent can carry evaluators at different scopes, and that is often the right answer: trace-scope for answer accuracy, session-scope for whether the conversation resolved.
They contribute to one blended quality score, so read the evaluator table rather than the headline number when scopes are mixed — the blend hides which dimension moved.
Important considerations
Scores at different scopes are not comparable — Do not chart them together or treat a change of scope as a change in quality.
Coverage is counted in the scope's unit — "Records scored of records sampled" means sessions for a session evaluator, traces for a trace one.
Session scope costs less per conversation — One score per conversation rather than one per turn — worth knowing when the agent is chatty.