Evaluation continuously scores the Enterprise Search agent's real sessions so you can watch answer quality over time.
What Online Evaluation Does
Online evaluation scores live production sessions on a schedule, instead of running the agent against a fixed test set. Turn it on with Enable Online Evaluation, choose which sessions to sample and how often, and pick the metrics to apply — the scores then show up under Sessions › Evaluated Sessions and can trigger alerts.
Session Settings
| Setting | Purpose |
| Filter sessions for online evaluation | Narrows which live sessions are eligible for scoring. |
| Sampling Rate | What share of eligible sessions to evaluate (0–100; default 50). |
| Mark Session as Closed After | How long to wait before a session counts as finished (default 24 hours). |
| Evaluation Check after every | How often the evaluation job runs (default 5 minutes). |
Metrics and Alerts
Under Configure Metrics, pick from a list already filtered to metrics that match your session's input and output fields — only AI and Automation metric types appear, and any metric needing ground truth a live session can't supply is excluded. Under Configure Alerts, attach a metric first — with none selected the section reads Alerts are disabled — then set a threshold so a metric falling below it notifies you.
How It's Different From an Experiment
An experiment runs the agent against a fixed dataset on demand, good for comparing versions before you ship. Online evaluation continuously samples real, live conversations after the agent is deployed, good for monitoring quality in production.