Introduction

Agent Optimizer looks for changes that make an agent cheaper, faster or better, tries them, and shows you which ones worked. Each run scores a baseline, then proposes a change, replays the evaluation cases on a copy of the agent, and scores the result — round after round, until a limit or a stopping rule ends it.

The live agent is never touched while a run is in progress. When you like a result, Apply recommendations adds a new version to the agent, and nothing goes live until you publish that version.

Where to find it

AI Governance > Agent Optimizer, in the AI Evaluation group.

The list shows one row per optimization: Optimization, Agent, Goal, Status, Owner and Updated, newest first. Search by name; the row menu has Delete.

Access to AI Governance is granted through platform roles. Ask your workspace administrator, or see Settings > Governance > Roles for how permissions are assigned.

Which agents it works on

UnifyApps agents, both Simple and Advanced. Click New optimization and pick the agent — search by name, or filter by Created by me and Type. The goal, data and limits come next.

How a run works

1. Baseline — The run copies the agent and scores the copy on your evaluation data. This is iteration 0, the number every later round is compared with.

2. Propose — The optimizer model studies where the agent loses cost, time or quality and proposes a change to one or more parts it is allowed to edit.

3. Replay and score — The changed copy replays the evaluation cases and is scored against the goal and constraints. A round that breaks a constraint is rejected.

4. Repeat — Rounds continue until the target is met, a round stops improving, or a limit is reached. The best round becomes the recommendation.

Note: A run spends real money. It replays sessions, calls models and runs evaluators every round. The run shows its total spend against Maximum spend, and stops when it reaches it. Run again costs what the first run cost.

Set up a run

The Setup tab has seven steps — Goal, Target, Evaluation data, Optimization constraints, Allowed changes, Model pool and Optimizer settings. With the Auto goal, Target is hidden, so there are six. Start optimization stays disabled until every required field is filled — hover it to see what is missing.

Step 1: Goal

Choose what to improve, so every round can be scored against it. The default is Reduce cost.

Step 2: Target

For every goal except Auto, set the Stopping condition (and, for Evaluator based, the evaluator): No limit runs until a round stops beating the one before it; Fixed target ends on the first round that reaches the number you set.

Step 3: Evaluation data

Choose the cases each round is scored against.

With Agent sessions, Filter sessions decides which ones are used:

Enable validation data (off by default) holds part of the evaluation data back. The run improves the agent on the rest, then scores it on the held-out cases, so you can tell a real improvement from one that only fits the cases it tuned on. Validation split is the share held back, and must be above 0 when validation is on.

Note: Sessions carry no expected answers. Without an evaluator, rounds are compared on cost and latency only, so a change that makes answers wrong is not caught. Use a dataset or experiment with evaluators, or add an evaluator constraint, when quality matters.

Step 4: Optimization constraints

Set limits the optimizer must respect while it chases the goal. Each row is a metric, a bound — At least or At most — and a value. Each metric can appear in one row only. Metrics include Cost / session, p95 latency and the evaluators that can score your data.

Fill in the values, or remove rows you do not need. With no constraints, every result is judged on the goal alone. The report marks each constraint Held or Breached.

Step 5: Allowed changes

Choose which parts of the agent the optimizer may edit.

Note: Why three changes are marked risky. Changing a tool's structure changes its schema for everything else that calls it. A task's structure is a contract with whatever triggers it, so new inputs or outputs can break callers the run never tests. Skills are shared across agents, so adding or deleting one changes other agents too, and a deletion cannot be undone from the run.

Step 6: Model pool

Add the models the optimizer may try. Each round runs on one of them. Only text-generation models are offered. The pool is empty by default; add models when you want the run to compare the agent on other models.

On Advanced agents, the model is set per job — Response generation, Task selection and Task execution — and a round can change each one separately.

Step 7: Optimizer settings

Before you can start, the run needs: an agent that loads; an experiment, or a dataset with at least one evaluator, if you chose those sources; at least one session id or condition, and a sampling rate, for Sessions or Custom; a validation split above 0 if validation is on; an evaluator and a target value where the goal needs them; complete constraint rows; and an optimizer model.

Follow a run

A run has three tabs.

Open an iteration to see its Overview and the Sessions it scored. Its changes are grouped by surface — System prompt, Model, Tools, Tasks, Compaction settings, Prompt caching — with Show diff to compare against the baseline (iteration 0) or another iteration you choose.

While a run is going you can Pause, Resume or Stop it. After it finishes you can Run again, and on a Success run you can Apply recommendations. From More actions you can Duplicate run to try a variation, or Delete it — which removes the run with its iterations and baseline and cannot be undone.

Statuses

A finished run also records why it ended — for example max iterations reached.

Apply the result

Step 1: Choose the round

The run recommends its best round. To ship a different one, open it and click Use this iteration; that overrides the recommendation, and you can change your mind before applying.

Step 2: Click Apply recommendations

Under What to apply, select the changes to ship — all of them, or only some surfaces.

Step 3: Test, then publish

Applying adds a new version to the agent and does not publish it. Check the new version — an experiment on a fixed dataset is the most reliable way, see Create an Experiment — then publish it when you are satisfied.

Best Practices

1. Add an evaluator whenever quality matters — A cheaper agent that answers wrongly is not an improvement, and sessions alone cannot catch it.

2. Turn on validation data for larger sets — It shows whether the gain holds on cases the run did not tune on.

3. Keep risky changes off at first — Turn them on deliberately, for an agent you are ready to restructure.

4. Read every diff before applying — A recommendation is a proposed edit to your agent; review it like code.

5. Start with a small budget — A few iterations and a modest spend show whether the agent has headroom before you commit more.

Related

• Evaluation Overview

• Create an Experiment

• Create a Dataset

• Compare Experiment Runs