Compare experiments and promote a variant

Once you have results, decide what to keep. Inside one experiment, its variants sit side by side. Across experiments, a baseline and the compare page show what changed, and Use it saves a winning variant to the agent.

Set a baseline

On a completed experiment, choose Set as baseline. An agent or a workflow has one baseline at a time; setting another replaces it. Later experiments on the same dataset can filter to Regressed vs baseline and offer Compare with baseline.

Steps: compare experiments

1. 1

   Pick them

   On the Experiments list, tick two to four experiments and choose Compare, or choose Compare with baseline on an experiment.

2. 2

   Read the table

   Each experiment's first variant is set against the first one: its scores, with changes shown as points (such as "+12 pts"), latency p50 and errors.

3. 3

   Run the pairwise judge

   For two completed experiments on the same dataset, pick a pairwise evaluator (make one from the Pairwise preference template if you have none) and choose Run pairwise judge. A model reads both replies to each example and picks the better one, or a tie. The order is shuffled for each example so neither side is favoured.

The pairwise judge reports "A wins · B wins · Ties" over the examples it judged. It runs on the evaluator's judge model, or the first experiment's agent model when that's Same as the agent, and costs no credits.

Steps: promote a variant

1. 1

   Find the winner

   In the experiment's summary, a variant that changed something has a Use it button.

2. 2

   Use it

   Choose Use it, then Use this variant. Only what the variant changed is written to the agent: its instructions, provider and model, temperature or key.

3. 3

   Check the version

   It takes effect at once, on every channel, and is recorded on the agent's Versions tab, so you can roll it back.