Run an experiment

An experiment sends every example in a dataset to what you're testing, scores each reply with your evaluators and adds the scores up per variant. Use it before you change an agent's instructions or model, to see whether the change helps.

Before you start

• A dataset with examples. See Datasets and examples.
• One or more evaluators. See Evaluators.
• Credits: 1 for each agent reply the experiment makes.

Steps

1. 1

   Start a new experiment

   Choose Run experiment on a dataset, or New experiment under Datasets → Experiments.

2. 2

   Choose what to test

   Under Target, pick an agent or a workflow. For an agent, Variants starts with Current, the agent as saved. Add a variant for each change you want to try (up to 4 in all): give it a name and change its Instructions, or its Model (key, model and temperature).

3. 3

   Pick the dataset and evaluators

   Choose the Dataset, and All examples or one Split. Tick the Evaluators to score with.

4. 4

   Set how it runs

   Repetitions: once to 5 times, to see how much replies vary. At a time: how many replies run at once (4 by default, up to 10).

5. 5

   Check the estimate and start

   The footer shows the runs and credits, such as "24 runs · 24 credits". Choose Start experiment.

Runs, limits and credits

• Runs = examples × variants × repetitions, at most 500 per experiment. Use a split, fewer variants or fewer repetitions to stay under it.
• Each agent run is a real reply on the agent's own model and key, and costs 1 credit, like any reply; failures are refunded. The credits are checked before it starts, and if they run out on the way, the experiment stops with the results so far.
• Replies in an experiment don't create conversations and never answer through the agent's workflow. Each one has a trace. The agent gets only the built-in tools, so an experiment never posts or creates anything in your apps.
• Experiment traffic counts against the agent's requests per minute, and waits when it hits the limit.
• Two experiments run at once per account. A third waits and starts by itself when one finishes.
• An experiment stops after 14 minutes with the results so far, and a single reply has 240 seconds.

Testing a workflow

Pick a workflow as the target and tick Published, Draft (unpublished changes) or both. Each example runs as a test run of the workflow, with the example's input as message, so the Start block needs no other required fields. Its Agent, Classify and Extract blocks cost what they always do, and its other blocks really run, emails and app actions included. Up to 3 runs go at a time. A workflow that was never published can't be tested here yet: publish it first.

Reading the results

• While it runs, the page fills in as each reply finishes. You can leave; it keeps running. Cancel stops it within a few seconds and keeps what finished.
• The summary has one tile per variant: each evaluator's average and pass rate, latency p50 and p95, replies and errors, tokens and cost.
• Below it, each example shows every variant's reply with its score chips. Filter by Failed only, Errors or Regressed vs baseline, and pick a repetition.
• On a reply: Add to dataset (with that reply as the reference answer), Send to review and Open trace.

If something goes wrong