Evaluation
How evaluation works
Datasets, evaluators, experiments, live scoring, review queues and feedback: what each one does, how they fit together, and what they cost.
Concept2 min readUpdated 31 October 2026
Evaluation tells you whether a change to an agent makes its answers better or worse before your users find out. You keep a set of real questions, decide how a good answer is scored, run the agent over every question, and compare the scores. It's under Evaluate in the left menu: Datasets (with Experiments as its second tab), Evaluators, Review queues and Insights.
The pieces
| Piece | What it is |
|---|---|
| Dataset | A list of examples: a question, optionally the answer you expect, and a split. Every change makes a new version. See Datasets and examples. |
| Evaluator | One way to score a reply from 0 to 1: a model judge with a rubric, a code check, or a pairwise judge that picks the better of two replies. See Evaluators. |
| Experiment | Runs an agent (as saved, and variants with other instructions or models) or a workflow over a dataset and scores every reply. See Run an experiment. |
| Live scoring | An evaluator scoring a sample of an agent's live replies every 5 minutes. See Score live replies. |
| Review queue | Replies waiting for a person to score them against a rubric and write what the agent should have said. See Review queues. |
| Feedback | Every score on a reply: from people, evaluators, your end users and your own code through POST /v1/feedback. See Feedback and scores. |
A first loop
- Collect examples. Send real questions from Conversations to a dataset with Add to dataset, upload a CSV, or let Generate examples write some.
- Choose evaluators. Correctness when your examples have the answers you expect, Groundedness for an agent with knowledge, a code check such as Contains for phrases a reply must include.
- Run an experiment. The agent as saved (Current) against a variant with your change.
- Compare and promote. Read the scores per variant, then Use it to save the better variant to the agent. See Compare experiments and promote a variant.
- Keep watching. Score live replies with the same evaluators, and let a review queue collect thumbs down and low scores. Corrections from the queue go back into the dataset.
What it costs
| What | Credits |
|---|---|
| An agent reply in an experiment (one example, one variant, one repetition) | 1, like any reply. Failed replies are refunded |
| A workflow run in an experiment | What its blocks cost as they run: Agent, Classify and Extract blocks 1 each |
| A model judge's score (experiments, live replies, Run evaluator, Try it) | None. The judge runs on your own LLM key, and your provider bills its tokens |
| Code checks | None |
| Generate examples | 5 per batch of 10 |
| An Insights report | 15 |
Review queues and POST /v1/feedback | None |
Scores are 0 to 1
A pass or a thumbs up is 1, a fail or a thumbs down is 0, and 4 on a 1 to 5 scale is 0.75. Pages show them as percentages or as the value, such as "4/5".
Related
Still stuck?
Email us with what you tried and any request id. A person replies.