Datasets and examples

A dataset is a list of questions, each with the answer you expect if you have one. Experiments send every question to an agent or a workflow and score the replies. The more your examples look like real traffic, the more the scores mean.

Steps: create a dataset

1. 1

   Open Datasets

   Click Datasets under Evaluate in the left menu and choose New dataset.

2. 2

   Name it and pick the target

   Name (up to 200 characters) and an optional Description. Target is the agent or workflow it tests, or Any agent. New experiments start with it, and generated examples are written from it; it doesn't stop you running the dataset on something else.

3. 3

   Choose how to start

   Empty to type examples yourself, Paste or upload CSV, Recent conversations to pick real replies from the agent's chats, or Generate with AI.

4. 4

   Create it

   Choose Create dataset. It opens on Examples, at version 1.

Examples

Add example types one in. The list shows where each example came from: Typed, CSV, Conversation, Trace, Generated, Review or Experiment.

Import a CSV

Import CSV takes a file or pasted text: one row per example, with the columns input, expected_output (optional) and split (optional). A header row is optional. Download a template gives you a file to start from.

input,expected_output,splitCan I return a worn jacket?,Only if it's faulty. Unworn items can be returned within 30 days.,testHow long does delivery take?,Standard delivery takes 3 to 5 working days.,trainDo you sell gift cards?,Yes: digital gift cards from $25 to $500.,

• Headers are matched loosely: question, prompt, message or query count as the input, and expected, reference, answer, output or ideal as the reference answer.
• Up to 1,000 rows at a time. Rows without an input are skipped, and each field is cut at 20,000 characters.
• There's no export yet.

Add to dataset from anywhere

The Add to dataset sheet lets you edit the input and the reference answer, pick the dataset (or New dataset: <agent> examples) and a split before you add it. When the reply wasn't good, write the right answer in Reference answer first.

Generate examples

Generate examples (or Generate with AI on a new dataset) has SentientOne's own model write realistic questions, with reference answers, from the target: an agent's name, description, instructions, knowledge titles and Q&A pairs, or a workflow's blocks. It avoids questions the dataset already has, and Focus can steer it, for example "refund questions".

• Choose 10 examples or 20 examples. It costs 5 credits for each 10, charged only once the examples are written. If anything fails, you aren't charged.
• Drafts marked Edge case or Out of scope are included on purpose. Pick the ones you want and choose Add; nothing is saved until you do.
• It needs the dataset's target to be an agent or a workflow, and you can generate 10 batches a minute.

Versions

• Every add, edit or delete makes a new version of the dataset, however many examples it touches. Renaming it doesn't.
• A deleted example leaves the current version but stays in older ones, so past experiments keep what they ran on.
• Pick an older version above the list to see the examples as they were then; it's read only.
• An experiment runs on the dataset's current version and shows it in its header, such as v3.

Splits

A split is a label you choose: test and train are suggested, any word works. Set it on an example, in the CSV's split column, or when you add from elsewhere. Filter the list by split, and run an experiment on All examples or one split. Generated examples and examples from conversations start with no split.

If something goes wrong