Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

Build an Evaluation DatasetBeta

Beta
This is coming — this page describes a feature that's still being rolled out and may not be available to you yet.

Keep the conversations that matter to you in a Dataset, so you can reuse them whenever you measure a change.

About This Task​

A Dataset holds conversations you want to compare pipeline versions on. It can include conversations from any pipeline in the workspace. To open the Datasets page, click Evaluation in the sidebar and go to Datasets, or open a pipeline's Evaluation tab and go to Datasets. Both show every Dataset in the workspace. On a pipeline's Evaluation tab, a Dataset lists only the test cases of that pipeline.

For background, see Datasets. To reuse a Dataset from another workspace, export it there and import it here. For steps, see Import and Export a Dataset. If you have no recorded conversations, you can also build a Dataset from a CSV or XLSX table. See Add Test Cases from a Table.

Add Conversations from Search History​

  1. Open a pipeline's search history and select the queries you want to keep. Each conversation becomes one test case, even if you select several of its queries.
  2. Click Add to Dataset.
  3. In Add to, choose Existing Dataset and select the Dataset, or choose New Dataset and type a name, for example refund regressions. If your workspace has no Datasets yet, the dialog starts on New Dataset.
  4. Click Add. A notification confirms how many test cases were added and links to the Dataset.

Each test case is named after its conversation's first query and records the pipeline you added it from.

Create an Empty Dataset​

  1. Go to Datasets and click New Dataset.
  2. Type a name and click Save. The new Dataset opens with no test cases. Add conversations from search history, or copy test cases from another Dataset.

Add Test Cases from a Table​

To measure your pipeline on questions you keep in a spreadsheet, add them to a Dataset as table-based test cases. Each row of a CSV or XLSX file becomes one test case, and each column is either a pipeline input or expected data that your evaluator reads. For background, see Table-Based Test Cases.

Prepare the File​

  • Use a CSV or XLSX file with a header row. Each following row is one test case.
  • Include a column for each pipeline input you want to send, for example query. If a pipeline input has another name, name the column after it.
  • Add a column for each value your evaluator needs to judge the answer, for example expected_answer.
  • A Dataset holds at most 100 test cases, so keep the file within the room the Dataset has left. If the file has more rows, split it and add the parts to separate Datasets.

What Happens When You Replay a Row​

A replay of a row sends one request to the pipeline. The query value is sent as the query, and every other input goes to the pipeline input with the same name. If the test case has a setup script, the inputs it returns take precedence over the row's inputs. Your evaluator can read the row's inputs and expected data from session["test_case"].

For how to use them in a metric, see Write and Test an Evaluation Metric.

Optimization runs don't accept a Dataset that holds table-based test cases yet. To measure a Service on a Dataset, use one with recorded conversations only.

Copy Test Cases from Another Dataset​

Copy test cases to reuse them in another Dataset, including test cases recorded on other pipelines, without adding the conversations from search history again.

  1. Open the Dataset that should receive the test cases and click Copy test cases.
  2. In From Dataset, choose the source Dataset.
  3. In Test cases, select the test cases to copy.
  4. Click Copy. The button shows the number of selected test cases.

Each copy keeps the conversation, its pipeline, name, and scripts. The copy is independent of the original, so editing one leaves the other as it was.

Review and Edit Test Cases​

Open a Dataset to see its test cases. The table shows these columns:

  • Test case: The name. Click it to edit the test case.
  • Conversation: The ID of the conversation.
  • Pipeline: The pipeline the conversation was recorded on.
  • Scripts: Whether the test case has a Setup script, a Cleanup script, or none.
  • Actions: View chat opens the conversation, Edit opens the test case editor, and Remove deletes the test case after you confirm. View chat also works for test cases that came from an imported Dataset. For details, see View an Imported Conversation and Its Traces.

To edit a test case, click Edit. You can change its name. You can also write a setup script that runs before every replay, for example to give the replay environment variables or inputs. A cleanup script runs after every replay, even a failed one. A setup script defines one public function that takes no arguments and returns a dictionary with the optional keys env, inputs, and logs. A cleanup script defines one public function that takes the setup result (or None) and returns a dictionary with the optional key logs. Leave a script empty to skip that step. To reuse scripts, click Copy scripts from another test case. Nothing is saved until you click Save.

To remove several test cases at once, select them and click Remove selected. Removing a test case doesn't delete the conversation from search history, and Experiments that already ran it keep their results.

To rename a Dataset, click Rename. To delete it, click Delete and confirm. Deleting a Dataset deletes its test cases, but Experiments that already ran it keep their results.

View an Imported Conversation and Its Traces​

When you import a Dataset from an archive, each conversation is stored with the Dataset under a new Session ID. Search history doesn't hold these conversations, because they never ran in your workspace. You can still read them, and open the trace of each turn, to check what a test case replays before you run an Experiment.

To view an imported conversation:

  1. Open the imported Dataset.
  2. In the row of the test case, click View chat. The imported conversation drawer opens. It lists each turn with the question and the final answer.
  3. To see how the pipeline produced an answer, click Go to trace on a turn. The turn opens in the Session inspector, where you can step through its components, inputs, and outputs.

Each answer in the drawer comes from the turn's trace, so it matches the answer in the trace header. If the drawer shows "This conversation could not be loaded", the Dataset has no imported Session with this ID or the turn is missing from it.

Imported conversations have these limits:

  • Archives don't contain retrieved documents, feedback, or logs, so imported turns show none. The drawer tells you this.
  • In the Session inspector, actions that apply only to a pipeline are hidden for imported conversations.
  • The Session inspector on a scoring run page can't load imported conversations yet.

In an Experiment, the recorded side of an imported test case opens the same imported conversation drawer. Conversations replayed by the Experiment are still read from search history.

What To Do Next​

Use your Dataset to compare pipeline versions. For details, see Run an Experiment.