Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

Build an Evaluation DatasetBeta

Beta
This is coming — this page describes a feature that's still being rolled out and may not be available to you yet.

Keep the conversations that matter to you in a Dataset, so you can reuse them whenever you measure a change.

About This Task​

A Dataset holds conversations you want to compare pipeline versions on. It can include conversations from any pipeline in the workspace. To open the Datasets page, click Evaluation in the sidebar and go to Datasets, or open a pipeline's Evaluation tab and go to Datasets. Both show every Dataset in the workspace. On a pipeline's Evaluation tab, a Dataset lists only the test cases of that pipeline.

For background, see Datasets.

Add Conversations from Search History​

  1. Open a pipeline's search history and select the queries you want to keep. Each conversation becomes one test case, even if you select several of its queries.
  2. Click Add to Dataset.
  3. In Add to, choose Existing Dataset and select the Dataset, or choose New Dataset and type a name, for example refund regressions. If your workspace has no Datasets yet, the dialog starts on New Dataset.
  4. Click Add. A notification confirms how many test cases were added and links to the Dataset.

Each test case is named after its conversation's first query and records the pipeline you added it from.

Create an Empty Dataset​

  1. Go to Datasets and click New Dataset.
  2. Type a name and click Save. The new Dataset opens with no test cases. Add conversations from search history, or copy test cases from another Dataset.

Copy Test Cases from Another Dataset​

Copy test cases to reuse them in another Dataset, including test cases recorded on other pipelines, without adding the conversations from search history again.

  1. Open the Dataset that should receive the test cases and click Copy test cases.
  2. In From Dataset, choose the source Dataset.
  3. In Test cases, select the test cases to copy.
  4. Click Copy. The button shows the number of selected test cases.

Each copy keeps the conversation, its pipeline, name, and scripts. The copy is independent of the original, so editing one leaves the other as it was.

Review and Edit Test Cases​

Open a Dataset to see its test cases. The table shows these columns:

  • Test case: The name. Click it to edit the test case.
  • Conversation: The ID of the conversation.
  • Pipeline: The pipeline the conversation was recorded on.
  • Scripts: Whether the test case has a Setup script, a Cleanup script, or none.
  • Actions: View chat opens the conversation, Edit opens the test case editor, and Remove deletes the test case after you confirm.

To edit a test case, click Edit. You can change its name. You can also write a setup script that runs before every replay, for example to give the replay environment variables or inputs. A cleanup script runs after every replay, even a failed one. A setup script defines one public function that takes no arguments and returns a dictionary with the optional keys env, inputs, and logs. A cleanup script defines one public function that takes the setup result (or None) and returns a dictionary with the optional key logs. Leave a script empty to skip that step. To reuse scripts, click Copy scripts from another test case. Nothing is saved until you click Save.

To remove several test cases at once, select them and click Remove selected. Removing a test case doesn't delete the conversation from search history, and Experiments that already ran it keep their results.

To rename a Dataset, click Rename. To delete it, click Delete and confirm. Deleting a Dataset deletes its test cases, but Experiments that already ran it keep their results.

What To Do Next​

Use your Dataset to compare pipeline versions. For details, see Run an Experiment.