Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

Evaluation Datasets and Metric GroupsBeta

Beta
This is coming — this page describes a feature that's still being rolled out and may not be available to you yet.

A Dataset holds the test cases you measure your pipelines on, and a Metric Group holds the metrics you score them with. Together they are the inputs of an Experiment that compares two pipeline versions.

Datasets​

A Dataset is a collection of test cases that you keep for measuring your pipelines. Datasets belong to your workspace, not to a single pipeline, so one Dataset can hold test cases from several pipelines. Use a Dataset to keep the queries that matter to you, for example questions your users often ask or conversations where the pipeline failed, and reuse them whenever you measure a change.

Each test case shows the pipeline it comes from, so you can tell test cases apart when a Dataset mixes pipelines. To work with the test cases of one pipeline only, open the Datasets pages on that pipeline's Evaluation tab. They show only that pipeline's test cases.

You can build a Dataset in these ways. For steps, see Build an Evaluation Dataset.

  • Add from Search History: In Search History, select Add to Dataset on a query. The test case keeps a reference to the pipeline the query ran on.
  • Copy from another Dataset: Copy test cases from any Dataset in the workspace into the Dataset you're working on. Use this to reuse a good set of test cases instead of building it again.

To see how many Datasets, metrics, and Experiments your workspace has, check the tiles on the Evaluation Overview page.

Metric Groups​

A Metric Group is a named, ordered set of metrics in your workspace, for example "Answer quality" or "Cost and speed". Group metrics that belong together so you can pick the whole set by name instead of choosing metrics one by one each time.

Metric Groups work like this:

  • A Metric Group holds metrics, not pinned versions. When you save a new version of a metric, every group that contains it uses the latest version.
  • Results show metrics in the order you pick them.
  • A group needs at least one metric and can hold up to 50. Scoring with an empty group would judge nothing, so the platform refuses it.
  • A metric can belong to several groups.
  • A new metric must join at least one Metric Group before you can save it, either an existing group or one you create together with the metric. This keeps every metric in your workspace reachable through a group.
  • You can't delete a Metric Group while an optimization goal uses it.

You manage Metric Groups on the Metric Groups tab of the Metrics page. The AI assistant can also propose a Metric Group as part of a suggested experiment, and opens the same form pre-filled so you can review and adjust it before you save. For steps, see Manage Metric Groups.