Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

Evaluation Datasets and Metric GroupsBeta

Beta
This is coming — this page describes a feature that's still being rolled out and may not be available to you yet.

A Dataset holds the test cases you measure your pipelines on, and a Metric Group holds the metrics you score them with. Together they are the inputs of an Experiment that compares two pipeline versions.

Datasets​

A Dataset is a collection of test cases that you keep for measuring your pipelines. Datasets belong to your workspace, not to a single pipeline, so one Dataset can hold test cases from several pipelines. Use a Dataset to keep the queries that matter to you, for example questions your users often ask or conversations where the pipeline failed, and reuse them whenever you measure a change.

Each test case shows the pipeline it comes from, so you can tell test cases apart when a Dataset mixes pipelines. To work with the test cases of one pipeline only, open the Datasets pages on that pipeline's Evaluation tab. They show only that pipeline's test cases.

You can build a Dataset in these ways. For steps, see Build an Evaluation Dataset.

  • Add from Search History: In Search History, select Add to Dataset on a query. The test case keeps a reference to the pipeline the query ran on.
  • Copy from another Dataset: Copy test cases from any Dataset in the workspace into the Dataset you're working on. Use this to reuse a good set of test cases instead of building it again.
  • Import from an archive: Import a Dataset from an archive. Each imported conversation is stored with the Dataset under a new Session ID, so search history doesn't hold it. You can still read it and open the trace of each turn.
  • Add from a table: Upload a CSV or XLSX file and turn each row into a test case. Use this when you have questions and expected answers in a spreadsheet but no recorded conversations.

To see how many Datasets, metrics, and Experiments your workspace has, check the tiles on the Evaluation Overview page. The metrics tile leaves out archived metrics.

You can export a Dataset as a .zip archive and import it into another workspace, for example to run the same regression checks on a staging pipeline. The archive holds the conversations themselves, so imported test cases replay like any other test case. For steps, see Import and Export a Dataset.

An archive is imported as it is, without choosing a pipeline. Its test cases belong to no pipeline, so each one runs on whichever pipeline version an Experiment picks.

Table-Based Test Cases​

A table-based test case is one row of a CSV or XLSX file instead of a recorded conversation. Use it to measure a pipeline on questions you wrote down, for example a spreadsheet of questions with the answers you expect. Each column of the file has one of two roles:

  • Input: A value sent to the pipeline, for example query.
  • Expected: Data the evaluator reads to judge the answer, for example expected_answer. The pipeline never receives it.

A test case that replays a recorded conversation can carry expected data too.

Table-based test cases work like this:

  • A row is named after its query input. If the row has no query, it is named after its first text input.
  • A replay of a row sends one request. The query input is sent as the query, and every other input is passed to the pipeline input with the same name. Inputs returned by a test case's setup script take precedence over the row's inputs.
  • Because a row has no recorded conversation, the Whole conversation replay option doesn't apply to it.
  • An evaluator reads the test case as session["test_case"]. It holds the test case's name, inputs, and expected. For example, a code metric reads session["test_case"]["expected"]["answer"] to compare the pipeline's answer with the expected one. For how to write such a metric, see Write and Test an Evaluation Metric.
  • An Experiment never reuses an earlier evaluation for a replay of a row, because that evaluation might have read different expected data.
  • If an Experiment compares a Candidate with the Recorded conversations Baseline on a Dataset that holds only rows, the Baseline is absent: nothing is replayed or scored for it, and the results show the Candidate alone, with no Baseline values or changes. On a Dataset that mixes rows and recorded conversations, the comparison works as usual.
  • A Dataset holds at most 100 test cases.
  • Optimization runs don't accept Datasets that hold rows yet.

To add a table to a Dataset, see Add Test Cases from a Table.

Metric Groups​

A Metric Group is a named, ordered set of metrics in your workspace, for example "Answer quality" or "Cost and speed". Group metrics that belong together so you can pick the whole set by name instead of choosing metrics one by one each time.

Metric Groups work like this:

  • A Metric Group holds metrics, not pinned versions. When you save a new version of a metric, every group that contains it uses the latest version.

  • Results show metrics in the order you pick them.

  • A group needs at least one metric and can hold up to 50. Scoring with an empty group would judge nothing, so the platform refuses it.

  • A metric can belong to several groups.

  • A new metric must join at least one Metric Group before you can save it, either an existing group or one you create together with the metric. This keeps every metric in your workspace reachable through a group.

  • You can't delete a Metric Group while an optimization goal uses it.

  • Archiving a metric doesn't remove it from its groups. Those groups still score with it, and you can't add it to a new group while it's archived. For details, see Archived Metrics.

You manage Metric Groups on the Metric Groups tab of the Metrics page. The AI assistant can also propose a Metric Group as part of a suggested experiment, and opens the same form pre-filled so you can review and adjust it before you save. For steps, see Manage Metric Groups.

Metric Weights​

A Metric Group can weight its metrics, so that some count more than others toward the group's Composite, the single number that combines the group's metrics. For example, in an "Answer quality" group you might weight groundedness at 3 and politeness at 1. Weights are relative: only their proportions matter, so weights of 3 and 1 count the same as 6 and 2.

You set weights through the REST API or in the Weights section of the Metric Group form when you create or update a Metric Group. Each weight names a metric by its Evaluator and its metric key, and gives a number above 0. In the form, each metric that can be weighted has a weight field that shows the metric's share of the total weight, for example 75%. Leave a weight empty to keep its metric out of the Composite. A weight of 0 or less, or one that isn't a number, blocks Save until you fix it.

A metric can be weighted only when it has a normalized mean, which is its average result on a common 0 to 1 scale. This is what lets the Composite combine metrics that are measured differently. These metrics have one:

  • A score that declares both a range and a direction (better).
  • A label that declares a Scale.

A metric without one can't be weighted, including a metric that only has a legacy declaration. To add a range, a direction, or a Scale, see Evaluation Metrics. In the Metric Group form, a metric that can't be weighted has no weight field. It shows what its declaration is missing, for example that you need to declare which way is better for the score.

The platform checks weights when you save the group and refuses the whole save, with one error that names every problem, if:

  • A weight is 0, negative, or not a finite number.
  • The same Evaluator and metric key is weighted twice.
  • A weight names an Evaluator that isn't in the group.
  • A weight names a key that the Evaluator's latest version doesn't declare.
  • A weighted metric has no normalized mean.

Saving weights behaves like this:

  • Sending weights replaces the stored ones. Sending an empty list clears them.

  • Leaving weights out of an update keeps the stored weights of the Evaluators that stay in the group and drops the weights of Evaluators you remove. This is why saving a group in the UI doesn't clear its weights.

  • Leaving weights out when you create a group means the group has no weights.

  • The Metric Group form sends weights only when you change them, so renaming a group never drops its weights.

Weights are stored as you send them, and they stay with the metric's Evaluator, not with a version. When you read a group, each weight reports whether it's still usable against the Evaluator's latest version. A newer version can drop the key or remove its range, direction, or Scale. In that case the weight is kept and reads back as not usable, with the reason. Reading a group never fails because of a stale weight. To make the weight usable again, restore the declaration in a new version of the metric, or replace the weights. In the Metric Group form, such a weight shows the saved value and the reason it doesn't count. It isn't sent back when you save.

Normalized Means and the Composite​

To see how a Metric Group did in a run with one number, read the run's Composite. The scoring-run read of the REST API reports each Metric's normalized mean in normalized_means and the group's Composite in composite. The platform computes both when you read the run, from the run's stored results, the declarations of the Evaluator versions that ran, and the group's current weights. Because the weights are current, changing them changes the Composite of earlier runs, and every reader gets the same number.

A Metric's normalized mean is its whole-run value on a scale from 0 to 1, where 1 is best:

  • Score: The mean is placed in the Score's declared range. If the Score declares better="lower", the result is turned around so that a lower mean gives a higher value. The result is clipped to 0 and 1.
  • Label: Each label's share of the Sessions is multiplied by that label's value on the Label's Scale, and the products are added up. The Scale counts only when every Evaluator version behind the result declares the same one.

Each entry of normalized_means has a kind. With kind set to value, it holds the normalized mean. With kind set to missing, it holds the reason the Metric has none, for example that a Score declares no range, that no Session has a value for it, or that the Evaluator versions that ran disagree on its range, direction, or Scale. The list is empty until the run has stored its results, which happens when the run ends.

The Composite is the weighted mean of the weighted Metrics' normalized means. The platform divides the weights by their sum, so weights of 2, 1, and 1 count as 0.5, 0.25, and 0.25. With kind set to value, the Composite also reports coverage: the lowest coverage among the weighted Metrics, which is the share of judged Sessions that have a value for the Metric with the fewest.

A run has no partial Composite. If any weighted Metric has no normalized mean, the Composite is missing. A Composite over only the Metrics that happen to have a value would move when one of them dropped out, and that would look like a change in quality. Instead, the response has kind set to missing, a reason, and a missing list that names each weighted Metric without a normalized mean, with its own reason. Fixing those Metrics brings the Composite back. A weighted Metric that is no longer usable, or that the run doesn't report, counts as missing too. The Composite is also missing when the Metric Group has no weights, or when the run has no stored results yet.

Where You See the Composite​

You don't need the REST API to read a Composite. Use it to judge a run or an Experiment at a glance, then open the individual Metrics to find out what moved it.

  • Scoring-run page: The page leads with the run's Composite, a number from 0 to 1, and its coverage. Coverage tells you how much of the run the number is based on.
  • Experiment page: The page leads with the Composite of the Candidate and of the Baseline, and the change between them. The change is in absolute points, for example +0.04. It is green when the Candidate scores higher and red when it scores lower. If the Experiment has no Baseline, the page shows only the Candidate's Composite.

When a Composite is missing, the page says why and lists each weighted metric that has no normalized mean, with its reason. Fix those metrics, for example by declaring a range and a direction for a score, and the Composite comes back. If the reason is that the group has no weights, add weights in the Metric Group form.