Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

Run an ExperimentBeta

Beta
This is coming — this page describes a feature that's still being rolled out and may not be available to you yet.

An experiment measures your pipeline with a set of metrics on the sessions you choose. Use it to see how your pipeline performs overall, find the conversations where it struggles, and check whether a change made it better.

About This Task​

An experiment is a question you come back to. You create it once, with the metrics that matter for your use case, and run it whenever you want an up-to-date answer: after you change a prompt, switch models, or when new production traffic comes in.

An experiment over sessions doesn't run your pipeline again. It measures sessions your pipeline already had. To measure a change this way, send queries to the changed pipeline first, and then run the experiment on those sessions. To have the platform replay test cases on two pipeline versions and compare them, see Compare Two Pipeline Versions.

To run the same test cases in another workspace, for example on a staging pipeline, export the Dataset and import it there. For steps, see Import and Export a Dataset.

For background, see Evaluation Experiments and Evaluation Metrics.

Prerequisites​

  • You have at least one metric you've tested and trust. To create a metric, see Write and Test an Evaluation Metric.
  • Your pipeline has sessions to measure. Every query sent to the pipeline, from Playground, a deployed app, or the API, creates one.

Create an Experiment​

  1. In the sidebar, under Testing, click Evaluation. To see only one pipeline's data, open that pipeline and click Evaluation instead.
  2. Go to Experiments>New experiment.
  1. Give your experiment a name that describes what the experiment answers, for example Groundedness and cost.
  2. In Metrics, choose the metrics to run. Pick a small set that covers what matters most, for example one metric for answer quality and one for cost. The order you pick them in is the order of the columns in the results.
  1. Click Save.

Experiments always use the latest version of each metric. When you improve a metric, the next run picks up the change, and past runs keep their original results.

The set of metrics you save is a Metric Group. It belongs to your workspace, not to this pipeline, so you can run the same Metric Group on any pipeline and compare the results. Before you create a new experiment, check whether a Metric Group with the metrics you need already exists.

Run the Experiment​

  1. Open the experiment and click Run experiment.
  2. In Sessions to judge, choose the sessions to measure. Choose sessions that represent what you want to learn about. To see how the pipeline does for real users, pick recent production sessions. To check a change, pick the sessions you created after the change.
  3. If the experiment has already measured some of these sessions, choose what to do under Sessions this experiment has already judged:
    • Fill in the gaps only: Measure only sessions without a result. Use this to add new sessions to the picture without repeating work.
    • Judge everything again: Measure all selected sessions. Use this after you change a metric and want every session measured with the new version.
  4. Check the number of results the run produces. It's the number of sessions times the number of metric results. LLM judges call a model for each result, so a large run takes longer and costs more.
  5. Click Run.

A run over hand-picked sessions can measure up to 100 sessions and produce up to 500 results. Your workspace can have up to three runs in progress at once.

To measure every session in a time window, such as the last seven days, see Run an Experiment Over a Time Window with the REST API.

Runs Waiting for Capacity and Transient Failures​

When you start many runs at once, for example a large experiment or a comparison, your organization's runtime and your judge's model provider can only handle so many calls at the same time. To keep runs from failing under that load, the platform limits how many Evaluation Runs of one organization execute at once. By default, up to six runs execute at the same time. Experiments, comparisons, and AI Operator runs all count toward this limit, so a busy AI Operator can make your experiment runs wait. The rest wait in the queue and start as earlier runs finish.

A run that waits for capacity stays in the Created state. The wait doesn't count against the run's time budget. If a run still hasn't started after 6 hours, the platform marks it as failed with a capacity error. The experiment run still closes, so the results you have remain available.

If a call fails for a temporary reason, the platform retries it automatically, up to three attempts in total and within the run's time budget. It retries when:

  • The runtime has no worker free, is unavailable, or asks the platform to slow down.
  • The model provider that powers an LLM judge rate limits the request, is overloaded, or times out.

The platform doesn't retry a metric whose code is broken, because it fails the same way every time. If all attempts fail, the cell shows the last error. Read the Results explains how scorecards flag sessions that broke.

A retry calls your LLM judge again, so it can add to your model provider costs. The limit of three attempts keeps this cost bounded.

Read the Results​

When the run finishes, each metric gets its own scorecard:

  • For a Score, the scorecard shows the mean and the p10 — the value below which the worst 10% of sessions fall — so you see the typical answer and how bad the worst ones get.
  • For a Label, the scorecard shows the most common label.

A scorecard also flags anything worth a closer look: how many sessions broke because the metric's code raised an error or the run failed, a source gap between where sessions came from, such as different Services or pipeline versions, any rare answers for a Label metric, and how many sessions are N/A because the metric didn't apply.

Click a scorecard to open its breakdown:

  • Distribution: the same chart as before, paired with a table that breaks the mean down by source, so you can tell whether a difference comes from the questions or from the pipeline that answered them. Click a source row to inspect the sessions behind it.
  • Evidence: the sessions most worth a look for this metric. It shows the sessions where the metric broke; if none broke, it shows the rarest labels for a Label metric or the lowest-scoring sessions for a Score metric. Click a row to inspect that session.
  • Click Inspect all Sessions to open every session this metric measured.

Any of these opens the session inspector: pick a session, then a turn, to see the value and the rationale behind it, along with the trace. A session whose traces are no longer available shows that it has no traces instead.

For metrics that answer per turn, select how to roll up each conversation: Mean per conversation for the typical answer, or Worst turn per conversation to find conversations where a single answer went wrong.

Sessions where a metric doesn't apply are counted as N/A and left out of the distribution.

Read How Label Metrics Changed Against the Baseline​

To see which labels the Candidate produces more or less often than the Baseline, for example whether more answers are now partial instead of pass, check the Δ vs Baseline column in the results table of an Experiment that compares a Candidate with a Baseline. A label Metric has no mean, so the column shows how the share of each label moved.

Each line in the column is one label:

  • The label name, on the left.
  • A bar that starts at a zero line in the middle. The bar grows to the right in blue for a label whose share grew, and to the left in orange for a label whose share shrank.
  • The change in share, beside the bar, for example partial +21%.

Blue and orange don't mean good and bad. Whether a label growing or shrinking is better depends on the Metric. Score Metrics use neutral arrows for the same reason.

To read the bars:

  • Length: A bar's length is the change in share. All label Metrics in the table use one scale, so the same change has the same length in every row. The scale never drops below ±10%, so a small change doesn't fill a bar.
  • Hover: To see the full change including the count, point to a line, for example partial +3 (+21%). Screen readers read each line the same way.
  • Order: The largest shifts come first.
  • Unchanged labels: Labels with the same share on both sides aren't listed. If no label changed, the column shows No change. If a side has no value, it shows —.

To see the exact label counts on each side as text, check the Candidate and Baseline columns or expand a test case's row.

Read Not-Applicable Counts in a Baseline and Candidate Comparison​

When an Experiment compares a Candidate with a Baseline, an Evaluator can declare a result not applicable: it returned neither a value nor a failure. For example, a tool-call Metric has nothing to judge on a session where no tool was needed. To see how much of a side's result is missing for this reason, and not because something broke, look for the n/a count:

  • In the results table of the Experiment, the count appears under each side's value, for example 2 n/a. It appears next to the errored count when a side has both.
  • In the test cases list, expand a test case's row. The same count appears under each side's value for that test case.

The n/a count appears in secondary text, not in the warning color used for errored counts, because not applying is an answer, not a failure. Nothing appears when a side has no not-applicable results.

A test case whose Metrics were all not applicable on one side is counted as scored, not Not scored yet.

Compare Runs​

Each run is kept in the experiment's history. Choose a run from the run picker to see its results, along with who started it, when, and which metric versions it used.

To compare two runs, make sure they measured similar sessions. If one run covers easy questions and the other covers hard ones, the difference comes from the questions, not your pipeline. To compare two versions of your pipeline fairly:

  1. Run the same set of queries on each version.
  2. Run the experiment once on the sessions from each version.
  3. Compare the distributions and look at sessions that changed.

To compare two versions on the same test cases without sending queries by hand, see Compare Two Pipeline Versions.

Evaluate a Single Turn From a Trace​

When you spot a problem while inspecting a trace, you can measure the whole conversation it belongs to on the spot:

  1. Open your pipeline and click Analytics > Traces, then open a trace.
  2. Click Evaluate this conversation.
  3. In Choose an Experiment, choose the experiment to run, and click Run on this conversation. If the pipeline has no experiments yet, click Go to Experiments to create one.
  4. Click Open in Experiment to see the result with its rationale.

The run judges every turn of the conversation. Runs started with Evaluate this conversation on a trace appear in the Experiment's run history tagged hand-picked. They don't feed the Experiment's trends. When you follow an Open in Experiment link, the Experiment opens on that run instead of the latest one. To learn more about working with traces, see Evaluate a Conversation from a Trace.

What To Do Next​