Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

Define Evaluation Goals on a PipelineBeta

Beta
This is coming β€” this page describes a feature that's still being rolled out and may not be available to you yet.

An Evaluation goal states what good enough looks like for a pipeline: which metric to read, what a single session must reach to pass, and how many sessions must pass.


About This Task​

A metric gives you a value for every session. A goal turns that value into a pass or fail rule, so you can tell whether the pipeline meets your bar before you ship a change or compare versions. For background, see Evaluation Goals. A goal reads one metric from one Evaluator in your workspace. You define goals per pipeline, so two pipelines can hold different goals on the same Evaluator.

A goal has these parts:

  • Name and description: What the goal checks and why it matters.
  • Evaluator: The Evaluator whose metric the goal reads.
  • Pass rule: The condition a single session must meet. It depends on the kind of metric (see below).
  • Minimum pass rate: The share of sessions that must pass for the goal to be met, from 0 to 1. The default is 1, which means every session must pass.

The goal is met when the number of passing sessions divided by the number of sessions with a value is at least the minimum pass rate. Sessions where the metric was not applicable, ended in error, or hasn't finished don't count either way.

How the Pass Rule Works​

The pass rule follows how the metric is declared:

  • Score metric: Set a threshold. A session passes when the mean of its values is at or above the threshold if the score declares that higher is better, or at or below the threshold if it declares that lower is better. The goal reads the direction from the metric, so if a later version of the metric flips it, its goals flip with it. The threshold must fall inside the score's declared range, if it has one.
  • Label metric: Choose the passing values, the words that count as a pass. A session passes when every one of its rows has one of these words.

For background on scores, labels, and direction, see Evaluation Metrics.

Prerequisites​

  • An Evaluator whose code declares exactly one metric. You write it as part of creating the goal. For a Score, the metric must declare which way is better (better="higher" or better="lower"). For a Label, the metric must have a fixed list of words.
  • Permission to edit the pipeline. Deleting a goal needs the same permission you need to delete other pipeline resources.

Add a Goal​

  1. Open your pipeline and click Evaluation. The Evaluation goals page opens.
  2. Click New Evaluation goal. The goal form opens as a page of its own.
  1. Enter a name. Optionally, add a description.
  2. Write the Evaluator for the goal, as Code or as an LLM-as-a-judge. Saving the goal creates a new Evaluator and doesn't change any existing one. For how to write each kind, see Write an LLM Judge and Write a Code Metric.
  3. Set the pass rule. For Code, the form reads the metric your code declares as you type, so you can set the rule without saving the Evaluator first:
    • For a Score, enter the threshold.
    • For a Label, select the passing values.
  4. Optionally, change the minimum pass rate. Leave it at 1 if every session must pass.
  5. Optionally, click Try to run your unsaved Evaluator on recent sessions before you save.
  6. Click Save. The platform creates the Evaluator, and then the goal.

Result: The goal appears in the list for this pipeline. To see how your sessions score against it, click Evaluate now. For details, see Evaluating Step-by-Step.

Evaluate Goals on a Window, a Dataset, or an Experiment​

Evaluate now judges every goal on the Sessions of the last 7 days. To check goals on other Sessions, to judge fewer Sessions, or to compare two versions of the pipeline under the same rules, use the chooser. For background, see Sources of an Evaluation Run.

  1. On the pipeline's Evaluation tab, click the arrow beside Evaluate now. The Evaluate on… dialog opens.
  2. Choose what to do:
    • Run all Evaluation goals, and then choose the Sessions:
      • Sessions in a time window: Pick Last 24 hours, Last 7 days, or Last 30 days, or set a custom range. Optionally, set a maximum number of Sessions to judge only the newest ones in the window. A run with a maximum counts as a custom run, even over the last 7 days, so it doesn't feed the default verdicts or the trend.
      • A Dataset: Pick a Dataset, and then choose Score its recorded Sessions or Replay on the draft, then score. Replaying sends each Session to your draft and scores the draft and the recorded Sessions side by side. It needs a draft version of the pipeline.
      • A new Dataset: Pick a time window and name the Dataset. The run saves the window's Sessions as a Dataset, so you can evaluate them again later. The name defaults to the pipeline name and the window.
    • Run an Experiment, and then set up both sides:
      1. Optionally, enter an Experiment name.
      2. Under Baseline, keep the draft or pick another version.
      3. Under Candidate, pick the version you compare with the Baseline.
      4. Give each side a label, for example "Shorter prompt". Labels are at most 64 characters.
      5. Choose the Sessions the same way as for a run: a time window, a Dataset, or a new Dataset.
      6. Optionally, choose the replay mode, which sets what each replay sends (the default is the first user message of each Session), and the number of Sessions to score per test case and side (the default is 1). A time window keeps that many times fewer test cases.
  3. Click Start evaluation, or Start Experiment for an Experiment.

Result: Progress shows on the Evaluate now button, as for any run. When the run finishes, use the Showing switcher to pick which run or Experiment the cards' Pass or Fail comes from. With an Experiment selected, each card shows both sides, for example "Draft Fail Β· Shorter prompt Pass", and marks a side that is still scoring as partial.

If the arrow beside Evaluate now or the Showing switcher doesn't appear, the platform can't list past runs yet. Evaluate now still runs on the last 7 days.

Check the Size of a Run First​

A run is refused when it would make more judgments than the per-run limit allows. A judgment is one Session judged by one Evaluator, so the count is the Sessions judged times the distinct Evaluators of your goals, and for an Experiment, also times the Sessions per test case. The chooser sizes the run before you start it when the platform supports it.

If a run is refused for size, make it smaller:

  • For a time window, set a lower maximum number of Sessions.
  • For an Experiment, lower the Sessions per test case. If even one Session is over the limit, the goals use too many Evaluators for that number of replays, so remove goals that use other Evaluators.
  • For a Dataset, pick a Dataset with fewer test cases.

A window or a new Dataset is trimmed to stay within the limit, so only a Dataset run is refused for size.

Edit or Delete a Goal​

Each goal is a card in the list. Click a card to unfold it and change the goal. You can:

  • Change the name and description in place. They turn into editors when the card is unfolded: click the text, edit it, and the pen icon after the last character shows that it's editable.
  • Change the passing rule and the minimum pass rate in the card's body.
  • Change the Evaluator in a dialog. See Edit and Try the Evaluator Behind a Goal.

The actions in the card's body are Drop, which stands apart from the others, and then

Evaluator, and Save. Save saves the passing rule only. It stays unavailable until the goal has a name, a pass rule, and a minimum pass rate. The save replaces the whole goal, so every field is checked again. A new Evaluator version isn't saved with it, because the Evaluator has its own Save in its dialog.

Result: The goal keeps the new rule, and its verdict reflects it. A change to the threshold or pass rate shows the new verdict right away, because the platform recomputes verdicts from the stored results. If the last run can't answer the rule as you've set it, or didn't judge the latest version of the Evaluator, the goal shows Not evaluated yet or a Stale tag. Click Evaluate now to get a verdict.

To delete a goal, unfold its card, click Drop, and confirm. Deleting removes only the rule: the metric and its results stay. The goal no longer appears in the list.

If you delete the Evaluator a goal reads, or delete the pipeline, its goals are deleted with it.

Open the Run Behind a Trend Point​

When a goal was missed on a past run, open that run to see which other goals moved with it. The trend on an unfolded goal card shows a chart once the goal has two or more runs with enough Sessions.

  1. Unfold the goal's card.
  2. In the trend chart, click the point for the run you want. Points grow when you hover over them.

Result: The run opens in a panel titled with the goal's name and the run's date. It is the full run, so it covers every goal judged in it, not only the goal you clicked. The goal you came from is selected.

In the panel:

  • The Goals in this run list shows each goal's pass rate against its target. Goals that need attention come first, sorted by how far they are from their target. Other metrics lists metrics in the run that no goal targets.
  • The details beside the list show the selected goal's headline value, how many Sessions were judged, how many broke, and how many were not applicable, followed by its breakdown. Click another goal in the list to see its details.

On narrow screens, the list stacks above the details.

Edit and Try the Evaluator Behind a Goal​

If a goal is missed and you suspect the Evaluator, not the pipeline, change the Evaluator without leaving the goal. To open it, unfold the goal's card and click Evaluator. The dialog lists the Evaluator's versions, newest first, and shows the selected version in the form that fits it:

  • LLM judge: An Evaluator whose code has the shape of a judge written in the judge form opens as that form. You edit the criteria, which are the prompt the judge follows, and select the model that judges. For what each field does, see Write an LLM Judge.
  • Code: Any other Evaluator opens in the code editor. For how to write the function, see Write a Code Metric.

The platform tells the two apart from the Evaluator's code, so an Evaluator doesn't have a separate type. If you edit a judge's code by hand elsewhere so that it no longer has the judge shape, it opens as code from then on.

To test your change before you save it:

  1. In the dialog, edit the criteria, the model, or the code. To start from an older version, pick it in the list of versions first.
  2. Click Try. A panel slides in from the right. The current draft runs on your pipeline's recent sessions, and the panel lists the verdict and rationale for each. Nothing is saved.
  3. Read the rationales. If a verdict isn't what you'd decide yourself, refine the draft and try again.

Tries aren't stored, so the list is empty again when you close the panel and leave the goal.

To keep your change, click Save in the dialog and confirm. Save stays unavailable until the code differs from the latest version.

Result: The platform adds a new version of the Evaluator, and it becomes the latest. If you edited from an older version, saving restores that version's content as a new latest version. The Evaluator is shared: goals read the latest version of their Evaluator, so other goals that use the same Evaluator, including goals on other pipelines, use the new version too.

If saving fails, the dialog shows the reason. Fix the cause and save again.

Fix a Goal That Can't Be Saved​

A goal is refused when its rule doesn't fit the metric. The message names the field to fix. The most common causes are:

  • The Evaluator doesn't meet the prerequisites: it must declare exactly one metric, with better for a Score or a fixed list of words for a Label. Fix the Evaluator code and save again.
  • The rule field doesn't match the metric: you set a threshold for a Label, or passing values for a Score.
  • The threshold is outside the score's declared range.
  • A passing value isn't one of the label's words.
  • The Evaluator code declares a metric that can't be read as written, for example one that uses a variable where a fixed value is needed. Write the declaration with plain values.