Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

Write and Test an Evaluation MetricBeta

Beta
This is coming — this page describes a feature that's still being rolled out and may not be available to you yet.

A metric is a question you ask about every conversation your pipeline has, such as "Was the answer grounded in the documents?" or "Did the agent use the search tool?" Write the metric, test it on real sessions until its answers match your own judgment, and then use it in experiments to measure your pipeline.

About This Task​

A metric is only useful if you trust it. A metric that marks good answers as bad, or the other way round, leads you to the wrong conclusions about your pipeline. That's why you test every metric on real sessions before you save it, and read the rationale it gives for each result.

For background on metrics, scores, labels, and LLM judges, see Evaluation Metrics.

Prerequisites​

  • Your pipeline has sessions to evaluate. Every query sent to the pipeline creates one. If you have none yet, run a few queries in Playground. Pick queries that represent what your users ask, including ones you expect the pipeline to struggle with.
  • To use an LLM judge, your workspace must have a model connected. You can add one in Settings. For details, see Using Hosted Models and External Services.

Decide What to Measure​

Before you open the editor, write down the question your metric answers, in one sentence. A good metric question is:

  • Specific: "Does the answer cite at least one retrieved document?" is easier to measure than "Is the answer good?"
  • About one thing: If you want to measure groundedness and tone, create two criteria or two metrics. When one metric mixes several qualities, a low result doesn't tell you what to fix.
  • Answerable from the conversation: The metric reads what the pipeline received, retrieved, called, and replied. It can't see anything outside the session.

Then decide how to measure it:

  • Choose LLM-as-a-judge for qualities that need judgment, such as groundedness, relevance, helpfulness, or tone.
  • Choose Code for things you can check exactly, such as whether a tool was called, whether a turn failed, or how many tokens a conversation used. Code is faster, cheaper, and returns the same result every time.

Create the Metric​

  1. In the sidebar, go to Evaluation. To work on a single pipeline, open the pipeline and click Evaluation instead.
  2. Go to Metrics>New Metric.
  3. Give the metric a name that says what it measures, for example grounded_answer. You'll see this name as a column in your experiment results.
  4. Under Method, select LLM-as-a-judge or Code, and then follow the matching section below. ::: Tip LLM Judges vs Code Metrics Use LLM-as-a-judge when you need to judge a quality that's hard to measure exactly, such as groundedness, helpfulness, or tone. Use code for qualities you can measure exactly, such as whether a tool was called or how many tokens a conversation used. :::

Write an LLM Judge​

  1. In What should the model judge?, describe the criterion as instructions to the model.
    The model can read the whole session: the turns, the retrieved documents, and the tool calls. Tell it what to look at instead of pasting content in. For example: "Check whether every claim in the final answer is supported by the retrieved documents. Ignore greetings and follow-up questions."
  2. In What should it answer?, choose how the model answers:
    • A label, from a list: Add at least two labels, for example grounded, partly grounded, and not grounded. Use this when you can.
      A model picks between clearly described options more reliably than it picks a number.
    • A score, within a range: Set Lowest and Highest under Range for every number. The range defaults to 0 to 1.
      The model is told the range, so scores stay comparable between sessions.
  3. In Answer it, select Once per conversation to judge the whole conversation, or Per turn to judge each answer separately.
    Per turn shows you exactly which answer in a conversation went wrong.
  4. Optional: Click Add criterion to judge more qualities in the same metric.
    Criteria judged together tend to influence each other. If you need independent results, create a separate metric for each criterion.
  5. In Which model should judge?, choose the model. Select Connected only to show only models you can use right now.
    The model's configuration is saved in the metric, so later changes to the model don't change this metric's results.

The form writes the metric as Python code. Select Code to read it. If you edit that code by hand, the metric stays code from then on.

Write a Code Metric​

  1. In the Evaluation Function editor, write exactly one public function that takes a session. Prefix any helper functions with an underscore so they're not treated as the metric.
    A new metric opens with starter code that checks whether an agent called a tool. Change it to fit your question.
  2. Return a result with a rationale:
    • Score(score=..., rationale=...) for a number.
    • Label(label=..., rationale=...) for a category.
    • NotApplicable(reason=...) when the question doesn't apply to a session, so it doesn't count toward your results.
  3. To read the session, import helpers from dc_haystack_utilities.evaluation. Click Helpers to see the full list. For example, traces(session) returns the session's turns, replies(trace) returns the pipeline's answers, component_output(trace, component) returns what a component produced, and failed(trace) tells you whether a turn failed.
    Click the info icon next to the editor to see the full contract, including how to return several metrics from one function with @metrics.

This example scores a session by the share of turns that ran without failing:

from dc_haystack_utilities.evaluation import Score, failed, traces


def turns_without_failure(session):
    runs = traces(session)
    failures = sum(1 for run in runs if failed(run))
    return Score(
        score=1 - failures / len(runs),
        rationale=f"{failures} of {len(runs)} turns failed",
    )

The Returns field above the editor shows what your metric produces, read from your code or from your last try. Check that it matches what you intended.

Declare a Score Range​

When a function returns several metrics with @metrics, you can declare the scale of each Score metric with range. Each entry in @metrics names a metric key and its type: Score(grain=..., range=...) or Label(grain=..., vocabulary=...). grain sets whether the metric answers once per session ("session") or once per turn ("turn"). Pass the lowest and highest value as a pair of numbers. This way, a metric that scores from 1 to 5 and one that scores from 0 to 1 each keep their own scale.

@metrics(
    helpful=Score(grain="session", range=(1, 5)),
    grounded=Score(grain="turn", range=[0, 1]),
)
def quality(session):
    ...

The platform reads the declaration without running your code, so write the range as literal numbers. When you save the metric, these rules apply:

  • You can set range on a Score only. A Label doesn't accept it.
  • The range must be two finite numbers, and the first must be smaller than the second, for example range=(1, 5). Booleans aren't valid bounds.

Test the Metric on Real Sessions​

Test before you save. A try runs whatever is in the editor, saved or not. It doesn't create a version or add results to any experiment, so you can experiment freely.

  1. In Try it, choose a mode:
    • One conversation: Use this while you write, to get quick feedback on a single conversation.
    • Several conversations: Use this to check the metric on a range of conversations at once. Each conversation is a row, and every try adds a column.
  2. Pick sessions from the list. It shows your pipeline's most recent sessions.
    Choose sessions where you already know the right answer: a clearly good one, a clearly bad one, and a borderline case. If the metric gets those right, you can trust it on the rest.
  3. Click Try, or Try on all in Several conversations mode. You can also press Ctrl+Enter (Command+Enter on Mac).
  4. Read the result and the rationale for each session. Ask yourself whether you'd give the same answer. If not, refine the criterion or the code, and try again.

If your code raises an error, the editor highlights the line that failed. Open the try to see the full traceback. An error in your function says nothing about your pipeline, so fix the function and try again. If a try couldn't run at all, for example because the session is too large, click Retry.

Compare Your Drafts​

Every try appears in Try history, grouped into Current draft and Earlier draft. When you try a session again after an edit, the history tells you whether the result changed, so you know which edit made the difference.

  • In Several conversations mode, a cell is flagged when its result changed from the previous column.
  • Click Compare to see two tries side by side.
  • Click Restore, in the history or from any cell in the results, to bring an earlier draft back into the editor. If your current edits aren't saved anywhere else, they're kept in the history first, so you don't lose them.

Tries last only until you leave the page, and the try history can't be kept across visits. If you have unsaved changes, the editor warns you before you leave. Save the metric when you're happy with it.

Choose Metric Groups for a New Metric​

A new metric must join at least one Metric Group before you can save it. A Metric Group is a named set of metrics in your workspace. For background, see Metric Groups.

In the metric editor, find the Metric Groups field and do one or both of the following:

  • Select one or more existing Metric Groups from the list. You can type to filter the list by name.
  • Add a new Metric Group and type its name. The platform creates the group together with the metric. If your workspace has no Metric Groups yet, the editor asks you to name one right away, because there is nothing to select.

To remove a new group you haven't saved yet, click the close icon next to its name. This option is available when your workspace already has other Metric Groups.

Save stays unavailable until the metric joins at least one group.

Save the Metric​

  1. Click Save metric. For an existing metric, the button shows the next version, for example Save as v3.
    If the code hasn't changed, there's nothing new to save, and the button stays disabled. Saving code that matches an existing version doesn't add a version, and going back to older code returns to the version that already holds it.
  2. If experiments already use this metric, a message tells you how many. Their next run uses the version you're saving. Past runs keep the results of the version they used, so you can still read them.

To look at an earlier version, choose it from Version at the top of the page. Saving changes to an older version creates a new version and doesn't overwrite anything. Click Back to latest to return. To drop unsaved edits, click Discard.

Keep a Declared Range Stable​

A metric key keeps one measure across versions, so results chart as one line. If a previous version declared a range for a Score, you can't save a version that changes or drops that range. To use a different scale, for example 1–5 instead of 0–1, save the metric under a new key.

You can add a range to a Score whose previous version declared none.

Manage Metric Groups​

To create, edit, or delete Metric Groups without creating a metric, see Manage Metric Groups.

What To Do Next​

Add your metric to an experiment and run it on your pipeline's sessions to see how your pipeline performs. When you create a metric, you can add it to existing Metric Groups, or start a new Metric Group with it, in the same step, so you don't have to edit the group afterward.