Evaluation MetricsBeta
Metrics are how Evaluation measures your pipeline.
A metric is a question you ask about every session, answered the same way each time. For example:
- Did the answer stay grounded in the retrieved documents?
- Did the agent call the search tool before answering?
- How polite was the reply, on a scale from 0 to 1?
- How many tokens did the conversation use?
A metric returns a value and a rationale: the measurement and the reason for it. The rationale lets you check whether the metric judged correctly, which is how you learn to trust it.
Scores and Labels
Each metric returns one of two kinds of value. A score is a number, such as 0.8. Use it for things that come in degrees, such as relevance or cost. Scores let you see a distribution and spot sessions at the low end.
To chart a score against the scale it's measured on, declare its range, for example 0 to 1 or 1 to 5. Each score in a metric can have its own range.
A label is one value from a list you define, such as called or skipped, or grounded, partly grounded, and not grounded. Use it for things that fall into categories. A yes-or-no question is a label with two values.
When you ask a model to judge, prefer labels. A model picks between clearly described categories more reliably than it places a value on a numeric scale.
Give a Label a Scale
A label on its own has no order and no direction. To say how good each word is, declare a Scale on a closed label, which is a label with a fixed list of words. Write its vocabulary in the @metrics entry as a mapping from each word to a number from 0 to 1, where higher is better. For example: @metrics(verdict=Label(grain="session", vocabulary={"wrong": 0, "partial": 0.5, "correct": 1})). The platform keeps the words in the order you wrote them, and returns the Scale with the metric's declaration.
Write the mapping as literals, because the platform reads the declaration without running your code. When you save the metric, these rules apply:
- Each value must be a finite number from 0 to 1. Booleans aren't valid values.
- No two words can have the same value.
- A Scale applies to a label with a fixed vocabulary only.
If a value breaks a rule, the save is refused, and the message names the metric key and the word to fix.
Like a score's direction, a Scale describes how to read a label, not what the metric measures. Any version of a metric can add, change, or drop its Scale. Narrowing a label's vocabulary is still refused, whether or not it has a Scale.
In an LLM-as-a-judge metric that answers with a label, you rank the answers in the judge form instead of writing code. Each word gets a value from 0 to 1, prefilled evenly in the order the words are listed, and the form writes the mapping for you. A new label starts ranked, with fail at 0 and pass at 1. A word you add to a ranked label starts without a value, so you place it yourself. If the answers have no order, select These answers don't rank (categories). The form then writes a plain list of words with no Scale.
The form doesn't let you save while a value is empty, outside 0 to 1, or repeated, or while a word is empty or repeated on a ranked label. Repeated words matter because a Python dictionary silently collapses them, so the form refuses them whenever a Scale is set. Fix the highlighted fields to save.
While you edit, words stay where you put them. They're sorted by value when you open a judge and when you save it. To sort them sooner, click Sort by value. It appears only when the words are out of order and every value is valid. A judge you wrote by hand keeps its own layout when you save. A judge written with the mapping form opens in the judge form, with the words in Scale order.
On the run page of a scoring run, a label with a Scale is shown from worst to best. Each bar is tinted from red at 0 to green at 1, with its word beside it. A label without a Scale keeps grey bars.
A metric can also report that it doesn't apply to a session, for example a tool-call metric on a session where no tool was needed. The platform counts those sessions separately, so they don't distort your results.
Not applicable is an answer, not a failure: the metric had nothing to judge.
Metric Direction
To see whether a change in a score is an improvement, declare which way is better: better="higher" or better="lower". Labels don't take a direction. Declare it in the score's @metrics entry, for example @metrics(helpful=Score(grain="session", range=(1, 5), better="higher")). Without a direction, comparisons show only the numeric change.
In an LLM-as-a-judge metric that answers with a score, you don't write code for this. Under the score's range, choose Higher is better, Lower is better, or Not set when neither direction is better.
When you compare experiments, or review the candidates of a pipeline optimization, the platform classifies how the candidate moved each metric against the baseline:
- Improved: the value moved in the direction you declared as better.
- Regressed: the value moved the opposite way.
- Unchanged: the difference is exactly zero. There's no tolerance.
- Unknown: there's no difference to read, or the score declares no direction.
Direction describes how to read a change, not whether a value is good, so the platform never judges an absolute value. The built-in pipeline metrics, such as cost, duration, and tokens, always count lower as better.
Any version of a metric can set, flip, or drop its direction, because direction isn't part of what the metric measures. The newest version that judged the metric on either side of a comparison decides the direction. Changing it in a new version therefore also changes how earlier results read.
Session, Turn, or Component
A metric can answer once for the whole session, once for each turn, or for a component within a turn. A component is one building block of your pipeline, such as the retriever, which finds documents. Choose the level that matches your question:
- Per session: "Did the conversation reach a resolution?"
- Per turn: "Was this answer grounded?"
- Per component: "Did the retriever return relevant documents?"
When a metric answers per turn, you can roll the results up per session using the average or the lowest-scoring turn. The lowest-scoring turn is useful when a single bad answer ruins the conversation.
Metric Versions
Each time you save a metric with new code, you get a new version. Experiments always use the latest version and record which version produced each result. You can improve a metric over time and still read past results exactly as they were measured.
When you save a new version, keep each metric key's shape, level, and declared score range, so results for that key still chart as one line. To change a score's scale, for example from 0–1 to 1–5, use a new key. You can add a range to a score that didn't have one.
A label's Scale isn't part of this rule. You can add, change, or drop it in any version.
Metrics belong to your workspace, so you can reuse one metric across several pipelines.
Archived Metrics
To tidy the Metrics page without changing what your experiments measure, archive a metric you no longer use. An archived metric is hidden from the Metrics page by default. It keeps its name, its versions, and its place in every Metric Group, and those groups still score with it. Archiving therefore never changes what a run measures, and you don't need to confirm anything.
To archive a metric:
- Open your pipeline, click Evaluation, and go to the Metrics page.
- In the metric's row, click Archive.
To list archived metrics, turn on Show archived, which also shows how many metrics are archived. The switch is off by default and appears only when at least one metric is archived. Archived metrics appear with an Archived tag. To put a metric back on the page, click Unarchive in its row.
Archived metrics behave like this elsewhere:
- A new Metric Group doesn't offer archived metrics. A group that already holds an archived metric keeps showing it by name.
- The Metric count on the Evaluation Overview page leaves archived metrics out.
- An archived metric keeps its name, so you can't give the name to a new metric.
LLM-as-a-Judge and Code
LLM-as-a-judge asks a language model to rate the session against criteria you describe in plain language. Use it for qualities that need judgment: groundedness, helpfulness, tone, or whether the answer addresses the question. You write the criteria, choose the answers the model can give, and pick the model, without writing code.
A code evaluation relies on a Python function that reads the session and computes the value. Use it for things you can check exactly: whether a tool was called, whether a turn failed, how many tokens the conversation used, or whether the answer contains a required phrase. Code is fast, cheap, and gives the same result every time.
Many teams combine both: code for the facts and an LLM judge for the qualities.
Both methods produce one metric that the platform stores as code. The LLM-as-a-judge form writes that code for you. You can switch to the code view to see it, and edit it by hand if you need more control.
An LLM judge can be wrong, so check its rationales.
Knowing how an LLM judge works helps you understand why some sessions end in error for a metric, and how to write criteria the judge can finish. The judge works in steps:
- It reads the parts of the session it needs, such as turns, retrieved documents, and tool calls.
- It sends one report that answers every criterion at once. If the report is incomplete or in the wrong shape, it's sent back and the judge can correct it.
- If the judge hasn't sent a valid report when its limited reading steps run out, it gets two more steps in which it can only report. It answers from what it has read and uses not applicable with a reason where it can't answer.
- If it still doesn't report, the session ends in error for that metric.
A specific criterion with few questions to answer helps the judge finish. For how to fix these errors, see Fix Sessions Where the Judge Didn't Conclude.
Read a Session in a Code Metric
A code metric reads a session through helpers you import from dc_haystack_utilities.evaluation. The helpers let you score the answer a RAG pipeline gave and the context it was given, without parsing a trace yourself. Tracing of content can be disabled per Run, so every reader returns an empty value when the data is missing.
Reach Turns by Number
Turns are numbered from 1, the same way as the focus argument of your function and turn/<n> result addresses.
turns(session, focus=None)returns a list of(n, trace)pairs for the turns your function is asked about. Withoutfocus, it returns every turn.turn(session, n)returns the trace of one turn.
If you ask for a turn the session doesn't have, both helpers raise NoSuchTurn, and the message lists the turns that exist. This way, a wrong turn number fails visibly instead of scoring an empty turn. Every other reader takes the trace of one turn.
Read the Answer and Its Context
To judge whether an answer is faithful to what the pipeline retrieved, read both from the same trace:
get_last_reply(trace)returns the answer: the reply of the last LLM that answered. For an Agent, that is its final step. For a pipeline without an Agent, it's the last chat generator. It returns an empty string when nothing replied or the last step only called tools.get_intermediate_replies(trace)returns the replies before the answer, such as an Agent's "Let me look that up." before it calls a tool.get_last_context(trace)returns everything the answering LLM was given. By default it keeps only user messages and tool results, so system prompts and the LLM's own earlier replies don't count as context. Passroles=to change which message roles it keeps.
Read Documents, Tools, and Skills
documents(trace)returns the documents that retrievers and rankers produced. By default, each document is listed once, matched byid, in its last recorded version. A reranked document therefore carries the ranker's score.tool_calls(trace)returns the tool calls of the turn.available_tools(trace)returns the tools the Agent could use.available_skills(trace)returns the skills the Agent could load.
Limit Readers to Some Components
replies, token_usage, models, and the readers above take a components= filter with one component name or several. A span matches when its own component name or the name of any component it ran inside is in the filter. An Agent's name therefore covers its LLM steps and tool calls. Use the filter to read one part of a pipeline, for example the tokens one Agent used, or the answer of one LLM when another LLM runs after it.
Update Existing Code Metrics
documents_for_turn, tool_calls_for_turn, and context_for_turn are no longer available in dc_haystack_utilities.evaluation. A metric that imports them fails with an ImportError. Use documents, tool_calls, and get_last_context on turn(session, n) instead. To read turns, use turns(session, focus) or turn(session, n) instead of traces(session)[n - 1].
What an LLM Judge Reads
When an LLM judge reads a turn, it sees the turn's inputs by name, in the form component.socket, with chat input cut down to the turn's new messages. It also sees the turn's answer and its intermediate replies separately. This way, a judge doesn't mistake an Agent's "Let me look that up." for the answer. To judge whether the answer is faithful, the judge can also read the context the answering LLM was given.
Related Information
Was this page helpful?