Evaluation MetricsBeta
Metrics are how Evaluation measures your pipeline.
A metric is a question you ask about every session, answered the same way each time. For example:
- Did the answer stay grounded in the retrieved documents?
- Did the agent call the search tool before answering?
- How polite was the reply, on a scale from 0 to 1?
- How many tokens did the conversation use?
A metric returns a value and a rationale: the measurement and the reason for it. The rationale lets you check whether the metric judged correctly, which is how you learn to trust it.
Scores and Labels
Each metric returns one of two kinds of value. A score is a number, such as 0.8. Use it for things that come in degrees, such as relevance or cost. Scores let you see a distribution and spot sessions at the low end.
To chart a score against the scale it's measured on, declare its range, for example 0 to 1 or 1 to 5. Each score in a metric can have its own range.
A label is one value from a list you define, such as called or skipped, or grounded, partly grounded, and not grounded. Use it for things that fall into categories. A yes-or-no question is a label with two values.
When you ask a model to judge, prefer labels. A model picks between clearly described categories more reliably than it places a value on a numeric scale.
A metric can also report that it doesn't apply to a session, for example a tool-call metric on a session where no tool was needed. The platform counts those sessions separately, so they don't distort your results.
Session, Turn, or Component
A metric can answer once for the whole session, once for each turn, or for a component within a turn. A component is one building block of your pipeline, such as the retriever, which finds documents. Choose the level that matches your question:
- Per session: "Did the conversation reach a resolution?"
- Per turn: "Was this answer grounded?"
- Per component: "Did the retriever return relevant documents?"
When a metric answers per turn, you can roll the results up per session using the average or the lowest-scoring turn. The lowest-scoring turn is useful when a single bad answer ruins the conversation.
Metric Versions
Each time you save a metric with new code, you get a new version. Experiments always use the latest version and record which version produced each result. You can improve a metric over time and still read past results exactly as they were measured.
When you save a new version, keep each metric key's shape, level, and declared score range, so results for that key still chart as one line. To change a score's scale, for example from 0–1 to 1–5, use a new key. You can add a range to a score that didn't have one.
Metrics belong to your workspace, so you can reuse one metric across several pipelines.
LLM-as-a-Judge and Code
LLM-as-a-judge asks a language model to rate the session against criteria you describe in plain language. Use it for qualities that need judgment: groundedness, helpfulness, tone, or whether the answer addresses the question. You write the criteria, choose the answers the model can give, and pick the model, without writing code.
A code evaluation relies on a Python function that reads the session and computes the value. Use it for things you can check exactly: whether a tool was called, whether a turn failed, how many tokens the conversation used, or whether the answer contains a required phrase. Code is fast, cheap, and gives the same result every time.
Many teams combine both: code for the facts and an LLM judge for the qualities.
Both methods produce one metric that the platform stores as code. The LLM-as-a-judge form writes that code for you. You can switch to the code view to see it, and edit it by hand if you need more control.
An LLM judge can be wrong, so check its rationales.
Knowing how an LLM judge works helps you understand why some sessions end in error for a metric, and how to write criteria the judge can finish. An LLM judge works in steps. It reads the parts of the session it needs, such as turns, retrieved documents, and tool calls, and then sends one report that answers every criterion at once. A report that's incomplete or in the wrong shape is sent back, and the judge can correct it. The judge has a limited number of steps for reading. If it hasn't sent a valid report when those steps run out, it gets two more steps in which it can only report. In those steps, it answers from what it has read and uses not applicable with a reason where it can't answer. If it still doesn't report, the session ends in error for that metric. A specific criterion with few questions to answer helps the judge finish. For how to fix these errors, see Fix Sessions Where the Judge Didn't Conclude.
Related Information
Was this page helpful?