Add Built-In MetricsBeta
Add ready-made Metrics to your workspace to measure agent efficiency, agent quality, RAG quality, and retrieval quality, so you can start evaluating without writing a Metric first.
About This Task
A built-in Metric is a template. When you add it, the platform creates an ordinary Evaluator in your workspace from the template's name and code. The Evaluator has no link back to the template, so later improvements to a template never change what your existing Metric measures. That keeps the history of your results comparable. You can open the added Metric, read its code, and edit it like any Metric you wrote yourself. For how to edit one, see Write and Test an Evaluation Metric.
Metrics that use an LLM judge run on the platform's GPT-6 Luna model and open in the LLM-as-a-judge form, where you can swap the model. Metrics that read the trace don't use a model.
Available Built-In Metrics
The Metrics come in four categories.
Agent Efficiency
These Metrics read the trace and don't use a model. For all of them, lower is better.
- Tool calls: How many tools the Agent called in each turn.
- Failed tool calls: How many of the Agent's tool calls failed in each turn. A turn in which no Agent ran is not applicable.
- Repeated tool calls: How often the Agent called the same tool with the same arguments again within one turn.
- Agent iterations: How many times the Agent called its LLM in each turn. Many iterations for one answer often mean the Agent is retrying or can't find a tool.
- User turns: How many turns a conversation took.
Agent Quality
These Metrics use an LLM judge.
- Goal reached: Whether the conversation reached the user's goal: reached, partially reached, or not reached.
- Helpfulness: How helpful the assistant was, from 1 to 5.
- User frustration: How frustrated the user was in each turn.
- User approval: Whether the user approved or declined, for each turn that answers an approval request. The share of
approvedresults is the approval rate.
RAG Quality
These Metrics use an LLM judge and score from 0 to 1, unless noted.
- Faithfulness (RAGAS): The share of claims in each reply that the retrieved documents support.
- Attribution faithfulness: The share of the reply's citations that point to a source that supports the cited statement. A reply that cites nothing is not applicable.
- Answer relevance: How well each reply addresses what the user asked. It doesn't measure correctness.
- Answer correctness: Whether the conversation arrived at the expected answer: correct, partially correct, or incorrect. It needs a test case with an expected
answercolumn. - Faithfulness (ragas): The same quality as Faithfulness (RAGAS), but computed by the ragas library, which extracts the statements, checks each one, and calculates the ratio itself. It needs an
OPENAI_API_KEYsecret in your workspace.
Retrieval Quality
- Recall: Whether the documents the pipeline retrieved include the ones the test case expects. It uses no model and needs a test case with expected
documents. An expected entry matches a document by its ID, by one of its metadata values such as the file name, or, if the entry contains a space, by its text. - Context precision: The share of retrieved documents that are relevant to the question. It uses an LLM judge. A turn that retrieved nothing is not applicable.
Add Built-In Metrics
Before you start, make sure your workspace has a pipeline. If you plan to add Faithfulness (ragas), add an OPENAI_API_KEY secret to the workspace first.
- Open your pipeline, click Evaluation, and go to Metrics.
- Click Add built-in Metrics. If the button doesn't appear, your environment doesn't support built-in Metrics yet.
- Select the Metrics you want.
- Choose where the new Metrics go. By default, the Metrics of each category are added to a Metric Group with the category's name. The platform creates the group if it doesn't exist.
- Optional: Select Combine the Metrics of a category into one to merge a category's judged Metrics that use the same model into one Evaluator named <category> judge. One model run then reads each conversation and answers every criterion. With the built-in Metrics, this gives Agent quality judge (four criteria) and RAG quality judge (three criteria). The combined Evaluator opens in the LLM-as-a-judge form.
Criteria judged together tend to influence each other. If you need independent results, keep the option off.
Code Metrics, Answer correctness, and a judge that is alone in its category stay separate, except that the code Metrics of a category can combine into one <category> metrics Evaluator (see Combine Metrics You Already Added). - Confirm.
The new Metrics appear on the Metrics tab, and the groups on the Metric Groups tab.
Combine Metrics You Already Added
Combining also works on Metrics you added earlier on their own, so you can merge Metrics you already use into one Evaluator and run them together. With the combining option selected (labeled Combine the Metrics of a category into one), the dialog lists every Metric, including ones you added on their own (tagged Added) and ones already in a combined Metric (tagged In a combined Metric). Select the ones you want to merge. With combining off, Metrics you added on their own aren't listed.
Combining adds the combined Metric next to the individual ones. Nothing is archived, renamed, or moved between Metric Groups.
Combining also covers code Metrics. The code of a category's code Metrics is merged into one Evaluation Function that reports all their Metrics. For example, the five agent efficiency Metrics become one Metric named Agent efficiency metrics, next to Agent quality judge. Each Metric keeps its own code inside the combined function, so results match the individual Metrics.
A category can have several combined Metrics. If you pick a different set of Metrics than an existing combined Metric holds, the platform adds another one and numbers its name, for example Agent efficiency metrics 2. Nothing is added when your pick matches exactly what a listed combined Metric already holds, or when an added Metric has nothing to combine with.
Use the Metrics in an Experiment
The added Metrics are ready to use. Select their Metric Group when you set up an experiment. For more information, see Run an Experiment.
Metrics that need expected data, Answer correctness and Recall, are not applicable to sessions that no test case stands behind, or whose test case lacks the expected column. To add test cases that carry expected data, see Build an Evaluation Dataset.
What To Do Next
Was this page helpful?