Skip to main content
For the complete documentation index for agents and LLMs, see llms.txt.

Add Built-In MetricsBeta

Beta
This is coming — this page describes a feature that's still being rolled out and may not be available to you yet.

Add ready-made Metrics to your workspace to measure agent efficiency, agent quality, RAG quality, and retrieval quality, so you can start evaluating without writing a Metric first.

About This Task​

A built-in Metric is a template. When you add it, the platform creates an ordinary Evaluator in your workspace from the template's name and code. The Evaluator has no link back to the template, so later improvements to a template never change what your existing Metric measures. That keeps the history of your results comparable. You can open the added Metric, read its code, and edit it like any Metric you wrote yourself. For how to edit one, see Write and Test an Evaluation Metric.

Metrics that use an LLM judge run on the platform's GPT-6 Luna model and open in the LLM-as-a-judge form, where you can swap the model. Metrics that read the trace don't use a model.

Available Built-In Metrics​

The Metrics come in four categories.

Agent Efficiency​

These Metrics read the trace and don't use a model. For all of them, lower is better.

  • Tool calls: How many tools the Agent called in each turn.
  • Failed tool calls: How many of the Agent's tool calls failed in each turn. A turn in which no Agent ran is not applicable.
  • Repeated tool calls: How often the Agent called the same tool with the same arguments again within one turn.
  • Agent iterations: How many times the Agent called its LLM in each turn. Many iterations for one answer often mean the Agent is retrying or can't find a tool.
  • User turns: How many turns a conversation took.

Agent Quality​

These Metrics use an LLM judge.

  • Goal reached: Whether the conversation reached the user's goal: reached, partially reached, or not reached.
  • Helpfulness: How helpful the assistant was, from 1 to 5.
  • User frustration: How frustrated the user was in each turn.
  • User approval: Whether the user approved or declined, for each turn that answers an approval request. The share of approved results is the approval rate.

RAG Quality​

These Metrics use an LLM judge and score from 0 to 1, unless noted.

  • Faithfulness (RAGAS): The share of claims in each reply that the retrieved documents support.
  • Attribution faithfulness: The share of the reply's citations that point to a source that supports the cited statement. A reply that cites nothing is not applicable.
  • Answer relevance: How well each reply addresses what the user asked. It doesn't measure correctness.
  • Answer correctness: Whether the conversation arrived at the expected answer: correct, partially correct, or incorrect. It needs a test case with an expected answer column.
  • Faithfulness (ragas): The same quality as Faithfulness (RAGAS), but computed by the ragas library, which extracts the statements, checks each one, and calculates the ratio itself. It needs an OPENAI_API_KEY secret in your workspace.

Retrieval Quality​

  • Recall: Whether the documents the pipeline retrieved include the ones the test case expects. It uses no model and needs a test case with expected documents. An expected entry matches a document by its ID, by one of its metadata values such as the file name, or, if the entry contains a space, by its text.
  • Context precision: The share of retrieved documents that are relevant to the question. It uses an LLM judge. A turn that retrieved nothing is not applicable.

Add Built-In Metrics​

  1. Open your pipeline, click Evaluation, and go to Metrics.
  2. Click Add built-in Metrics.
  3. Select the Metrics you want. The list is grouped by category and shows the model each Metric asks and the expected columns it needs. A Metric that your workspace already has is marked as added.
  4. Choose where the new Metrics go. By default, the Metrics of each category are added to a Metric Group with the category's name. The platform creates the group if it doesn't exist.
  5. Optional: Select Combine judged Metrics of a category into one to merge a category's judged Metrics that use the same model into one Evaluator named <category> judge. One model run then reads each conversation and answers every criterion. With the built-in Metrics, this gives Agent quality judge (four criteria) and RAG quality judge (three criteria). The combined Evaluator opens in the LLM-as-a-judge form.
    Criteria judged together tend to influence each other. If you need independent results, keep the option off.
    Code Metrics, Answer correctness, and a judge that is alone in its category stay separate.
  6. Confirm.
    The new Metrics appear on the Metrics tab, and the groups on the Metric Groups tab. If the Add built-in Metrics button doesn't appear, your environment doesn't support built-in Metrics yet.

Use the Metrics in an Experiment​

The added Metrics are ready to use. Select their Metric Group when you set up an experiment. For more information, see Run an Experiment.

Metrics that need expected data, Answer correctness and Recall, are not applicable to sessions that no test case stands behind, or whose test case lacks the expected column. To add test cases that carry expected data, see Build an Evaluation Dataset.

What To Do Next​