RagasEvaluator

A component that uses the Ragas framework to evaluate

Basic Information

• Type: haystack_integrations.ragas.src.haystack_integrations.components.evaluators.ragas.evaluator.RagasEvaluator

Inputs

Parameter	Type	Default	Description
query	Optional[str]	None	The input query from the user.
response	Optional[Union[List[ChatMessage], str]]	None	A list of ChatMessage responses (typically from a language model or agent).
documents	Optional[List[Union[Document, str]]]	None	A list of Haystack Document or strings that were retrieved for the query.
reference_contexts	Optional[List[str]]	None	A list of reference contexts that should have been retrieved for the query.
multi_responses	Optional[List[str]]	None	List of multiple responses generated for the query.
reference	Optional[str]	None	A string reference answer for the query.
rubrics	Optional[Dict[str, str]]	None	A dictionary of evaluation rubric, where keys represent the score and the values represent the corresponding evaluation criteria.

Outputs

Parameter	Type	Default	Description
result	EvaluationResult		A dictionary containing the evaluation result.

Overview

Work in Progress

Bear with us while we're working on adding pipeline examples and most common components connections.

A component that uses the Ragas framework to evaluate inputs against specified Ragas metrics.

Usage example:

from haystack.components.generators import OpenAIGenerator
from haystack_integrations.components.evaluators.ragas import RagasEvaluator
from ragas.metrics import ContextPrecision
from ragas.llms import HaystackLLMWrapper

llm = OpenAIGenerator(model="gpt-4o-mini")
evaluator_llm = HaystackLLMWrapper(llm)

evaluator = RagasEvaluator(
    ragas_metrics=[ContextPrecision()],
    evaluator_llm=evaluator_llm
)
output = evaluator.run(
    query="Which is the most popular global sport?",
    documents=[
        "Football is undoubtedly the world's most popular sport with"
        " major events like the FIFA World Cup and sports personalities"
        " like Ronaldo and Messi, drawing a followership of more than 4"
        " billion people."
    ],
    reference="Football is the most popular sport with around 4 billion"
              " followers worldwide",
)

output['result']

Usage Example

components:
  RagasEvaluator:
    type: ragas.src.haystack_integrations.components.evaluators.ragas.evaluator.RagasEvaluator
    init_parameters:

Parameters

Init Parameters

These are the parameters you can configure in Pipeline Builder:

Parameter	Type	Default	Description
ragas_metrics	List[Metric]		A list of evaluation metrics from the Ragas library.
evaluator_llm	Optional[BaseRagasLLM]	None	A language model used by metrics that require LLMs for evaluation.
evaluator_embedding	Optional[BaseRagasEmbeddings]	None	An embedding model used by metrics that require embeddings for evaluation.

Run Method Parameters

These are the parameters you can configure for the component's run() method. This means you can pass these parameters at query time through the API, in Playground, or when running a job. For details, see Modify Pipeline Parameters at Query Time.

Parameter	Type	Default	Description
query	Optional[str]	None	The input query from the user.
response	Optional[Union[List[ChatMessage], str]]	None	A list of ChatMessage responses (typically from a language model or agent).
documents	Optional[List[Union[Document, str]]]	None	A list of Haystack Document or strings that were retrieved for the query.
reference_contexts	Optional[List[str]]	None	A list of reference contexts that should have been retrieved for the query.
multi_responses	Optional[List[str]]	None	List of multiple responses generated for the query.
reference	Optional[str]	None	A string reference answer for the query.
rubrics	Optional[Dict[str, str]]	None	A dictionary of evaluation rubric, where keys represent the score and the values represent the corresponding evaluation criteria.