EvaluationDataset and evaluate(): the classic path

Search for RAGAS anywhere and you will find this: build an EvaluationDataset, call evaluate, read a dictionary of averages. It is the older API, it is everywhere, and it still works.

Example

from pretend_ragas import LegacyJudge
from ragas import EvaluationDataset, SingleTurnSample, evaluate
from ragas.metrics import Faithfulness, LLMContextRecall

context = ["Order A17 shipped on 3 March by courier."]
samples = [
    SingleTurnSample(user_input="Where is order A17?", response="Order A17 shipped on 3 March by courier.",
                     retrieved_contexts=context, reference=context[0]),
    SingleTurnSample(user_input="Where is order A17?", response="Order A17 was cancelled.",
                     retrieved_contexts=context, reference=context[0]),
]
result = evaluate(dataset=EvaluationDataset(samples=samples),
                  metrics=[Faithfulness(), LLMContextRecall()], llm=LegacyJudge())
print(result)

One call, every metric on every sample, and averages back. The faithful answer scores 1 and the contradicting one scores 0, so the mean is 0.5.

This path needs a different judge. The metrics in ragas.metrics expect a model that returns text containing JSON, not a Pydantic object, which is why the stand-in file has a second class for it.

python

def legacy_answer(data):
    """The same rules, chosen by what the older prompts put in their input."""
    if "statements" in data and "context" in data:
        return {"statements": [verdict_for(s, data["context"]) for s in data["statements"]]}
    if "context" in data and "answer" in data:
        return {"classifications": [attributed_for(s, data["context"])
                                    for s in sentences(data["answer"])]}
    return {"statements": sentences(data.get("answer", ""))}

The rules are the ones from part 2, picked by what the older prompts put in their input block.

python

class LegacyJudge(BaseRagasLLM):
    """What ragas.metrics expects: text back, holding JSON."""

    def generate_text(self, prompt, n=1, temperature=1e-8, stop=None, callbacks=None):
        from langchain_core.outputs import Generation, LLMResult
        text = prompt.to_string() if hasattr(prompt, "to_string") else str(prompt)
        answer = json.dumps(legacy_answer(asked(text)))
        return LLMResult(generations=[[Generation(text=answer)]])

    async def agenerate_text(self, prompt, n=1, temperature=None, stop=None, callbacks=None):
        return self.generate_text(prompt)

    def is_finished(self, response):
        return True

Example

frame = result.to_pandas()
print(frame[["response", "faithfulness", "context_recall"]].to_string())

to_pandas is the reason people like this API: one table, one row per sample, one column per metric, ready to sort by the worst.

It is on the way out

Importing these metrics prints a deprecation warning: they are removed in version 1.0. Use them to read other people's code and to score an existing dataset quickly; write new suites with the collections metrics and experiments from part 4.

Try it yourself

Add a third sample and read the new averages.
Sort the table by faithfulness and print the worst two rows.

PreviousScores inside an experiment, and comparing runs

Every expert started right here.