Factual correctness: claims against a reference

Faithfulness compared the answer with the documents. Factual correctness compares it with the answer you say is right, which is the closest thing RAGAS has to marking homework.

from pretend_ragas import PretendJudge
from ragas.metrics.collections import FactualCorrectness

factual = FactualCorrectness(llm=PretendJudge())
reference = "Order A17 shipped on 3 March by courier."

for answer in ["Order A17 shipped on 3 March by courier.",
               "Order A17 shipped on 3 March.",
               "Order A17 was cancelled."]:
    print(factual.score(response=answer, reference=reference).value, "|", answer)

The judge breaks both texts into claims and checks them against each other, so this metric needs a reference and does not need any documents at all. It is the one to use when you have written down what a good answer contains.

The second answer scores lower here only because the stand-in judge matches words: it never says courier, so that claim counts as missing. A real judge reads it as a shorter but still correct answer, which is a useful reminder that a judged score is one judge's opinion.

What the judge was asked

@answers("ClaimDecompositionOutput")
def split_into_claims(data, model):
    return {"claims": sentences(data.get("response", ""))}

Both texts go through this one, so the metric compares claim lists rather than sentences.