Faithfulness: is the answer grounded in the context

The first judged metric, and the one people reach for first. Faithfulness asks how much of the answer is actually supported by the documents the bot was given.

from pretend_ragas import PretendJudge
from ragas.metrics.collections import Faithfulness

faithfulness = Faithfulness(llm=PretendJudge())
context = ["Order A17 shipped on 3 March by courier."]

for answer in ["Order A17 shipped on 3 March by courier.",
               "Order A17 shipped on 3 March by courier. It arrives tomorrow.",
               "Order A17 was cancelled."]:
    result = faithfulness.score(user_input="Where is order A17?", response=answer, retrieved_contexts=context)
    print(result.value, "|", answer)

The first answer says only what the document says, so every statement is supported and the score is 1. The second adds a delivery date nobody retrieved, so one of its two statements is unsupported and the score halves. The third contradicts the document outright and scores 0.

How the score is made

Faithfulness is two questions to the judge, not one. First it asks for the statements in the answer. Then it asks, for each statement, whether the context supports it. The score is the share of statements that came back supported.

Lesson 4 wrote the first of those two answers. This is the second: one verdict per statement.

@answers("NLIStatementOutput")
def check_each_statement(data, model):
    context = data.get("context", "")
    return {"statements": [verdict_for(s, context) for s in data.get("statements", [])]}

It hands each statement to a rule of its own, which keeps the answer short and the rule readable.

def verdict_for(statement, context):
    """One statement, judged against the context it should have come from."""
    held = supported(statement, context)
    return {"statement": statement, "verdict": int(held),
            "reason": "every word of it is in the context" if held else "the context does not say this"}

supported is the rule from lesson 4: every word that matters in the statement has to be in the context. A real judge reads instead of counting, and disagrees with this one on wording.

Reading the algorithm matters more than it sounds. Faithfulness cannot tell you whether an answer is right, only whether it stayed inside the documents. An answer that copies a wrong document scores 1.