Faithfulness: did it stick to what it read

Until now the bot answered from nothing. Most real bots read something first: a policy page, a help article, a row from a database. That changes what can go wrong, and what to measure.

POLICY = [
    "Order A17 shipped on 3 March by courier.",
    "Refunds are paid within 5 working days.",
]


def retrieve(question):
    asked = set(question.lower().strip("?").split())
    return [doc for doc in POLICY if asked & set(doc.lower().strip(".").split())]

Two policy lines and a search that keeps any line sharing a word with the question. It is a crude retriever, which is useful, because three lessons from now the metrics have to show which half of the bot gets things wrong.

print(retrieve("Where is order A17?"))

One line came back for that question. What the bot read is as much a part of the test case as what it said, and it goes in a field of its own: retrieval_context.

An answer that stays with the documents

from deepeval.metrics import FaithfulnessMetric
from deepeval.test_case import LLMTestCase
from pretend_judge import PretendJudge

question = "Where is order A17?"
grounded = LLMTestCase(input=question, actual_output="Order A17 shipped on 3 March.",
                       retrieval_context=retrieve(question))
faithfulness = FaithfulnessMetric(model=PretendJudge())
faithfulness.measure(grounded)
print(faithfulness.score, faithfulness.reason)

Faithfulness scores how much of the answer is supported by what was read. Everything the answer says is in the document it read, so it scores 1.

An answer that adds something

invented = LLMTestCase(input=question, retrieval_context=retrieve(question),
                       actual_output="Order A17 shipped on 3 March. It arrives tomorrow.")
faithfulness.measure(invented)
print(faithfulness.score, faithfulness.reason)

The bot added a delivery date that no document mentions, and faithfulness still scores 1. This is the part that surprises people, and it is deliberate: the metric only lowers the score for a claim the documents contradict. A claim they merely do not mention is recorded as unknown, and unknown does not count against the answer.

strict = FaithfulnessMetric(model=PretendJudge(), penalize_ambiguous_claims=True)
strict.measure(invented)
print(strict.score, strict.reason)

penalize_ambiguous_claims=True changes that, and now the invented sentence costs half the score. Which one you want depends on your bot: for a support bot that must not invent policy, the strict one is closer to what you mean.

An answer that contradicts them

wrong = LLMTestCase(input=question, retrieval_context=retrieve(question),
                    actual_output="Order A17 was cancelled. We are sorry.")
faithfulness.measure(wrong)
print(faithfulness.score, faithfulness.reason)

Cancelled, when the document says shipped. The judge marks that claim as a contradiction, so half of the answer's claims are unfaithful and the score halves.

What the judge was asked

Faithfulness asks three questions. The facts in the documents, the claims in the answer, and then a verdict on each claim.

@answers("faithfulness.Truths")
def faithfulness_truths(prompt, judge):
    return {"truths": sentences(section(prompt, "Text"))}

@answers("faithfulness.Claims")
def faithfulness_claims(prompt, judge):
    return {"claims": sentences(section(prompt, "AI Output"))}

@answers("faithfulness.Verdicts")
def faithfulness_verdicts(prompt, judge):
    truths = section(prompt, "Retrieval Contexts")
    verdicts = []
    for claim in listed(prompt, "Claims"):
        found = share(claim, truths)
        if found == 1:
            verdicts.append({"verdict": "yes"})
        elif found > 0:
            verdicts.append({"verdict": "no", "reason": f"The context says otherwise about: {claim}"})
        else:
            verdicts.append({"verdict": "idk", "reason": f"The context does not mention: {claim}"})
    return {"verdicts": verdicts}

yes when every word that matters is in the documents, no when the claim is about something the documents cover and says something else, and idk when they are silent. The score is the share of claims that are not no.

The fourth question is the reason, which every judged metric asks last. This judge reads the score back out of the prompt rather than writing prose about it.

def read_back_score(prompt):
    score = re.findall(r"Score:\s*\n([\d.]+)", prompt)
    return f"The score is {score[-1]}." if score else "No score was given."

def sentences(text):
    return [s.strip() for s in re.split(r"(?<=[.!?])\s+", text) if s.strip()]

def listed(prompt, name):
    return ast.literal_eval(section(prompt, name) or "[]")