Answer relevancy: did it answer the question

Faithfulness asks whether the answer stayed with the documents. It says nothing about whether the answer was about the question at all, and that is the other half of grading a bot's reply.

from deepeval.metrics import AnswerRelevancyMetric

padded = LLMTestCase(
    input=question,
    actual_output="Thanks for getting in touch. We also sell gift cards. Order A17 shipped on 3 March.",
)
relevancy = AnswerRelevancyMetric(model=PretendJudge())
relevancy.measure(padded)
print(relevancy.score, relevancy.reason)

Every sentence in that answer is true, and the bot still buried the one the customer asked for. Answer relevancy splits the answer into statements and asks which of them address the question, so padding costs score.

It needs only the question and the answer. No expected answer, and no documents, which makes it one of the few metrics you can run on live traffic where nobody has written down the right answer.

tight = LLMTestCase(input=question, actual_output="Order A17 shipped on 3 March.")
relevancy.measure(tight)
print(relevancy.score, relevancy.reason)

The same fact on its own scores 1. Nothing about the answer got better; the noise around it went.

What the judge was asked

@answers("answer_relevancy.Statements")
def relevancy_statements(prompt, judge):
    return {"statements": sentences(section(prompt, "Text"))}

@answers("answer_relevancy.Verdicts")
def relevancy_verdicts(prompt, judge):
    question = section(prompt, "Input")
    return {"verdicts": [
        {"verdict": "yes"} if words(s) & words(question)
        else {"verdict": "no", "reason": f"Nothing in it is about: {question}"}
        for s in listed(prompt, "Statements")]}

Split into sentences, then one verdict per sentence: relevant when it shares a word that matters with the question. The score is the share of statements marked relevant.

relevancy.verbose_mode = True
relevancy.measure(padded)

Verbose mode shows the split and the verdicts, which is the fastest way to see why a good answer scored badly: usually a greeting, a caveat, or an apology counted as its own statement.