Context assertions: faithfulness and recall

A bot that answers from documents can fail in two different places, and one score cannot tell them apart. Promptfoo has a check for each.

When a retrieval bot gives a wrong answer, either the search fetched the wrong documents or the writing drifted away from the right ones. context-recall is about the first and context-faithfulness is about the second.

The two questions

Assertion	Asks	A low score means
context-recall	do the documents support the right answer	the search missed
context-faithfulness	is every claim in the answer in the documents	the bot made something up

Both are model-graded, so both go to the judge from part 4, and both need two extra branches in it because they ask in their own formats.

python

    # context-faithfulness asks twice: split the answer up, then judge each part.
    if "create one or more statements" in text:
        return "\n".join(sentences(after(text, "answer:").split("statements:")[0]))
    if "Natural language inference" in text:
        block = after(text, "context:")
        source = re.split("statements:", block, flags=re.I)[0]
        lines, final = [], []
        for i, claim in enumerate(sentences(after(block, "statements:").split("Answer:")[0]), 1):
            ok = share(claim, source) == 1.0
            lines.append("%d. %s Verdict: %s." % (i, claim, "Yes" if ok else "No"))
            final.append("Yes." if ok else "No.")
        return "\n".join(lines) + "\nFinal verdict for each statement in order: " + " ".join(final)

And the recall branch, which reads the reference answer rather than what the bot said.

python

    # context-recall: is each part of the right answer in the documents?
    if "can be attributed to the given context" in text:
        block = after(text, "context:")
        source = re.split("answer:", block, flags=re.I)[0]
        verdicts = []
        for i, claim in enumerate(sentences(after(block, "answer:").split("classification")[0]), 1):
            ok = share(claim, source) == 1.0
            verdicts.append("%d. %s So [%s]" % (i, claim, "Attributed" if ok else "Not Attributed"))
        return "\n".join(verdicts)

Faithfulness asks twice: split the answer into claims, then say whether each one is in the documents. Recall asks once, over the reference answer instead of the bot's answer. The word rule does not change; only the shape of the question does.

Documents in the test

The simplest version puts the documents in the test, as a variable called context, with the question in one called query.

yaml

tests:
  - vars:
      query: Where is order A17?
      context: Order A17 shipped on 3 March by courier.
    assert:
      - type: context-faithfulness
        threshold: 0.9
      - type: context-recall
        value: Order A17 shipped on 3 March.

Example

promptfoo eval

These two names are not optional. The context assertions look for query and context in the test's variables. Call the question question, as every earlier lesson in this course does, and the assertion stops with requires a "query" variable rather than failing. It is an error, not a low score.

Documents from the bot

Fixed documents test the writing but not the search. For the real thing the bot has to say which documents it used, and contextTransform pulls them out of its answer.

python

import json

HANDBOOK = {
    "a17": "Order A17 shipped on 3 March by courier.",
    "refund": "Refunds take five working days.",
}

def search(question):
    asked = question.lower()
    return [line for key, line in HANDBOOK.items() if key in asked]

search is the retrieval step, crude but real: it keeps every handbook line that shares a word with the question. Then the provider answers and reports what it used.

python

def call_api(prompt, options, context):
    found = search(prompt)
    answer = found[0] if found else "I could not find that in the handbook."
    return {"output": json.dumps({"answer": answer, "context": found})}

yaml

defaultTest:
  transform: JSON.parse(output).answer
tests:
  - vars:
      query: Where is order A17?
    assert:
      - type: context-faithfulness
        contextTransform: JSON.parse(output).context.join(" ")
        threshold: 0.9

transform takes the answer out of the JSON so every other assertion sees a sentence, and contextTransform takes the documents out for the context check. Now a retrieval change moves the score.

Example

promptfoo eval

The trap in the defaults

context-faithfulness has a default threshold of 0, which means it passes at any score at all. A suite with it added and no threshold set reports green forever while recording the number you wanted to watch. Always give it one.

The other surprise is that context-recall does not look at the bot's answer. It compares the documents to the reference answer, so a bot that answers completely wrongly still scores 1 for recall if the search did its job. That is the point of having both, and it is confusing the first time you see it.

Try it yourself

Remove the threshold from the faithfulness check and watch a bad answer pass.
Change the bot to add a sentence that is not in the documents, and watch faithfulness drop.
Rename query to question and read the error.

PreviousA real model: the swap

You understood something today that you didn't yesterday.