Context assertions: faithfulness and recall
Context assertions judge a retrieval answer against the very documents it was built from: context-faithfulness asks whether the answer stays inside those documents, and context-recall asks whether the documents held the right answer at all.
Last updated: 15 Oct, 2026 · Promptfoo 0.124.1
Part four built a judge that runs with no model key, the Python grader in A grading script as the stand-in judge, and watched it pass and fail in When the judge is wrong. Those checks read one answer on its own. A retrieval bot answers from documents it looked up, and the two context checks compare the answer to those documents. Both still grade through the same keyless file.
The retrieval bot and the judge
The bot searches a tiny handbook and returns two things in one JSON string: the answer it chose, and the lines it retrieved. The retrieved lines are the context a context check needs. Each config in this part is its own promptfooconfig.yaml, changed as the questions and checks change, with the bot and the grader saved beside it.
import json
# a tiny retrieval bot: it searches a handbook and returns both its answer
# and the lines it retrieved, so a context check has something to compare against
HANDBOOK = {
"a17": "Order A17 shipped on 3 March by courier.",
"refund": "Refunds take five working days.",
}
def search(question):
asked = question.lower()
return [line for key, line in HANDBOOK.items() if key in asked]
def call_api(prompt, options, context):
found = search(prompt)
answer = found[0] if found else "I could not find that in the handbook."
return {"output": json.dumps({"answer": answer, "context": found})}
The grader is pretend_promptfoo.py, the same stand-in judge from A grading script as the stand-in judge, wired in as the grading provider so every context check runs with no key. Here it is once more, so this part stands on its own.
"""A grader for promptfoo, written in Python.
Every model-graded assertion sends a prompt to a provider and reads the reply.
This file is that provider. It never calls a model. It decides with one rule:
a claim is supported when every long word in it also appears in the text
it is checked against.
Each assertion type asks for a different answer, so `grade` looks at the prompt
it was given and answers in the format that assertion expects.
"""
import json
import re
WORD = re.compile(r"[a-z0-9]+")
# A word is cut to its first four letters, so `shipped` and `shipping` count
# as the same word. These stems tell the grader what to do rather than what to
# look for, so they are dropped before anything is compared.
INSTRUCTIONS = {"repl", "answ", "outp", "resp", "says", "said", "ment", "cont",
"incl", "shou", "must", "text", "that", "this", "when", "wher",
"with", "from", "give", "tell", "show", "stat", "name", "user"}
def words(text):
"""The word stems worth comparing."""
found = {w[:4] for w in WORD.findall(text.lower()) if len(w) > 3}
return found - INSTRUCTIONS or found
def share(claim, source):
"""How much of the claim the source covers, 0 to 1."""
wanted = words(claim)
return len(wanted & words(source)) / len(wanted) if wanted else 0.0
def tag(text, name):
"""The text inside the LAST <name>...</name>, or an empty string."""
found = re.findall(r"<%s>\s*(.*?)\s*</%s>" % (name, name), text, re.S)
return found[-1] if found else ""
def flatten(prompt):
"""Chat graders send a JSON list of messages. Join them into one string."""
try:
loaded = json.loads(prompt)
except ValueError:
return prompt
if isinstance(loaded, list):
return "\n".join(str(m.get("content", "")) for m in loaded)
return prompt
def after(text, marker):
"""Everything past the last marker, ignoring case.
Every grader prompt opens with worked examples that use the same markers,
so the real input is always the last one.
"""
parts = re.split(re.escape(marker), text, flags=re.I)
return parts[-1].strip() if len(parts) > 1 else ""
def sentences(text):
return [s.strip() for s in re.split(r"(?<=[.!?])\s+", text.strip()) if s.strip()]
def grade(prompt):
text = flatten(prompt)
# factuality: one letter for how the answer relates to the reference.
if "factuality evaluator" in text:
expert, submitted = tag(text, "expert_answer"), tag(text, "submitted_answer")
if share(expert, submitted) < 1.0:
return json.dumps({"category": "D", "reason": "the answer misses part of it"})
return json.dumps({"category": "C", "reason": "the reference is covered"})
# g-eval asks twice: first for the steps, then for a score out of ten.
if "EVALUATION CRITERIA" in text and '"steps"' in text:
criteria = after(text.split("**OUTPUT FORMAT**")[0], "**EVALUATION CRITERIA**")
return json.dumps({"steps": ["Check that the reply satisfies: " + criteria]})
if "Evaluation Steps" in text and '"score"' in text:
criteria = text.split("**Evaluation Criteria**")[1].split("**")[0]
reply = text.split("**Reply**")[1].split("**OUTPUT FORMAT**")[0]
hit = share(criteria, reply)
return json.dumps({"score": round(hit * 10), "reason": "word overlap %.2f" % hit})
# context-faithfulness asks twice: split the answer up, then judge each part.
if "create one or more statements" in text:
return "\n".join(sentences(after(text, "answer:").split("statements:")[0]))
if "Natural language inference" in text:
block = after(text, "context:")
source = re.split("statements:", block, flags=re.I)[0]
lines, final = [], []
for i, claim in enumerate(sentences(after(block, "statements:").split("Answer:")[0]), 1):
ok = share(claim, source) == 1.0
lines.append("%d. %s Verdict: %s." % (i, claim, "Yes" if ok else "No"))
final.append("Yes." if ok else "No.")
return "\n".join(lines) + "\nFinal verdict for each statement in order: " + " ".join(final)
# context-recall: is each part of the right answer in the documents?
if "can be attributed to the given context" in text:
block = after(text, "context:")
source = re.split("answer:", block, flags=re.I)[0]
verdicts = []
for i, claim in enumerate(sentences(after(block, "answer:").split("classification")[0]), 1):
ok = share(claim, source) == 1.0
verdicts.append("%d. %s So [%s]" % (i, claim, "Attributed" if ok else "Not Attributed"))
return "\n".join(verdicts)
# llm-rubric, and everything built on it: pass, score and a reason.
output, rubric = tag(text, "Output"), tag(text, "Rubric")
hit = share(rubric, output)
return json.dumps({"reason": "word overlap %.2f" % hit,
"pass": hit >= 0.5, "score": hit})
def call_api(prompt, options, context):
return {"output": grade(prompt)}
context-faithfulness: does the answer stay in the documents
The provider under test is the bot, the prompt is {{query}}, and the grader is set under defaultTest.options.provider. The context is not a var here: contextTransform pulls the retrieved lines out of the bot's JSON output and joins them into one string. context-faithfulness breaks the answer into claims and checks each against that context; context-recall takes a reference answer in value and checks the documents hold it. A threshold is set on both.
providers:
- file://rag_bot.py
prompts:
- "{{query}}"
defaultTest:
options:
provider: file://pretend_promptfoo.py
tests:
- vars:
query: Where is order A17?
assert:
- type: context-faithfulness
contextTransform: JSON.parse(output).context.join(" ")
threshold: 0.9
- type: context-recall
value: Order A17 shipped on 3 March.
contextTransform: JSON.parse(output).context.join(" ")
threshold: 0.5
With the bot and the grader in the folder, both checks grade through the Python file. The answer about order A17 came straight from the retrieved line, so it is fully supported, and the line itself carries the reference, so recall is complete.
promptfoo eval --no-tableStarting evaluation eval-wiY-2026-10-14T19:24:22 Running 1 test cases (up to 4 at a time)... Creating cache folder at ~/.promptfoo/cache. ✓ Eval complete (ID: eval-wiY-2026-10-14T19:24:22) » View results: promptfoo view » Share with your team: https://promptfoo.app » Feedback: https://promptfoo.dev/feedback Results: ✓ 1 passed (100%) 0 failed (0%) 0 errors (0%) Duration: 4s (concurrency: 4)
What the two checks measured
- Faithfulness scored the answer against the documents. Every claim in the answer appeared in the retrieved line, so faithfulness was 1.00, clear of the 0.9 bar.
- Recall scored the documents against a reference. The reference Order A17 shipped on 3 March. is attributable to the retrieved line, so recall was 1.00.
- Neither check read a model. The grading provider was the Python file, so a retrieval suite runs on a laptop or in a job that holds no credentials.
Both checks need a query and a context
These are the only assertions that demand named inputs. Each one needs a query var holding the user question, and a context, supplied either by a context var or by contextTransform. Take either away and the run stops with an Invariant failed message. This is an error, reported before any grading, not a low score.
# rename the query var: the context checks can no longer find the question
sed 's/query/question/g' promptfooconfig.yaml > broken.yaml
promptfoo eval -c broken.yaml --no-table 2>&1 | grep "Invariant failed"
# drop the contextTransform: now there is no context to compare against
grep -v contextTransform promptfooconfig.yaml > nocontext.yaml
promptfoo eval -c nocontext.yaml --no-table 2>&1 | grep "Invariant failed" "message": "Invariant failed: context-faithfulness assertion requires a \"query\" variable with the user question"
"message": "Invariant failed: Context is required for context-based assertions. Provide either a \"context\" variable (string or array of strings) in your test case or use \"contextTransform\" to extract context from the provider response."query var and a missing context source each raise their own message, so read which input it is asking for.The 0.5 threshold, and what recall ignores
Set the two checks against a wrong answer to see how each one behaves. Here the provider is echo so the answer can be stated directly as a var, the context is the correct retrieved line, and the answer is the bot's handbook-miss reply. Faithfulness carries no threshold, and recall takes the right reference in value.
providers:
- echo
prompts:
- "{{answer}}"
defaultTest:
options:
provider: file://pretend_promptfoo.py
tests:
- vars:
query: Where is order A17?
context: Order A17 shipped on 3 March by courier.
answer: I could not find that in the handbook.
assert:
- type: context-faithfulness
- type: context-recall
value: Order A17 shipped on 3 March.
promptfoo eval -o out.json --no-table >/dev/null 2>&1
# the same miss answer, seen by both context checks
python3 - <<'PY'
import json
r = json.load(open("out.json"))["results"]["results"][0]
print("answer:", r["vars"]["answer"])
for c in r["gradingResult"]["componentResults"]:
print(" %-20s %s (%s)" % (c["assertion"]["type"], c["pass"], c["reason"]))
PYanswer: I could not find that in the handbook. context-faithfulness False (Faithfulness 0.00 is < 0.5) context-recall True (Recall 1.00 is >= 0.5)
- Faithfulness failed at a threshold nobody set. The miss shares no words with the context, so its score was 0.00, and with no
thresholdthe default is now 0.5, so a zero fails. On earlier promptfoo the default was 0 and a zero passed. - Recall passed on the same wrong answer. Recall never reads the answer. It checks whether the reference in
valueis attributable to the context, and the context was right, so recall stayed 1.0 while faithfulness was 0.
threshold on context-faithfulness. The default moved from 0 to 0.5, so a suite written against an older version, counting on a zero to pass, now fails. State the bar you mean.Where you use context assertions
- Faithfulness against hallucination, flagging an answer that adds a claim the retrieved documents do not support.
- Recall against a thin index, catching a reference answer that the retrieval step failed to surface at all.
- A retrieval regression gate, run on every change to the search step to show whether the right documents still come back.
Related
- Previous: Moving the suite to a hosted model
- Next: Red teaming: what promptfoo redteam does
- Reference: Promptfoo documentation
- Lower the faithfulness
thresholdto 0 and confirm the miss answer passes, the old default behaviour. - Ask the bot a question it cannot answer and watch
contextTransformreturn an empty context, which raises the Invariant. - Add a second question with its own
queryand reference and confirm both context checks run from the one grader.
Every expert started right here.