Testing the retrieval support bot
The support bot suite is this course's finished artifact: one config that tests the retrieval bot end to end, with deterministic checks, the keyless model judge and context assertions over several questions, and it fails the build when the bot gets one wrong.
Last updated: 15 Oct, 2026 · Promptfoo 0.124.1
The course opened with a promise: test your own app, with no model key, behind a gate that fails a bad build. This suite keeps all three. The app under test is the retrieval bot. The checks run keyless through the stand-in grader. And a failing question makes the run exit non-zero, the signal Failing a build: exit codes and GitHub Actions turns into a gate.
One config, every kind of check
The suite layers the checks the course built. A structural is-json: checking the shape of an answer check and a contains, equals and regex check decide in code; an llm-rubric: the check that needs a model check reads the answer through the keyless judge; and Context assertions: faithfulness and recall weighs it against the retrieved documents. Three questions run. The third is one the bot cannot answer.
description: Retrieval support bot suite
providers:
- file://rag_bot.py
prompts:
- "{{query}}"
defaultTest:
options:
provider: file://pretend_promptfoo.py
tests:
- vars:
query: Where is order A17?
assert:
- type: is-json
- type: contains
value: "3 March"
- type: llm-rubric
value: the answer says the order shipped in March
- type: context-faithfulness
contextTransform: JSON.parse(output).context.join(" ")
threshold: 0.5
- vars:
query: How long does a refund take?
assert:
- type: contains
value: five working days
- type: llm-rubric
value: the answer says a refund takes five working days
- type: context-faithfulness
contextTransform: JSON.parse(output).context.join(" ")
threshold: 0.5
- vars:
query: How long for my money back?
assert:
- type: contains
value: five working days
View the code here
import json
# a tiny retrieval bot: it searches a handbook and returns both its answer
# and the lines it retrieved, so a context check has something to compare against
HANDBOOK = {
"a17": "Order A17 shipped on 3 March by courier.",
"refund": "Refunds take five working days.",
}
def search(question):
asked = question.lower()
return [line for key, line in HANDBOOK.items() if key in asked]
def call_api(prompt, options, context):
found = search(prompt)
answer = found[0] if found else "I could not find that in the handbook."
return {"output": json.dumps({"answer": answer, "context": found})}
"""A grader for promptfoo, written in Python.
Every model-graded assertion sends a prompt to a provider and reads the reply.
This file is that provider. It never calls a model. It decides with one rule:
a claim is supported when every long word in it also appears in the text
it is checked against.
Each assertion type asks for a different answer, so `grade` looks at the prompt
it was given and answers in the format that assertion expects.
"""
import json
import re
WORD = re.compile(r"[a-z0-9]+")
# A word is cut to its first four letters, so `shipped` and `shipping` count
# as the same word. These stems tell the grader what to do rather than what to
# look for, so they are dropped before anything is compared.
INSTRUCTIONS = {"repl", "answ", "outp", "resp", "says", "said", "ment", "cont",
"incl", "shou", "must", "text", "that", "this", "when", "wher",
"with", "from", "give", "tell", "show", "stat", "name", "user"}
def words(text):
"""The word stems worth comparing."""
found = {w[:4] for w in WORD.findall(text.lower()) if len(w) > 3}
return found - INSTRUCTIONS or found
def share(claim, source):
"""How much of the claim the source covers, 0 to 1."""
wanted = words(claim)
return len(wanted & words(source)) / len(wanted) if wanted else 0.0
def tag(text, name):
"""The text inside the LAST <name>...</name>, or an empty string."""
found = re.findall(r"<%s>\s*(.*?)\s*</%s>" % (name, name), text, re.S)
return found[-1] if found else ""
def flatten(prompt):
"""Chat graders send a JSON list of messages. Join them into one string."""
try:
loaded = json.loads(prompt)
except ValueError:
return prompt
if isinstance(loaded, list):
return "\n".join(str(m.get("content", "")) for m in loaded)
return prompt
def after(text, marker):
"""Everything past the last marker, ignoring case.
Every grader prompt opens with worked examples that use the same markers,
so the real input is always the last one.
"""
parts = re.split(re.escape(marker), text, flags=re.I)
return parts[-1].strip() if len(parts) > 1 else ""
def sentences(text):
return [s.strip() for s in re.split(r"(?<=[.!?])\s+", text.strip()) if s.strip()]
def grade(prompt):
text = flatten(prompt)
# factuality: one letter for how the answer relates to the reference.
if "factuality evaluator" in text:
expert, submitted = tag(text, "expert_answer"), tag(text, "submitted_answer")
if share(expert, submitted) < 1.0:
return json.dumps({"category": "D", "reason": "the answer misses part of it"})
return json.dumps({"category": "C", "reason": "the reference is covered"})
# g-eval asks twice: first for the steps, then for a score out of ten.
if "EVALUATION CRITERIA" in text and '"steps"' in text:
criteria = after(text.split("**OUTPUT FORMAT**")[0], "**EVALUATION CRITERIA**")
return json.dumps({"steps": ["Check that the reply satisfies: " + criteria]})
if "Evaluation Steps" in text and '"score"' in text:
criteria = text.split("**Evaluation Criteria**")[1].split("**")[0]
reply = text.split("**Reply**")[1].split("**OUTPUT FORMAT**")[0]
hit = share(criteria, reply)
return json.dumps({"score": round(hit * 10), "reason": "word overlap %.2f" % hit})
# context-faithfulness asks twice: split the answer up, then judge each part.
if "create one or more statements" in text:
return "\n".join(sentences(after(text, "answer:").split("statements:")[0]))
if "Natural language inference" in text:
block = after(text, "context:")
source = re.split("statements:", block, flags=re.I)[0]
lines, final = [], []
for i, claim in enumerate(sentences(after(block, "statements:").split("Answer:")[0]), 1):
ok = share(claim, source) == 1.0
lines.append("%d. %s Verdict: %s." % (i, claim, "Yes" if ok else "No"))
final.append("Yes." if ok else "No.")
return "\n".join(lines) + "\nFinal verdict for each statement in order: " + " ".join(final)
# context-recall: is each part of the right answer in the documents?
if "can be attributed to the given context" in text:
block = after(text, "context:")
source = re.split("answer:", block, flags=re.I)[0]
verdicts = []
for i, claim in enumerate(sentences(after(block, "answer:").split("classification")[0]), 1):
ok = share(claim, source) == 1.0
verdicts.append("%d. %s So [%s]" % (i, claim, "Attributed" if ok else "Not Attributed"))
return "\n".join(verdicts)
# llm-rubric, and everything built on it: pass, score and a reason.
output, rubric = tag(text, "Output"), tag(text, "Rubric")
hit = share(rubric, output)
return json.dumps({"reason": "word overlap %.2f" % hit,
"pass": hit >= 0.5, "score": hit})
def call_api(prompt, options, context):
return {"output": grade(prompt)}
Running it grades every check through the Python file, so the whole suite needs no key. Two questions pass all of their checks; the third fails, so the run reports a pass rate below 100 and exits non-zero.
promptfoo eval --no-tableStarting evaluation eval-Why-2026-10-14T19:24:33 Running 3 test cases (up to 4 at a time)... Creating cache folder at ~/.promptfoo/cache. ✓ Eval complete (ID: eval-Why-2026-10-14T19:24:33) » View results: promptfoo view » Share with your team: https://promptfoo.app » Feedback: https://promptfoo.dev/feedback Results: ✓ 2 passed (66.67%) ✗ 1 failed (33.33%) 0 errors (0%) Duration: 1s (concurrency: 4)
The failing case
The third question asks for a refund in words the bot's keyword search does not match, so it retrieves nothing and returns its handbook-miss line. The contains check for the refund window fails. The suite found a real gap in the bot, that its search misses a question a customer would plausibly ask.
promptfoo eval -o out.json --no-table >/dev/null 2>&1
# the one question that failed, and the check that caught it
python3 - <<'PY'
import json
for r in json.load(open("out.json"))["results"]["results"]:
if not r["success"]:
print("failed:", r["vars"]["query"])
print(" bot said:", json.loads(r["response"]["output"])["answer"])
for c in r["gradingResult"]["componentResults"]:
if not c["pass"]:
print(" %s: %s" % (c["assertion"]["type"], c["reason"]))
PYfailed: How long for my money back? bot said: I could not find that in the handbook. contains: Expected output to contain "five working days"
What this suite keeps
- It tests your own app. The provider is the retrieval bot you wrote, not a vendor endpoint, so the suite measures your code.
- It needs no key. Deterministic checks decide in code and every model-graded check runs through the stand-in grader, so the suite runs anywhere.
- It gates a build. One failing question drops the pass rate and exits 100, which a continuous-integration job reads as a failure.
defaultTest.options.provider for a hosted model, which Moving the suite to a hosted model covers. The config is otherwise unchanged; the deterministic and context checks stay exactly as they are.Where you take it next
- Grow the question set, adding the real queries your support inbox sees so the suite covers them.
- Split the grade, running the keyless suite on every commit and a model-graded pass nightly.
- Fix the bug this run found, widening the retrieval step so the refund question matches, then watching the suite turn green.
Related
- Previous: Red teaming: what promptfoo redteam does
- Reference: Promptfoo documentation
- Add a
context-recallcheck with a reference to one of the passing questions and confirm it scores against the retrieved documents. - Widen the bot's search so money back matches the refund policy, then run the suite and watch the pass rate reach 100.
- Set
PROMPTFOO_PASS_RATE_THRESHOLDto 60 and confirm the two-of-three run now exits zero.
You understood something today that you didn't yesterday.