Promptfoopromptfoo 0.124.1 · Node 24 · Python 3
Dashboard
0%

Testing the retrieval support bot

The support bot suite is this course's finished artifact: one config that tests the retrieval bot end to end, with deterministic checks, the keyless model judge and context assertions over several questions, and it fails the build when the bot gets one wrong.

Last updated: 15 Oct, 2026 · Promptfoo 0.124.1

The course opened with a promise: test your own app, with no model key, behind a gate that fails a bad build. This suite keeps all three. The app under test is the retrieval bot. The checks run keyless through the stand-in grader. And a failing question makes the run exit non-zero, the signal Failing a build: exit codes and GitHub Actions turns into a gate.

One config, every kind of check

The suite layers the checks the course built. A structural is-json: checking the shape of an answer check and a contains, equals and regex check decide in code; an llm-rubric: the check that needs a model check reads the answer through the keyless judge; and Context assertions: faithfulness and recall weighs it against the retrieved documents. Three questions run. The third is one the bot cannot answer.

yaml
description: Retrieval support bot suite
providers:
  - file://rag_bot.py
prompts:
  - "{{query}}"
defaultTest:
  options:
    provider: file://pretend_promptfoo.py
tests:
  - vars:
      query: Where is order A17?
    assert:
      - type: is-json
      - type: contains
        value: "3 March"
      - type: llm-rubric
        value: the answer says the order shipped in March
      - type: context-faithfulness
        contextTransform: JSON.parse(output).context.join(" ")
        threshold: 0.5
  - vars:
      query: How long does a refund take?
    assert:
      - type: contains
        value: five working days
      - type: llm-rubric
        value: the answer says a refund takes five working days
      - type: context-faithfulness
        contextTransform: JSON.parse(output).context.join(" ")
        threshold: 0.5
  - vars:
      query: How long for my money back?
    assert:
      - type: contains
        value: five working days
Project files used on this pageThis lesson builds on a project from earlier lessons. The code below imports these files. Click a file to see its code, or follow the link to the lesson that wrote it. To run the code yourself, keep them in the same folder.
View the code here
rag_bot.py
import json

# a tiny retrieval bot: it searches a handbook and returns both its answer
# and the lines it retrieved, so a context check has something to compare against
HANDBOOK = {
    "a17": "Order A17 shipped on 3 March by courier.",
    "refund": "Refunds take five working days.",
}

def search(question):
    asked = question.lower()
    return [line for key, line in HANDBOOK.items() if key in asked]

def call_api(prompt, options, context):
    found = search(prompt)
    answer = found[0] if found else "I could not find that in the handbook."
    return {"output": json.dumps({"answer": answer, "context": found})}
pretend_promptfoo.py
"""A grader for promptfoo, written in Python.

Every model-graded assertion sends a prompt to a provider and reads the reply.
This file is that provider. It never calls a model. It decides with one rule:

    a claim is supported when every long word in it also appears in the text
    it is checked against.

Each assertion type asks for a different answer, so `grade` looks at the prompt
it was given and answers in the format that assertion expects.
"""
import json
import re

WORD = re.compile(r"[a-z0-9]+")
# A word is cut to its first four letters, so `shipped` and `shipping` count
# as the same word. These stems tell the grader what to do rather than what to
# look for, so they are dropped before anything is compared.
INSTRUCTIONS = {"repl", "answ", "outp", "resp", "says", "said", "ment", "cont",
                "incl", "shou", "must", "text", "that", "this", "when", "wher",
                "with", "from", "give", "tell", "show", "stat", "name", "user"}


def words(text):
    """The word stems worth comparing."""
    found = {w[:4] for w in WORD.findall(text.lower()) if len(w) > 3}
    return found - INSTRUCTIONS or found


def share(claim, source):
    """How much of the claim the source covers, 0 to 1."""
    wanted = words(claim)
    return len(wanted & words(source)) / len(wanted) if wanted else 0.0

def tag(text, name):
    """The text inside the LAST <name>...</name>, or an empty string."""
    found = re.findall(r"<%s>\s*(.*?)\s*</%s>" % (name, name), text, re.S)
    return found[-1] if found else ""


def flatten(prompt):
    """Chat graders send a JSON list of messages. Join them into one string."""
    try:
        loaded = json.loads(prompt)
    except ValueError:
        return prompt
    if isinstance(loaded, list):
        return "\n".join(str(m.get("content", "")) for m in loaded)
    return prompt

def after(text, marker):
    """Everything past the last marker, ignoring case.

    Every grader prompt opens with worked examples that use the same markers,
    so the real input is always the last one.
    """
    parts = re.split(re.escape(marker), text, flags=re.I)
    return parts[-1].strip() if len(parts) > 1 else ""


def sentences(text):
    return [s.strip() for s in re.split(r"(?<=[.!?])\s+", text.strip()) if s.strip()]


def grade(prompt):
    text = flatten(prompt)

    # factuality: one letter for how the answer relates to the reference.
    if "factuality evaluator" in text:
        expert, submitted = tag(text, "expert_answer"), tag(text, "submitted_answer")
        if share(expert, submitted) < 1.0:
            return json.dumps({"category": "D", "reason": "the answer misses part of it"})
        return json.dumps({"category": "C", "reason": "the reference is covered"})

    # g-eval asks twice: first for the steps, then for a score out of ten.
    if "EVALUATION CRITERIA" in text and '"steps"' in text:
        criteria = after(text.split("**OUTPUT FORMAT**")[0], "**EVALUATION CRITERIA**")
        return json.dumps({"steps": ["Check that the reply satisfies: " + criteria]})
    if "Evaluation Steps" in text and '"score"' in text:
        criteria = text.split("**Evaluation Criteria**")[1].split("**")[0]
        reply = text.split("**Reply**")[1].split("**OUTPUT FORMAT**")[0]
        hit = share(criteria, reply)
        return json.dumps({"score": round(hit * 10), "reason": "word overlap %.2f" % hit})

    # context-faithfulness asks twice: split the answer up, then judge each part.
    if "create one or more statements" in text:
        return "\n".join(sentences(after(text, "answer:").split("statements:")[0]))
    if "Natural language inference" in text:
        block = after(text, "context:")
        source = re.split("statements:", block, flags=re.I)[0]
        lines, final = [], []
        for i, claim in enumerate(sentences(after(block, "statements:").split("Answer:")[0]), 1):
            ok = share(claim, source) == 1.0
            lines.append("%d. %s Verdict: %s." % (i, claim, "Yes" if ok else "No"))
            final.append("Yes." if ok else "No.")
        return "\n".join(lines) + "\nFinal verdict for each statement in order: " + " ".join(final)

    # context-recall: is each part of the right answer in the documents?
    if "can be attributed to the given context" in text:
        block = after(text, "context:")
        source = re.split("answer:", block, flags=re.I)[0]
        verdicts = []
        for i, claim in enumerate(sentences(after(block, "answer:").split("classification")[0]), 1):
            ok = share(claim, source) == 1.0
            verdicts.append("%d. %s So [%s]" % (i, claim, "Attributed" if ok else "Not Attributed"))
        return "\n".join(verdicts)

    # llm-rubric, and everything built on it: pass, score and a reason.
    output, rubric = tag(text, "Output"), tag(text, "Rubric")
    hit = share(rubric, output)
    return json.dumps({"reason": "word overlap %.2f" % hit,
                       "pass": hit >= 0.5, "score": hit})


def call_api(prompt, options, context):
    return {"output": grade(prompt)}

Running it grades every check through the Python file, so the whole suite needs no key. Two questions pass all of their checks; the third fails, so the run reports a pass rate below 100 and exits non-zero.

Examplethe suite passes two questions and fails the third, exiting non-zero for a build gate
promptfoo eval --no-table

The failing case

The third question asks for a refund in words the bot's keyword search does not match, so it retrieves nothing and returns its handbook-miss line. The contains check for the refund window fails. The suite found a real gap in the bot, that its search misses a question a customer would plausibly ask.

Examplereading back which question failed and the check that caught it
promptfoo eval -o out.json --no-table >/dev/null 2>&1

# the one question that failed, and the check that caught it
python3 - <<'PY'
import json
for r in json.load(open("out.json"))["results"]["results"]:
    if not r["success"]:
        print("failed:", r["vars"]["query"])
        print("  bot said:", json.loads(r["response"]["output"])["answer"])
        for c in r["gradingResult"]["componentResults"]:
            if not c["pass"]:
                print("  %s: %s" % (c["assertion"]["type"], c["reason"]))
PY

What this suite keeps

  • It tests your own app. The provider is the retrieval bot you wrote, not a vendor endpoint, so the suite measures your code.
  • It needs no key. Deterministic checks decide in code and every model-graded check runs through the stand-in grader, so the suite runs anywhere.
  • It gates a build. One failing question drops the pass rate and exits 100, which a continuous-integration job reads as a failure.
Moving to a real model. When a verdict turns on meaning rather than shared words, swap the one line under defaultTest.options.provider for a hosted model, which Moving the suite to a hosted model covers. The config is otherwise unchanged; the deterministic and context checks stay exactly as they are.

Where you take it next

  • Grow the question set, adding the real queries your support inbox sees so the suite covers them.
  • Split the grade, running the keyless suite on every commit and a model-graded pass nightly.
  • Fix the bug this run found, widening the retrieval step so the refund question matches, then watching the suite turn green.
Try it yourself
  • Add a context-recall check with a reference to one of the passing questions and confirm it scores against the retrieved documents.
  • Widen the bot's search so money back matches the refund policy, then run the suite and watch the pass rate reach 100.
  • Set PROMPTFOO_PASS_RATE_THRESHOLD to 60 and confirm the two-of-three run now exits zero.

You understood something today that you didn't yesterday.