Checking a support bot's answers in Python
Checking an answer by hand is running the bot on a question and testing its reply with a plain Python assert, which is the job promptfoo will later do for you.
Last updated: 15 Oct, 2026 · Promptfoo 0.124.1
Before any config file, it helps to do the work promptfoo automates, by hand, once. This lesson builds the shop support bot as a few lines of Python and checks two of its answers with assert. Doing it this way shows what a test is before a tool hides it, and it builds the exact bot every later lesson reuses.
The handbook the bot answers from
The bot knows two facts, kept in a dictionary keyed by a word that appears in the question. These two facts, the order and the refund, stay the same for the whole course.
HANDBOOK = {
"a17": "Order A17 shipped on 3 March by courier.",
"refund": "Refunds take five working days.",
}Answering a question
The reply function takes the shape promptfoo expects from a Python provider: a prompt plus two dicts it does not need yet, returning a dict with an output key. It lowercases the question, looks for a handbook word in it, and falls back to a miss.
def call_api(prompt, options, context):
asked = prompt.lower() # promptfoo passes the rendered prompt here
for key, line in HANDBOOK.items():
if key in asked:
return {"output": line}
return {"output": "I could not find that in the handbook."}Checking two answers with assert
Calling the bot and testing its reply is three lines per question: ask it, assert the answer contains the fact, and print it. An assert that holds is silent; one that fails stops the program and shows the reply.
# promptfoo will call this with the prompt and two dicts; do the same by hand
reply = call_api("Where is order A17?", {}, {})["output"]
assert "3 March" in reply, reply
print("A17:", reply)
reply = call_api("How long does a refund take?", {}, {})["output"]
assert "five working days" in reply
print("refund:", reply)
A17: Order A17 shipped on 3 March by courier. refund: Refunds take five working days.
What the checks showed
- Both asserts held, so the program ran to the end and printed each reply. A test that passes is quiet.
- The reply is passed as the assert's message (
assert ... , reply), so a failure would print the wrong answer, not only the line number. - The bot is deterministic: the same question gives the same answer every run, which is what lets a test check it at all.
By hand vs promptfoo
| Checking by hand | With promptfoo | |
|---|---|---|
| Where the questions live | Scattered through a script | In one config file |
| Where the checks live | Python assert lines | assert entries in the config |
| When one fails | The run stops at the first | Every test still runs |
| The result | Whatever you print | A pass rate and a table |
Where a by-hand check still helps
- A one-off sanity check while writing the bot, before there is a suite worth the setup.
- A failing case you are debugging, where you want to stop on the first break and read the reply.
- Explaining what a test is to someone new, with no tool in the way.
assert stops the program at the first failure, so later checks never run and you see one break at a time. That is the opposite of what you want from a test suite, where every case should run so you see the whole picture at once. That is the first thing promptfoo gives you.Related
- Previous: Installation and setup
- Next: promptfoo eval with a config file
- Add a third check for
Where is order B99?and assert the reply containscould not find. - Break a check: assert the A17 reply contains
warehouseand read the failure and the printed reply. - Call
call_apiwith an empty question and print what the fallback line returns.
You understood something today that you didn't yesterday.