Promptfoopromptfoo 0.124.1 · Node 24 · Python 3
Dashboard
0%

Checking a support bot's answers in Python

Checking an answer by hand is running the bot on a question and testing its reply with a plain Python assert, which is the job promptfoo will later do for you.

Last updated: 15 Oct, 2026 · Promptfoo 0.124.1

Before any config file, it helps to do the work promptfoo automates, by hand, once. This lesson builds the shop support bot as a few lines of Python and checks two of its answers with assert. Doing it this way shows what a test is before a tool hides it, and it builds the exact bot every later lesson reuses.

The handbook the bot answers from

The bot knows two facts, kept in a dictionary keyed by a word that appears in the question. These two facts, the order and the refund, stay the same for the whole course.

python
HANDBOOK = {
    "a17": "Order A17 shipped on 3 March by courier.",
    "refund": "Refunds take five working days.",
}

Answering a question

The reply function takes the shape promptfoo expects from a Python provider: a prompt plus two dicts it does not need yet, returning a dict with an output key. It lowercases the question, looks for a handbook word in it, and falls back to a miss.

python
def call_api(prompt, options, context):
    asked = prompt.lower()              # promptfoo passes the rendered prompt here
    for key, line in HANDBOOK.items():
        if key in asked:
            return {"output": line}
    return {"output": "I could not find that in the handbook."}

Checking two answers with assert

Calling the bot and testing its reply is three lines per question: ask it, assert the answer contains the fact, and print it. An assert that holds is silent; one that fails stops the program and shows the reply.

ExampleTwo answers checked by hand
# promptfoo will call this with the prompt and two dicts; do the same by hand
reply = call_api("Where is order A17?", {}, {})["output"]
assert "3 March" in reply, reply
print("A17:", reply)

reply = call_api("How long does a refund take?", {}, {})["output"]
assert "five working days" in reply
print("refund:", reply)

What the checks showed

  • Both asserts held, so the program ran to the end and printed each reply. A test that passes is quiet.
  • The reply is passed as the assert's message (assert ... , reply), so a failure would print the wrong answer, not only the line number.
  • The bot is deterministic: the same question gives the same answer every run, which is what lets a test check it at all.

By hand vs promptfoo

Checking by handWith promptfoo
Where the questions liveScattered through a scriptIn one config file
Where the checks livePython assert linesassert entries in the config
When one failsThe run stops at the firstEvery test still runs
The resultWhatever you printA pass rate and a table

Where a by-hand check still helps

  • A one-off sanity check while writing the bot, before there is a suite worth the setup.
  • A failing case you are debugging, where you want to stop on the first break and read the reply.
  • Explaining what a test is to someone new, with no tool in the way.
Watch out. An assert stops the program at the first failure, so later checks never run and you see one break at a time. That is the opposite of what you want from a test suite, where every case should run so you see the whole picture at once. That is the first thing promptfoo gives you.
Try it yourself
  • Add a third check for Where is order B99? and assert the reply contains could not find.
  • Break a check: assert the A17 reply contains warehouse and read the failure and the printed reply.
  • Call call_api with an empty question and print what the fallback line returns.

You understood something today that you didn't yesterday.