The support bot suite, end to end

Everything in the course, in one directory, gating one build. This is the suite promised in lesson 0.

Four files. The bot, the judge, the questions and the config. Nothing here is new: every piece was built alone in an earlier lesson and the list below says which.

The pieces

The questions

query,__expected
Where is order A17?,contains: 3 March
How long does a refund take?,contains: five working days
Where is order B99?,contains: could not find

Three rows, each with a deterministic check. The variable is query rather than question, because the context assertions from lesson 22 require that name.

The config

description: Support bot suite
providers:
  - file://rag_bot.py
prompts:
  - "{{query}}"
defaultTest:
  transform: JSON.parse(output).answer
  options:
    provider: file://pretend_promptfoo.py
  assert:
    - type: python
      value: file://house_style.py
      metric: Style
tests: file://tests.csv

defaultTest does three jobs at once here. It unwraps the bot's JSON so every check sees the answer, it points the model-graded checks at the local judge, and it adds the house style rule to every test without repeating it three times.

Running it

promptfoo eval

Three questions, each checked twice: once for the fact it has to contain and once against the house style. All three pass, and the run took no key and no money.

The cells show the raw JSON the bot returned, because the table prints what the provider said. transform runs on the way into the assertions, so the checks saw the sentence rather than the wrapper.

The failure that matters

A suite is only worth what it catches. Change the bot to answer confidently when it has found nothing, which is the failure a retrieval application actually has.

promptfoo eval

One row goes red, and it names two separate problems. The answer for B99 no longer says it could not find anything, so the check from the CSV fails, and it uses a word the shop does not use, so the house style rule fails as well. Neither of those is a crash, and neither would have been noticed by a test that was not looking for it.

Where to go next

Add a real model with the two lines from lesson 21 and keep this one as the fast gate. Add the workflow from lesson 20. Then hand tests.csv to the person who answers these questions for a living and let them add twenty rows.

What this course left out

Promptfoo publishes 385 documentation pages. This course taught the part you need to test an application; here is what is real and not here.