promptfoo eval with a config file
promptfooconfig.yaml is the file that names the provider under test, the prompt it answers and the checks each answer must pass, so one promptfoo eval runs them all.
Last updated: 15 Oct, 2026 · Promptfoo 0.124.1
The checking-answers-by-hand lesson tested the bot with Python asserts that stop at the first failure. Promptfoo moves those questions and checks into one file and runs every one of them with a single command. The bot does not change; only who calls it does.
Pointing the config at the bot
The config is YAML with three top-level keys. The first two name the provider and the prompt.
providers:
- file://support_bot.py # the bot from the last lesson, now the app under test
prompts:
- "{{question}}" # each test fills this in
file://support_bot.py tells promptfoo to import that file and call its call_api function as the app under test. The prompt is a template; {{question}} is filled from each test's vars.
Two tests the bot passes
Each test gives a question and one assert. Both checks here are chosen to be answers the bot gets right, so the first run is a clean pass and you learn to read a passing report before a failing one.
tests:
- vars:
question: Where is order A17?
assert:
- type: contains
value: 3 March
- vars:
question: How long does a refund take?
assert:
- type: contains
value: five working days
View the code here
# the shop support bot: find the answer in the handbook, or say it is missing
HANDBOOK = {
"a17": "Order A17 shipped on 3 March by courier.",
"refund": "Refunds take five working days.",
}
def call_api(prompt, options, context):
asked = prompt.lower() # the rendered prompt is the question
for key, line in HANDBOOK.items():
if key in asked:
return {"output": line}
return {"output": "I could not find that in the handbook."}
Running the eval
With the config and the bot in the same folder, one command runs both tests. The --no-table flag prints the clean summary instead of the full grid, which the next lesson reads.
promptfoo eval --no-tableStarting evaluation eval-E2r-2026-10-14T19:03:35 Running 2 test cases (up to 4 at a time)... Creating cache folder at ~/.promptfoo/cache. ✓ Eval complete (ID: eval-E2r-2026-10-14T19:03:35) » View results: promptfoo view » Share with your team: https://promptfoo.app » Feedback: https://promptfoo.dev/feedback Results: ✓ 2 passed (100%) 0 failed (0%) 0 errors (0%) Duration: 0s (concurrency: 4)
What the summary says
- The run has an id (
eval-...with a timestamp), which promptfoo saves so you can list and reopen the run later. - Two passed, none failed, with the pass rate as a percentage. This is the line a build would check.
- The three
»lines point at the browser view, team sharing and feedback;View results: promptfoo viewopens this run in a browser. - Creating cache folder appears on the first run in a fresh home directory, where promptfoo makes a place to cache provider replies.
A config file vs Python asserts
| Python asserts | promptfooconfig.yaml | |
|---|---|---|
| Adding a question | Write more code | Add a vars block |
| Running all cases | Stops at the first failure | Runs every test |
| The report | Whatever you print | A pass rate and an id |
| Who can edit it | Someone who writes Python | Anyone who can edit YAML |
Where a config file helps
- A suite that lives beside the code, committed and run the same way by everyone.
- A non-programmer adding cases, by copying a
varsblock in a file. - A run you want kept and compared, because each eval is saved with an id.
file://support_bot.py. Drop the file:// and promptfoo reads it as the name of a hosted provider, not a local file, and the run errors instead of calling your bot.Related
- Previous: Checking a support bot's answers in Python
- Next: Reading the result table
- Reference: Promptfoo configuration guide
- Change the expected value for the refund test to
ten working daysand rerun; watch the pass count drop. - Add a third
varsblock for a new question the bot answers and rerun. - Run without
--no-tableand notice the full grid the next lesson explains.
Slow is fine. Stopping is the only problem.