Promptfoopromptfoo 0.124.1 · Node 24 · Python 3
Dashboard
0%

promptfoo eval with a config file

promptfooconfig.yaml is the file that names the provider under test, the prompt it answers and the checks each answer must pass, so one promptfoo eval runs them all.

Last updated: 15 Oct, 2026 · Promptfoo 0.124.1

The checking-answers-by-hand lesson tested the bot with Python asserts that stop at the first failure. Promptfoo moves those questions and checks into one file and runs every one of them with a single command. The bot does not change; only who calls it does.

Pointing the config at the bot

The config is YAML with three top-level keys. The first two name the provider and the prompt.

yaml
providers:
  - file://support_bot.py       # the bot from the last lesson, now the app under test
prompts:
  - "{{question}}"              # each test fills this in

file://support_bot.py tells promptfoo to import that file and call its call_api function as the app under test. The prompt is a template; {{question}} is filled from each test's vars.

Two tests the bot passes

Each test gives a question and one assert. Both checks here are chosen to be answers the bot gets right, so the first run is a clean pass and you learn to read a passing report before a failing one.

yaml
tests:
  - vars:
      question: Where is order A17?
    assert:
      - type: contains
        value: 3 March
  - vars:
      question: How long does a refund take?
    assert:
      - type: contains
        value: five working days
Project files used on this pageThis lesson builds on a project from earlier lessons. The code below imports this file. Click a file to see its code, or follow the link to the lesson that wrote it. To run the code yourself, keep it in the same folder.
View the code here
support_bot.py
# the shop support bot: find the answer in the handbook, or say it is missing
HANDBOOK = {
    "a17": "Order A17 shipped on 3 March by courier.",
    "refund": "Refunds take five working days.",
}

def call_api(prompt, options, context):
    asked = prompt.lower()                      # the rendered prompt is the question
    for key, line in HANDBOOK.items():
        if key in asked:
            return {"output": line}
    return {"output": "I could not find that in the handbook."}

Running the eval

With the config and the bot in the same folder, one command runs both tests. The --no-table flag prints the clean summary instead of the full grid, which the next lesson reads.

ExampleThe first run: two tests, both pass
promptfoo eval --no-table

What the summary says

  • The run has an id (eval-... with a timestamp), which promptfoo saves so you can list and reopen the run later.
  • Two passed, none failed, with the pass rate as a percentage. This is the line a build would check.
  • The three » lines point at the browser view, team sharing and feedback; View results: promptfoo view opens this run in a browser.
  • Creating cache folder appears on the first run in a fresh home directory, where promptfoo makes a place to cache provider replies.

A config file vs Python asserts

Python assertspromptfooconfig.yaml
Adding a questionWrite more codeAdd a vars block
Running all casesStops at the first failureRuns every test
The reportWhatever you printA pass rate and an id
Who can edit itSomeone who writes PythonAnyone who can edit YAML

Where a config file helps

  • A suite that lives beside the code, committed and run the same way by everyone.
  • A non-programmer adding cases, by copying a vars block in a file.
  • A run you want kept and compared, because each eval is saved with an id.
Watch out. The provider must be written file://support_bot.py. Drop the file:// and promptfoo reads it as the name of a hosted provider, not a local file, and the run errors instead of calling your bot.
Try it yourself
  • Change the expected value for the refund test to ten working days and rerun; watch the pass count drop.
  • Add a third vars block for a new question the bot answers and rerun.
  • Run without --no-table and notice the full grid the next lesson explains.

Slow is fine. Stopping is the only problem.