Promptfoopromptfoo 0.124.1 · Node 24 · Python 3
Dashboard
0%

Tests in a CSV file

A CSV of tests moves the questions and their checks out of the YAML and into a spreadsheet, one test per row, that anyone can edit without touching the config.

Last updated: 15 Oct, 2026 · Promptfoo 0.124.1

The Prompts and vars: one prompt, many questions lesson wrote each test as a vars block in the config. When the questions grow into dozens, a spreadsheet reads better than nested YAML, and a colleague who does not write YAML can still add a row. Promptfoo loads tests straight from a CSV.

The shape of the CSV

The header row names the columns. Each var in the prompt gets a column, and a special __expected column holds one inline assert per row, written as a type and a value. Every other row is one test.

text
question,__expected
Where is order A17?,contains: 3 March
How long does a refund take?,contains: five working days
Where is order B99?,contains: could not find

Here question is the only var, so there is one var column, and __expected carries a contains: check per row. The last row asks about an order the handbook does not know and expects the fallback wording, so the bot's miss still counts as a pass.

Pointing the config at the CSV

The tests key takes a single file:// path instead of an inline list. The provider and prompt stay in the config; only the tests move out.

yaml
providers:
  - file://support_bot.py
prompts:
  - "{{question}}"
tests: file://tests.csv
Project files used on this pageThis lesson builds on a project from earlier lessons. The code below imports this file. Click a file to see its code, or follow the link to the lesson that wrote it. To run the code yourself, keep it in the same folder.
View the code here
support_bot.py
# the shop support bot: find the answer in the handbook, or say it is missing
HANDBOOK = {
    "a17": "Order A17 shipped on 3 March by courier.",
    "refund": "Refunds take five working days.",
}

def call_api(prompt, options, context):
    asked = prompt.lower()                      # the rendered prompt is the question
    for key, line in HANDBOOK.items():
        if key in asked:
            return {"output": line}
    return {"output": "I could not find that in the handbook."}
Examplethree tests loaded from a CSV, each with an inline check
promptfoo eval --no-table

What the CSV run shows

  • Three rows became three tests, run exactly as if they had been written inline in the config.
  • The __expected column was the assert. Each row's contains: check ran against that row's answer.
  • Adding a test is adding a row. No YAML, no indentation to get right, only one more line in the file.

A CSV vs inline tests

Inline testsA CSV file
Where tests liveIn the config YAMLIn a separate spreadsheet
Who can add oneSomeone who writes YAMLAnyone who edits a spreadsheet
Asserts per rowA full assert listOne inline __expected check
Best forA few tests, rich checksMany tests, one check each

Where a CSV of tests helps

  • A long list of questions that reads better as rows than as nested YAML.
  • A non-programmer owning the cases, adding and editing rows in a tool they already use.
  • Importing tests from elsewhere, such as a support log exported to a spreadsheet.
Watch out. The assert column must be named __expected, with two leading underscores, and its value is a type and a value separated by a colon, such as contains: 3 March. A plain column name or a missing type makes the column a var, not a check.
Try it yourself
  • Add a fourth row for a refund question and rerun; watch four tests run.
  • Change the B99 row's check to contains: shipped and watch that row fail.
  • Add a second var column and a {{name}} to the prompt, then fill it per row.

Slow is fine. Stopping is the only problem.