Tests in a CSV file
A CSV of tests moves the questions and their checks out of the YAML and into a spreadsheet, one test per row, that anyone can edit without touching the config.
Last updated: 15 Oct, 2026 · Promptfoo 0.124.1
The Prompts and vars: one prompt, many questions lesson wrote each test as a vars block in the config. When the questions grow into dozens, a spreadsheet reads better than nested YAML, and a colleague who does not write YAML can still add a row. Promptfoo loads tests straight from a CSV.
The shape of the CSV
The header row names the columns. Each var in the prompt gets a column, and a special __expected column holds one inline assert per row, written as a type and a value. Every other row is one test.
question,__expected
Where is order A17?,contains: 3 March
How long does a refund take?,contains: five working days
Where is order B99?,contains: could not find
Here question is the only var, so there is one var column, and __expected carries a contains: check per row. The last row asks about an order the handbook does not know and expects the fallback wording, so the bot's miss still counts as a pass.
Pointing the config at the CSV
The tests key takes a single file:// path instead of an inline list. The provider and prompt stay in the config; only the tests move out.
providers:
- file://support_bot.py
prompts:
- "{{question}}"
tests: file://tests.csv
View the code here
# the shop support bot: find the answer in the handbook, or say it is missing
HANDBOOK = {
"a17": "Order A17 shipped on 3 March by courier.",
"refund": "Refunds take five working days.",
}
def call_api(prompt, options, context):
asked = prompt.lower() # the rendered prompt is the question
for key, line in HANDBOOK.items():
if key in asked:
return {"output": line}
return {"output": "I could not find that in the handbook."}
promptfoo eval --no-tableStarting evaluation eval-h9J-2026-10-14T19:17:04 Running 3 test cases (up to 4 at a time)... Creating cache folder at ~/.promptfoo/cache. ✓ Eval complete (ID: eval-h9J-2026-10-14T19:17:04) » View results: promptfoo view » Share with your team: https://promptfoo.app » Feedback: https://promptfoo.dev/feedback Results: ✓ 3 passed (100%) 0 failed (0%) 0 errors (0%) Duration: 0s (concurrency: 4)
What the CSV run shows
- Three rows became three tests, run exactly as if they had been written inline in the config.
- The
__expectedcolumn was the assert. Each row'scontains:check ran against that row's answer. - Adding a test is adding a row. No YAML, no indentation to get right, only one more line in the file.
A CSV vs inline tests
| Inline tests | A CSV file | |
|---|---|---|
| Where tests live | In the config YAML | In a separate spreadsheet |
| Who can add one | Someone who writes YAML | Anyone who edits a spreadsheet |
| Asserts per row | A full assert list | One inline __expected check |
| Best for | A few tests, rich checks | Many tests, one check each |
Where a CSV of tests helps
- A long list of questions that reads better as rows than as nested YAML.
- A non-programmer owning the cases, adding and editing rows in a tool they already use.
- Importing tests from elsewhere, such as a support log exported to a spreadsheet.
__expected, with two leading underscores, and its value is a type and a value separated by a colon, such as contains: 3 March. A plain column name or a missing type makes the column a var, not a check.Related
- Previous: Prompts and vars: one prompt, many questions
- Next: Loading test datasets from a file glob
- Reference: Promptfoo test cases docs
- Add a fourth row for a refund question and rerun; watch four tests run.
- Change the B99 row's check to
contains: shippedand watch that row fail. - Add a second var column and a
{{name}}to the prompt, then fill it per row.
Slow is fine. Stopping is the only problem.