Tests in a CSV file

Three tests fit in the config. Forty do not, and the people who know which forty questions matter usually do not want to edit YAML.

tests can be a path instead of a list. Every row of the file becomes one test, every column becomes a variable, and one magic column carries the checks.

question,__expected
Where is order A17?,contains: 3 March
How long does a refund take?,contains: five working days
Where is order B99?,contains: could not find

question is a variable because the prompt uses that name. __expected is the exception: promptfoo reads it as an assertion, written as the type, a colon, and the value.

providers:
  - file://support_bot.py
prompts:
  - "{{question}}"
tests: file://tests.csv

promptfoo eval

The same three tests as the previous lesson, now in a file a spreadsheet can open. The config has shrunk to four lines and stopped changing when tests are added.

More than one check per row

For a second assertion on the same row, add a column called __expected2, then __expected3. They are checked in order and the row passes only if all of them pass.

Which file types work

CSV is the common one. Promptfoo also reads tests from JSON, JSONL and YAML files, and from a glob that matches several at once, which is how a large suite gets split by topic.

tests:
  - file://tests/orders.csv
  - file://tests/refunds.csv