Promptfoopromptfoo 0.124.1 · Node 24 · Python 3
Dashboard
0%
1
Curious builder0 XP earned · 300 to level 2
0 daysFinish a lesson to begin
Badge collection0 of 6 unlocked
33 small wins to finish your pathNext lesson →

Loading test datasets from a file glob

A file glob lets the tests key load several test files at once, so a suite can be split into small datasets that promptfoo gathers with one pattern.

Last updated: 15 Oct, 2026 · Promptfoo 0.124.1

The Tests in a CSV file lesson kept every test in one file. A real suite grows past that: orders in one dataset, refunds in another, each owned by a different person. Promptfoo lets tests point at a glob, a path with a * wildcard, and loads every file it matches.

Splitting the tests into datasets

Each file is a plain list of test cases, the same entries that would sit inline under tests. Here the orders dataset holds the A17 question.

yaml
- vars:
    question: Where is order A17?
  assert:
    - type: contains
      value: 3 March

The refunds dataset is a second file in the same shape, holding the refund question.

yaml
- vars:
    question: How long does a refund take?
  assert:
    - type: contains
      value: five working days

Gathering them with one pattern

The tests key takes a glob. The pattern file://tests_*.yaml matches every file whose name starts with tests_, so both datasets load and their cases run together. A glob into a folder, such as file://tests/*.yaml, works the same way when the files sit in a directory.

yaml
providers:
  - file://support_bot.py
prompts:
  - "{{question}}"
tests: file://tests_*.yaml
Project files used on this pageThis lesson builds on a project from earlier lessons. The code below imports this file. Click a file to see its code, or follow the link to the lesson that wrote it. To run the code yourself, keep it in the same folder.
View the code here
support_bot.py
# the shop support bot: find the answer in the handbook, or say it is missing
HANDBOOK = {
    "a17": "Order A17 shipped on 3 March by courier.",
    "refund": "Refunds take five working days.",
}

def call_api(prompt, options, context):
    asked = prompt.lower()                      # the rendered prompt is the question
    for key, line in HANDBOOK.items():
        if key in asked:
            return {"output": line}
    return {"output": "I could not find that in the handbook."}
Exampletwo datasets loaded by one glob pattern
promptfoo eval --no-table

What the glob loaded

  • Two files, two tests, one run. The glob matched both datasets and promptfoo ran every case it found.
  • The order follows the match. Files are gathered in the order the pattern expands, and their cases run in that order.
  • Each file stands alone. A dataset is only a list of tests, so one can be edited or added without touching the others or the config.

A glob vs a single file

A single test fileA glob of files
Where tests liveOne fileMany files, gathered by a pattern
Adding a datasetAppend to the one fileDrop in a file the pattern matches
OwnershipOne file, one ownerA file per team or topic
Config changeNoneNone, the pattern already matches

Where loading from a glob helps

  • A suite split by topic, orders in one dataset and refunds in another, each editable on its own.
  • Tests owned by different people, where a new file is added without a merge conflict in a shared one.
  • Generated datasets, where a script writes a file per batch and the glob picks them all up.
Watch out. A glob matches by name, so a pattern like file://*.yaml would also sweep up promptfooconfig.yaml and try to read it as tests. Give the datasets a shared prefix, such as tests_, and point the pattern at that, so the config is never caught in the match.
Try it yourself
  • Add a tests_shipping.yaml dataset with one case and rerun; watch three tests run with no config change.
  • List the two files as explicit file:// entries under tests instead of a glob and confirm the same run.
  • Rename the pattern to file://*.yaml and read the error when the config is loaded as tests.
PreviousTests in a CSV file

This is what real progress feels like.