Promptfoopromptfoo 0.124.1 · Node 24 · Python 3
Dashboard
0%
1
Curious builder0 XP earned · 300 to level 2
0 daysFinish a lesson to begin
Badge collection0 of 6 unlocked
33 small wins to finish your pathNext lesson →

Reusing setup across tests with scenarios

A scenario pairs a set of data rows with a set of checks, so one group of asserts runs across every row without being written out once per row.

Last updated: 15 Oct, 2026 · Promptfoo 0.124.1

The Loading test datasets from a file glob lesson gathered many tests, but each still carried its own assert. Often several rows should face the same checks: every answer from the handbook should carry a fact, every miss should say so. A scenario states the rows once and the checks once, and promptfoo runs each check over each row.

The shape of a scenario

The scenarios key is a list. Each scenario has a config, a list of data rows that each set the vars for a run, and a tests list holding the asserts. Promptfoo pairs every row in config with every entry in tests, so one set of checks covers all the rows.

yaml
providers:
  - file://support_bot.py
prompts:
  - "{{question}}"
scenarios:
  - description: questions the handbook answers
    config:
      - vars:
          question: Where is order A17?
      - vars:
          question: How long does a refund take?
    tests:
      - assert:
          - type: not-contains
            value: could not find
  - description: a question the handbook misses
    config:
      - vars:
          question: Where is order B99?
    tests:
      - assert:
          - type: contains
            value: could not find

The first scenario lists two questions the handbook answers and one check, that the reply is not the fallback, so that one assert runs over both rows. The second scenario is a single miss with the opposite check. Three rows in all, each graded by its scenario's test.

Project files used on this pageThis lesson builds on a project from earlier lessons. The code below imports this file. Click a file to see its code, or follow the link to the lesson that wrote it. To run the code yourself, keep it in the same folder.
View the code here
support_bot.py
# the shop support bot: find the answer in the handbook, or say it is missing
HANDBOOK = {
    "a17": "Order A17 shipped on 3 March by courier.",
    "refund": "Refunds take five working days.",
}

def call_api(prompt, options, context):
    asked = prompt.lower()                      # the rendered prompt is the question
    for key, line in HANDBOOK.items():
        if key in asked:
            return {"output": line}
    return {"output": "I could not find that in the handbook."}

Running the scenarios

One eval runs both scenarios. The two rows of the first and the one row of the second become three test cases, each checked by the asserts of its own scenario.

Exampletwo scenarios, three rows, each row checked by its scenario's asserts
promptfoo eval --no-table

What the scenarios ran

  • The first scenario's check ran twice, once per row in its config, without being written out for each.
  • The second scenario had its own check, the opposite one, so the miss was expected to say it could not find the order.
  • Three cases in all. Each scenario multiplied its rows by its tests, and the two scenarios' results were reported together.

Scenarios vs plain tests

Plain testsA scenario
AssertsWritten per testWritten once, run over every row
Data rowsOne vars block eachA config list of rows
RepetitionSame check copied per rowCheck stated once
Best forRows with different checksRows sharing one set of checks

Where scenarios help

  • One set of checks over many rows, such as a correctness and a tone assert applied to every answer.
  • Grouping rows by expectation, a scenario for answers the bot should know and another for ones it should not.
  • Keeping a suite short, stating a shared check once rather than copying it into every test.
Watch out. A scenario runs every row in config against every entry in tests, so the count is the two multiplied. A scenario with four rows and three tests runs twelve cases, not four, which is easy to forget when a scenario grows.
Try it yourself
  • Add a second check to the first scenario's tests and watch both run over each of its rows.
  • Add a third row to the first scenario's config and count the cases the run reports.
  • Move the B99 row into the first scenario and read why its check now fails there.
PreviousLoading test datasets from a file glob

Every expert started right here.