Promptfoopromptfoo 0.124.1 · Node 24 · Python 3
Dashboard
0%
1
Curious builder0 XP earned · 300 to level 2
0 daysFinish a lesson to begin
Badge collection0 of 6 unlocked
33 small wins to finish your pathNext lesson →

Filtering runs: re-running the failures

The --filter flags pick a subset of a suite to run, so you can re-run only the tests that failed last time, only the first few, or only those whose description matches a pattern.

Last updated: 15 Oct, 2026 · Promptfoo 0.124.1

A big suite is slow to run whole while you fix one broken case. Promptfoo's --filter options narrow a run to the tests you care about right now. The most useful is --filter-failing, which reads a past run and re-runs only its non-passing tests. This suite has three tests, one of which the bot cannot answer.

yaml
providers:
  - file://support_bot.py
prompts:
  - "{{question}}"
tests:
  - description: order A17            # --filter-pattern matches this description
    vars:
      question: Where is order A17?
    assert:
      - type: contains
        value: 3 March
  - description: refund window
    vars:
      question: How long does a refund take?
    assert:
      - type: contains
        value: five working days
  - description: order B99            # the one the bot cannot answer
    vars:
      question: Where is order B99?
    assert:
      - type: contains
        value: shipped
Project files used on this pageThis lesson builds on a project from earlier lessons. The code below imports this file. Click a file to see its code, or follow the link to the lesson that wrote it. To run the code yourself, keep it in the same folder.
View the code here
support_bot.py
# the shop support bot: find the answer in the handbook, or say it is missing
HANDBOOK = {
    "a17": "Order A17 shipped on 3 March by courier.",
    "refund": "Refunds take five working days.",
}

def call_api(prompt, options, context):
    asked = prompt.lower()                      # the rendered prompt is the question
    for key, line in HANDBOOK.items():
        if key in asked:
            return {"output": line}
    return {"output": "I could not find that in the handbook."}

A full run with one failure

Running the whole suite and writing it to a file gives both the result and the record --filter-failing reads next. Order B99 is not in the handbook, so its test fails and the command exits non-zero.

Examplethe full suite: two pass, one fails, saved to out.json
promptfoo eval -o out.json

Re-running only the failures

--filter-failing takes the path to that JSON file and runs only the tests that did not pass in it. Here that is the single B99 test, so the rerun is one row instead of three. It still fails, so it still exits non-zero, which is exactly what you want while you work on the fix.

Exampleonly the one failing test runs again
promptfoo eval --filter-failing out.json

Narrowing a run other ways

Two more flags pick a subset without a prior run. -n (the long form is --filter-first-n) runs only the first few tests, and --filter-pattern runs only those whose description matches a regular expression.

Examplefirst-n and a description pattern each narrow the run
# -n runs only the first test; --filter-pattern matches a test description
promptfoo eval --no-table -n 1
promptfoo eval --no-table --filter-pattern refund

What each filter selected

  • --filter-failing out.json re-ran the one non-passing test from the saved run, the B99 order.
  • -n 1 ran only the first test in the file, the A17 order, and passed.
  • --filter-pattern refund ran only the test whose description contains refund, the refund window, and passed.

The filter flags at a glance

FlagRunsNeeds a prior run
--filter-failingThe non-passing tests of a saved runYes, a JSON file or eval id
-n / --filter-first-nThe first N testsNo
--filter-patternTests whose description matches a regexNo

Where filtering a run helps

  • Fixing a failing test, by re-running only it until it passes, then the whole suite once to confirm.
  • A quick smoke run, with -n to try the first few cases before the full suite.
  • Working on one area, with --filter-pattern to run only the tests you named for it.
Watch out. --filter-pattern matches the test's description, not its vars or its question. A test with no description can never be selected by it, so give every test a short description if you want to filter on it. --filter-failing also accepts an eval id in place of a file path, so you can re-run the failures of a kept run without writing it out first.
Try it yourself
  • Fix the B99 test's expected value to could not find and rerun with --filter-failing out.json against the old file; it still re-runs B99, now passing.
  • Run -n 2 and confirm the first two tests run and the third is skipped.
  • Change --filter-pattern to order and watch both order tests run while the refund one is left out.
PreviousThe cache, --repeat, and scores that move

This is what real progress feels like.