Promptfoopromptfoo 0.124.1 · Node 24 · Python 3
Dashboard
0%

Reading the result table

The result table is the grid promptfoo prints after a run: one row per test, one column per provider, each cell marked [PASS] or [FAIL] next to the answer.

Last updated: 15 Oct, 2026 · Promptfoo 0.124.1

The first-config lesson ran with --no-table for a clean summary. Drop that flag and promptfoo draws the full grid, which is where you see which answer failed and what the bot said in reply. To read a failure, the suite needs one.

Adding a test the bot fails

The promptfooconfig.yaml here keeps the two passing tests and adds a third whose expectation the bot cannot meet: it asks about an order the handbook does not know and expects the word shipped.

yaml
  - vars:
      question: Where is order B99?
    assert:
      - type: contains
        value: shipped

The bot has no entry for order B99, so it returns its fallback line, which does not contain shipped. That test will fail, and the grid will show why.

Project files used on this pageThis lesson builds on a project from earlier lessons. The code below imports these files. Click a file to see its code, or follow the link to the lesson that wrote it. To run the code yourself, keep them in the same folder.
View the code here
support_bot.py
# the shop support bot: find the answer in the handbook, or say it is missing
HANDBOOK = {
    "a17": "Order A17 shipped on 3 March by courier.",
    "refund": "Refunds take five working days.",
}

def call_api(prompt, options, context):
    asked = prompt.lower()                      # the rendered prompt is the question
    for key, line in HANDBOOK.items():
        if key in asked:
            return {"output": line}
    return {"output": "I could not find that in the handbook."}
promptfooconfig.yaml
providers:
  - file://support_bot.py
prompts:
  - "{{question}}"
tests:
  - vars:
      question: Where is order A17?
    assert:
      - type: contains
        value: 3 March
  - vars:
      question: How long does a refund take?
    assert:
      - type: contains
        value: five working days
  - vars:
      question: Where is order B99?
    assert:
      - type: contains
        value: shipped

Running with the table

Running promptfoo eval without --no-table prints the grid. Because one test fails, the command exits with code 100 rather than 0, which is why this block is marked as one that is expected to exit non-zero.

ExampleThe full grid, with one failing test
promptfoo eval

Reading the cells

  • The left column is the var, the question each row was run with.
  • The right column is the provider; its data cells start with [PASS] or [FAIL] and then the bot's answer.
  • Two rows pass and one fails, and the summary underneath reports 66.67% passed and 33.33% failed.
  • The failing row shows the fallback line, the answer that did not contain shipped.

The header shows the template, not the value

The provider column's header reads [python:support_bot.py:default] {{question}}. That {{question}} is the raw prompt template, not the question that ran; only the data cells below it show the rendered value. If a var were mistyped the prompt would render empty and the header would still show the template, so it cannot tell you the prompt that ran. To read the exact rendered prompt, write the run to a file with -o out.json and read results.results[].prompt.raw there.

The table vs the summary

The summary (--no-table)The full table
What it showsCounts and the pass rateEvery answer, cell by cell
Best forA quick check or a buildSeeing why a test failed
WidthA few short linesA fixed wide grid

Where the table helps

  • Debugging a failure, where you need the exact answer the bot gave.
  • Comparing providers later, when each gets its own column side by side.
  • A small suite you read by eye, before it grows too wide for the terminal.
Watch out. A failed test makes promptfoo eval exit with code 100, not 0. In a terminal that is easy to miss, but a build job reads the exit code and will fail on it. The failing-a-build lesson turns that exit code into a build gate.
Try it yourself
  • Change the third test's expected value to could not find and rerun; watch all three pass.
  • Run the same config with -o out.json and open the file to find results.results[0].prompt.raw.
  • Run promptfoo eval; echo $? and read the exit code after a failing run.

This is what real progress feels like.