Reading the result table
Two runs have printed a table and nobody has said what the columns are. They are worth a lesson, because this table is the thing you will read a hundred times.
The last run is the one to look at. One column per variable, then one column per provider, then a summary at the bottom.
promptfoo evalWhat each part is
The left columns are your variables. There is one for every name used in vars, here just question. If a test set two variables there would be two columns, which is how you tell rows apart when there are forty of them.
The right column is one provider. Its heading is the provider's name and the prompt it was given, so when two providers run the same prompt you get two columns side by side. That is lesson 16.
Each cell starts with the verdict. [PASS] or [FAIL], then the answer itself. A cell that failed carries the reason underneath the answer when there is room for it.
The summary counts tests, not assertions. A test with four checks in it counts once, and it passes only if every check in it passed. That matters when a suite looks worse than it is: one bad check can fail a row whose answer was fine.
The eval id
Every run prints a line like Eval complete (ID: eval-...). That id is how promptfoo keeps runs apart. It is stored locally, which means you can go back to a run from last week and compare, and it is what the next lesson is about.
- Add a third test whose question is
Where is order B99?and whose check iscontains: B99, then predict the result before you run it. - Run
promptfoo eval --no-tableand see what is left. That is the shape you want in CI. - Run it twice and compare the two eval ids.
Slow is fine. Stopping is the only problem.