Promptfoopromptfoo 0.124.1 · Node 24 · Python 3
Dashboard
0%
1
Curious builder0 XP earned · 300 to level 2
0 daysFinish a lesson to begin
Badge collection0 of 6 unlocked
33 small wins to finish your pathNext lesson →

Promptfoo logoPromptfoo overview

Promptfoo is a command-line tool that tests an LLM application from a config file: it lists the questions, the app that answers them and what a good answer looks like, then runs the grid and prints a pass rate.

Last updated: 15 Oct, 2026 · Promptfoo 0.124.1

An LLM application is hard to test by clicking through it. The answer changes with the prompt, the model and the question, and a change that fixes one case can break another. Promptfoo puts the questions and the checks in a file next to your code, so one command re-runs them all and tells you what broke. This course builds that file for a shop support bot and grows it into a full test suite, and every page runs with no API key.

The ideas this course is built on

Promptfoo is a small number of pieces that fit together. Each gets its own lessons later; here is the whole picture in one place.

  • The config file (promptfooconfig.yaml) lists everything a run needs: the provider, the prompt and the tests.
  • A provider is the application under test. It can be the built-in echo, a Python file you write, or a hosted model.
  • Prompts and vars turn one template into many questions, so a single prompt is run against a table of inputs.
  • Asserts say what counts as a good answer, from a plain substring to a schema to a model acting as a judge.
  • One command, promptfoo eval, runs every prompt against every test and reports a pass rate.
  • Every run is kept in a local database, so you can list past runs, reopen one in a browser and compare them.

What you will build

One example grows through the whole course: a support bot for a shop that answers from a short handbook. It starts as a few lines of Python checked by hand, becomes a provider promptfoo calls, gains deterministic checks and then a model judge that needs no key, is compared across two prompts and two models, and ends as a retrieval bot whose answer and whose search are scored apart. The two facts it knows, where order A17 is and how long a refund takes, stay the same from the first lesson to the last.

The learning path of the Promptfoo course in seven parts, read top to bottom: An eval with no model, The thing being tested, Checks that need no model, A judge of your own, Comparing, Out of the terminal, and A RAG bot end to end, with every lesson listed under its part.

The seven parts

PartLessons
1. An eval with no modelPromptfoo overview, Installation and setup, Checking a support bot's answers in Python, promptfoo eval with a config file, Reading the result table, promptfoo view: every run kept
2. The thing being testedA Python provider: your own app under test, Prompts and vars: one prompt, many questions, Tests in a CSV file, Loading test datasets from a file glob, Reusing setup across tests with scenarios
3. Checks that need no modelDeterministic checks: contains, equals, regex, is-json: checking the shape of an answer, A Python assertion: a check that is a function, A JavaScript assertion, Transforms: reshaping output before a check, threshold, weight and named metrics
4. A judge of your ownllm-rubric: the check that needs a model, A grading script as the stand-in judge, factuality and g-eval: two more formats, When the judge is wrong
5. ComparingTwo providers side by side, Two prompts side by side, The cache, --repeat, and scores that move, Filtering runs: re-running the failures
6. Out of the terminalOutput files: json, csv and junit, Failing a build: exit codes and GitHub Actions, promptfoo from Node, not the CLI, Sharing, self-hosting and tracing, Moving the suite to a hosted model
7. A RAG bot, end to endContext assertions: faithfulness and recall, Red teaming: what promptfoo redteam does, Testing the retrieval support bot
Read this first. Everything in this course runs with no API key. The app under test is the built-in echo provider or a Python file you write, and the model judge in part four is a grading script promptfoo calls instead of a hosted model. The course is pinned to promptfoo 0.124.1 on Node 24, and where the installed tool behaves differently from the published docs, the lesson shows what the tool does and says so.
Back toAll frameworks

Every expert started right here.