What you are going to build

Promptfoo is a command that runs your prompts and your application against a list of questions, checks every answer, and tells you which ones broke. This course builds one suite for one support bot, from a single question to a run that fails a build. Every lesson runs on your machine with no API key.

You already test the rest of your code. An answer from a model is harder: it is a paragraph, not a number, and it changes when you touch the prompt. Promptfoo's answer is a file that lists the questions, the thing that answers them, and what counts as a good answer. Running it is one command.

The finished suite

promptfoo eval

Three questions, three answers, three checks, and a line saying how many passed. That is the whole idea of the tool, and the rest of the course is about what you put in that file.

The bot being tested here is an ordinary Python file that you write in lesson 5. Nothing in this run reached the internet, which is the second thing this course is about.

No API key, anywhere

Every other promptfoo tutorial starts by asking for an OPENAI_API_KEY. That is a reasonable thing to need eventually, and a bad thing to need in lesson one. So this course uses providers that run on your machine: a built in one called echo, and Python files you write yourself. The last part shows the one line that swaps in a real model.

Some checks in promptfoo ask a model to judge an answer. Those need a key too, so in lesson 13 you write the judge: a Python file that promptfoo talks to exactly as it would talk to a model, which decides by comparing words instead of by thinking. It is about sixty lines by the end of that lesson, and it grows twice more as new kinds of check arrive.

What you will have built

What you need

• Node.js 22.22 or newer. Node 24 is what the promptfoo docs recommend.
• Python 3, for the provider and the judge you write.
• A terminal. No account, no key, no card.