Install promptfoo and run the same check

The loop in lesson 1 did three things: name the questions, get an answer, check it. Promptfoo wants the same three things, written in a file instead of in code.

Installing it

Promptfoo is a Node command. It needs Node 22.22 or newer, and the docs recommend Node 24; on an older Node it refuses to start and tells you so.

npm install -g promptfoo

If you would rather not install anything globally, npx promptfoo@latest works everywhere this course writes promptfoo.

promptfoo --version

The config file

Promptfoo looks for a file called promptfooconfig.yaml in the directory you run it from. It has three parts, and they are the same three things the loop had.

providers:
  - echo
prompts:
  - "{{question}}"
tests:
  - vars:
      question: Where is order A17?
    assert:
      - type: contains
        value: A17

providers is what answers. prompts is what gets sent to it, with {{question}} standing in for a value that changes. tests is the list of runs: each one sets the variables and says what has to be true of the answer.

The provider here is echo, which is built in and returns the prompt it was given, unchanged. Nothing is being answered yet. That is deliberate: the first run should show you the machinery with nothing mysterious inside it.

Running it

promptfoo eval

promptfoo eval read the file, ran one test, and printed a table with one row. The answer was the question, because echo gives back what it is given, and the check passed because the word A17 was in it.

A second question

Tests is a list, so a second question is another entry under it. Nothing else in the file changes.

  - vars:
      question: How long does a refund take?
    assert:
      - type: contains
        value: five working days

promptfoo eval

Two rows, and the second one failed. echo returned the question How long does a refund take?, which does not contain five working days, so the check was not satisfied. A failing test on the second run is the right way round: you have now seen what a failure looks like before you had anything real to lose.