Prompts and vars: one prompt, many questions

Every test so far has set one variable called question. That name is not special, and understanding why is what lets a suite grow.

A prompt is a template. Anything inside double braces is looked up in the test's vars when the test runs, so one prompt line serves every test in the file.

yaml

providers:
  - file://support_bot.py
prompts:
  - "A customer asks: {{question}}"

The prompt now wraps the question in a sentence. Under it go the tests, which supply the variable it needs.

yaml

tests:
  - vars:
      question: Where is order A17?
    assert:
      - type: contains
        value: 3 March
  - vars:
      question: How long does a refund take?
    assert:
      - type: contains
        value: five working days

Each test sets only question, and promptfoo builds the whole prompt once per test.

Example

promptfoo eval

Two rows, both passing. The left column is the variable and the right column is the answer, which is why the table stays readable as tests are added: one row per test, always.

Why the wrapper matters

The prompt is part of what is being tested. When you change A customer asks: to something else, every answer can change, and the whole point of keeping the questions in a list is that you can make that change and immediately see what it cost. Lesson 17 runs two versions of the prompt side by side for exactly that reason.

More than one variable

A test can set as many variables as the prompt uses. Promptfoo gives each one its own column.

yaml

prompts:
  - "You are {{tone}}. A customer asks: {{question}}"
tests:
  - vars:
      tone: brief
      question: Where is order A17?

A variable missing from vars renders as nothing. Promptfoo fills the template with what the test gives it, so a typo in a variable name does not error, it quietly produces a prompt with a hole in it. If an answer looks strange, read the prompt column in the viewer before suspecting the model.

Prompts in their own file

Once a prompt is longer than a line, it belongs outside the YAML. A prompt entry that starts with file:// is read from disk, and a .txt file is used as it is.

yaml

prompts:
  - file://support_prompt.txt

Try it yourself

Add a tone variable to the prompt but leave it out of one test, then read that row's prompt in promptfoo view.
Put two entries under prompts and run the eval. Count the rows you get.
Rename question to ask in one test only, and see which column appears.

PreviousA Python provider: your own app under test

Little by little, you're building something great.