Two prompts side by side

Providers are not the only thing there can be two of. The prompt is usually the thing you are really testing, and it is the cheapest thing to change.

prompts is a list, and promptfoo runs every prompt against every provider against every test.

providers:
  - file://support_bot.py
prompts:
  - "{{question}}"
  - "Answer briefly. {{question}}"
tests:
  - vars:
      question: Where is order A17?
    assert:
      - type: contains
        value: 3 March

promptfoo eval

One test, two prompts, two columns. The bot ignores the instruction because it is a dictionary lookup, so both answers are the same; with a real model behind it the second column is where you would see the wording take effect.

The grid

It is worth knowing exactly how many runs a config asks for, because it grows faster than it looks.

Every one of those is a call. On a real provider the last row is a coffee break and a bill, which is the practical reason to keep one variable moving at a time.

Prompts per provider

Sometimes a prompt only makes sense for one provider, because the models take different formats. A prompt can name the providers it belongs to. The provider named below is a real model rather than a file, which is what lesson 21 swaps in.

prompts:
  - id: file://chat_prompt.json
    label: chat
    providers:
      - openai:gpt-5