Promptfoopromptfoo 0.124.1 · Node 24 · Python 3
Dashboard
0%

Two prompts side by side

Two prompts in one config keep the provider fixed and vary the wording, so you can read off how a change to the prompt moves the answer and the pass rate.

Last updated: 15 Oct, 2026 · Promptfoo 0.124.1

The Two providers side by side lesson varied the app and held the prompt. This one does the reverse: one bot, two prompt templates. It is how you test a change to your prompt the same way you test a change to your code, with a column for each wording.

A question the bot cannot place

The support bot finds a fact only when a handbook word appears in the prompt it is handed. The question here, How long until my money comes back?, never says refund, so a bare prompt gives the bot nothing to match. The two prompts differ in whether they name the topic for it.

yaml
providers:
  - file://support_bot.py
prompts:
  - "{{question}}"                                      # the bare question
  - "The customer is asking about a refund. {{question}}"   # the same question, with the topic named
tests:
  - vars:
      question: How long until my money comes back?
    assert:
      - type: contains
        value: five working days
Project files used on this pageThis lesson builds on a project from earlier lessons. The code below imports this file. Click a file to see its code, or follow the link to the lesson that wrote it. To run the code yourself, keep it in the same folder.
View the code here
support_bot.py
# the shop support bot: find the answer in the handbook, or say it is missing
HANDBOOK = {
    "a17": "Order A17 shipped on 3 March by courier.",
    "refund": "Refunds take five working days.",
}

def call_api(prompt, options, context):
    asked = prompt.lower()                      # the rendered prompt is the question
    for key, line in HANDBOOK.items():
        if key in asked:
            return {"output": line}
    return {"output": "I could not find that in the handbook."}

Running both prompts

Each prompt becomes a column. The bare prompt passes the question straight through; the second prepends a line naming the refund topic. Both answer the same question against the same check, so the only difference is the wording.

Exampleone bot, two prompts, only the wording changes
promptfoo eval

Why the wording decided the answer

  • The bare prompt failed, because the question never contains refund, so the bot fell back to its miss line.
  • The prompt that named the topic passed, because adding a refund gave the bot the word it looks up on.
  • The bot never changed, only the text it was handed, which is the whole point of comparing prompts.

Two prompts vs two providers

Two promptsTwo providers
What variesThe wording sent inThe app under test
What is held fixedThe providerThe prompt
The question it answersDid the rewording helpDid the code change help

Where two prompts side by side help

  • Tuning a system prompt, by running two versions over the same questions.
  • Measuring a reword, so a prompt change is judged by its pass rate, not by eye.
  • Finding a brittle prompt, one that passes only when the input is phrased a certain way.
Watch out. This bot matches on a literal word, so the prompt that names the topic wins. A hosted model would read the bare question and often still answer, so the gap between two prompts is usually smaller and more about tone and reliability than a single missing word. The method is the same; only how much the wording matters changes.
Try it yourself
  • Change the question to How long does a refund take? and rerun; watch both prompts pass once the word is in the question itself.
  • Add a third prompt that names the order instead of the refund and read its column.
  • Swap the second prompt's wording for one that still omits refund and confirm it fails like the bare one.

You understood something today that you didn't yesterday.