Two prompts side by side
Two prompts in one config keep the provider fixed and vary the wording, so you can read off how a change to the prompt moves the answer and the pass rate.
Last updated: 15 Oct, 2026 · Promptfoo 0.124.1
The Two providers side by side lesson varied the app and held the prompt. This one does the reverse: one bot, two prompt templates. It is how you test a change to your prompt the same way you test a change to your code, with a column for each wording.
A question the bot cannot place
The support bot finds a fact only when a handbook word appears in the prompt it is handed. The question here, How long until my money comes back?, never says refund, so a bare prompt gives the bot nothing to match. The two prompts differ in whether they name the topic for it.
providers:
- file://support_bot.py
prompts:
- "{{question}}" # the bare question
- "The customer is asking about a refund. {{question}}" # the same question, with the topic named
tests:
- vars:
question: How long until my money comes back?
assert:
- type: contains
value: five working days
View the code here
# the shop support bot: find the answer in the handbook, or say it is missing
HANDBOOK = {
"a17": "Order A17 shipped on 3 March by courier.",
"refund": "Refunds take five working days.",
}
def call_api(prompt, options, context):
asked = prompt.lower() # the rendered prompt is the question
for key, line in HANDBOOK.items():
if key in asked:
return {"output": line}
return {"output": "I could not find that in the handbook."}
Running both prompts
Each prompt becomes a column. The bare prompt passes the question straight through; the second prepends a line naming the refund topic. Both answer the same question against the same check, so the only difference is the wording.
promptfoo evalStarting evaluation eval-8FF-2026-10-14T19:18:18
Running 2 test cases (up to 4 at a time)...
Creating cache folder at ~/.promptfoo/cache.
┌────────────────────────────────────────┬────────────────────────────────────────┬────────────────────────────────────────┐
│ question │ [python:support_bot.py:default] │ [python:support_bot.py:default] The │
│ │ {{question}} │ customer is asking about a refund. │
│ │ │ {{question}} │
├────────────────────────────────────────┼────────────────────────────────────────┼────────────────────────────────────────┤
│ How long until my money comes back? │ [FAIL] I could not find that in the │ [PASS] Refunds take five working days. │
│ │ handbook. │ │
└────────────────────────────────────────┴────────────────────────────────────────┴────────────────────────────────────────┘
✓ Eval complete (ID: eval-8FF-2026-10-14T19:18:18)
» View results: promptfoo view
» Share with your team: https://promptfoo.app
» Feedback: https://promptfoo.dev/feedback
Results:
✓ 1 passed (50.00%)
✗ 1 failed (50.00%)
0 errors (0%)
Duration: 0s (concurrency: 4)Why the wording decided the answer
- The bare prompt failed, because the question never contains
refund, so the bot fell back to its miss line. - The prompt that named the topic passed, because adding a refund gave the bot the word it looks up on.
- The bot never changed, only the text it was handed, which is the whole point of comparing prompts.
Two prompts vs two providers
| Two prompts | Two providers | |
|---|---|---|
| What varies | The wording sent in | The app under test |
| What is held fixed | The provider | The prompt |
| The question it answers | Did the rewording help | Did the code change help |
Where two prompts side by side help
- Tuning a system prompt, by running two versions over the same questions.
- Measuring a reword, so a prompt change is judged by its pass rate, not by eye.
- Finding a brittle prompt, one that passes only when the input is phrased a certain way.
Related
- Previous: Two providers side by side
- Next: The cache, --repeat, and scores that move
- Reference: Promptfoo documentation
- Change the question to How long does a refund take? and rerun; watch both prompts pass once the word is in the question itself.
- Add a third prompt that names the order instead of the refund and read its column.
- Swap the second prompt's wording for one that still omits
refundand confirm it fails like the bare one.
You understood something today that you didn't yesterday.