Two providers side by side
Two providers in one config run the same tests against two versions of your app and draw them as columns side by side, which is the one thing promptfoo is built to do best.
Last updated: 15 Oct, 2026 · Promptfoo 0.124.1
Every run so far tested one bot. The real use of an eval is to answer did my change help. List two providers in the same config and promptfoo runs each test against both, puts a column for each in the grid, and reports a pass rate per column. This lesson compares the support bot from A Python provider: your own app under test against a tweaked version.
Each lesson in this part changes the promptfooconfig.yaml it runs, so the config shown with every example is the one that produced its output.
The bot and its tweak
The first provider is the bot as it has been all along. The second, support_bot_v2.py, is the same bot with one change: it opens every reply with Thanks for asking. Comparing them answers whether that change helped or hurt.
HANDBOOK = {
"a17": "Order A17 shipped on 3 March by courier.",
"refund": "Refunds take five working days.",
}
def call_api(prompt, options, context): # promptfoo calls this with the rendered prompt
asked = prompt.lower()
for key, line in HANDBOOK.items(): # match a handbook word in the question
if key in asked:
return {"output": line}
return {"output": "I could not find that in the handbook."}The tweaked bot differs by a single line, the polite preamble on each answer.
HANDBOOK = {
"a17": "Order A17 shipped on 3 March by courier.",
"refund": "Refunds take five working days.",
}
def call_api(prompt, options, context):
asked = prompt.lower()
for key, line in HANDBOOK.items():
if key in asked:
return {"output": "Thanks for asking. " + line} # v2 adds a polite preamble
return {"output": "Thanks for asking. I could not find that in the handbook."}Listing both providers
Both files go under providers. Each test carries two checks: a factual one the bot must still pass, and a contains: Thanks that only the polite version can meet. That second check is what makes the two columns diverge.
providers:
- file://support_bot.py # the bot from part one
- file://support_bot_v2.py # the tweaked bot
prompts:
- "{{question}}"
tests:
- vars:
question: Where is order A17?
assert:
- type: contains
value: 3 March
- type: contains
value: Thanks
- vars:
question: How long does a refund take?
assert:
- type: contains
value: five working days
- type: contains
value: Thanks
Running the comparison
One promptfoo eval runs every test against both providers and draws a column for each. Because the first bot never says Thanks, its cells fail, so the command exits non-zero even though the second bot passes everything.
promptfoo evalStarting evaluation eval-pEn-2026-10-14T19:18:16
Running 4 test cases (up to 4 at a time)...
Creating cache folder at ~/.promptfoo/cache.
┌────────────────────────────────────────┬────────────────────────────────────────┬────────────────────────────────────────┐
│ question │ [python:support_bot.py:default] │ [python:support_bot_v2.py:default] │
│ │ {{question}} │ {{question}} │
├────────────────────────────────────────┼────────────────────────────────────────┼────────────────────────────────────────┤
│ Where is order A17? │ [FAIL] Order A17 shipped on 3 March by │ [PASS] Thanks for asking. Order A17 │
│ │ courier. │ shipped on 3 March by courier. │
├────────────────────────────────────────┼────────────────────────────────────────┼────────────────────────────────────────┤
│ How long does a refund take? │ [FAIL] Refunds take five working days. │ [PASS] Thanks for asking. Refunds take │
│ │ │ five working days. │
└────────────────────────────────────────┴────────────────────────────────────────┴────────────────────────────────────────┘
✓ Eval complete (ID: eval-pEn-2026-10-14T19:18:16)
» View results: promptfoo view
» Share with your team: https://promptfoo.app
» Feedback: https://promptfoo.dev/feedback
Results:
✓ 2 passed (50.00%)
✗ 2 failed (50.00%)
0 errors (0%)
Duration: 0s (concurrency: 4)Reading the two columns
- Each provider is its own column, headed with the file it came from, so the same question sits in one row across both.
- The first bot fails both rows, because its answers carry no
Thanks, while the tweaked bot passes both. - The pass rate is for the whole grid, two of four cells, which is why the summary reads 50 percent even though one bot is perfect and the other is not.
A pass rate for each provider
The grid pass rate mixes both bots together. To compare them you want a count per provider, which the kept JSON holds under results.prompts[].metrics. Writing the run to a file and reading those numbers shows each bot on its own.
promptfoo eval -o out.json >/dev/null 2>&1
# the pass count for each provider, read from the kept run
python3 - <<'PY'
import json
d = json.load(open("out.json"))
for pr in d["results"]["prompts"]:
m = pr["metrics"]
total = m["testPassCount"] + m["testFailCount"]
print("%-32s %s/%s passed" % (pr["provider"], m["testPassCount"], total))
PYpython:support_bot.py:default 0/2 passed python:support_bot_v2.py:default 2/2 passed
What the per-provider counts show
- The first bot passed none of its two tests, because the polite check caught every reply.
- The tweaked bot passed both, so the change moved it from zero to a clean pass on this suite.
- The numbers come from one run, so the two bots were tested on identical questions and checks, which is what makes the comparison fair.
Where two providers side by side help
- Judging a change to your app, by running the old and new versions on one suite.
- Choosing between two models later, with a column each and a pass rate to compare.
- Catching a regression, where a tweak that helps one case quietly breaks another.
Related
- Previous: When the judge is wrong
- Next: Two prompts side by side
- Reference: Promptfoo documentation
- Drop the
contains: Thankschecks and rerun; watch both columns pass and the rate climb to 100 percent. - Add a third provider, a copy of the bot under a new name, and read the third column.
- Open the run in a browser with
promptfoo viewand compare the two columns there.
Little by little, you're building something great.