Promptfoopromptfoo 0.124.1 · Node 24 · Python 3
Dashboard
0%

Two providers side by side

Two providers in one config run the same tests against two versions of your app and draw them as columns side by side, which is the one thing promptfoo is built to do best.

Last updated: 15 Oct, 2026 · Promptfoo 0.124.1

Every run so far tested one bot. The real use of an eval is to answer did my change help. List two providers in the same config and promptfoo runs each test against both, puts a column for each in the grid, and reports a pass rate per column. This lesson compares the support bot from A Python provider: your own app under test against a tweaked version.

Each lesson in this part changes the promptfooconfig.yaml it runs, so the config shown with every example is the one that produced its output.

The bot and its tweak

The first provider is the bot as it has been all along. The second, support_bot_v2.py, is the same bot with one change: it opens every reply with Thanks for asking. Comparing them answers whether that change helped or hurt.

python
HANDBOOK = {
    "a17": "Order A17 shipped on 3 March by courier.",
    "refund": "Refunds take five working days.",
}

def call_api(prompt, options, context):   # promptfoo calls this with the rendered prompt
    asked = prompt.lower()
    for key, line in HANDBOOK.items():    # match a handbook word in the question
        if key in asked:
            return {"output": line}
    return {"output": "I could not find that in the handbook."}

The tweaked bot differs by a single line, the polite preamble on each answer.

python
HANDBOOK = {
    "a17": "Order A17 shipped on 3 March by courier.",
    "refund": "Refunds take five working days.",
}

def call_api(prompt, options, context):
    asked = prompt.lower()
    for key, line in HANDBOOK.items():
        if key in asked:
            return {"output": "Thanks for asking. " + line}   # v2 adds a polite preamble
    return {"output": "Thanks for asking. I could not find that in the handbook."}

Listing both providers

Both files go under providers. Each test carries two checks: a factual one the bot must still pass, and a contains: Thanks that only the polite version can meet. That second check is what makes the two columns diverge.

yaml
providers:
  - file://support_bot.py          # the bot from part one
  - file://support_bot_v2.py       # the tweaked bot
prompts:
  - "{{question}}"
tests:
  - vars:
      question: Where is order A17?
    assert:
      - type: contains
        value: 3 March
      - type: contains
        value: Thanks
  - vars:
      question: How long does a refund take?
    assert:
      - type: contains
        value: five working days
      - type: contains
        value: Thanks

Running the comparison

One promptfoo eval runs every test against both providers and draws a column for each. Because the first bot never says Thanks, its cells fail, so the command exits non-zero even though the second bot passes everything.

Examplethe same tests against two bots, one column each
promptfoo eval

Reading the two columns

  • Each provider is its own column, headed with the file it came from, so the same question sits in one row across both.
  • The first bot fails both rows, because its answers carry no Thanks, while the tweaked bot passes both.
  • The pass rate is for the whole grid, two of four cells, which is why the summary reads 50 percent even though one bot is perfect and the other is not.

A pass rate for each provider

The grid pass rate mixes both bots together. To compare them you want a count per provider, which the kept JSON holds under results.prompts[].metrics. Writing the run to a file and reading those numbers shows each bot on its own.

Exampleeach provider's pass count, read from the run
promptfoo eval -o out.json >/dev/null 2>&1

# the pass count for each provider, read from the kept run
python3 - <<'PY'
import json
d = json.load(open("out.json"))
for pr in d["results"]["prompts"]:
    m = pr["metrics"]
    total = m["testPassCount"] + m["testFailCount"]
    print("%-32s %s/%s passed" % (pr["provider"], m["testPassCount"], total))
PY

What the per-provider counts show

  • The first bot passed none of its two tests, because the polite check caught every reply.
  • The tweaked bot passed both, so the change moved it from zero to a clean pass on this suite.
  • The numbers come from one run, so the two bots were tested on identical questions and checks, which is what makes the comparison fair.

Where two providers side by side help

  • Judging a change to your app, by running the old and new versions on one suite.
  • Choosing between two models later, with a column each and a pass rate to compare.
  • Catching a regression, where a tweak that helps one case quietly breaks another.
Watch out. The grid pass rate is across every cell, not per provider, so a strong bot and a weak one average into one middling number. Read the per-provider counts from the JSON, or the browser view, when you want to tell the two apart.
Try it yourself
  • Drop the contains: Thanks checks and rerun; watch both columns pass and the rate climb to 100 percent.
  • Add a third provider, a copy of the bot under a new name, and read the third column.
  • Open the run in a browser with promptfoo view and compare the two columns there.

Little by little, you're building something great.