Two providers side by side

Every run so far has had one provider. The reason promptfoo draws a table rather than a list is that it expects more than one, and running two is one line.

Here is the support bot and a second version of it that opens with a courtesy line. Two files, same interface.

HANDBOOK = {
    "a17": "Order A17 shipped on 3 March by courier.",
    "refund": "Refunds take five working days.",
}

def call_api(prompt, options, context):
    asked = prompt.lower()
    for key, line in HANDBOOK.items():
        if key in asked:
            return {"output": "Thanks for asking. " + line}
    return {"output": "Thanks for asking. I could not find that in the handbook."}

providers:
  - file://support_bot.py
  - file://support_bot_v2.py
prompts:
  - "{{question}}"
tests: file://tests.csv

promptfoo eval

One column per provider, one row per question, and every test run against both. The counts at the bottom cover the whole grid, so six results rather than three.

Both versions pass every check, which is the useful answer: the courtesy line did not break anything. Had one column gone red while the other stayed green, the table would have told you which change caused it without any further work.

What this is for

The obvious use is choosing between two models. The better use is the one above: the same application before and after a change. A prompt edit, a new retrieval step, a cheaper model, a different temperature. Put the old one and the new one in the list together and the comparison is done by the run rather than by memory.

Naming the columns

A provider entry can carry a label, which is what the column heading uses. With two files whose names differ by one character, that is worth doing.

providers:
  - id: file://support_bot.py
    label: current
  - id: file://support_bot_v2.py
    label: with greeting