Promptfoopromptfoo 0.124.1 · Node 24 · Python 3
Dashboard
0%

A Python provider: your own app under test

A Python provider is a file of your own that promptfoo calls as the thing under test, so the eval runs your application's answers rather than a raw model's.

Last updated: 15 Oct, 2026 · Promptfoo 0.124.1

Part one pointed the config at file://support_bot.py without stopping to say what that file has to look like. It is the running example's whole app, and promptfoo treats it as the provider: the code that turns a prompt into an answer. This lesson reads the contract that file obeys, then runs an eval against it.

The call_api contract

A Python provider is any file that defines a call_api function. Promptfoo imports the file and calls that function once per test, passing the rendered prompt and two dictionaries, and expects back a dict with an output key. The support bot meets the contract: call_api takes the prompt and two dicts it does not need yet, and returns {"output": ...}.

python
HANDBOOK = {
    "a17": "Order A17 shipped on 3 March by courier.",
    "refund": "Refunds take five working days.",
}

def call_api(prompt, options, context):   # promptfoo calls this for every test
    asked = prompt.lower()
    for key, line in HANDBOOK.items():
        if key in asked:
            return {"output": line}
    return {"output": "I could not find that in the handbook."}
  • prompt is the rendered prompt, the template with every var filled in.
  • options and context carry provider config and the test's vars; this bot ignores them.
  • The return is a dict with output, the answer promptfoo then checks. Anything else in the dict, such as a token count, is optional.

Running the eval against the file

The config names the provider as file://support_bot.py and runs two questions the bot answers from its handbook. Each lesson in this part is its own promptfooconfig.yaml, changed as the tests change, with the bot saved beside it.

yaml
providers:
  - file://support_bot.py
prompts:
  - "{{question}}"
tests:
  - vars:
      question: Where is order A17?
    assert:
      - type: contains
        value: 3 March
  - vars:
      question: How long does a refund take?
    assert:
      - type: contains
        value: five working days
Examplethe eval runs against your own Python file, no model involved
promptfoo eval --no-table

What the run shows

  • Both questions passed. Promptfoo imported the file, called call_api twice, and matched each answer against its contains check.
  • No model and no key were involved. The provider is your code, so the answer is whatever the function returns.
  • The answer is deterministic. The same question gives the same reply every run, which is what lets a test pin it down.

A Python provider vs a hosted model

A Python providerA hosted model
What answersYour call_api functionA vendor's model over the network
Needs a keyNoYes
Same answer each runYes, if your code isNo, usually not
Good forTesting your own app end to endTesting the model itself

Where a Python provider helps

  • Putting your real app under test, so the eval checks the answers users would get, not a model in isolation.
  • Wrapping a pipeline that retrieves, prompts and post-processes, all behind one call_api.
  • A deterministic stand-in for a model while you build the suite, swapped for the real thing later.
Watch out. The function must be named call_api exactly, and it must return a dict with an output key. A function with another name, or a bare string return, makes the run error rather than grade an answer.
Try it yourself
  • Add a third order to the HANDBOOK dict and a test that asks about it.
  • Return an extra key alongside output and confirm the run ignores it.
  • Rename call_api to answer and read the error promptfoo prints.

Little by little, you're building something great.