A Python provider: your own app under test
A Python provider is a file of your own that promptfoo calls as the thing under test, so the eval runs your application's answers rather than a raw model's.
Last updated: 15 Oct, 2026 · Promptfoo 0.124.1
Part one pointed the config at file://support_bot.py without stopping to say what that file has to look like. It is the running example's whole app, and promptfoo treats it as the provider: the code that turns a prompt into an answer. This lesson reads the contract that file obeys, then runs an eval against it.
The call_api contract
A Python provider is any file that defines a call_api function. Promptfoo imports the file and calls that function once per test, passing the rendered prompt and two dictionaries, and expects back a dict with an output key. The support bot meets the contract: call_api takes the prompt and two dicts it does not need yet, and returns {"output": ...}.
HANDBOOK = {
"a17": "Order A17 shipped on 3 March by courier.",
"refund": "Refunds take five working days.",
}
def call_api(prompt, options, context): # promptfoo calls this for every test
asked = prompt.lower()
for key, line in HANDBOOK.items():
if key in asked:
return {"output": line}
return {"output": "I could not find that in the handbook."}promptis the rendered prompt, the template with every var filled in.optionsandcontextcarry provider config and the test's vars; this bot ignores them.- The return is a dict with
output, the answer promptfoo then checks. Anything else in the dict, such as a token count, is optional.
Running the eval against the file
The config names the provider as file://support_bot.py and runs two questions the bot answers from its handbook. Each lesson in this part is its own promptfooconfig.yaml, changed as the tests change, with the bot saved beside it.
providers:
- file://support_bot.py
prompts:
- "{{question}}"
tests:
- vars:
question: Where is order A17?
assert:
- type: contains
value: 3 March
- vars:
question: How long does a refund take?
assert:
- type: contains
value: five working days
promptfoo eval --no-tableStarting evaluation eval-24W-2026-10-14T19:17:00 Running 2 test cases (up to 4 at a time)... Creating cache folder at ~/.promptfoo/cache. ✓ Eval complete (ID: eval-24W-2026-10-14T19:17:00) » View results: promptfoo view » Share with your team: https://promptfoo.app » Feedback: https://promptfoo.dev/feedback Results: ✓ 2 passed (100%) 0 failed (0%) 0 errors (0%) Duration: 0s (concurrency: 4)
What the run shows
- Both questions passed. Promptfoo imported the file, called
call_apitwice, and matched each answer against itscontainscheck. - No model and no key were involved. The provider is your code, so the answer is whatever the function returns.
- The answer is deterministic. The same question gives the same reply every run, which is what lets a test pin it down.
A Python provider vs a hosted model
| A Python provider | A hosted model | |
|---|---|---|
| What answers | Your call_api function | A vendor's model over the network |
| Needs a key | No | Yes |
| Same answer each run | Yes, if your code is | No, usually not |
| Good for | Testing your own app end to end | Testing the model itself |
Where a Python provider helps
- Putting your real app under test, so the eval checks the answers users would get, not a model in isolation.
- Wrapping a pipeline that retrieves, prompts and post-processes, all behind one
call_api. - A deterministic stand-in for a model while you build the suite, swapped for the real thing later.
call_api exactly, and it must return a dict with an output key. A function with another name, or a bare string return, makes the run error rather than grade an answer.Related
- Previous: promptfoo view: every run kept
- Next: Prompts and vars: one prompt, many questions
- Reference: Promptfoo Python provider docs
- Add a third order to the
HANDBOOKdict and a test that asks about it. - Return an extra key alongside
outputand confirm the run ignores it. - Rename
call_apitoanswerand read the error promptfoo prints.
Little by little, you're building something great.