A Python provider: your own app under test

Echo has done its job. The thing you actually want to test is your application, and promptfoo will call any Python file that offers it one function.

A provider is anything that takes a prompt and gives back an answer. The built in ones wrap model APIs. A file provider wraps whatever you write, which for most real work is your own application: the prompt goes in, your retrieval and your model and your formatting all run, and one answer comes out.

The one function

Promptfoo looks for a function called call_api, and calls it with three arguments: the prompt after the variables have been filled in, the provider's own configuration, and the test case it came from. It must return a dictionary with an output key.

python

def call_api(prompt, options, context):
    return {"output": "Order A17 shipped on 3 March by courier."}

That is a complete provider. It ignores the question and always says the same thing, which makes it useless and legal.

The support bot

Here is the one this course uses for the rest of the way. It is the bot from lesson 1 with the function renamed and the answer wrapped in a dictionary.

python

HANDBOOK = {
    "a17": "Order A17 shipped on 3 March by courier.",
    "refund": "Refunds take five working days.",
}

def call_api(prompt, options, context):
    asked = prompt.lower()
    for key, line in HANDBOOK.items():
        if key in asked:
            return {"output": line}
    return {"output": "I could not find that in the handbook."}

Two facts, a lookup by keyword, and a sentence for everything else. No model, no key, no network. Everything you learn about testing it applies unchanged when the body of that function becomes a real call.

Pointing the config at it

A provider that lives in a file is written as file:// and the path. The rest of the config does not change.

yaml

providers:
  - file://support_bot.py
prompts:
  - "{{question}}"
tests:
  - vars:
      question: Where is order A17?
    assert:
      - type: contains
        value: 3 March

Example

promptfoo eval

The column heading changed. Instead of [echo] it now names the file and the function, and the answer in the cell came from your code.

Promptfoo runs python, not python3. On a machine where only python3 exists you get a Python 3 not found error. The fix is either PROMPTFOO_PYTHON=/usr/bin/python3 in the environment, or pythonExecutable on the provider in the config, which is how you point one provider at a virtual environment.

What else it can return

output is the only key you need. The rest are optional and worth knowing about, because they are how the numbers in later lessons get filled in: tokenUsage and cost feed the cost checks, latencyMs overrides the measured time, and error instead of output marks the row as an error rather than a failure.

Try it yourself

Make call_api return {"error": "handbook offline"} and run the eval. Note that the summary counts it as an error, not a failure.
Print prompt inside call_api and run again, to see exactly what promptfoo sends.
Add a third fact to HANDBOOK and a test that asks about it.

Previouspromptfoo view: every run kept

Every expert started right here.