Promptfoopromptfoo 0.124.1 · Node 24 · Python 3
Dashboard
0%

A Python assertion: a check that is a function

A Python assertion is a check you write as a function: type: python with value: file://assert.py, and promptfoo calls that file for every answer.

Last updated: 15 Oct, 2026 · Promptfoo 0.124.1

The built-in checks cover text and shape. When a rule is specific to your bot, write it in Python instead. The one thing you have to get right is the name of the function promptfoo calls, and getting it wrong fails in a way worth seeing first.

The entry point has a fixed name

Promptfoo loads the file and calls one function, get_assert, passing the answer and a context object. Any other name is not found. Here is the check written with the wrong name, so you can read the error it raises.

python
# WRONG: promptfoo looks for get_assert, not this name
def check_answer(output, context):
    return {"pass": "A17" in output, "score": 1.0, "reason": "mentions the order id"}

The config wires the file in as a python assertion.

yaml
providers:
  - file://support_bot.py
prompts:
  - "{{question}}"
tests:
  - vars: { question: "Where is order A17?" }
    assert:
      - type: python
        value: file://assert.py
Project files used on this pageThis lesson builds on a project from earlier lessons. The code below imports these files. Click a file to see its code, or follow the link to the lesson that wrote it. To run the code yourself, keep them in the same folder.
View the code here
support_bot.py
# the shop support bot: find the answer in the handbook, or say it is missing
HANDBOOK = {
    "a17": "Order A17 shipped on 3 March by courier.",
    "refund": "Refunds take five working days.",
}

def call_api(prompt, options, context):
    asked = prompt.lower()                      # the rendered prompt is the question
    for key, line in HANDBOOK.items():
        if key in asked:
            return {"output": line}
    return {"output": "I could not find that in the handbook."}
show.py
# read out.json: each answer, its checks, and any metric scores
import json

for r in json.load(open("out.json"))["results"]["results"]:
    print(r["response"]["output"], "->", "PASS" if r["success"] else "FAIL")

    for c in r["gradingResult"]["componentResults"]:
        print("  ", "pass" if c["pass"] else "FAIL", c["assertion"]["type"], "-", c["reason"])

    if r["namedScores"]:
        scores = {k: round(v, 3) for k, v in r["namedScores"].items()}
        print("   metrics", scores, "| test score", round(r["score"], 3))

Run it. Promptfoo imports the file, looks for get_assert, and does not find it, so the test errors and the run fails.

ExampleThe wrong function name raises AttributeError
promptfoo eval -c pyassert.yaml --no-table 2>&1 | grep -E "no attribute|1 failed"

The same file with the right name

The fix is one line: change the function name in assert.py to get_assert, keeping the same body. It returns a dictionary with pass, a score and a reason, which is what shows up in the results.

python
# promptfoo calls get_assert(output, context) in the file value points at
def get_assert(output, context):
    passed = "A17" in output

    return {
        "pass": passed,
        "score": 1.0 if passed else 0.0,
        "reason": "mentions the order id" if passed else "order id missing",
    }
Exampleget_assert runs and reports its reason
promptfoo eval -c pyassert.yaml -o out.json --no-table >/dev/null 2>&1
python3 show.py

What get_assert receives and returns

PartWhat it is
outputThe answer string to check
contextAn object with vars, the prompt and the provider response
Return a boolTrue passes, False fails
Return a dict{pass, score, reason} for a graded result with a reason

Where a Python assertion fits

  • A rule the built-ins do not have, such as a cap on a number parsed out of the answer.
  • A check that reads a var, comparing the answer against context rather than a fixed value.
  • A reusable check, kept in one file and pointed at from many tests.
Watch out. The function must be named get_assert. A wrong name does not warn at load time; it raises AttributeError: module '...' has no attribute 'get_assert' when the test runs, and the test is reported as failed.
Try it yourself
  • Return a plain True from get_assert and confirm the test passes with no reason line.
  • Read context["vars"]["question"] inside the function and include it in the reason.
  • Rename the function back to the wrong name and read the AttributeError in full without the grep filter.

Slow is fine. Stopping is the only problem.