A Python assertion: a check that is a function
A Python assertion is a check you write as a function: type: python with value: file://assert.py, and promptfoo calls that file for every answer.
Last updated: 15 Oct, 2026 · Promptfoo 0.124.1
The built-in checks cover text and shape. When a rule is specific to your bot, write it in Python instead. The one thing you have to get right is the name of the function promptfoo calls, and getting it wrong fails in a way worth seeing first.
The entry point has a fixed name
Promptfoo loads the file and calls one function, get_assert, passing the answer and a context object. Any other name is not found. Here is the check written with the wrong name, so you can read the error it raises.
# WRONG: promptfoo looks for get_assert, not this name
def check_answer(output, context):
return {"pass": "A17" in output, "score": 1.0, "reason": "mentions the order id"}
The config wires the file in as a python assertion.
providers:
- file://support_bot.py
prompts:
- "{{question}}"
tests:
- vars: { question: "Where is order A17?" }
assert:
- type: python
value: file://assert.py
View the code here
# the shop support bot: find the answer in the handbook, or say it is missing
HANDBOOK = {
"a17": "Order A17 shipped on 3 March by courier.",
"refund": "Refunds take five working days.",
}
def call_api(prompt, options, context):
asked = prompt.lower() # the rendered prompt is the question
for key, line in HANDBOOK.items():
if key in asked:
return {"output": line}
return {"output": "I could not find that in the handbook."}
# read out.json: each answer, its checks, and any metric scores
import json
for r in json.load(open("out.json"))["results"]["results"]:
print(r["response"]["output"], "->", "PASS" if r["success"] else "FAIL")
for c in r["gradingResult"]["componentResults"]:
print(" ", "pass" if c["pass"] else "FAIL", c["assertion"]["type"], "-", c["reason"])
if r["namedScores"]:
scores = {k: round(v, 3) for k, v in r["namedScores"].items()}
print(" metrics", scores, "| test score", round(r["score"], 3))
Run it. Promptfoo imports the file, looks for get_assert, and does not find it, so the test errors and the run fails.
promptfoo eval -c pyassert.yaml --no-table 2>&1 | grep -E "no attribute|1 failed"AttributeError: module 'assert' has no attribute 'get_assert' Error running Python script: AttributeError: module 'assert' has no attribute 'get_assert' Stack Trace: Error: AttributeError: module 'assert' has no attribute 'get_assert' ✗ 1 failed (100%)
The same file with the right name
The fix is one line: change the function name in assert.py to get_assert, keeping the same body. It returns a dictionary with pass, a score and a reason, which is what shows up in the results.
# promptfoo calls get_assert(output, context) in the file value points at
def get_assert(output, context):
passed = "A17" in output
return {
"pass": passed,
"score": 1.0 if passed else 0.0,
"reason": "mentions the order id" if passed else "order id missing",
}
promptfoo eval -c pyassert.yaml -o out.json --no-table >/dev/null 2>&1
python3 show.pyOrder A17 shipped on 3 March by courier. -> PASS pass python - mentions the order id
What get_assert receives and returns
| Part | What it is |
|---|---|
output | The answer string to check |
context | An object with vars, the prompt and the provider response |
| Return a bool | True passes, False fails |
| Return a dict | {pass, score, reason} for a graded result with a reason |
Where a Python assertion fits
- A rule the built-ins do not have, such as a cap on a number parsed out of the answer.
- A check that reads a var, comparing the answer against
contextrather than a fixed value. - A reusable check, kept in one file and pointed at from many tests.
get_assert. A wrong name does not warn at load time; it raises AttributeError: module '...' has no attribute 'get_assert' when the test runs, and the test is reported as failed.Related
- Previous: is-json: checking the shape of an answer
- Next: A JavaScript assertion
- Reference: Promptfoo Python assertions
- Return a plain
Truefromget_assertand confirm the test passes with no reason line. - Read
context["vars"]["question"]inside the function and include it in the reason. - Rename the function back to the wrong name and read the AttributeError in full without the grep filter.
Slow is fine. Stopping is the only problem.