Deterministic checks: contains, equals, regex
A deterministic check compares the answer to a value you already know, with no model and no key: contains, equals, regex and their kin either match the text or they do not.
Last updated: 15 Oct, 2026 · Promptfoo 0.124.1
Most of what a support bot has to get right can be checked by reading the text. Did the answer mention the shipping date? Does it start with the order line? Is a refund figure in there? Promptfoo has a family of assertions for exactly these questions, and none of them calls a model, so they run anywhere, for free, and give the same verdict every time.
The bot under test
The running example is a shop support bot. It is an ordinary Python file: promptfoo calls call_api with the rendered prompt, and the function looks the question up in a two-line handbook. A question it cannot place gets a fixed miss reply.
# the shop support bot: find the answer in the handbook, or say it is missing
HANDBOOK = {
"a17": "Order A17 shipped on 3 March by courier.",
"refund": "Refunds take five working days.",
}
def call_api(prompt, options, context):
asked = prompt.lower() # the rendered prompt is the question
for key, line in HANDBOOK.items():
if key in asked:
return {"output": line}
return {"output": "I could not find that in the handbook."}
One config, several checks
The config names the bot as the provider, sends each question through the {{question}} prompt, and lists the assertions under each test. The assertion types here cover the common shapes: an exact equals, a substring contains and its case-insensitive icontains, a starts-with, a regex, and contains-all / contains-any for a set of words.
providers:
- file://support_bot.py
prompts:
- "{{question}}"
tests:
- vars: { question: "Where is order A17?" }
assert:
- type: contains
value: "3 March"
- type: starts-with
value: "Order"
- type: regex
value: "A[0-9]+"
- vars: { question: "Tell me the refund window." }
assert:
- type: icontains
value: "REFUNDS"
- type: equals
value: "Refunds take five working days."
- type: contains-all
value: ["Refunds", "working", "days"]
- vars: { question: "Is the store open on Sunday?" }
assert:
- type: contains
value: "shipped"
- type: contains-any
value: ["handbook", "manual"]
Running the checks
Run the eval. The summary counts the tests that passed and failed. The third question is one the handbook does not cover, so its contains check for the word shipped fails, and a failed test is why the whole run reports a failure.
promptfoo eval -c det.yaml --no-tableStarting evaluation eval-h7Z-2026-10-14T19:13:02 Running 3 test cases (up to 4 at a time)... Creating cache folder at ~/.promptfoo/cache. ✓ Eval complete (ID: eval-h7Z-2026-10-14T19:13:02) » View results: promptfoo view » Share with your team: https://promptfoo.app » Feedback: https://promptfoo.dev/feedback Results: ✓ 2 passed (66.67%) ✗ 1 failed (33.33%) 0 errors (0%) Duration: 0s (concurrency: 4)
Which check decided each answer
The summary gives counts, not detail. To see the verdict per assertion, write the results to JSON with -o and read them back. A tiny reader prints each answer, then a line per check.
# read out.json: each answer, its checks, and any metric scores
import json
for r in json.load(open("out.json"))["results"]["results"]:
print(r["response"]["output"], "->", "PASS" if r["success"] else "FAIL")
for c in r["gradingResult"]["componentResults"]:
print(" ", "pass" if c["pass"] else "FAIL", c["assertion"]["type"], "-", c["reason"])
if r["namedScores"]:
scores = {k: round(v, 3) for k, v in r["namedScores"].items()}
print(" metrics", scores, "| test score", round(r["score"], 3))
Point it at the same config and every assertion shows its own pass or fail, with the reason promptfoo recorded.
promptfoo eval -c det.yaml -o out.json --no-table >/dev/null 2>&1 # run, then read the JSON
python3 show.pyOrder A17 shipped on 3 March by courier. -> PASS pass contains - Assertion passed pass starts-with - Assertion passed pass regex - Assertion passed Refunds take five working days. -> PASS pass icontains - Assertion passed pass equals - Assertion passed pass contains-all - Assertion passed I could not find that in the handbook. -> FAIL FAIL contains - Expected output to contain "shipped" pass contains-any - Assertion passed
The deterministic assertion types
| Type | Passes when the output |
|---|---|
contains / icontains | Holds the value as a substring (icontains ignores case) |
equals | Equals the value exactly |
starts-with | Begins with the value |
regex | Matches the regular expression |
contains-all | Holds every string in the list |
contains-any | Holds at least one string in the list |
Where deterministic checks fit
- A fact the answer must carry, such as an order id or a date, caught by
containsorregex. - A fixed reply you can pin exactly, where
equalsproves the wording did not drift. - A keyword set, where
contains-allasks for every term andcontains-anyfor one of them.
contains is a plain substring test, not a word test, so contains: "cat" passes on escalate. Reach for regex with a word boundary when a loose substring would match more than you meant.Related
- Previous: Reusing setup across tests with scenarios
- Next: is-json: checking the shape of an answer
- Reference: Promptfoo deterministic assertions
- Add an
icontainscheck forcourieron the first answer and confirm it passes whatever the case. - Change the
equalsvalue by one word and watch that test fail while the substring checks still pass. - Swap the regex for
\bcat\bon an answer that holds escalate and see the word boundary refuse the match.
Little by little, you're building something great.