threshold, weight and named metrics
Scores turn a set of checks into one number: weight decides how much each check counts, threshold sets the bar a test must clear, and metric groups checks into named scores.
Last updated: 15 Oct, 2026 · Promptfoo 0.124.1
A test can pass or fail on all-or-nothing, but often some checks matter more than others and a little slack is fine. Promptfoo combines a test's assertions into a weighted score, lets you set the bar that score has to clear, and can tag checks so related ones roll up under a name you choose.
Weights, a threshold and two metrics
This test has three checks. Two are tagged Accuracy and one Format, and the accuracy content check carries twice the weight. One accuracy check looks for a word the answer does not have, so it fails. The test threshold is 0.6.
providers:
- file://support_bot.py
prompts:
- "{{question}}"
tests:
- vars: { question: "Where is order A17?" }
threshold: 0.6
assert:
- type: contains
value: "3 March"
weight: 2
metric: Accuracy
- type: starts-with
value: "Order"
weight: 1
metric: Format
- type: contains
value: "tomorrow"
weight: 1
metric: Accuracy
View the code here
# read out.json: each answer, its checks, and any metric scores
import json
for r in json.load(open("out.json"))["results"]["results"]:
print(r["response"]["output"], "->", "PASS" if r["success"] else "FAIL")
for c in r["gradingResult"]["componentResults"]:
print(" ", "pass" if c["pass"] else "FAIL", c["assertion"]["type"], "-", c["reason"])
if r["namedScores"]:
scores = {k: round(v, 3) for k, v in r["namedScores"].items()}
print(" metrics", scores, "| test score", round(r["score"], 3))
# the shop support bot: find the answer in the handbook, or say it is missing
HANDBOOK = {
"a17": "Order A17 shipped on 3 March by courier.",
"refund": "Refunds take five working days.",
}
def call_api(prompt, options, context):
asked = prompt.lower() # the rendered prompt is the question
for key, line in HANDBOOK.items():
if key in asked:
return {"output": line}
return {"output": "I could not find that in the handbook."}
The weighted score is the passing weight over the total weight. Two of the four weight units pass on accuracy and the format check passes, giving 0.75, above the 0.6 bar, so the test passes even with one check failing. The reader prints the per-metric scores and the test score.
promptfoo eval -c scores.yaml -o out.json --no-table >/dev/null 2>&1
python3 show.pyOrder A17 shipped on 3 March by courier. -> PASS
pass contains - Assertion passed
pass starts-with - Assertion passed
FAIL contains - Expected output to contain "tomorrow"
metrics {'Accuracy': 0.667, 'Format': 1} | test score 0.75Raising the bar fails the same answer
Nothing about the answer changed. Change scores.yaml to raise the threshold to 0.8 and the same 0.75 now sits below the bar, so the test fails. The score is the same; the bar is what moved.
threshold: 0.8promptfoo eval -c scores.yaml -o out.json --no-table >/dev/null 2>&1
python3 show.pyOrder A17 shipped on 3 March by courier. -> FAIL
pass contains - Assertion passed
pass starts-with - Assertion passed
FAIL contains - Expected output to contain "tomorrow"
metrics {'Accuracy': 0.667, 'Format': 1} | test score 0.75What each key does
| Key | Effect |
|---|---|
weight | How much a check counts in the test's weighted score (default 1) |
threshold | The score a test must reach to pass |
metric | A name that groups checks into a reported score |
Where scores and metrics fit
- A test with a must-have and a nice-to-have, weighted so the must-have carries the verdict.
- A pass bar below 100%, where a threshold lets a strong-but-imperfect answer pass.
- A report by theme, where
metricrolls accuracy and format checks into separate named scores.
Related
- Previous: Transforms: reshaping output before a check
- Next: llm-rubric: the check that needs a model
- Reference: Promptfoo weighted assertions and metrics
- Drop the failing accuracy check and watch the score rise to 1.0 and both metrics reach 1.0.
- Give the format check a weight of 3 and see the test score shift toward format.
- Set the threshold to exactly 0.75 and confirm the test passes on the bar.
Little by little, you're building something great.