Promptfoopromptfoo 0.124.1 · Node 24 · Python 3
Dashboard
0%

threshold, weight and named metrics

Scores turn a set of checks into one number: weight decides how much each check counts, threshold sets the bar a test must clear, and metric groups checks into named scores.

Last updated: 15 Oct, 2026 · Promptfoo 0.124.1

A test can pass or fail on all-or-nothing, but often some checks matter more than others and a little slack is fine. Promptfoo combines a test's assertions into a weighted score, lets you set the bar that score has to clear, and can tag checks so related ones roll up under a name you choose.

Weights, a threshold and two metrics

This test has three checks. Two are tagged Accuracy and one Format, and the accuracy content check carries twice the weight. One accuracy check looks for a word the answer does not have, so it fails. The test threshold is 0.6.

yaml
providers:
  - file://support_bot.py
prompts:
  - "{{question}}"
tests:
  - vars: { question: "Where is order A17?" }
    threshold: 0.6
    assert:
      - type: contains
        value: "3 March"
        weight: 2
        metric: Accuracy
      - type: starts-with
        value: "Order"
        weight: 1
        metric: Format
      - type: contains
        value: "tomorrow"
        weight: 1
        metric: Accuracy
Project files used on this pageThis lesson builds on a project from earlier lessons. The code below imports these files. Click a file to see its code, or follow the link to the lesson that wrote it. To run the code yourself, keep them in the same folder.
View the code here
show.py
# read out.json: each answer, its checks, and any metric scores
import json

for r in json.load(open("out.json"))["results"]["results"]:
    print(r["response"]["output"], "->", "PASS" if r["success"] else "FAIL")

    for c in r["gradingResult"]["componentResults"]:
        print("  ", "pass" if c["pass"] else "FAIL", c["assertion"]["type"], "-", c["reason"])

    if r["namedScores"]:
        scores = {k: round(v, 3) for k, v in r["namedScores"].items()}
        print("   metrics", scores, "| test score", round(r["score"], 3))
support_bot.py
# the shop support bot: find the answer in the handbook, or say it is missing
HANDBOOK = {
    "a17": "Order A17 shipped on 3 March by courier.",
    "refund": "Refunds take five working days.",
}

def call_api(prompt, options, context):
    asked = prompt.lower()                      # the rendered prompt is the question
    for key, line in HANDBOOK.items():
        if key in asked:
            return {"output": line}
    return {"output": "I could not find that in the handbook."}

The weighted score is the passing weight over the total weight. Two of the four weight units pass on accuracy and the format check passes, giving 0.75, above the 0.6 bar, so the test passes even with one check failing. The reader prints the per-metric scores and the test score.

ExampleA weighted score of 0.75 clears the 0.6 bar
promptfoo eval -c scores.yaml -o out.json --no-table >/dev/null 2>&1
python3 show.py

Raising the bar fails the same answer

Nothing about the answer changed. Change scores.yaml to raise the threshold to 0.8 and the same 0.75 now sits below the bar, so the test fails. The score is the same; the bar is what moved.

yaml
    threshold: 0.8
ExampleThe same 0.75 now falls short of 0.8
promptfoo eval -c scores.yaml -o out.json --no-table >/dev/null 2>&1
python3 show.py

What each key does

KeyEffect
weightHow much a check counts in the test's weighted score (default 1)
thresholdThe score a test must reach to pass
metricA name that groups checks into a reported score

Where scores and metrics fit

  • A test with a must-have and a nice-to-have, weighted so the must-have carries the verdict.
  • A pass bar below 100%, where a threshold lets a strong-but-imperfect answer pass.
  • A report by theme, where metric rolls accuracy and format checks into separate named scores.
Watch out. A threshold is a floor on a weighted average, so one failing check can still pass a test when the rest outweigh it. If a check must never fail, keep it on its own test or give it the whole weight, rather than leaning on a threshold.
Try it yourself
  • Drop the failing accuracy check and watch the score rise to 1.0 and both metrics reach 1.0.
  • Give the format check a weight of 3 and see the test score shift toward format.
  • Set the threshold to exactly 0.75 and confirm the test passes on the bar.

Little by little, you're building something great.