threshold, weight and named metrics
Every test so far has needed every check to pass. Real suites are not like that: some checks matter more than others, and a run is worth grading rather than failing.
Three settings turn a list of checks into a score. weight says how much an assertion counts, threshold says what score the test needs, and metric gives an assertion a name so scores can be added up across the suite.
tests:
- vars:
question: Where is order A17?
assert:
- type: contains
value: 3 March
metric: Correct
weight: 3
- type: contains
value: please
metric: Polite
weight: 1
threshold: 0.7Being correct is worth three and being polite is worth one. The bot's answer contains the date but not the word please, so it scores three out of four, which is 0.75 and clears the threshold of 0.7.
promptfoo evalThe row passes even though one of its two checks did not. That is the difference between an eval and a unit test, expressed in one number.
Named metrics
The metric field is the part that pays off as a suite grows. Every assertion carrying the same name is pooled, so after forty tests you do not have one pass rate, you have a score for correctness and a score for politeness, and you can see which one moved.
Promptfoo prints them per provider in the viewer, and writes them into the output file from lesson 19, which is where a dashboard or a spreadsheet picks them up.
Where thresholds live
A threshold on a test governs that test. The same key under defaultTest governs every test in the file, which is the usual way to write it once.
defaultTest:
threshold: 0.7- Raise the threshold to 0.8 and watch the same answer start failing.
- Change the weights to 1 and 1, and work out the new score before running.
- Give both assertions the same metric name and see them pooled.
Little by little, you're building something great.