When the judge is wrong
A stand-in judge is a fallible program: it agrees when its rule agrees, so it rewards an answer that reuses the rubric's words even when that answer states the opposite of the truth.
Last updated: 15 Oct, 2026 · Promptfoo 0.124.1
The factuality and g-eval: two more formats lesson passed grounded answers through the grader. A grader earns trust by disagreeing in the right places. This lesson watches it reject a bad answer correctly, then watches it accept one it should have caught.
The judge disagreeing: grounded against a miss
One suite, two answers. The first is the grounded line; the second is the support bot's reply when it finds nothing, I could not find that in the handbook. The three checks are set once under defaultTest so both answers face the same rubric, reference and criterion.
providers:
- echo
prompts:
- "{{answer}}"
defaultTest:
options:
provider: file://pretend_promptfoo.py
assert: # every test is checked the same three ways
- type: llm-rubric
value: the answer says the order shipped in March
- type: factuality
value: Order A17 shipped on 3 March by courier.
- type: g-eval
value: the answer mentions a courier
tests:
- vars:
answer: Order A17 shipped on 3 March by courier.
- vars:
answer: I could not find that in the handbook.
View the code here
"""A grader for promptfoo, written in Python.
Every model-graded assertion sends a prompt to a provider and reads the reply.
This file is that provider. It never calls a model. It decides with one rule:
a claim is supported when every long word in it also appears in the text
it is checked against.
Each assertion type asks for a different answer, so `grade` looks at the prompt
it was given and answers in the format that assertion expects.
"""
import json
import re
WORD = re.compile(r"[a-z0-9]+")
# A word is cut to its first four letters, so `shipped` and `shipping` count
# as the same word. These stems tell the grader what to do rather than what to
# look for, so they are dropped before anything is compared.
INSTRUCTIONS = {"repl", "answ", "outp", "resp", "says", "said", "ment", "cont",
"incl", "shou", "must", "text", "that", "this", "when", "wher",
"with", "from", "give", "tell", "show", "stat", "name", "user"}
def words(text):
"""The word stems worth comparing."""
found = {w[:4] for w in WORD.findall(text.lower()) if len(w) > 3}
return found - INSTRUCTIONS or found
def share(claim, source):
"""How much of the claim the source covers, 0 to 1."""
wanted = words(claim)
return len(wanted & words(source)) / len(wanted) if wanted else 0.0
def tag(text, name):
"""The text inside the LAST <name>...</name>, or an empty string."""
found = re.findall(r"<%s>\s*(.*?)\s*</%s>" % (name, name), text, re.S)
return found[-1] if found else ""
def flatten(prompt):
"""Chat graders send a JSON list of messages. Join them into one string."""
try:
loaded = json.loads(prompt)
except ValueError:
return prompt
if isinstance(loaded, list):
return "\n".join(str(m.get("content", "")) for m in loaded)
return prompt
def after(text, marker):
"""Everything past the last marker, ignoring case.
Every grader prompt opens with worked examples that use the same markers,
so the real input is always the last one.
"""
parts = re.split(re.escape(marker), text, flags=re.I)
return parts[-1].strip() if len(parts) > 1 else ""
def sentences(text):
return [s.strip() for s in re.split(r"(?<=[.!?])\s+", text.strip()) if s.strip()]
def grade(prompt):
text = flatten(prompt)
# factuality: one letter for how the answer relates to the reference.
if "factuality evaluator" in text:
expert, submitted = tag(text, "expert_answer"), tag(text, "submitted_answer")
if share(expert, submitted) < 1.0:
return json.dumps({"category": "D", "reason": "the answer misses part of it"})
return json.dumps({"category": "C", "reason": "the reference is covered"})
# g-eval asks twice: first for the steps, then for a score out of ten.
if "EVALUATION CRITERIA" in text and '"steps"' in text:
criteria = after(text.split("**OUTPUT FORMAT**")[0], "**EVALUATION CRITERIA**")
return json.dumps({"steps": ["Check that the reply satisfies: " + criteria]})
if "Evaluation Steps" in text and '"score"' in text:
criteria = text.split("**Evaluation Criteria**")[1].split("**")[0]
reply = text.split("**Reply**")[1].split("**OUTPUT FORMAT**")[0]
hit = share(criteria, reply)
return json.dumps({"score": round(hit * 10), "reason": "word overlap %.2f" % hit})
# context-faithfulness asks twice: split the answer up, then judge each part.
if "create one or more statements" in text:
return "\n".join(sentences(after(text, "answer:").split("statements:")[0]))
if "Natural language inference" in text:
block = after(text, "context:")
source = re.split("statements:", block, flags=re.I)[0]
lines, final = [], []
for i, claim in enumerate(sentences(after(block, "statements:").split("Answer:")[0]), 1):
ok = share(claim, source) == 1.0
lines.append("%d. %s Verdict: %s." % (i, claim, "Yes" if ok else "No"))
final.append("Yes." if ok else "No.")
return "\n".join(lines) + "\nFinal verdict for each statement in order: " + " ".join(final)
# context-recall: is each part of the right answer in the documents?
if "can be attributed to the given context" in text:
block = after(text, "context:")
source = re.split("answer:", block, flags=re.I)[0]
verdicts = []
for i, claim in enumerate(sentences(after(block, "answer:").split("classification")[0]), 1):
ok = share(claim, source) == 1.0
verdicts.append("%d. %s So [%s]" % (i, claim, "Attributed" if ok else "Not Attributed"))
return "\n".join(verdicts)
# llm-rubric, and everything built on it: pass, score and a reason.
output, rubric = tag(text, "Output"), tag(text, "Rubric")
hit = share(rubric, output)
return json.dumps({"reason": "word overlap %.2f" % hit,
"pass": hit >= 0.5, "score": hit})
def call_api(prompt, options, context):
return {"output": grade(prompt)}
Writing the run to a file and reading it back shows each answer with how every check scored it.
promptfoo eval -o out.json >/dev/null 2>&1
# read each answer back and how every check scored it
python3 - <<'PY'
import json
d = json.load(open("out.json"))
for r in d["results"]["results"]:
print(r["vars"]["answer"])
for c in r["gradingResult"]["componentResults"]:
print(" %-11s pass=%s" % (c["assertion"]["type"], c["pass"]))
PYOrder A17 shipped on 3 March by courier. llm-rubric pass=True factuality pass=True g-eval pass=True I could not find that in the handbook. llm-rubric pass=False factuality pass=False g-eval pass=False
Why the miss failed
- The miss shared none of the rubric's words. I could not find that in the handbook has no
order,shippedorMarch, so the overlap was zero andllm-rubricfailed. - factuality saw the reference uncovered and answered the miss category, a fail.
- g-eval scored zero, because none of the criterion's words were in the reply.
The run as a whole exits non-zero because a test failed; what that exit code means for a build is the subject of Failing a build: exit codes and GitHub Actions.
A contradiction the lexical judge lets through
The grader decides on shared words, not on meaning, and a contradicted answer can share the very words the rubric names. Take Order A17 was cancelled and never shipped. against the rubric the answer says the order shipped in March. It carries order and a shipped stem, misses only March, and an overlap of two words out of three clears the 0.5 bar.
providers:
- echo
prompts:
- "{{answer}}"
defaultTest:
options:
provider: file://pretend_promptfoo.py
tests:
- vars:
answer: Order A17 was cancelled and never shipped.
assert:
- type: llm-rubric
value: the answer says the order shipped in March
promptfoo eval -o out.json >/dev/null 2>&1
# a contradicted answer that still reuses the rubric words
python3 - <<'PY'
import json
r = json.load(open("out.json"))["results"]["results"][0]
c = r["gradingResult"]["componentResults"][0]
print("answer :", r["vars"]["answer"])
print("llm-rubric pass=%s (%s)" % (c["pass"], c["reason"]))
PYanswer : Order A17 was cancelled and never shipped. llm-rubric pass=True (word overlap 0.67)
Picking examples a lexical judge can catch
- Choose contradicted answers that avoid the rubric's key words. The handbook miss failed cleanly because it shared nothing; the cancelled-order line passed because it shared almost everything.
- Read the reason, not only the verdict. A pass at an overlap of 0.67 is a warning the grade rode on words, not on sense.
- Treat the stand-in as a floor. It is honest about words and blind to meaning, so it catches gross misses and lets a clever contradiction by.
defaultTest.options.provider changes.Where a fallible judge still helps
- A regression floor, catching an answer that drops the facts the rubric names.
- A fast, keyless gate, run on every commit before a slower model grade runs nightly.
- A teaching harness, where a deterministic grader makes the pass and fail predictable while you learn the formats.
Related
- Previous: factuality and g-eval: two more formats
- Next: Context assertions: faithfulness and recall
- See also: Moving the suite to a hosted model
- Reference: Promptfoo documentation
- Write a contradicted answer that avoids
orderandshippedand confirm it fails as it should. - Lower the pass bar in
sharefrom 0.5 and see which borderline answers flip. - Add the cancelled-order line as a test and assert it should fail, documenting the known blind spot.
Every expert started right here.