Promptfoopromptfoo 0.124.1 · Node 24 · Python 3
Dashboard
0%

When the judge is wrong

A stand-in judge is a fallible program: it agrees when its rule agrees, so it rewards an answer that reuses the rubric's words even when that answer states the opposite of the truth.

Last updated: 15 Oct, 2026 · Promptfoo 0.124.1

The factuality and g-eval: two more formats lesson passed grounded answers through the grader. A grader earns trust by disagreeing in the right places. This lesson watches it reject a bad answer correctly, then watches it accept one it should have caught.

The judge disagreeing: grounded against a miss

One suite, two answers. The first is the grounded line; the second is the support bot's reply when it finds nothing, I could not find that in the handbook. The three checks are set once under defaultTest so both answers face the same rubric, reference and criterion.

yaml
providers:
  - echo
prompts:
  - "{{answer}}"
defaultTest:
  options:
    provider: file://pretend_promptfoo.py
  assert:                       # every test is checked the same three ways
    - type: llm-rubric
      value: the answer says the order shipped in March
    - type: factuality
      value: Order A17 shipped on 3 March by courier.
    - type: g-eval
      value: the answer mentions a courier
tests:
  - vars:
      answer: Order A17 shipped on 3 March by courier.
  - vars:
      answer: I could not find that in the handbook.
Project files used on this pageThis lesson builds on a project from earlier lessons. The code below imports this file. Click a file to see its code, or follow the link to the lesson that wrote it. To run the code yourself, keep it in the same folder.
View the code here
pretend_promptfoo.py
"""A grader for promptfoo, written in Python.

Every model-graded assertion sends a prompt to a provider and reads the reply.
This file is that provider. It never calls a model. It decides with one rule:

    a claim is supported when every long word in it also appears in the text
    it is checked against.

Each assertion type asks for a different answer, so `grade` looks at the prompt
it was given and answers in the format that assertion expects.
"""
import json
import re

WORD = re.compile(r"[a-z0-9]+")
# A word is cut to its first four letters, so `shipped` and `shipping` count
# as the same word. These stems tell the grader what to do rather than what to
# look for, so they are dropped before anything is compared.
INSTRUCTIONS = {"repl", "answ", "outp", "resp", "says", "said", "ment", "cont",
                "incl", "shou", "must", "text", "that", "this", "when", "wher",
                "with", "from", "give", "tell", "show", "stat", "name", "user"}


def words(text):
    """The word stems worth comparing."""
    found = {w[:4] for w in WORD.findall(text.lower()) if len(w) > 3}
    return found - INSTRUCTIONS or found


def share(claim, source):
    """How much of the claim the source covers, 0 to 1."""
    wanted = words(claim)
    return len(wanted & words(source)) / len(wanted) if wanted else 0.0

def tag(text, name):
    """The text inside the LAST <name>...</name>, or an empty string."""
    found = re.findall(r"<%s>\s*(.*?)\s*</%s>" % (name, name), text, re.S)
    return found[-1] if found else ""


def flatten(prompt):
    """Chat graders send a JSON list of messages. Join them into one string."""
    try:
        loaded = json.loads(prompt)
    except ValueError:
        return prompt
    if isinstance(loaded, list):
        return "\n".join(str(m.get("content", "")) for m in loaded)
    return prompt

def after(text, marker):
    """Everything past the last marker, ignoring case.

    Every grader prompt opens with worked examples that use the same markers,
    so the real input is always the last one.
    """
    parts = re.split(re.escape(marker), text, flags=re.I)
    return parts[-1].strip() if len(parts) > 1 else ""


def sentences(text):
    return [s.strip() for s in re.split(r"(?<=[.!?])\s+", text.strip()) if s.strip()]


def grade(prompt):
    text = flatten(prompt)

    # factuality: one letter for how the answer relates to the reference.
    if "factuality evaluator" in text:
        expert, submitted = tag(text, "expert_answer"), tag(text, "submitted_answer")
        if share(expert, submitted) < 1.0:
            return json.dumps({"category": "D", "reason": "the answer misses part of it"})
        return json.dumps({"category": "C", "reason": "the reference is covered"})

    # g-eval asks twice: first for the steps, then for a score out of ten.
    if "EVALUATION CRITERIA" in text and '"steps"' in text:
        criteria = after(text.split("**OUTPUT FORMAT**")[0], "**EVALUATION CRITERIA**")
        return json.dumps({"steps": ["Check that the reply satisfies: " + criteria]})
    if "Evaluation Steps" in text and '"score"' in text:
        criteria = text.split("**Evaluation Criteria**")[1].split("**")[0]
        reply = text.split("**Reply**")[1].split("**OUTPUT FORMAT**")[0]
        hit = share(criteria, reply)
        return json.dumps({"score": round(hit * 10), "reason": "word overlap %.2f" % hit})

    # context-faithfulness asks twice: split the answer up, then judge each part.
    if "create one or more statements" in text:
        return "\n".join(sentences(after(text, "answer:").split("statements:")[0]))
    if "Natural language inference" in text:
        block = after(text, "context:")
        source = re.split("statements:", block, flags=re.I)[0]
        lines, final = [], []
        for i, claim in enumerate(sentences(after(block, "statements:").split("Answer:")[0]), 1):
            ok = share(claim, source) == 1.0
            lines.append("%d. %s Verdict: %s." % (i, claim, "Yes" if ok else "No"))
            final.append("Yes." if ok else "No.")
        return "\n".join(lines) + "\nFinal verdict for each statement in order: " + " ".join(final)

    # context-recall: is each part of the right answer in the documents?
    if "can be attributed to the given context" in text:
        block = after(text, "context:")
        source = re.split("answer:", block, flags=re.I)[0]
        verdicts = []
        for i, claim in enumerate(sentences(after(block, "answer:").split("classification")[0]), 1):
            ok = share(claim, source) == 1.0
            verdicts.append("%d. %s So [%s]" % (i, claim, "Attributed" if ok else "Not Attributed"))
        return "\n".join(verdicts)

    # llm-rubric, and everything built on it: pass, score and a reason.
    output, rubric = tag(text, "Output"), tag(text, "Rubric")
    hit = share(rubric, output)
    return json.dumps({"reason": "word overlap %.2f" % hit,
                       "pass": hit >= 0.5, "score": hit})


def call_api(prompt, options, context):
    return {"output": grade(prompt)}

Writing the run to a file and reading it back shows each answer with how every check scored it.

Examplethe same three checks pass a grounded answer and fail a miss
promptfoo eval -o out.json >/dev/null 2>&1

# read each answer back and how every check scored it
python3 - <<'PY'
import json
d = json.load(open("out.json"))

for r in d["results"]["results"]:
    print(r["vars"]["answer"])
    for c in r["gradingResult"]["componentResults"]:
        print("  %-11s pass=%s" % (c["assertion"]["type"], c["pass"]))
PY

Why the miss failed

  • The miss shared none of the rubric's words. I could not find that in the handbook has no order, shipped or March, so the overlap was zero and llm-rubric failed.
  • factuality saw the reference uncovered and answered the miss category, a fail.
  • g-eval scored zero, because none of the criterion's words were in the reply.

The run as a whole exits non-zero because a test failed; what that exit code means for a build is the subject of Failing a build: exit codes and GitHub Actions.

A contradiction the lexical judge lets through

The grader decides on shared words, not on meaning, and a contradicted answer can share the very words the rubric names. Take Order A17 was cancelled and never shipped. against the rubric the answer says the order shipped in March. It carries order and a shipped stem, misses only March, and an overlap of two words out of three clears the 0.5 bar.

yaml
providers:
  - echo
prompts:
  - "{{answer}}"
defaultTest:
  options:
    provider: file://pretend_promptfoo.py
tests:
  - vars:
      answer: Order A17 was cancelled and never shipped.
    assert:
      - type: llm-rubric
        value: the answer says the order shipped in March
Examplea contradicted answer passes llm-rubric on shared words
promptfoo eval -o out.json >/dev/null 2>&1

# a contradicted answer that still reuses the rubric words
python3 - <<'PY'
import json
r = json.load(open("out.json"))["results"]["results"][0]
c = r["gradingResult"]["componentResults"][0]
print("answer :", r["vars"]["answer"])
print("llm-rubric pass=%s  (%s)" % (c["pass"], c["reason"]))
PY

Picking examples a lexical judge can catch

  • Choose contradicted answers that avoid the rubric's key words. The handbook miss failed cleanly because it shared nothing; the cancelled-order line passed because it shared almost everything.
  • Read the reason, not only the verdict. A pass at an overlap of 0.67 is a warning the grade rode on words, not on sense.
  • Treat the stand-in as a floor. It is honest about words and blind to meaning, so it catches gross misses and lets a clever contradiction by.
Watch out. A word-overlap grader is for learning the mechanics and for running a suite with no key. When a verdict turns on meaning rather than wording, move the grade to a model, which Moving the suite to a hosted model covers. The wiring is identical; only the provider behind defaultTest.options.provider changes.

Where a fallible judge still helps

  • A regression floor, catching an answer that drops the facts the rubric names.
  • A fast, keyless gate, run on every commit before a slower model grade runs nightly.
  • A teaching harness, where a deterministic grader makes the pass and fail predictable while you learn the formats.
Try it yourself
  • Write a contradicted answer that avoids order and shipped and confirm it fails as it should.
  • Lower the pass bar in share from 0.5 and see which borderline answers flip.
  • Add the cancelled-order line as a test and assert it should fail, documenting the known blind spot.

Every expert started right here.