Promptfoopromptfoo 0.124.1 · Node 24 · Python 3
Dashboard
0%

A grading script as the stand-in judge

A stand-in judge is a Python file wired in as the grading provider, so llm-rubric runs through your own code and needs no model key.

Last updated: 15 Oct, 2026 · Promptfoo 0.124.1

The llm-rubric: the check that needs a model lesson left the check erroring for a key. Promptfoo lets you replace the grading provider with any provider, a Python file included. This lesson builds that file, pretend_promptfoo.py, a grader that decides by reading words instead of by calling a model.

What the judge has to answer

Every model-graded assertion sends the grader a prompt and reads its reply. For llm-rubric, the prompt wraps the answer in <Output> tags and the rubric in <Rubric> tags, and the reply is a JSON object with pass, score and reason. The grader's one rule: a rubric holds when its words appear in the output.

Reading the words that matter

words cuts each term to a four-letter stem, so shipped and shipping count as one, and drops a set of instruction words like answer and says that describe the task rather than the facts. share is the fraction of a claim's words the source also carries.

python
WORD = re.compile(r"[a-z0-9]+")

def words(text):
    # keep 4-letter stems, drop instruction words like "answer" and "says"
    found = {w[:4] for w in WORD.findall(text.lower()) if len(w) > 3}
    return found - INSTRUCTIONS or found

def share(claim, source):
    # the fraction of the claim's words the source also carries, 0 to 1
    wanted = words(claim)
    return len(wanted & words(source)) / len(wanted) if wanted else 0.0

Finding the answer and the rubric in the prompt

tag pulls the text out of a pair of tags, and it takes the last match deliberately. The real llm-rubric prompt opens with two worked examples, a greeting and a pirate line, wrapped in the same <Output> and <Rubric> tags as the input. A grader that took the first match would grade one of those examples instead of the answer. The last match is always the real one. flatten joins the chat message list the grader is handed into a single string to search.

python
def tag(text, name):
    # the text inside the LAST <name>...</name>, or ""
    found = re.findall(r"<%s>\s*(.*?)\s*</%s>" % (name, name), text, re.S)
    return found[-1] if found else ""

def flatten(prompt):
    # a chat grader sends a JSON list of messages; join them into one string
    try:
        loaded = json.loads(prompt)
    except ValueError:
        return prompt
    if isinstance(loaded, list):
        return "\n".join(str(m.get("content", "")) for m in loaded)
    return prompt

Dispatching to the right format

grade flattens the prompt, lets the format-specific handlers catch their own assertions, and falls through to llm-rubric: compare the rubric to the output, pass at an overlap of 0.5 or more. call_api is the entry point promptfoo calls for every graded assertion.

python
def grade(prompt):
    text = flatten(prompt)
    # the handlers above catch factuality, g-eval and context by a phrase;
    # llm-rubric is the fallback: compare the rubric to the output
    output, rubric = tag(text, "Output"), tag(text, "Rubric")
    hit = share(rubric, output)
    return json.dumps({"reason": "word overlap %.2f" % hit,
                       "pass": hit >= 0.5, "score": hit})

def call_api(prompt, options, context):
    # promptfoo calls this for every model-graded assertion
    return {"output": grade(prompt)}

The whole grader

Here is the file in full. The handlers for factuality, g-eval and the context checks sit between the dispatch and the llm-rubric fallback; the next lessons use them. Save it next to the config as pretend_promptfoo.py.

python
"""A grader for promptfoo, written in Python.

Every model-graded assertion sends a prompt to a provider and reads the reply.
This file is that provider. It never calls a model. It decides with one rule:

    a claim is supported when every long word in it also appears in the text
    it is checked against.

Each assertion type asks for a different answer, so `grade` looks at the prompt
it was given and answers in the format that assertion expects.
"""
import json
import re

WORD = re.compile(r"[a-z0-9]+")
# A word is cut to its first four letters, so `shipped` and `shipping` count
# as the same word. These stems tell the grader what to do rather than what to
# look for, so they are dropped before anything is compared.
INSTRUCTIONS = {"repl", "answ", "outp", "resp", "says", "said", "ment", "cont",
                "incl", "shou", "must", "text", "that", "this", "when", "wher",
                "with", "from", "give", "tell", "show", "stat", "name", "user"}


def words(text):
    """The word stems worth comparing."""
    found = {w[:4] for w in WORD.findall(text.lower()) if len(w) > 3}
    return found - INSTRUCTIONS or found


def share(claim, source):
    """How much of the claim the source covers, 0 to 1."""
    wanted = words(claim)
    return len(wanted & words(source)) / len(wanted) if wanted else 0.0

def tag(text, name):
    """The text inside the LAST <name>...</name>, or an empty string."""
    found = re.findall(r"<%s>\s*(.*?)\s*</%s>" % (name, name), text, re.S)
    return found[-1] if found else ""


def flatten(prompt):
    """Chat graders send a JSON list of messages. Join them into one string."""
    try:
        loaded = json.loads(prompt)
    except ValueError:
        return prompt
    if isinstance(loaded, list):
        return "\n".join(str(m.get("content", "")) for m in loaded)
    return prompt

def after(text, marker):
    """Everything past the last marker, ignoring case.

    Every grader prompt opens with worked examples that use the same markers,
    so the real input is always the last one.
    """
    parts = re.split(re.escape(marker), text, flags=re.I)
    return parts[-1].strip() if len(parts) > 1 else ""


def sentences(text):
    return [s.strip() for s in re.split(r"(?<=[.!?])\s+", text.strip()) if s.strip()]


def grade(prompt):
    text = flatten(prompt)

    # factuality: one letter for how the answer relates to the reference.
    if "factuality evaluator" in text:
        expert, submitted = tag(text, "expert_answer"), tag(text, "submitted_answer")
        if share(expert, submitted) < 1.0:
            return json.dumps({"category": "D", "reason": "the answer misses part of it"})
        return json.dumps({"category": "C", "reason": "the reference is covered"})

    # g-eval asks twice: first for the steps, then for a score out of ten.
    if "EVALUATION CRITERIA" in text and '"steps"' in text:
        criteria = after(text.split("**OUTPUT FORMAT**")[0], "**EVALUATION CRITERIA**")
        return json.dumps({"steps": ["Check that the reply satisfies: " + criteria]})
    if "Evaluation Steps" in text and '"score"' in text:
        criteria = text.split("**Evaluation Criteria**")[1].split("**")[0]
        reply = text.split("**Reply**")[1].split("**OUTPUT FORMAT**")[0]
        hit = share(criteria, reply)
        return json.dumps({"score": round(hit * 10), "reason": "word overlap %.2f" % hit})

    # context-faithfulness asks twice: split the answer up, then judge each part.
    if "create one or more statements" in text:
        return "\n".join(sentences(after(text, "answer:").split("statements:")[0]))
    if "Natural language inference" in text:
        block = after(text, "context:")
        source = re.split("statements:", block, flags=re.I)[0]
        lines, final = [], []
        for i, claim in enumerate(sentences(after(block, "statements:").split("Answer:")[0]), 1):
            ok = share(claim, source) == 1.0
            lines.append("%d. %s Verdict: %s." % (i, claim, "Yes" if ok else "No"))
            final.append("Yes." if ok else "No.")
        return "\n".join(lines) + "\nFinal verdict for each statement in order: " + " ".join(final)

    # context-recall: is each part of the right answer in the documents?
    if "can be attributed to the given context" in text:
        block = after(text, "context:")
        source = re.split("answer:", block, flags=re.I)[0]
        verdicts = []
        for i, claim in enumerate(sentences(after(block, "answer:").split("classification")[0]), 1):
            ok = share(claim, source) == 1.0
            verdicts.append("%d. %s So [%s]" % (i, claim, "Attributed" if ok else "Not Attributed"))
        return "\n".join(verdicts)

    # llm-rubric, and everything built on it: pass, score and a reason.
    output, rubric = tag(text, "Output"), tag(text, "Rubric")
    hit = share(rubric, output)
    return json.dumps({"reason": "word overlap %.2f" % hit,
                       "pass": hit >= 0.5, "score": hit})


def call_api(prompt, options, context):
    return {"output": grade(prompt)}

Wiring it as the grader

defaultTest.options.provider sets the grading provider for every test, pointing it at the file. The provider under test stays echo, and the prompt is {{answer}}, so echo returns the answer itself for the grader to judge. Each suite in this part is its own promptfooconfig.yaml, changed as the checks change, with the grader file saved beside it.

yaml
providers:
  - echo
prompts:
  - "{{answer}}"
defaultTest:
  options:
    provider: file://pretend_promptfoo.py
tests:
  - vars:
      answer: Order A17 shipped on 3 March by courier.
    assert:
      - type: llm-rubric
        value: the answer says the order shipped in March

Grading a grounded answer with no key

With both files in the folder, the run grades llm-rubric through the Python file. No key is read, and the grounded answer carries the rubric's words, so it passes.

Examplellm-rubric passes with no key, judged by your own file
promptfoo eval --no-table

What the run shows

  • The test passed without a model key. The grading provider was the Python file, not a vendor chosen from the environment.
  • The grader returned a real score. Every word of the rubric appeared in the answer, so the overlap was 1.00 and the verdict was a pass.
  • The provider under test did no work. echo handed its prompt straight back, which let the answer be the thing graded.
Watch out. If tag took the first match instead of the last, it would read the greeting or pirate example baked into the grading prompt and grade that, not your answer. The last match is the one that matters; a greedy first match is a quiet, wrong pass.

Where you use a stand-in judge

  • A suite that must run with no key, in a local checkout or a CI job that holds no credentials.
  • A deterministic grade, where the same answer must get the same verdict every run.
  • A placeholder grader, standing in while a real model grader is wired up later.
Try it yourself
  • Change the answer to a line that misses the word March and watch the overlap and the verdict drop.
  • Add a print to call_api that writes the prompt to a file, and read the tags promptfoo sent.
  • Point the grader at a per-assertion provider on one assert instead of defaultTest.

Slow is fine. Stopping is the only problem.