Writing the stand-in judge

The stub answered without reading the question. To do better, the first thing to find out is what promptfoo actually sends a grader.

Look at what it asks

A provider can do anything before it answers, including write down what it was given. This one keeps the prompt and then agrees, exactly like the stub.

import json
import os


def call_api(prompt, options, context):
    """Write down what promptfoo asked, then agree with it."""
    with open(os.path.join(os.path.dirname(__file__), "asked.txt"), "a") as f:
        f.write(prompt + "\n")
    return {"output": json.dumps({"pass": True, "score": 1, "reason": "looks fine"})}

promptfoo eval > /dev/null; cat asked.txt

That is the whole grading prompt, and it is worth reading twice. It arrives as a JSON list of chat messages. The system message explains the job and shows two worked examples. The last message carries the real work, in two tags: <Output> holding the answer and <Rubric> holding your sentence.

It also tells you the exact format expected back: a JSON object with reason, pass and score. Nothing here is a secret, and nothing here requires a model. It requires something that can compare two pieces of text.

One rule

The judge in this course decides by words. A claim is supported when the words in it also appear in the text being judged. Words are cut to their first four letters so that shipped and shipping count as the same word, and the words that tell a grader what to do are thrown away, because the answer mentions a courier is a rule about couriers, not about answers or mentions.

WORD = re.compile(r"[a-z0-9]+")
# A word is cut to its first four letters, so `shipped` and `shipping` count
# as the same word. These stems tell the grader what to do rather than what to
# look for, so they are dropped before anything is compared.
INSTRUCTIONS = {"repl", "answ", "outp", "resp", "says", "said", "ment", "cont",
                "incl", "shou", "must", "text", "that", "this", "when", "wher",
                "with", "from", "give", "tell", "show", "stat", "name", "user"}

With the noise gone, two small functions do the deciding.

def words(text):
    """The word stems worth comparing."""
    found = {w[:4] for w in WORD.findall(text.lower()) if len(w) > 3}
    return found - INSTRUCTIONS or found


def share(claim, source):
    """How much of the claim the source covers, 0 to 1."""
    wanted = words(claim)
    return len(wanted & words(source)) / len(wanted) if wanted else 0.0

share is the whole decision. One when the text covers everything the rule asked for, zero when it covers none of it, and something in between when it is partly there.

Reading the prompt

Two more small functions. One pulls the text out of a tag, taking the last match because the system message uses the same tags for its examples. The other turns the list of chat messages back into one string.

def tag(text, name):
    """The text inside the LAST <name>...</name>, or an empty string."""
    found = re.findall(r"<%s>\s*(.*?)\s*</%s>" % (name, name), text, re.S)
    return found[-1] if found else ""

And one that turns the list of chat messages back into a single string to search.

def flatten(prompt):
    """Chat graders send a JSON list of messages. Join them into one string."""
    try:
        loaded = json.loads(prompt)
    except ValueError:
        return prompt
    if isinstance(loaded, list):
        return "\n".join(str(m.get("content", "")) for m in loaded)
    return prompt

Answering

Now the judge itself: flatten the messages, take the two tags, compare them, and reply in the format the prompt asked for.

def grade(prompt):
    text = flatten(prompt)
    output, rubric = tag(text, "Output"), tag(text, "Rubric")
    hit = share(rubric, output)
    return json.dumps({"reason": "word overlap %.2f" % hit,
                       "pass": hit >= 0.5, "score": hit})


def call_api(prompt, options, context):
    return {"output": grade(prompt)}

Half the rule's words present is enough to pass. That number is a judgement call and it is yours to move.

promptfoo eval

The first row passes and the second fails, which is the result the stub could not produce. The reason column carries the overlap, so when a verdict surprises you the number tells you why.

What it is not

It cannot tell you whether an answer is polite, because politeness has no words to match. It is fooled by an answer that repeats the rule's words in a wrong sentence. A model judge is better at both. What this one gives you is every lesson in this course running for free, and a clear head about what a judge is: a program that reads text and returns a number.