Promptfoopromptfoo 0.124.1 · Node 24 · Python 3
Dashboard
0%

factuality and g-eval: two more formats

factuality and g-eval are two more model-graded checks, and the one stand-in judge answers both by reading the prompt and replying in the shape each one expects.

Last updated: 15 Oct, 2026 · Promptfoo 0.124.1

The A grading script as the stand-in judge lesson wired a Python file as the grader and used it for llm-rubric, which wants a pass, score and reason. factuality and g-eval each ask for something different. The grader tells them apart by a phrase in the prompt and answers each in its own format.

factuality: one letter for the relationship

factuality compares the answer to a reference answer and asks for a single category, A through E, for how the two relate: a superset, a subset, agreement, disagreement, or a difference that does not matter. The grader returns a {category, reason} object. When the reference is fully covered it answers C; when the answer misses part of it, D.

python
# factuality: one letter for how the answer relates to the reference
if "factuality evaluator" in text:
    expert, submitted = tag(text, "expert_answer"), tag(text, "submitted_answer")
    if share(expert, submitted) < 1.0:
        return json.dumps({"category": "D", "reason": "the answer misses part of it"})
    return json.dumps({"category": "C", "reason": "the reference is covered"})

g-eval: steps, then a score

g-eval runs in two calls. The first asks the grader to turn the criteria into evaluation steps; the second asks it to score the reply from 0 to 10 against those steps. The grader answers {steps: [...]} to the first and {score, reason} to the second, spotting which call it is by the markers in the prompt.

python
# g-eval asks twice: first for the steps, then a score out of ten
if "EVALUATION CRITERIA" in text and '"steps"' in text:
    criteria = after(text.split("**OUTPUT FORMAT**")[0], "**EVALUATION CRITERIA**")
    return json.dumps({"steps": ["Check that the reply satisfies: " + criteria]})

if "Evaluation Steps" in text and '"score"' in text:
    criteria = text.split("**Evaluation Criteria**")[1].split("**")[0]
    reply = text.split("**Reply**")[1].split("**OUTPUT FORMAT**")[0]
    hit = share(criteria, reply)
    return json.dumps({"score": round(hit * 10), "reason": "word overlap %.2f" % hit})

Running both through the stand-in

This suite puts a factuality check and a g-eval check on the grounded answer, both graded by the same file. The reference and the criteria are carried in each assertion's value.

yaml
providers:
  - echo
prompts:
  - "{{answer}}"
defaultTest:
  options:
    provider: file://pretend_promptfoo.py
tests:
  - vars:
      answer: Order A17 shipped on 3 March by courier.
    assert:
      - type: factuality
        value: Order A17 shipped on 3 March by courier.
      - type: g-eval
        value: the answer mentions a courier
Project files used on this pageThis lesson builds on a project from earlier lessons. The code below imports this file. Click a file to see its code, or follow the link to the lesson that wrote it. To run the code yourself, keep it in the same folder.
View the code here
pretend_promptfoo.py
"""A grader for promptfoo, written in Python.

Every model-graded assertion sends a prompt to a provider and reads the reply.
This file is that provider. It never calls a model. It decides with one rule:

    a claim is supported when every long word in it also appears in the text
    it is checked against.

Each assertion type asks for a different answer, so `grade` looks at the prompt
it was given and answers in the format that assertion expects.
"""
import json
import re

WORD = re.compile(r"[a-z0-9]+")
# A word is cut to its first four letters, so `shipped` and `shipping` count
# as the same word. These stems tell the grader what to do rather than what to
# look for, so they are dropped before anything is compared.
INSTRUCTIONS = {"repl", "answ", "outp", "resp", "says", "said", "ment", "cont",
                "incl", "shou", "must", "text", "that", "this", "when", "wher",
                "with", "from", "give", "tell", "show", "stat", "name", "user"}


def words(text):
    """The word stems worth comparing."""
    found = {w[:4] for w in WORD.findall(text.lower()) if len(w) > 3}
    return found - INSTRUCTIONS or found


def share(claim, source):
    """How much of the claim the source covers, 0 to 1."""
    wanted = words(claim)
    return len(wanted & words(source)) / len(wanted) if wanted else 0.0

def tag(text, name):
    """The text inside the LAST <name>...</name>, or an empty string."""
    found = re.findall(r"<%s>\s*(.*?)\s*</%s>" % (name, name), text, re.S)
    return found[-1] if found else ""


def flatten(prompt):
    """Chat graders send a JSON list of messages. Join them into one string."""
    try:
        loaded = json.loads(prompt)
    except ValueError:
        return prompt
    if isinstance(loaded, list):
        return "\n".join(str(m.get("content", "")) for m in loaded)
    return prompt

def after(text, marker):
    """Everything past the last marker, ignoring case.

    Every grader prompt opens with worked examples that use the same markers,
    so the real input is always the last one.
    """
    parts = re.split(re.escape(marker), text, flags=re.I)
    return parts[-1].strip() if len(parts) > 1 else ""


def sentences(text):
    return [s.strip() for s in re.split(r"(?<=[.!?])\s+", text.strip()) if s.strip()]


def grade(prompt):
    text = flatten(prompt)

    # factuality: one letter for how the answer relates to the reference.
    if "factuality evaluator" in text:
        expert, submitted = tag(text, "expert_answer"), tag(text, "submitted_answer")
        if share(expert, submitted) < 1.0:
            return json.dumps({"category": "D", "reason": "the answer misses part of it"})
        return json.dumps({"category": "C", "reason": "the reference is covered"})

    # g-eval asks twice: first for the steps, then for a score out of ten.
    if "EVALUATION CRITERIA" in text and '"steps"' in text:
        criteria = after(text.split("**OUTPUT FORMAT**")[0], "**EVALUATION CRITERIA**")
        return json.dumps({"steps": ["Check that the reply satisfies: " + criteria]})
    if "Evaluation Steps" in text and '"score"' in text:
        criteria = text.split("**Evaluation Criteria**")[1].split("**")[0]
        reply = text.split("**Reply**")[1].split("**OUTPUT FORMAT**")[0]
        hit = share(criteria, reply)
        return json.dumps({"score": round(hit * 10), "reason": "word overlap %.2f" % hit})

    # context-faithfulness asks twice: split the answer up, then judge each part.
    if "create one or more statements" in text:
        return "\n".join(sentences(after(text, "answer:").split("statements:")[0]))
    if "Natural language inference" in text:
        block = after(text, "context:")
        source = re.split("statements:", block, flags=re.I)[0]
        lines, final = [], []
        for i, claim in enumerate(sentences(after(block, "statements:").split("Answer:")[0]), 1):
            ok = share(claim, source) == 1.0
            lines.append("%d. %s Verdict: %s." % (i, claim, "Yes" if ok else "No"))
            final.append("Yes." if ok else "No.")
        return "\n".join(lines) + "\nFinal verdict for each statement in order: " + " ".join(final)

    # context-recall: is each part of the right answer in the documents?
    if "can be attributed to the given context" in text:
        block = after(text, "context:")
        source = re.split("answer:", block, flags=re.I)[0]
        verdicts = []
        for i, claim in enumerate(sentences(after(block, "answer:").split("classification")[0]), 1):
            ok = share(claim, source) == 1.0
            verdicts.append("%d. %s So [%s]" % (i, claim, "Attributed" if ok else "Not Attributed"))
        return "\n".join(verdicts)

    # llm-rubric, and everything built on it: pass, score and a reason.
    output, rubric = tag(text, "Output"), tag(text, "Rubric")
    hit = share(rubric, output)
    return json.dumps({"reason": "word overlap %.2f" % hit,
                       "pass": hit >= 0.5, "score": hit})


def call_api(prompt, options, context):
    return {"output": grade(prompt)}
Examplefactuality and g-eval both pass on a grounded answer, keyless
promptfoo eval --no-table

One judge, three shapes

The run passed both checks with no key, from the same grader that served llm-rubric. The difference is only in what each assertion asked for and what the grader wrote back.

AssertionWhat it asks forWhat the grader returns
llm-rubricDoes the rubric hold{pass, score, reason}
factualityHow the answer relates to a reference{category, reason}, one of A to E
g-evalSteps, then a 0 to 10 score{steps}, then {score, reason}

Where you use factuality and g-eval

  • factuality against a known-good answer, when you have a reference and want to flag drift from it.
  • g-eval for a graded criterion, where a plain pass or fail is too blunt and a score out of ten reads better.
  • A mix of all three, layering a rubric, a reference check and a scored criterion over one answer.
Watch out. Each format reads a different field. factuality needs its reference in value, and g-eval reads its criterion from value too, but a real model grader also expects the categories and the two-call shape it was trained on. A stand-in that answered every assertion with a pass object would break factuality and g-eval, which is why the one grader branches on the prompt.
Try it yourself
  • Give factuality a reference the answer only partly covers and watch the category fall to D.
  • Change the g-eval criterion to something the answer does not mention and read the lower score.
  • Add an llm-rubric back alongside these two and confirm all three run from one grader.

This is what real progress feels like.