factuality and g-eval: two more formats
factuality and g-eval are two more model-graded checks, and the one stand-in judge answers both by reading the prompt and replying in the shape each one expects.
Last updated: 15 Oct, 2026 · Promptfoo 0.124.1
The A grading script as the stand-in judge lesson wired a Python file as the grader and used it for llm-rubric, which wants a pass, score and reason. factuality and g-eval each ask for something different. The grader tells them apart by a phrase in the prompt and answers each in its own format.
factuality: one letter for the relationship
factuality compares the answer to a reference answer and asks for a single category, A through E, for how the two relate: a superset, a subset, agreement, disagreement, or a difference that does not matter. The grader returns a {category, reason} object. When the reference is fully covered it answers C; when the answer misses part of it, D.
# factuality: one letter for how the answer relates to the reference
if "factuality evaluator" in text:
expert, submitted = tag(text, "expert_answer"), tag(text, "submitted_answer")
if share(expert, submitted) < 1.0:
return json.dumps({"category": "D", "reason": "the answer misses part of it"})
return json.dumps({"category": "C", "reason": "the reference is covered"})g-eval: steps, then a score
g-eval runs in two calls. The first asks the grader to turn the criteria into evaluation steps; the second asks it to score the reply from 0 to 10 against those steps. The grader answers {steps: [...]} to the first and {score, reason} to the second, spotting which call it is by the markers in the prompt.
# g-eval asks twice: first for the steps, then a score out of ten
if "EVALUATION CRITERIA" in text and '"steps"' in text:
criteria = after(text.split("**OUTPUT FORMAT**")[0], "**EVALUATION CRITERIA**")
return json.dumps({"steps": ["Check that the reply satisfies: " + criteria]})
if "Evaluation Steps" in text and '"score"' in text:
criteria = text.split("**Evaluation Criteria**")[1].split("**")[0]
reply = text.split("**Reply**")[1].split("**OUTPUT FORMAT**")[0]
hit = share(criteria, reply)
return json.dumps({"score": round(hit * 10), "reason": "word overlap %.2f" % hit})Running both through the stand-in
This suite puts a factuality check and a g-eval check on the grounded answer, both graded by the same file. The reference and the criteria are carried in each assertion's value.
providers:
- echo
prompts:
- "{{answer}}"
defaultTest:
options:
provider: file://pretend_promptfoo.py
tests:
- vars:
answer: Order A17 shipped on 3 March by courier.
assert:
- type: factuality
value: Order A17 shipped on 3 March by courier.
- type: g-eval
value: the answer mentions a courier
View the code here
"""A grader for promptfoo, written in Python.
Every model-graded assertion sends a prompt to a provider and reads the reply.
This file is that provider. It never calls a model. It decides with one rule:
a claim is supported when every long word in it also appears in the text
it is checked against.
Each assertion type asks for a different answer, so `grade` looks at the prompt
it was given and answers in the format that assertion expects.
"""
import json
import re
WORD = re.compile(r"[a-z0-9]+")
# A word is cut to its first four letters, so `shipped` and `shipping` count
# as the same word. These stems tell the grader what to do rather than what to
# look for, so they are dropped before anything is compared.
INSTRUCTIONS = {"repl", "answ", "outp", "resp", "says", "said", "ment", "cont",
"incl", "shou", "must", "text", "that", "this", "when", "wher",
"with", "from", "give", "tell", "show", "stat", "name", "user"}
def words(text):
"""The word stems worth comparing."""
found = {w[:4] for w in WORD.findall(text.lower()) if len(w) > 3}
return found - INSTRUCTIONS or found
def share(claim, source):
"""How much of the claim the source covers, 0 to 1."""
wanted = words(claim)
return len(wanted & words(source)) / len(wanted) if wanted else 0.0
def tag(text, name):
"""The text inside the LAST <name>...</name>, or an empty string."""
found = re.findall(r"<%s>\s*(.*?)\s*</%s>" % (name, name), text, re.S)
return found[-1] if found else ""
def flatten(prompt):
"""Chat graders send a JSON list of messages. Join them into one string."""
try:
loaded = json.loads(prompt)
except ValueError:
return prompt
if isinstance(loaded, list):
return "\n".join(str(m.get("content", "")) for m in loaded)
return prompt
def after(text, marker):
"""Everything past the last marker, ignoring case.
Every grader prompt opens with worked examples that use the same markers,
so the real input is always the last one.
"""
parts = re.split(re.escape(marker), text, flags=re.I)
return parts[-1].strip() if len(parts) > 1 else ""
def sentences(text):
return [s.strip() for s in re.split(r"(?<=[.!?])\s+", text.strip()) if s.strip()]
def grade(prompt):
text = flatten(prompt)
# factuality: one letter for how the answer relates to the reference.
if "factuality evaluator" in text:
expert, submitted = tag(text, "expert_answer"), tag(text, "submitted_answer")
if share(expert, submitted) < 1.0:
return json.dumps({"category": "D", "reason": "the answer misses part of it"})
return json.dumps({"category": "C", "reason": "the reference is covered"})
# g-eval asks twice: first for the steps, then for a score out of ten.
if "EVALUATION CRITERIA" in text and '"steps"' in text:
criteria = after(text.split("**OUTPUT FORMAT**")[0], "**EVALUATION CRITERIA**")
return json.dumps({"steps": ["Check that the reply satisfies: " + criteria]})
if "Evaluation Steps" in text and '"score"' in text:
criteria = text.split("**Evaluation Criteria**")[1].split("**")[0]
reply = text.split("**Reply**")[1].split("**OUTPUT FORMAT**")[0]
hit = share(criteria, reply)
return json.dumps({"score": round(hit * 10), "reason": "word overlap %.2f" % hit})
# context-faithfulness asks twice: split the answer up, then judge each part.
if "create one or more statements" in text:
return "\n".join(sentences(after(text, "answer:").split("statements:")[0]))
if "Natural language inference" in text:
block = after(text, "context:")
source = re.split("statements:", block, flags=re.I)[0]
lines, final = [], []
for i, claim in enumerate(sentences(after(block, "statements:").split("Answer:")[0]), 1):
ok = share(claim, source) == 1.0
lines.append("%d. %s Verdict: %s." % (i, claim, "Yes" if ok else "No"))
final.append("Yes." if ok else "No.")
return "\n".join(lines) + "\nFinal verdict for each statement in order: " + " ".join(final)
# context-recall: is each part of the right answer in the documents?
if "can be attributed to the given context" in text:
block = after(text, "context:")
source = re.split("answer:", block, flags=re.I)[0]
verdicts = []
for i, claim in enumerate(sentences(after(block, "answer:").split("classification")[0]), 1):
ok = share(claim, source) == 1.0
verdicts.append("%d. %s So [%s]" % (i, claim, "Attributed" if ok else "Not Attributed"))
return "\n".join(verdicts)
# llm-rubric, and everything built on it: pass, score and a reason.
output, rubric = tag(text, "Output"), tag(text, "Rubric")
hit = share(rubric, output)
return json.dumps({"reason": "word overlap %.2f" % hit,
"pass": hit >= 0.5, "score": hit})
def call_api(prompt, options, context):
return {"output": grade(prompt)}
promptfoo eval --no-tableStarting evaluation eval-jOM-2026-10-14T19:06:22 Running 1 test cases (up to 4 at a time)... Creating cache folder at ~/.promptfoo/cache. ✓ Eval complete (ID: eval-jOM-2026-10-14T19:06:22) » View results: promptfoo view » Share with your team: https://promptfoo.app » Feedback: https://promptfoo.dev/feedback Results: ✓ 1 passed (100%) 0 failed (0%) 0 errors (0%) Duration: 3s (concurrency: 4)
One judge, three shapes
The run passed both checks with no key, from the same grader that served llm-rubric. The difference is only in what each assertion asked for and what the grader wrote back.
| Assertion | What it asks for | What the grader returns |
|---|---|---|
llm-rubric | Does the rubric hold | {pass, score, reason} |
factuality | How the answer relates to a reference | {category, reason}, one of A to E |
g-eval | Steps, then a 0 to 10 score | {steps}, then {score, reason} |
Where you use factuality and g-eval
- factuality against a known-good answer, when you have a reference and want to flag drift from it.
- g-eval for a graded criterion, where a plain pass or fail is too blunt and a score out of ten reads better.
- A mix of all three, layering a rubric, a reference check and a scored criterion over one answer.
factuality needs its reference in value, and g-eval reads its criterion from value too, but a real model grader also expects the categories and the two-call shape it was trained on. A stand-in that answered every assertion with a pass object would break factuality and g-eval, which is why the one grader branches on the prompt.Related
- Previous: A grading script as the stand-in judge
- Next: When the judge is wrong
- Reference: Promptfoo documentation
- Give
factualitya reference the answer only partly covers and watch the category fall toD. - Change the
g-evalcriterion to something the answer does not mention and read the lower score. - Add an
llm-rubricback alongside these two and confirm all three run from one grader.
This is what real progress feels like.