factuality and g-eval: two more formats

llm-rubric asks for a verdict. Two other model-graded checks ask the same judge for something else entirely, and a judge that only knows one format quietly breaks on them.

factuality

factuality compares the answer to a reference answer you supply, and it does not want a pass or a score. It wants one letter, from a list of five relationships: the answer is a subset of the reference, a superset, the same, a disagreement, or different in a way that does not matter. A, B, C and E pass by default. D fails.

That is a more useful question than is this right, because it separates an answer that is short from an answer that is wrong.

python

# factuality: one letter for how the answer relates to the reference.
    if "factuality evaluator" in text:
        expert, submitted = tag(text, "expert_answer"), tag(text, "submitted_answer")
        if share(expert, submitted) < 1.0:
            return json.dumps({"category": "D", "reason": "the answer misses part of it"})
        return json.dumps({"category": "C", "reason": "the reference is covered"})

If the reference says something the answer does not, the two disagree, which is D. Otherwise the answer covers the reference and the letter is C.

g-eval

g-eval asks twice. First it asks the judge to turn your criterion into a list of steps, then it sends the answer back with those steps and asks for a score out of ten, which promptfoo divides by ten and compares against a threshold that defaults to 0.7.

python

# g-eval asks twice: first for the steps, then for a score out of ten.
    if "EVALUATION CRITERIA" in text and '"steps"' in text:
        criteria = after(text.split("**OUTPUT FORMAT**")[0], "**EVALUATION CRITERIA**")
        return json.dumps({"steps": ["Check that the reply satisfies: " + criteria]})
    if "Evaluation Steps" in text and '"score"' in text:
        criteria = text.split("**Evaluation Criteria**")[1].split("**")[0]
        reply = text.split("**Reply**")[1].split("**OUTPUT FORMAT**")[0]
        hit = share(criteria, reply)
        return json.dumps({"score": round(hit * 10), "reason": "word overlap %.2f" % hit})

The first branch answers the steps question, the second scores. Both are recognised by a phrase in the prompt, which is how one file serves every kind of grader without being told which is which.

yaml

assert:
  - type: factuality
    value: Order A17 shipped on 3 March by courier.
  - type: g-eval
    value: the answer mentions a courier

Example

promptfoo eval

Both pass. The answer covers the reference, so factuality is C, and the criterion's words are in the answer, so g-eval scores ten out of ten.

The branch order matters. The rubric branch has no phrase of its own to look for, so it is the fallback at the bottom of grade. Put it first and every other grader gets a rubric verdict it cannot read, and the failures look like the judge disagreeing rather than the judge answering the wrong question.

The rest of the family

Assertion	What it asks the judge	Wants back
llm-rubric	does the answer satisfy this sentence	pass, score, reason
factuality	how does the answer relate to this reference	a letter A to E
g-eval	score the answer against these steps	a score out of ten
model-graded-closedqa	does the answer meet this requirement	a verdict
answer-relevance	is the answer about the question	a score, plus embeddings
select-best	which of these answers is best	an index

answer-relevance is the one to know about and avoid here: it needs an embedding model as well as a judge, so it needs a second stand-in. The context checks in lesson 22 need two more branches, and get them there.

Try it yourself

Change the reference so it mentions a date the answer does not, and confirm the letter becomes D.
Set threshold: 0.9 on the g-eval assertion and see whether ten out of ten still clears it.
Delete the factuality branch and run again: read the error the assertion gives when the judge answers in the wrong format.

PreviousWriting the stand-in judge

This is what real progress feels like.