What is the difference between reference-based and reference-free evaluation?

30-second answerSay your answer out loud first, then reveal.

Reference-based evaluation compares the output against a known correct answer: exact match, F1, semantic similarity, or a judge asking "does this match the reference?" Reference-free evaluation judges the output using only the input, context and criteria, for example "is this answer supported by the retrieved documents?" or "is the tone polite?" Reference-based is more objective but needs labelled data. Reference-free scales to live traffic and open-ended tasks but relies more on the judge.

Examples

Task	Reference-based	Reference-free
Extraction	Field values vs ground truth	Values present in source document?
RAG Q&A	Answer matches the gold answer	Answer grounded in retrieved context (faithfulness); relevant to question
Summarisation	Overlap with a human summary (weak)	Covers key points from the source; no unsupported claims
Chat	Matches the expected resolution	Helpful, polite, follows policy
Code	Output equals expected output	Passes tests (tests act as reference), lint clean

When to use which

Offline golden sets: mostly reference-based, plus reference-free criteria.
Online monitoring: mostly reference-free (there are no references for live queries).
Hybrid: reference-guided judging, where the judge sees the reference answer as a hint but grades meaning, not exact wording.

Caveat: "faithful to context" (reference-free) doesn't mean correct, because the context itself can be wrong. Combine both when stakes are high.

PreviousWhat are the main types of evals, and when do you use each?

Slow is fine. Stopping is the only problem.

What is the difference between reference-based and reference-free evaluation?

Examples

When to use which

Related