How do you evaluate summarisation quality?

30-second answerSay your answer out loud first, then reveal.

Use the dimensions that matter for summaries: faithfulness (no claims unsupported by the source), coverage (key information included), conciseness and relevance to the purpose, and format. Practical methods are claim-level faithfulness checks against the source, question-based coverage tests (generate key questions from the source and check whether the summary answers them), checklists of must-include items defined by SMEs, and rubric-based judging. ROUGE alone is inadequate.

Methods

Faithfulness: extract atomic claims from the summary, then verify each against the source (NLI / LLM judge). Report the % unsupported claims and flag critical errors (wrong numbers, names, negations).
Coverage via QA (QAG approach): generate N important questions from the source (or have SMEs write them), and check how many the summary answers correctly.
Must-include checklist: for domain summaries (e.g. a medical discharge summary must include medications, follow-up date, allergies), presence checks per item.
Purpose fit: a summary for an executive vs an engineer differs, so the rubric should reflect the audience.
Length/format: code checks (word limits, sections present).

Example rubric (binary per item)

Criterion	Pass condition
No unsupported claims	All claims entailed by source
Critical facts present	All checklist items included
No critical omissions	SME-defined key risks mentioned
Appropriate length	≤ 150 words

Watch out for: summaries that are faithful but omit the most important point. Coverage checks catch this.

PreviousWhat metrics do you use to evaluate an AI agent's tool use and trajectory?

Slow is fine. Stopping is the only problem.

How do you evaluate summarisation quality?

Methods

Example rubric (binary per item)

Related