What are BLEU, ROUGE, BERTScore, exact match and F1? Why are they often insufficient for LLM apps?

30-second answerSay your answer out loud first, then reveal.

BLEU and ROUGE measure n-gram overlap with reference texts (BLEU is precision-oriented, from translation; ROUGE is recall-oriented, from summarisation). BERTScore measures semantic similarity using embeddings. Exact match and token-level F1 are used for short answers. They're cheap and reproducible, but they penalise valid paraphrases, reward fluent but wrong text that shares words, ignore factual correctness and safety, and correlate poorly with human judgement on open-ended generation.

Metric	Measures	Good for	Weakness
Exact match	Identical (normalised) answer	Short factual answers, labels	Zero credit for paraphrases
Token F1	Word overlap precision/recall	Extractive QA	Surface-level
BLEU	n-gram precision vs references	Machine translation (historical)	Poor for open-ended tasks
ROUGE-L	Longest common subsequence / n-gram recall	Summarisation (historical)	Rewards copying, not correctness
BERTScore	Embedding similarity	Paraphrase-tolerant comparison	Can't detect factual errors with similar wording

Example failure. Reference "The refund window is 30 days." Output: "The refund window is 13 days." That gets high ROUGE and BERTScore despite being wrong.

When they're still useful: quick regression signals, translation benchmarks, extraction with exact values, and as cheap features alongside other checks.

Interview line. "For LLM apps I prefer task-specific criteria: exact checks where possible, plus calibrated rubric judging for meaning and faithfulness."

PreviousWhat is the difference between reference-based and reference-free evaluation?

This is what real progress feels like.

What are BLEU, ROUGE, BERTScore, exact match and F1? Why are they often insufficient for LLM apps?

Related