What is multi-head attention? What are Multi-Query Attention (MQA) and Grouped-Query Attention (GQA), and why do modern LLMs use GQA?

30-second answerSay your answer out loud first, then reveal.

Multi-head attention runs several attention operations in parallel on lower-dimensional projections, so different heads can capture different relationships (syntax, coreference, position), then concatenates them. In standard MHA every head has its own keys and values, which makes the KV cache large. MQA shares one K/V head across all query heads (smallest cache, some quality loss). GQA shares each K/V head within a group of query heads, getting most of MQA's memory savings with near-MHA quality, so most modern LLMs use it.

MQA shares one KV head across all 8 query heads, GQA shares two KV heads across groups of four query heads, and MHA gives each of the 8 query heads its own KV head.

Multi-head mechanics: with d_model = 4096 and 32 heads, each head works in 128 dimensions. Outputs are concatenated (back to 4096) and projected with W_O.

Why KV heads matter: during generation, the KV cache stores K and V for every layer, KV head and past token. Its size scales with the number of KV heads.

	MHA	GQA	MQA
KV heads	= query heads (e.g. 64)	Groups (e.g. 8)	1
KV cache size	1x	~1/8x	~1/64x
Quality	Baseline	≈ MHA	Slightly lower
Used by	Older models	Llama 2 70B, Llama 3, Mistral, many others	PaLM, Falcon (some)

Why it matters in production: a smaller KV cache means more concurrent users and longer contexts per GPU, and faster decoding (decoding is memory-bandwidth bound, Q20).

Related idea: Multi-head Latent Attention (MLA, DeepSeek) compresses K/V into a low-rank latent to shrink the cache even further.

PreviousWrite the scaled dot-product attention formula. Why divide by √d_k, and what is the causal mask?

You understood something today that you didn't yesterday.