What are the components of a transformer block in a modern LLM?

30-second answerSay your answer out loud first, then reveal.

Each block has two sub-layers: multi-head self-attention (tokens exchange information) and a feed-forward network or MLP (each token is processed independently, and much of the "knowledge" is thought to live here). Each sub-layer is wrapped with a residual connection and normalisation. Modern LLMs stack dozens of these blocks (e.g. 32–80+), typically with pre-normalisation (RMSNorm), SwiGLU activations, RoPE positions and grouped-query attention.

A pre-norm transformer block: input hidden states go through RMSNorm and multi-head self-attention, added back to the input by a residual connection, then through RMSNorm and a SwiGLU feed-forward MLP with a second residual add, and on to the next block.

Roles

Attention: mixes information across positions (context).
FFN / MLP: transforms each position independently, with an expansion then a projection back down. The FFN holds roughly two-thirds of a dense model's parameters.
Residual connections: each sub-layer adds to the input. This makes training deep networks stable (gradient flow) and creates a "residual stream" that layers read from and write to.
Normalisation: keeps activations in a stable range. Pre-norm (normalise before each sub-layer) trains more stably than the original post-norm (Q47).

After the last block: a final norm → a linear projection to vocabulary size → logits.

Parameter intuition: for hidden size d, attention has ~4d² parameters (Q, K, V, O) and the FFN ~8d² (more with SwiGLU), per layer.

PreviousExplain self-attention intuitively. What are queries, keys and values?

Little by little, you're building something great.

What are the components of a transformer block in a modern LLM?

Roles

Related