Encoder-only, decoder-only, encoder-decoder: what's the difference, and when is each used?

30-second answerSay your answer out loud first, then reveal.

Encoder-only models (BERT) see the whole input bidirectionally and are great for understanding tasks: classification, embeddings, NER. Decoder-only models (GPT, Llama, Claude, Gemini) use causal attention and generate text autoregressively. They dominate modern LLMs because one architecture handles every task as text generation. Encoder-decoder models (T5, original transformer) encode the input then decode an output, and are strong for translation and summarisation.

	Encoder-only	Decoder-only	Encoder-decoder
Attention	Bidirectional	Causal (left-to-right)	Encoder bidirectional; decoder causal + cross-attention
Training objective	Masked language modelling	Next-token prediction	Span corruption / seq2seq
Examples	BERT, RoBERTa, DeBERTa	GPT, Llama, Mistral, Qwen, Claude, Gemini	T5, BART, original Transformer, Whisper
Best for	Classification, embeddings, extraction	General generation, chat, code, agents	Translation, summarisation, speech-to-text

Why decoder-only won for general LLMs

Simple, uniform objective that scales well on raw text.
Any task can be phrased as "continue this text."
Efficient generation with a KV cache (Q18).

Encoder models are still everywhere: embedding models for RAG and rerankers are often encoder-based, because bidirectional attention gives better representations for understanding and retrieval.

PreviousWhat are the components of a transformer block in a modern LLM?

You understood something today that you didn't yesterday.

Encoder-only, decoder-only, encoder-decoder: what's the difference, and when is each used?

Why decoder-only won for general LLMs

Related