Walk through the LLM training lifecycle: pretraining, SFT, and preference alignment.

30-second answerSay your answer out loud first, then reveal.

Pretraining teaches language and knowledge through next-token prediction on trillions of tokens (the most expensive stage). Supervised fine-tuning (SFT) teaches the format and behaviour of an assistant from curated instruction–response pairs. Preference alignment (RLHF with a reward model and PPO, or DPO-style methods, plus RL on verifiable tasks) makes outputs more helpful, honest and safe, and improves reasoning. Evaluation and safety testing run throughout.

The LLM training lifecycle: web, books and code feed pretraining to give a base model, SFT on instruction-response pairs gives an instruction model, preference or RL tuning gives a chat assistant, then evals and red-teaming lead to deployment.

Stage	Data	Compute	What it adds
Pretraining	Trillions of tokens, mostly unlabelled	Huge (thousands of GPUs for weeks or months)	Language, knowledge, reasoning patterns
Mid-training / continued pretraining (optional)	Domain data, long documents, code, maths	Medium	Domain strength, longer context
SFT	~10K–1M high-quality demonstrations	Small	Following instructions, formats, tool use
Preference / RL	Comparisons (A better than B), reward models, verifiable tasks	Small–medium	Helpfulness, safety, style, reasoning

Base vs instruct models: ask a base model "What is the capital of France?" and it may continue with more quiz questions. An instruct model answers.

Follow-ups to expect

How does RLHF work in detail? (Q24.) What is DPO? (Q25.)

PreviousWhat is the context window, and what happens when you exceed it?

Every expert started right here.

Walk through the LLM training lifecycle: pretraining, SFT, and preference alignment.

Follow-ups to expect

Related