What are scaling laws? What did the Chinchilla paper show?

30-second answerSay your answer out loud first, then reveal.

Scaling laws are empirical power-law relationships: model loss falls predictably as you increase parameters, training data and compute. Kaplan et al. (2020) established them. Chinchilla (Hoffmann et al. 2022) showed that many large models were undertrained, and that for a fixed compute budget, parameters and training tokens should grow roughly equally, about 20 tokens per parameter for compute-optimal training. Today models are often trained far beyond that ratio, because inference cost favours smaller models trained on more data.

Key ideas

Loss ≈ a power law in N (params), D (tokens) and C (compute), so performance improves smoothly and predictably, which allows planning big training runs from small experiments.
Training compute ≈ 6 × N × D FLOPs (forward + backward pass).

Chinchilla result

Chinchilla (70B params, 1.4T tokens) outperformed Gopher (280B params, 300B tokens) with similar compute.
Compute-optimal: D ≈ 20 × N.

Why modern models "over-train"

Chinchilla optimises training compute, but a deployed model is run billions of times, so inference cost dominates.
A smaller model trained on far more tokens (e.g. 8B params on ~15T tokens, almost 2,000 tokens per parameter) is cheaper to serve with good quality.

Caveats

Loss improvements don't map linearly onto specific capabilities. Some abilities appear to emerge more abruptly (though this partly depends on the metric).
Data quality and mixture matter as much as quantity. High-quality data is becoming a bottleneck (synthetic data, filtering).
New scaling axes: test-time compute (Q15) and RL compute.

PreviousWhat is DPO, and how does it compare to RLHF with PPO?

Little by little, you're building something great.

What are scaling laws? What did the Chinchilla paper show?

Key ideas

Chinchilla result

Why modern models "over-train"

Caveats

Related