What is a Mixture of Experts (MoE) model?

30-second answerSay your answer out loud first, then reveal.

An MoE model replaces each dense feed-forward layer with many "expert" FFNs plus a router that sends each token to only a few experts (e.g. top-2 of 8). The model has a huge total parameter count (more knowledge capacity) but only a fraction is active per token, so compute per token is like a much smaller dense model. The trade-offs are high memory (all experts must be loaded), load-balancing difficulties, and more complex serving.

A token's hidden state goes to a router that picks the top-2 experts (Expert 2 and Expert 5) while the others (Expert 1, Expert 8) are not used; the chosen experts' outputs are combined in a weighted sum by router scores to give the output.

Example. Mixtral 8×7B has ~47B total parameters but ~13B active per token (2 of 8 experts per layer; the attention layers are shared). Several recent frontier and open models (e.g. DeepSeek-V3) use fine-grained MoE with many small experts plus shared experts.

Benefits

More capacity per unit of compute: better quality at the same inference FLOPs.
Faster training to a given loss for a fixed compute budget.

Challenges

Memory: all experts must be in memory, so a 47B MoE needs memory like a 47B dense model even though it computes like 13B.
Load balancing: the router may overuse a few experts. Fixed with auxiliary balancing losses or bias-based balancing.
Serving: expert parallelism across GPUs; all-to-all communication; batch efficiency depends on routing.
Fine-tuning can be less stable.

Interview nuance. MoE trades memory for compute. Great at large scale with high batch throughput; less attractive for single-GPU edge deployment.

PreviousWhat are scaling laws? What did the Chinchilla paper show?

You understood something today that you didn't yesterday.

What is a Mixture of Experts (MoE) model?

Benefits

Challenges

Related