What is the context window, and what happens when you exceed it?

Key points

• Input + output share the budget. A 128K model with a 120K-token prompt has at most 8K tokens for the answer (and providers often cap output separately).
• Why there's a limit: training length, positional encoding limits, and attention memory/compute that grows with sequence length (quadratic for standard attention, plus KV cache memory growing linearly per token).
• Effective vs advertised context: models may accept 1M tokens but use information less reliably deep in long contexts, especially with many similar distractors ("lost in the middle").

Handling long inputs

1. Retrieve only the relevant parts (RAG).
2. Summarise / compact older conversation turns.
3. Chunk and map-reduce: process parts separately, then combine.
4. Choose a long-context model when the task truly needs holistic reading.
5. Prompt caching for repeatedly used long prefixes.

Common mistakes

• Building chatbots that keep appending history until they crash or silently truncate the system prompt.

Related

• Previous: Q8. Explain temperature, top-k and top-p (nucleus) sampling.
• Next: Q10. Walk through the LLM training lifecycle: pretraining, SFT, and preference alignment.
• LLM Fundamentals
• LangChain