With 1M+ token context windows, is RAG dead? Argue both sides.

30-second answerSay your answer out loud first, then reveal.

No, but its role is shifting. Long context lets you skip retrieval for small corpora and lets you retrieve bigger units (whole documents). RAG remains necessary for large or changing corpora (far beyond any window), for cost and latency (you don't send millions of tokens per question), for access control, and for citations. Long-context quality also degrades with distractors. The practical answer is hybrid: retrieve coarser, then let the long-context model reason.

Arguments that long context reduces the need for RAG

Small corpora (one handbook, one codebase module, one contract) fit entirely, and no retrieval errors are possible.
Holistic reasoning across a whole document is better than across fragmented chunks.
Simpler architecture: no chunking or index tuning.
Prompt caching makes repeatedly sending the same large context cheaper.

Arguments that RAG remains essential

Scale: enterprise corpora are billions of tokens, orders of magnitude beyond any window.
Cost and latency: sending 1M tokens per query is slow and expensive compared with 5K well-chosen tokens, even with caching.
Freshness: RAG indexes update continuously; there's no need to rebuild a giant context.
Access control: each user must see a different subset, which retrieval filters handle naturally.
Quality: models still miss or misuse information in very long contexts, especially with many similar distractors ("needle in a haystack" tests are easier than real multi-fact reasoning).
Citations and auditability are cleaner with retrieved passages.

Synthesis (what senior candidates say)

Corpus fits and changes rarely → long context (+ caching).
Large or dynamic corpus → RAG, but with larger chunks or whole-document retrieval feeding a long-context model.
Decide with an eval comparing quality, latency and cost on your workload.

PreviousWhen and how would you fine-tune an embedding model or reranker for your domain?

Slow is fine. Stopping is the only problem.

With 1M+ token context windows, is RAG dead? Argue both sides.

Arguments that long context reduces the need for RAG

Arguments that RAG remains essential

Synthesis (what senior candidates say)

Related