How would you scale RAG to 100 million+ documents (billions of chunks)?

30-second answerSay your answer out loud first, then reveal.

Shard the index (by tenant, domain, time or hash), use memory-efficient ANN (quantization, IVF-PQ, disk-based graphs), use tiered and staged retrieval (cheap coarse retrieval → rerank → precise), route queries to relevant shards with metadata, build the index with distributed batch pipelines, and keep incremental updates efficient. Measure recall at each stage, because every approximation trades some of it away.

Offline, distributed batch embedding feeds an index build per shard; at query time a router sends the query to shards 1 to N, each returns top-k from a quantized ANN index, and the merged candidates are rescored with full-precision vectors and reranked by a cross-encoder before the LLM.

Design areas

Storage math: 2B chunks × 768 dims × 4 bytes ≈ 6 TB of raw vectors. In RAM, that's infeasible economically, so you need quantization (int8 → 1.5 TB, PQ or binary → much smaller) plus disk-based ANN or tiered memory.
Sharding strategies:
• By tenant / domain: queries hit one shard (best when filters are natural).
• By hash: even load, but every query fans out to all shards (scatter-gather), so tail latency matters.
• By time: recent data hot, old data cold.
Two-tier retrieval: a cheap first stage (BM25 or binary vectors) over everything → full-precision rescoring → cross-encoder on the top 100.
Indexing throughput: distributed embedding (GPU batch jobs), checkpointed pipelines, back-pressure. Re-embedding the whole corpus takes days and costs real money, so plan model migrations carefully (Q45).
Freshness: a small, fast "delta" index for recent documents merged with the large static index; periodic compaction.
Replication for QPS and availability; consistent snapshots.
Document-level vs chunk-level retrieval: first retrieve documents (using summaries), then chunks within the top documents. This reduces the search space.

Common mistakes

Designing one giant HNSW index in RAM without doing the memory math.

PreviousHow do you implement permission-aware (ACL-aware) retrieval correctly?

Slow is fine. Stopping is the only problem.

How would you scale RAG to 100 million+ documents (billions of chunks)?

Design areas

Common mistakes

Related