Design a RAG system over real-time data, e.g. a news or market-intelligence assistant that must reflect events from the last few minutes.

30-second answerSay your answer out loud first, then reveal.

Stream ingestion (a queue such as Kafka) into a low-latency pipeline: parse, deduplicate, chunk, embed, and write to an index supporting real-time upserts, with timestamps on everything. Retrieval combines relevance with recency (time filters, decay boosting) and handles fast-evolving stories (clustering duplicates, preferring the latest updates). Generation states the time context and cites timestamps. Monitor end-to-end freshness lag as a key SLA.

Real-time RAG: feeds flow through a stream queue, stream processors and GPU embedding workers into a hot real-time index that rolls over into a historical index; a query's time intent is parsed, both indexes are searched by relevance times recency, duplicate stories are clustered keeping the latest, and the LLM answers with timestamps and sources.

Key design points

Freshness SLA: e.g. an event becomes searchable within 60 seconds. Monitor lag from publish time to indexed time per source.
Dedup and story clustering: the same news appears from 50 outlets. Use near-duplicate detection (MinHash, or embedding similarity) and cluster stories so the answer isn't 10 copies of one fact.
Recency-aware ranking: score = relevance × time decay; parse temporal intent ("latest", "yesterday", "in 2024").
Hot / cold tiers: a small hot index with real-time inserts (fast HNSW updates) and a large historical index; query both and merge.
Entity tagging: companies, tickers and people as metadata for precise filtering.
Generation: include the current date and each source's timestamp in the prompt; distinguish confirmed facts from developing reports; avoid presenting stale information as current.
Source reliability scores, to limit misinformation amplification.

Trade-offs: embedding cost at high ingest volume (use smaller embedders and batching); index write amplification; consistency between the hot and cold tiers.

PreviousUsers ask "How many of our vendor contracts have an auto-renewal clause?" and RAG gives wrong answers. Why, and what would you build?

Slow is fine. Stopping is the only problem.

Design a RAG system over real-time data, e.g. a news or market-intelligence assistant that must reflect events from the last few minutes.

Key design points

Related