How do you build multimodal RAG over documents with images, charts, and scanned pages?

30-second answerSay your answer out loud first, then reveal.

Two main approaches. (1) Convert everything to text: OCR scanned pages, caption images and charts with a vision-language model, extract tables. Then use normal text RAG. (2) Retrieve on visuals directly: embed page images with a vision-based retriever (ColPali-style late interaction over page patches, or multimodal embeddings), then pass the retrieved page images to a vision-capable LLM. Many systems combine both.

Documents split into a text path (OCR and VLM captions into a text index) and a page-image path (a vision retriever into an image index); the query searches both, results are fused, and a vision-capable LLM reads text plus page images.

Path A: text conversion

Pros: reuses your text RAG stack; cheap retrieval; easy citations.
Cons: captions lose detail; complex charts and diagrams are hard to describe fully; a parsing pipeline per format.

Path B: visual retrieval

Pros: no fragile parsing; preserves layout, charts and visual tables; strong on visually rich documents (slides, reports).
Cons: larger indexes (multi-vector); a VLM is needed for generation (higher cost); harder to highlight exact text spans.

Practical design

Store page number + bounding boxes for citations ("see page 14, figure 3").
Use the VLM only for pages that matter (top 3–5 page images).
Evaluate with questions whose answers live in charts or tables, not just in text.

PreviousWhat is late interaction (ColBERT), and how does it compare to bi-encoders and cross-encoders?

This is what real progress feels like.

How do you build multimodal RAG over documents with images, charts, and scanned pages?

Path A: text conversion

Path B: visual retrieval

Practical design

Related