How do multimodal LLMs process images? Describe the typical architecture.

30-second answerSay your answer out loud first, then reveal.

Most vision-language models use a vision encoder (a ViT, often CLIP- or SigLIP-style) that turns image patches into embeddings, a projector (MLP or cross-attention resampler) that maps them into the LLM's token embedding space, and the LLM, which processes the "visual tokens" alongside text tokens. Training aligns the projector first (image–caption pairs), then fine-tunes on visual instruction data. Some models are natively multimodal, trained on interleaved image–text data from the start.

An image is split into patches, encoded by a ViT, mapped by a projector into visual tokens, and a decoder LLM attends over the visual tokens and the text token embeddings together to produce a text answer.

Design choices

Image resolution and tiling: high-resolution images (documents, screenshots) are split into tiles, plus a low-resolution global view. More visual tokens give better detail at higher cost (an image can be hundreds to thousands of tokens).
Projector type:
MLP (LLaVA-style): simple, keeps all patch tokens.
Resampler / Q-Former (Flamingo, BLIP-2): compresses to a fixed number of tokens.
Fusion: early fusion (visual tokens in the input sequence) vs cross-attention layers (Flamingo-style).
Training stages: (a) pretrain the projector on image–caption pairs with the encoder and LLM frozen; (b) visual instruction tuning (VQA, OCR, charts, grounding); (c) preference / RL tuning.

Practical considerations

Image token costs and latency (resize images sensibly).
OCR-heavy tasks (invoices, forms) need high resolution; check model support.
Hallucination about image contents ("object hallucination") needs evaluation.
Audio and video follow similar patterns: a modality encoder + projector + LLM, or native multimodal training.

PreviousAn enterprise asks: should we use LLM APIs or self-host open models? How do you analyse it?

This is what real progress feels like.

How do multimodal LLMs process images? Describe the typical architecture.

Design choices

Practical considerations

Related