What kinds of caching can you use in LLM systems?

30-second answerSay your answer out loud first, then reveal.

Several layers. An exact-match response cache (same normalised input gives the stored output). A semantic cache (similar meaning gives a reused answer, with a similarity threshold). Provider-side prompt/prefix caching (reuses the computed KV cache for identical prompt prefixes, which cuts input cost and TTFT). Embedding caches. Tool and retrieval result caches. Each needs invalidation rules and must respect personalisation and permissions.

A request checks the exact cache, then the semantic cache within the same scope, and only on two misses goes through the retrieval cache and an LLM call with prefix caching before results are stored with TTL and scope keys.

Cache types and their caveats

Cache	Saves	Risk / caveat
Exact response	Full LLM call	Low hit rate for free-text; good for FAQs and repeated workflows
Semantic	Full LLM call	Wrong answer if "similar" isn't equivalent ("cancel order" vs "cancel subscription"); tune threshold
Prompt / prefix (provider or vLLM)	Input token cost + TTFT	Prefix must be byte-identical; put static content first
Embedding	Embedding calls	Invalidate when the model changes
Retrieval / tool results	DB/API latency	Staleness; TTLs

Cache key scope (critical): include tenant, user permissions, locale, model version and prompt version. Never serve user A's personalised or permissioned answer to user B.

When not to cache: personalised content, time-sensitive answers ("today's balance"), high-stakes outputs without revalidation.

PreviousWhen do you use synchronous, streaming, or asynchronous/batch processing for LLM features?

Little by little, you're building something great.

What kinds of caching can you use in LLM systems?

Cache types and their caveats

Related