LiteLLMLiteLLM 1.105 · Python 3.10+
Dashboard
0%

Semantic caching

A semantic cache is a cache that matches requests by meaning instead of by exact text, so a paraphrase of an earlier question gets the earlier answer.

Last updated: 12 Oct, 2026 · LiteLLM 1.105

Cache keys showed that one changed character gives a new key. People rarely ask the same thing in the same words, so an exact cache misses most paraphrases. A semantic cache turns each question into a list of numbers that captures its meaning and answers from the stored entry whose numbers are close enough.

Showing the miss that motivates it

Two questions with one meaning, run against the exact in-memory cache, give two misses. The program prints the answer next to each one, so you can see that the second call paid for an answer it already had.

ExampleAPI keyTwo paraphrases against an exact cache
import litellm
from litellm import completion

litellm.cache = litellm.Cache()

for text in ["What is the capital of France?", "Which city is France's capital?"]:
    reply = completion(model="groq/openai/gpt-oss-20b", max_tokens=300,
                       messages=[{"role": "user", "content": text + " One word."}])
    hit = reply._hidden_params.get("cache_hit") is True
    print("hit " if hit else "miss", "|", reply.choices[0].message.content, "|", text)
  • Both lines say miss. Neither question was answered from the cache. The second one paid for a full call although it asked the same thing and got the same word.

Configuring a semantic cache

This part is not run here. A semantic cache needs two things the course keys do not provide. One is a Redis instance (or a Qdrant cluster) to store the vectors. The other is an embedding model to turn each question into one. The blocks below follow the example on the LiteLLM page for all cache types. The sentences right after them restate that page. The table and the warning further down are general reasoning, and nothing in them was measured here.

Redis semantic cache

python
import os
import litellm
from litellm import Cache

litellm.cache = Cache(
    type="redis-semantic",
    host=os.environ["REDIS_HOST"],
    port=os.environ["REDIS_PORT"],
    password=os.environ["REDIS_PASSWORD"],
    similarity_threshold=0.8,       # 1 means exact match, 0 means no similarity
    ttl=120,
    redis_semantic_cache_embedding_model="text-embedding-ada-002",
)

The docs list the requirements for this type: the redisvl package, a running Redis instance, and an embedding model. They say the model name is passed to litellm.embedding() and that any model that litellm.embedding() supports can be used. similarity_threshold is a number between 0 and 1: the closer to 1, the more alike two questions must be to share an answer.

Qdrant semantic cache

python
litellm.cache = Cache(
    type="qdrant-semantic",
    qdrant_api_base=os.environ["QDRANT_API_BASE"],
    qdrant_api_key=os.environ["QDRANT_API_KEY"],
    qdrant_collection_name="answers",
    similarity_threshold=0.7,
    qdrant_semantic_cache_embedding_model="text-embedding-ada-002",
    qdrant_semantic_cache_vector_size=1536,   # must match the embedding model
)

The docs call for a Qdrant cluster and an embedding model here too, and they say the vector size must match the embedding model's dimensions. No result of either configuration is shown in this lesson, and no similarity score is quoted, because none was measured here.

Exact cache vs semantic cache

Exact cacheSemantic cache
Matches onthe hash of the requestcloseness of meaning
Needsnothing outside Pythonan embedding model and a vector store
A paraphrasea misscan be a hit, depending on the threshold
Can return a wrong answer for a different questionnoyes, if the threshold is too low
Extra cost per requestnoneone embedding call

Where you use semantic caching

  • A public help bot where thousands of people ask the same ten questions in different words.
  • Search boxes backed by a model that see many near-duplicate queries.
  • Internal assistants with a stable set of policy questions.
Watch out. The threshold trades savings against wrong answers. Set too low, a question about cancelling an order can receive the stored answer to a question about returning one. Choose it by testing pairs of real questions, and keep personal or account-specific questions out of a shared semantic cache. Each lookup also costs an embedding call, so a cache that rarely hits can cost more than it saves.
Try it yourself
  • Run the first program with Cache(type="local") written out and confirm it still misses twice.
  • Add a third question that repeats the first one word for word. It hits, because exact matching still works for identical text.
  • Look up the embedding models your provider offers and write down the vector size of one, the number the Qdrant example needs.
PreviousCache keys

Every expert started right here.