Semantic caching
A semantic cache is a cache that matches requests by meaning instead of by exact text, so a paraphrase of an earlier question gets the earlier answer.
Last updated: 12 Oct, 2026 · LiteLLM 1.105
Cache keys showed that one changed character gives a new key. People rarely ask the same thing in the same words, so an exact cache misses most paraphrases. A semantic cache turns each question into a list of numbers that captures its meaning and answers from the stored entry whose numbers are close enough.
Showing the miss that motivates it
Two questions with one meaning, run against the exact in-memory cache, give two misses. The program prints the answer next to each one, so you can see that the second call paid for an answer it already had.
import litellm
from litellm import completion
litellm.cache = litellm.Cache()
for text in ["What is the capital of France?", "Which city is France's capital?"]:
reply = completion(model="groq/openai/gpt-oss-20b", max_tokens=300,
messages=[{"role": "user", "content": text + " One word."}])
hit = reply._hidden_params.get("cache_hit") is True
print("hit " if hit else "miss", "|", reply.choices[0].message.content, "|", text)miss | Paris | What is the capital of France? miss | Paris | Which city is France's capital?
- Both lines say miss. Neither question was answered from the cache. The second one paid for a full call although it asked the same thing and got the same word.
Configuring a semantic cache
This part is not run here. A semantic cache needs two things the course keys do not provide. One is a Redis instance (or a Qdrant cluster) to store the vectors. The other is an embedding model to turn each question into one. The blocks below follow the example on the LiteLLM page for all cache types. The sentences right after them restate that page. The table and the warning further down are general reasoning, and nothing in them was measured here.
Redis semantic cache
import os
import litellm
from litellm import Cache
litellm.cache = Cache(
type="redis-semantic",
host=os.environ["REDIS_HOST"],
port=os.environ["REDIS_PORT"],
password=os.environ["REDIS_PASSWORD"],
similarity_threshold=0.8, # 1 means exact match, 0 means no similarity
ttl=120,
redis_semantic_cache_embedding_model="text-embedding-ada-002",
)The docs list the requirements for this type: the redisvl package, a running Redis instance, and an embedding model. They say the model name is passed to litellm.embedding() and that any model that litellm.embedding() supports can be used. similarity_threshold is a number between 0 and 1: the closer to 1, the more alike two questions must be to share an answer.
Qdrant semantic cache
litellm.cache = Cache(
type="qdrant-semantic",
qdrant_api_base=os.environ["QDRANT_API_BASE"],
qdrant_api_key=os.environ["QDRANT_API_KEY"],
qdrant_collection_name="answers",
similarity_threshold=0.7,
qdrant_semantic_cache_embedding_model="text-embedding-ada-002",
qdrant_semantic_cache_vector_size=1536, # must match the embedding model
)The docs call for a Qdrant cluster and an embedding model here too, and they say the vector size must match the embedding model's dimensions. No result of either configuration is shown in this lesson, and no similarity score is quoted, because none was measured here.
Exact cache vs semantic cache
| Exact cache | Semantic cache | |
|---|---|---|
| Matches on | the hash of the request | closeness of meaning |
| Needs | nothing outside Python | an embedding model and a vector store |
| A paraphrase | a miss | can be a hit, depending on the threshold |
| Can return a wrong answer for a different question | no | yes, if the threshold is too low |
| Extra cost per request | none | one embedding call |
Where you use semantic caching
- A public help bot where thousands of people ask the same ten questions in different words.
- Search boxes backed by a model that see many near-duplicate queries.
- Internal assistants with a stable set of policy questions.
Related
- Previous: Cache keys
- Next: Gateway caching
- Reference: Caching with all cache types in the LiteLLM docs
- Run the first program with
Cache(type="local")written out and confirm it still misses twice. - Add a third question that repeats the first one word for word. It hits, because exact matching still works for identical text.
- Look up the embedding models your provider offers and write down the vector size of one, the number the Qdrant example needs.
Every expert started right here.