LiteLLMLiteLLM 1.105 · Python 3.10+
Dashboard
0%

Cache keys

A cache key is a fingerprint, computed from a request's model, messages and settings, that decides whether two requests count as the same one.

Last updated: 12 Oct, 2026 · LiteLLM 1.105

Caching responses showed an identical prompt hitting the cache. This lesson shows what identical means. Most cache misses come from a request that looks the same to a person and has a different fingerprint, and the fix starts with seeing the fingerprint.

Syntax: Cache.get_cache_key(**kwargs) takes the arguments you would give completion() and returns the key, a SHA-256 hash in hexadecimal. It makes no network call.

python
cache = litellm.Cache()
key = cache.get_cache_key(model=model, messages=messages, temperature=0.2)

Comparing keys of similar requests

Settings that change the key

The key is built from the model and the messages, plus the other arguments that shape an answer, such as temperature and max_tokens. Arguments that do not shape the answer, such as timeout and metadata, are left out. The program prints the first ten characters of each key, which is enough to compare them.

python
def key(**extra):
    return cache.get_cache_key(model=model, messages=messages, **extra)[:10]

print("temperature 0.2", key(temperature=0.2))
print("timeout 5      ", key(timeout=5))
ExampleKeys for one prompt with different settings
import litellm

cache = litellm.Cache()
model = "groq/openai/gpt-oss-20b"
messages = [{"role": "user", "content": "Name one planet."}]

def key(**extra):
    return cache.get_cache_key(model=model, messages=messages, **extra)[:10]

print("plain          ", key())
print("plain again    ", key())
print("temperature 0.2", key(temperature=0.2))
print("max_tokens 80  ", key(max_tokens=80))
print("timeout 5      ", key(timeout=5))
print("metadata       ", key(metadata={"team": "a"}))
  • plain and plain again print the same key, so those two requests share one cache entry.
  • temperature 0.2 and max_tokens 80 each print a new key. Each is a separate entry, because a different temperature or length can give a different answer.
  • timeout 5 and metadata print the plain key. A timeout and a label do not change the answer, so they do not change the entry.

Wording that changes the key

The text of the messages goes into the key exactly as written. A missing full stop, a capital letter or a timestamp in the prompt makes a new key.

ExampleKeys for five wordings
import litellm

cache = litellm.Cache()
model = "groq/openai/gpt-oss-20b"

def key(text):
    return cache.get_cache_key(model=model, messages=[{"role": "user", "content": text}])[:10]

print(key("Name one planet."))
print(key("Name one planet"))
print(key("name one planet."))
print(key("It is 10:41. Name one planet."))
print(key("It is 10:42. Name one planet."))
  • All five keys differ. A dropped full stop, a lower case letter and a different minute each give a new key, so each would be a separate request to the provider.
  • The two timestamp prompts are the costly case: a clock in the prompt means the same question never repeats and the cache never hits.

Running the keys against a real cache

The same rules decide hits. The program makes one request plain, one at temperature=0.5 and one in a namespace, then repeats each. A namespace is a prefix that keeps groups of entries apart, such as one per team. cache={"namespace": "team-b"} sets it on one call, as the cache controls page of the LiteLLM docs describes.

ExampleThree entries for one prompt
import litellm
from litellm import completion

litellm.cache = litellm.Cache()
messages = [{"role": "user", "content": "Name one desert. One word."}]

def ask(label, **options):
    reply = completion(model="groq/openai/gpt-oss-20b", messages=messages, max_tokens=300, **options)
    print(label, "hit" if reply._hidden_params.get("cache_hit") else "miss")

ask("plain             ")
ask("temperature 0.5   ", temperature=0.5)
ask("namespace team-b  ", cache={"namespace": "team-b"})
ask("plain again       ")
ask("temperature again ", temperature=0.5)
ask("team-b again      ", cache={"namespace": "team-b"})
  • The first three calls miss. They have three different keys, so three entries are made for one sentence.
  • The last three calls hit. Each repeats a request whose key is already stored.

Plain prompt vs changed prompt

Change to the requestNew cache entry?
Same messages, same settingsno, a hit
A different temperature or max_tokensyes
A different model nameyes
A different timeout or metadatano
A different namespaceyes
One changed character in a messageyes

Where you use cache keys

  • Raising the hit rate by stripping timestamps and request ids out of the prompt text.
  • Keeping teams apart with a namespace, so one team's cached answers are not served to another.
  • Debugging a miss by printing the keys of two requests that you expected to match.
Watch out. Which arguments count towards the key is decided by the installed version of LiteLLM, so check a request with get_cache_key instead of assuming. The key also covers the model name you pass: a request to a Router group name and a request to the real model name are different entries.
Try it yourself
  • Add user="u1" to key() in the first program. The key changes, which says the user field is part of it.
  • Add a fourth wording to the second program, with two spaces between two words, and print its key.
  • Run the third program with cache={"namespace": "team-a"} on the plain call. The plain-again call becomes a miss.

This is what real progress feels like.