Caching: not paying twice for the same answer

Support teams see the same questions all day. A cache stores an answer by the exact request, so a repeat is answered without calling the model at all.

class CountingLLM(ShopLLM):
    def __init__(self):
        super().__init__()
        self.calls = 0

    def completion(self, model, messages, *args, **kwargs):
        self.calls += 1
        return super().completion(model, messages)

A ShopLLM that counts how often it is really called, so a cache hit can be seen as a call that did not happen.

import litellm
from litellm import completion
from litellm.caching.caching import Cache

from shop_llm import ShopLLM

class CountingLLM(ShopLLM):
    def __init__(self):
        super().__init__()
        self.calls = 0

    def completion(self, model, messages, *args, **kwargs):
        self.calls += 1
        return super().completion(model, messages)


counting = CountingLLM()
litellm.custom_provider_map = [{"provider": "shop", "custom_handler": counting}]
litellm.cache = Cache()

for text in ["I was charged twice", "I was charged twice", "I was charged twice!"]:
    response = completion(model="shop/small", messages=[{"role": "user", "content": text}], caching=True)
    print(f"{text!r:24} cache_hit={response._hidden_params.get('cache_hit')} calls={counting.calls}")

litellm.cache = Cache() turns on the in-memory cache, and caching=True uses it for a call. The second request was identical and came from the cache: the provider count stayed at 1. The third differs by one character, so it missed. The caching docs call this exact match: the key is a hash of the whole request.

What a cache is good for

Repeated, identical requests: the same prompt run over the same document, a classification asked again, a test suite. A conversation where each message includes the history before it rarely repeats exactly, so it rarely hits.

The in-memory cache lives inside one Python process. The gateway's caching page recommends Redis once there is more than one worker, since each process would otherwise keep its own separate cache. Semantic caches, which match similar rather than identical requests, exist too, and the docs warn they go badly wrong for agent traffic.