Gateway caching
Gateway caching is a response cache that lives in the LiteLLM proxy server, so every application that calls the gateway shares one set of cached answers.
Last updated: 12 Oct, 2026 · LiteLLM 1.105
The cache in Caching responses lived inside one Python program. Put the cache in a server that all programs call and a question answered for one application is free for the next. This lesson turns the cache on in the proxy, which The LiteLLM proxy explains in full. Here the server is started with a config file and called with the OpenAI client.
Turning the cache on in the config
Syntax: two keys under litellm_settings in the proxy's YAML config.
litellm_settings:
cache: true
cache_params:
type: local # in memory, inside the proxy processThe config file and the starter
The setup program below writes two files. cache.yaml holds one model and the cache settings. master_key is the secret that callers must send, and os.environ/LITELLM_MASTER_KEY tells the proxy to read it from an environment variable. The proxy refuses to start without one, as The LiteLLM proxy shows. gateway.py is a helper that starts the litellm command on a port, waits until the server answers, hands back its address and stops the server when the with block ends, even after an error. That lesson explains it line by line. It is here so one program can run the server and call it.
import sys
from pathlib import Path
Path("cache.yaml").write_text("""model_list:
- model_name: fast
litellm_params:
model: groq/openai/gpt-oss-20b
api_key: os.environ/GROQ_API_KEY
general_settings:
master_key: os.environ/LITELLM_MASTER_KEY
litellm_settings:
cache: true
cache_params:
type: local
""")
Path("gateway.py").write_text("""import os, subprocess, sys, time, urllib.request
from contextlib import contextmanager
KEY = "sk-gateway-demo-0123456789abcdef"
@contextmanager
def running(port, config="config.yaml"):
env = {**os.environ, "LITELLM_MASTER_KEY": KEY}
litellm = os.path.join(os.path.dirname(sys.executable), "litellm")
cmd = [litellm, "--config", config, "--port", str(port)]
proc = subprocess.Popen(cmd, env=env, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
try:
for _ in range(60):
try:
urllib.request.urlopen(f"http://127.0.0.1:{port}/health/readiness")
break
except OSError:
time.sleep(1)
yield f"http://127.0.0.1:{port}"
finally:
proc.terminate()
proc.wait(10)
""")
sys.path.insert(0, ".") # lets the programs that follow import gateway from this folder
print("wrote cache.yaml and gateway.py")wrote cache.yaml and gateway.py
Calling the cached gateway from separate clients
Each call below builds a new OpenAI client, the way a separate application would. with_raw_response gives access to the HTTP headers, and a cached answer carries an x-litellm-cache-key header, as the proxy caching page of the docs says. The last call asks for a fresh answer with extra_body={"cache": {"no-cache": True}}.
import time
from openai import OpenAI
from gateway import running, KEY
question = [{"role": "user", "content": "Name one lake. One word."}]
with running(4124, config="cache.yaml") as url:
for who, extra in [("app A", {}), ("app A again", {}), ("app B", {}), ("app B, no-cache", {"cache": {"no-cache": True}})]:
client = OpenAI(base_url=url, api_key=KEY) # a new client each time
start = time.perf_counter()
raw = client.chat.completions.with_raw_response.create(
model="fast", messages=question, max_tokens=300, extra_body=extra)
ms = (time.perf_counter() - start) * 1000
hit = "x-litellm-cache-key" in raw.headers
print(f"{who:16} {'hit ' if hit else 'miss'} {ms:7.1f} ms", raw.parse().choices[0].message.content)app A miss 1074.0 ms Baikal app A again hit 11.7 ms Baikal app B hit 5.1 ms Baikal app B, no-cache miss 573.8 ms Tahoe
Reading the gateway's cache
- app A: miss is the first question. The gateway called Groq, so it took the time of a real call.
- app A again: hit carries the
x-litellm-cache-keyheader and returns in milliseconds with the same word. - app B: hit comes from a brand-new client object. The cache is in the server, so a client that never asked this question before still gets the stored answer.
- app B, no-cache: miss asked the provider again. It took the time of a real call, which shows the cache was skipped. The word can differ from the cached one.
- The times and the words change from run to run, so this block is marked live. The hit and miss labels do not.
Gateway cache vs SDK cache
| SDK cache | Gateway cache | |
|---|---|---|
| Where it lives | inside your Python process | inside the proxy server |
| Shared between programs | no | yes, every caller of the server |
| Works for non-Python callers | no | yes, any OpenAI client |
| Survives your program restarting | no | yes, while the server runs |
| Shared between several proxy workers or machines | no | only with a Redis cache |
Moving the gateway cache to Redis
This part is not run here. The in-memory type: local cache is for development: the docs say it is not shared between workers or replicas. Production setups use Redis, and the docs give the connection details as environment variables.
litellm_settings:
cache: true
cache_params:
type: redis # host, port and password come from REDIS_HOST, REDIS_PORT, REDIS_PASSWORDWhere you use gateway caching
- Several services that ask a model the same reference questions.
- A team where many people run the same demo prompts against one shared gateway.
- Load tests, where a repeated prompt should measure the gateway and not the provider.
Related
- Previous: Semantic caching
- Next: The LiteLLM proxy
- Reference: Caching on the proxy and Cache controls in the LiteLLM docs
- Change the question text in the program. The first call of the new text is a miss again.
- Add
"no-store": Trueto the first call'sextra_body. The second call is a miss, because nothing was saved. - Change
4124to another free port number in the program and run it again.
Little by little, you're building something great.