LiteLLMLiteLLM 1.105 · Python 3.10+
Dashboard
0%

Gateway caching

Gateway caching is a response cache that lives in the LiteLLM proxy server, so every application that calls the gateway shares one set of cached answers.

Last updated: 12 Oct, 2026 · LiteLLM 1.105

The cache in Caching responses lived inside one Python program. Put the cache in a server that all programs call and a question answered for one application is free for the next. This lesson turns the cache on in the proxy, which The LiteLLM proxy explains in full. Here the server is started with a config file and called with the OpenAI client.

Turning the cache on in the config

Syntax: two keys under litellm_settings in the proxy's YAML config.

yaml
litellm_settings:
  cache: true
  cache_params:
    type: local          # in memory, inside the proxy process

The config file and the starter

The setup program below writes two files. cache.yaml holds one model and the cache settings. master_key is the secret that callers must send, and os.environ/LITELLM_MASTER_KEY tells the proxy to read it from an environment variable. The proxy refuses to start without one, as The LiteLLM proxy shows. gateway.py is a helper that starts the litellm command on a port, waits until the server answers, hands back its address and stops the server when the with block ends, even after an error. That lesson explains it line by line. It is here so one program can run the server and call it.

ExampleWriting cache.yaml and gateway.py
import sys
from pathlib import Path

Path("cache.yaml").write_text("""model_list:
  - model_name: fast
    litellm_params:
      model: groq/openai/gpt-oss-20b
      api_key: os.environ/GROQ_API_KEY

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

litellm_settings:
  cache: true
  cache_params:
    type: local
""")
Path("gateway.py").write_text("""import os, subprocess, sys, time, urllib.request
from contextlib import contextmanager

KEY = "sk-gateway-demo-0123456789abcdef"

@contextmanager
def running(port, config="config.yaml"):
    env = {**os.environ, "LITELLM_MASTER_KEY": KEY}
    litellm = os.path.join(os.path.dirname(sys.executable), "litellm")
    cmd = [litellm, "--config", config, "--port", str(port)]
    proc = subprocess.Popen(cmd, env=env, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
    try:
        for _ in range(60):
            try:
                urllib.request.urlopen(f"http://127.0.0.1:{port}/health/readiness")
                break
            except OSError:
                time.sleep(1)
        yield f"http://127.0.0.1:{port}"
    finally:
        proc.terminate()
        proc.wait(10)
""")
sys.path.insert(0, ".")      # lets the programs that follow import gateway from this folder
print("wrote cache.yaml and gateway.py")

Calling the cached gateway from separate clients

Each call below builds a new OpenAI client, the way a separate application would. with_raw_response gives access to the HTTP headers, and a cached answer carries an x-litellm-cache-key header, as the proxy caching page of the docs says. The last call asks for a fresh answer with extra_body={"cache": {"no-cache": True}}.

ExampleAPI keyFour calls to one gateway
import time
from openai import OpenAI
from gateway import running, KEY

question = [{"role": "user", "content": "Name one lake. One word."}]

with running(4124, config="cache.yaml") as url:
    for who, extra in [("app A", {}), ("app A again", {}), ("app B", {}), ("app B, no-cache", {"cache": {"no-cache": True}})]:
        client = OpenAI(base_url=url, api_key=KEY)          # a new client each time
        start = time.perf_counter()
        raw = client.chat.completions.with_raw_response.create(
            model="fast", messages=question, max_tokens=300, extra_body=extra)
        ms = (time.perf_counter() - start) * 1000
        hit = "x-litellm-cache-key" in raw.headers
        print(f"{who:16} {'hit ' if hit else 'miss'} {ms:7.1f} ms", raw.parse().choices[0].message.content)

Reading the gateway's cache

  • app A: miss is the first question. The gateway called Groq, so it took the time of a real call.
  • app A again: hit carries the x-litellm-cache-key header and returns in milliseconds with the same word.
  • app B: hit comes from a brand-new client object. The cache is in the server, so a client that never asked this question before still gets the stored answer.
  • app B, no-cache: miss asked the provider again. It took the time of a real call, which shows the cache was skipped. The word can differ from the cached one.
  • The times and the words change from run to run, so this block is marked live. The hit and miss labels do not.

Gateway cache vs SDK cache

SDK cacheGateway cache
Where it livesinside your Python processinside the proxy server
Shared between programsnoyes, every caller of the server
Works for non-Python callersnoyes, any OpenAI client
Survives your program restartingnoyes, while the server runs
Shared between several proxy workers or machinesnoonly with a Redis cache

Moving the gateway cache to Redis

This part is not run here. The in-memory type: local cache is for development: the docs say it is not shared between workers or replicas. Production setups use Redis, and the docs give the connection details as environment variables.

yaml
litellm_settings:
  cache: true
  cache_params:
    type: redis          # host, port and password come from REDIS_HOST, REDIS_PORT, REDIS_PASSWORD

Where you use gateway caching

  • Several services that ask a model the same reference questions.
  • A team where many people run the same demo prompts against one shared gateway.
  • Load tests, where a repeated prompt should measure the gateway and not the provider.
Watch out. One cache for every caller also means one person's cached answer can be served to another. Do not cache prompts that carry private data, or give each team its own namespace as Cache keys shows. The in-memory gateway cache is lost when the server stops.
Try it yourself
  • Change the question text in the program. The first call of the new text is a miss again.
  • Add "no-store": True to the first call's extra_body. The second call is a miss, because nothing was saved.
  • Change 4124 to another free port number in the program and run it again.

Little by little, you're building something great.