LiteLLMLiteLLM 1.105 · Python 3.10+
Dashboard
0%

The LiteLLM proxy

The LiteLLM proxy is a server, started with the litellm command, that offers one OpenAI-compatible HTTP API and forwards each request to the provider that you configured.

Last updated: 12 Oct, 2026 · LiteLLM 1.105

Gateway caching already used a server to share a cache. This lesson explains that server. Until now every program imported litellm and held its own provider keys. With the proxy, the keys and the model list live in one place, and a program only needs the server's address and a key for the server.

Three applications, one written in Python, one in JavaScript and one a command line tool, all send OpenAI-style requests to the LiteLLM proxy. The proxy holds the Groq and Gemini keys and forwards each request to Groq or to Gemini.

Syntax: one command starts the server from a config file. --port is the port it listens on, and the docs name 4000 as the default.

bash
litellm --config config.yaml --port 4000

Writing the proxy config

Listing the models

model_list is the same idea as the Router's. Each entry has a model_name, the name that callers will put in their request, and litellm_params, which says which real model answers and which key it needs. os.environ/GROQ_API_KEY tells the proxy to read the key from that environment variable, so the config file holds no secret.

python
model_list:
  - model_name: fast                         # what callers ask for
    litellm_params:
      model: groq/openai/gpt-oss-20b         # what answers
      api_key: os.environ/GROQ_API_KEY       # read from the environment

Setting the master key

master_key is the secret that callers send to prove they may use the server. LiteLLM 1.105 refuses to start without one. The next two sections show why, and Gateway keys covers keys in depth.

python
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

Running the server from Python

An example needs to start the server, wait for it, call it and stop it. gateway.py does the first, second and last. It is the same file the previous lesson wrote.

Starting the process

subprocess.Popen runs the litellm command that pip installed next to your Python. The master key goes into the child's environment, and its log output is thrown away.

python
env = {**os.environ, "LITELLM_MASTER_KEY": KEY}
litellm = os.path.join(os.path.dirname(sys.executable), "litellm")
cmd = [litellm, "--config", config, "--port", str(port)]
proc = subprocess.Popen(cmd, env=env, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)

Waiting until the server answers

The server needs a few seconds to start. /health/readiness is a URL the proxy answers once it is ready, so the loop asks it once a second for up to a minute.

python
for _ in range(60):
    try:
        urllib.request.urlopen(f"http://127.0.0.1:{port}/health/readiness")
        break
    except OSError:
        time.sleep(1)

Stopping it in a finally block

A server that keeps running after an error holds its port. The finally block runs whether the code inside the with block succeeds or fails, so the server always stops.

python
try:
    yield f"http://127.0.0.1:{port}"     # the caller's code runs here
finally:
    proc.terminate()
    proc.wait(10)
ExampleWriting config.yaml and gateway.py
import sys
from pathlib import Path

Path("config.yaml").write_text("""model_list:
  - model_name: fast
    litellm_params:
      model: groq/openai/gpt-oss-20b
      api_key: os.environ/GROQ_API_KEY
  - model_name: smart
    litellm_params:
      model: groq/openai/gpt-oss-120b
      api_key: os.environ/GROQ_API_KEY

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
""")
Path("gateway.py").write_text("""import os, subprocess, sys, time, urllib.request
from contextlib import contextmanager

KEY = "sk-gateway-demo-0123456789abcdef"

@contextmanager
def running(port, config="config.yaml"):
    env = {**os.environ, "LITELLM_MASTER_KEY": KEY}
    litellm = os.path.join(os.path.dirname(sys.executable), "litellm")
    cmd = [litellm, "--config", config, "--port", str(port)]
    proc = subprocess.Popen(cmd, env=env, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
    try:
        for _ in range(60):
            try:
                urllib.request.urlopen(f"http://127.0.0.1:{port}/health/readiness")
                break
            except OSError:
                time.sleep(1)
        yield f"http://127.0.0.1:{port}"
    finally:
        proc.terminate()
        proc.wait(10)
""")
sys.path.insert(0, ".")      # lets the programs that follow import gateway from this folder
print("wrote config.yaml and gateway.py")

Starting without a master key

The first thing a new user meets is the refusal. The program removes the key from the environment and runs the server in the foreground. The server exits on its own, so the program reads what it printed and keeps only the lines that start with the refusal.

ExampleA proxy that refuses to start
import os, re, subprocess, sys

env = {k: v for k, v in os.environ.items() if k != "LITELLM_MASTER_KEY"}
litellm = os.path.join(os.path.dirname(sys.executable), "litellm")
run = subprocess.run([litellm, "--config", "config.yaml", "--port", "4125"],
                     env=env, capture_output=True, text=True, timeout=90)
text = re.sub(r"\x1b\[[0-9;]*m", "", run.stdout + run.stderr)     # drop colour codes
print("stopped by itself:", run.returncode != 0)
print(*[line for line in text.splitlines() if line.startswith("LiteLLM proxy refused")], sep="\n")
  • stopped by itself: True says the process ended with a non-zero exit code and nobody had to stop it.
  • The refusal line is the server's own message: with no master key every request would be accepted without authentication, so it will not start. The full message also prints a command that generates a key.

Calling the running proxy

With a key set, the program starts the server and sends two HTTP requests with the standard library: one for the model list and one chat request. Authorization: Bearer followed by the key is how the OpenAI API takes a key, and the proxy uses the same header.

ExampleAPI keyModels, then one chat request
import json, urllib.request
from gateway import running, KEY

def call(url, path, body=None):
    request = urllib.request.Request(
        url + path, data=json.dumps(body).encode() if body else None,
        headers={"Authorization": f"Bearer {KEY}", "Content-Type": "application/json"})
    return json.load(urllib.request.urlopen(request))

with running(4123) as url:
    print([m["id"] for m in call(url, "/v1/models")["data"]])
    reply = call(url, "/v1/chat/completions", {
        "model": "fast", "max_tokens": 300,
        "messages": [{"role": "user", "content": "Name one lake. One word."}]})
    print(reply["choices"][0]["message"]["content"], "|", reply["model"])
  • ['fast', 'smart'] is the list the proxy built from model_list. Each name is a model_name, not a provider model.
  • The word was written by the 20b model on Groq. It changes from run to run, so this block is marked live.
  • fast after the bar is the model name in the reply: the proxy echoes the name the caller asked for.

Proxy vs SDK

The SDK in your programThe proxy server
Who needs provider keysevery programonly the server
Languages that can call itPythonany language that can send HTTP
Where the model list livesin each program's codein one config file
Extra process to runnoyes
Shared cache, budgets, loggingper programone place for every caller

Where you use the proxy

  • A team where every service calls one address and the provider keys stay with the platform owner.
  • Mixed languages, where a Python worker and a JavaScript front end need the same models.
  • Swapping a provider by changing one line of config.yaml and leaving every caller alone.
Watch out. A server that is started and never stopped keeps its port and keeps running in the background. Start it in a finally block or under a process manager, and use a port that is free: a second server on a taken port fails to start. The proxy also needs litellm[proxy], which the setup lesson installs.
Try it yourself
  • Change "fast" to "smart" in the chat request. The 120b model answers.
  • Change --port in the refusal program to 4000. The refusal is the same, because it happens before any port is opened.
  • Add a third model to config.yaml using gemini/gemini-2.5-flash-lite and print the model list again.

You understood something today that you didn't yesterday.