LiteLLMLiteLLM 1.105 · Python 3.10+
Dashboard
0%

The gateway in a container

A gateway container is the LiteLLM proxy packaged as a Docker image, so the same server starts the same way on a laptop, a virtual machine or a cluster.

Last updated: 12 Oct, 2026 · LiteLLM 1.105

The LiteLLM proxy started the gateway with a command in your own Python environment. A company does not deploy a Python environment. It deploys an image, and this lesson shows the two ways the LiteLLM documentation starts that image: a plain container with no database, and Docker Compose with Postgres beside it. Docker is not run in this course, so the Docker commands are shown from the documentation and carry no output.

An application sends requests to a LiteLLM gateway running inside a Docker container on port 4000. The container reads config.yaml, mounted at /app/config.yaml, and environment variables such as <code>LITELLM_MASTER_KEY</code> and the provider keys. The gateway forwards requests to model providers. A dashed arrow from the gateway to a Postgres database holding keys, teams and spend is present only when the Compose file is used; a plain docker run has no database.

Syntax: a container is started with docker run. The config file is mounted with -v, secrets arrive with -e, the port is published with -p, and --config names the file inside the container.

bash
docker run \
  -v $(pwd)/litellm_config.yaml:/app/config.yaml \
  -e GROQ_API_KEY=<your-groq-key> \
  -e LITELLM_MASTER_KEY=sk-<a-long-random-key> \
  -p 4000:4000 \
  docker.litellm.ai/berriai/litellm:latest \
  --config /app/config.yaml

Not run here. This block needs Docker and shows no output.

Running the image with a config file

Mounting the config and passing keys

The command above is the one on the Docker quick start page. -v maps a file on your machine to /app/config.yaml inside the container, and --config /app/config.yaml makes the gateway read it. -e sets an environment variable inside the container. LITELLM_MASTER_KEY is the root credential of the gateway, and the page says the proxy will not start without it. -p 4000:4000 publishes port 4000, the port the gateway listens on.

The provider key stays out of the file. A config line such as api_key: os.environ/OPENAI_API_KEY tells the gateway to read the key from the container's environment when it starts, so the file can be committed without a secret in it.

Knowing what this container lacks

The same page says this form has no database. Without one there is no model management in the admin interface, no virtual keys and no spend tracking. A request that carries a virtual key fails with No connected db. The page also says litellm_settings.max_budget is not a spend cap on this path. The proxy logs a one-time warning at startup when a budget is set with no database, and then carries on. A container started this way answers requests and applies the config file, and nothing more.

Adding Postgres with Docker Compose

The documentation's second option brings up the gateway and a Postgres database together from a Compose file. The page says the database stores your models, keys and spend logs. These commands download the file, write a .env with two random keys, and start both services.

bash
curl -sSLO https://github.com/BerriAI/litellm/raw/main/docker/docker-compose.quickstart.yml
printf 'LITELLM_MASTER_KEY=sk-%s\nLITELLM_SALT_KEY=sk-%s\n' \
  "$(openssl rand -hex 32)" "$(openssl rand -hex 32)" > .env
docker compose -f docker-compose.quickstart.yml up -d

Not run here. This block needs Docker and shows no output.

  • LITELLM_MASTER_KEY is the root credential. The page says the admin interface at http://localhost:4000/ui accepts the user name admin and this key as the password.
  • LITELLM_SALT_KEY encrypts the provider API keys that you add in the admin interface. The page says it has no in-place rotation, so keep the generated value, which means keeping the .env file.
  • The database connection is set up by the Compose file. A gateway started another way names its database with the DATABASE_URL variable, as the virtual keys page describes.

A gateway started from pip needs one more package to talk to that database. The prisma package is listed under the extra-proxy extra of litellm, and it is not part of litellm[proxy]. The next lesson shows what a gateway without it does when asked for a key.

Testing the config outside Docker

Starting a real gateway from Python

A container cannot be started here, but the file inside it is the same config.yaml the pip-installed gateway reads. Every example in this part starts a real gateway from a config file, calls it and stops it. A small function does the starting: it runs the litellm command in a child process with config.yaml and a port, writes its log to gateway.log, and waits until the gateway answers at /health/liveliness. Each example repeats it, so that each one runs on its own.

python
def start(port=4400, env=None):
    proc = subprocess.Popen(
        ["litellm", "--config", "config.yaml", "--port", str(port)],
        stdout=open("gateway.log", "w"), stderr=subprocess.STDOUT, env=env,
    )
    while True:
        try:
            urllib.request.urlopen(f"http://127.0.0.1:{port}/health/liveliness")
            return proc
        except OSError:
            time.sleep(1)

The caller stops the process in a finally block with proc.terminate(), so a failed example cannot leave a server running. Each lesson writes its own config.yaml, replacing the one before it.

Reading the master key from the environment

The Docker page passes the master key as an environment variable and not in the file. The example sets the same variable for the child process, with a config that holds no master_key line, and asks the gateway which models it serves.

ExampleA real gateway started with the master key in its environment
import os
import subprocess
import time
import urllib.request
from pathlib import Path

import httpx


def start(port=4400, env=None):
    proc = subprocess.Popen(
        ["litellm", "--config", "config.yaml", "--port", str(port)],
        stdout=open("gateway.log", "w"), stderr=subprocess.STDOUT, env=env,
    )
    while True:
        try:
            urllib.request.urlopen(f"http://127.0.0.1:{port}/health/liveliness")
            return proc
        except OSError:
            time.sleep(1)


Path("config.yaml").write_text("""model_list:
  - model_name: careful
    litellm_params:
      model: groq/openai/gpt-oss-120b
      api_key: os.environ/GROQ_API_KEY
""")

proc = start(env={**os.environ, "LITELLM_MASTER_KEY": "sk-demo-1234"})
try:
    r = httpx.get("http://127.0.0.1:4400/v1/models", headers={"Authorization": "Bearer sk-demo-1234"})
    print(r.status_code, [m["id"] for m in r.json()["data"]])
finally:
    proc.terminate()
    proc.wait()

What the container run proved

  • The key came from the environment. The config had no master_key line, and the gateway accepted sk-demo-1234 because the process was started with LITELLM_MASTER_KEY set, the way -e sets it in a container.
  • The model group has a name of its own. The gateway lists careful, the model_name from the file, and not the provider's model string. Applications ask for the group.
  • The config is portable. The container reads the same file at /app/config.yaml. What worked here with --config config.yaml is what the image gets, with the same keys supplied through -e.

Plain container vs Compose

docker runDocker Compose quick start
DatabaseNonePostgres, started beside the gateway
Virtual keys and spend trackingNot availableAvailable
Admin interfaceNo model managementSign in at /ui
Budgetsmax_budget not enforcedChecked against spend stored in the database
Use it forA single shared model endpointA gateway several teams use

Where a container gateway fits

  • One shared endpoint for a team. A plain container with a config file gives every service the same model names and the same fallbacks.
  • A company gateway with accounts. Compose, or any container platform with a Postgres database, adds virtual keys, budgets and logs.
  • A reviewed deployment. The config file is the thing reviewed in a pull request, and the image tag is the thing promoted between environments.
Watch out. The Docker run example on the page uses the latest tag. latest moves when a new release is published. A service other applications depend on should run a release you chose and tested. The page tells you to edit the downloaded Compose file to pin a specific release tag, and it does not say how to find the tag names, so read the registry or the release notes.
Try it yourself
  • Change LITELLM_MASTER_KEY in the example to another value and call /v1/models with the old one. Print the status code and read what the gateway answers.
  • Add a second model group to the config, with the same Groq key, and print the list of ids again.
  • If you have Docker, run the docker run command with your own config and key, and call the container on port 4000 the same way.
PreviousGateway keys

Little by little, you're building something great.