Calling the gateway from any OpenAI client

Because the gateway speaks the OpenAI API, any tool built for OpenAI can use it by changing two settings: the address and the key.

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:4000", api_key="sk-gateway-1234")

print([model.id for model in client.models.list().data])

reply = client.chat.completions.create(
    model="quick",
    messages=[{"role": "user", "content": "I was charged twice for one order"}],
)
print(reply.choices[0].message.content)

This is the official openai package. base_url points it at the gateway instead of OpenAI, and api_key is the gateway's key, not a provider's. client.models.list() calls /models, which lists the groups this key can use.

litellm --config config.yaml --port 4000 > gateway.log 2>&1 &
until curl -s localhost:4000/health/liveliness > /dev/null; do sleep 1; done
python app.py
kill %1

The app is ordinary OpenAI code. Changing the provider behind quick, or adding fallbacks, is an edit to config.yaml and a gateway restart; no app changes.

The same idea everywhere

Anything with an OpenAI-compatible base URL setting works the same way: agent frameworks, IDE assistants, the Vercel AI SDK. The provider keys stay on the gateway, so an app, or a laptop, holding only a gateway key cannot call a provider directly.