Using the OpenAI client with the gateway
An OpenAI client is the openai library's OpenAI class, and because the LiteLLM gateway speaks the same HTTP API, the class works against the gateway when you change only its base_url and api_key.
Last updated: 12 Oct, 2026 · LiteLLM 1.105
The LiteLLM proxy called the server with raw HTTP requests. Real programs use a client library, and the one most programs already use is the OpenAI one. This lesson shows that such a program does not change, apart from where it points.
Syntax: two arguments on the client. Everything after the client is built stays as it was.
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:4123", api_key="<the gateway key>")Pointing the client at the gateway
What a program calls today
A program written for OpenAI creates the client with an OpenAI key and calls a model by an OpenAI model name. This code is not run here, because the course has no OpenAI key.
client = OpenAI(api_key="sk-...") # an OpenAI key
reply = client.chat.completions.create(model="gpt-4o", messages=messages)The two changes
base_url is the address the client sends requests to. api_key is now the gateway's key. The third change is not in the client: model must be a model_name from the gateway's config, such as fast, because the gateway has no model called gpt-4o.
client = OpenAI(base_url=url, api_key=KEY) # changed
reply = client.chat.completions.create(model="fast", messages=messages) # model name changedCalling the gateway with a model name it does not have
The program below keeps its OpenAI-style function ask unchanged and calls the gateway first with gpt-4o, then with fast. The function reads the failure from the exception's body.
from openai import OpenAI
from gateway import running, KEY
def ask(client, model):
reply = client.chat.completions.create(
model=model, max_tokens=300,
messages=[{"role": "user", "content": "Name one lake. One word."}])
return reply.choices[0].message.content
with running(4123) as url:
client = OpenAI(base_url=url, api_key=KEY)
try:
ask(client, "gpt-4o")
except Exception as err:
print(type(err).__name__, "|", err.body["message"])
print(ask(client, "fast"))BadRequestError | /chat/completions: Invalid model name passed in model=gpt-4o. Call `/v1/models` to view available models for your key. Baikal
- BadRequestError and its message come back from the gateway: the model name
gpt-4ois not in its list, and the message names the/v1/modelsroute that lists the valid ones. - The last line is the answer to the same function with
model="fast". The function itself did not change. - The word changes from run to run, so this block is marked live.
Changing no code at all
The OpenAI library also reads OPENAI_BASE_URL and OPENAI_API_KEY from the environment when you create OpenAI() with no arguments. A program that does this needs no edit: you set two variables where it runs. patch.dict from the standard library sets them for the length of the with block, as a shell would for one command. This program also streams, so the answer arrives in pieces.
import os
from unittest.mock import patch
from openai import OpenAI
from gateway import running, KEY
with running(4123) as url, patch.dict(os.environ, {"OPENAI_BASE_URL": url, "OPENAI_API_KEY": KEY}):
client = OpenAI() # no arguments: it reads the two variables
stream = client.chat.completions.create(
model="fast", max_tokens=300, stream=True,
messages=[{"role": "user", "content": "Count from 1 to 3, separated by commas."}])
print("".join(chunk.choices[0].delta.content or "" for chunk in stream if chunk.choices))1, 2, 3
- The line of numbers is the streamed answer, assembled from the
delta.contentof each chunk. Streaming works through the gateway as it does against the OpenAI API. - The client line has no arguments, so the program's source never mentions the gateway. The wording of the answer changes from run to run, so this block is marked live.
OpenAI directly vs through the gateway
| Calling OpenAI | Calling the gateway | |
|---|---|---|
base_url | the library's default | the gateway's address |
api_key | an OpenAI key | the gateway's key |
model | an OpenAI model name | a model_name from the gateway's config |
| The rest of the code | unchanged | unchanged |
| Which provider answers | OpenAI | whatever the config maps the name to |
Where you use the OpenAI client on the gateway
- An existing app built on the OpenAI library that should move to another provider without a rewrite.
- A tool with a base URL setting, where you paste the gateway's address and key.
- A test run that points a program at the gateway to see which models it can reach.
Related
- Previous: The LiteLLM proxy
- Next: Hybrid SDK, Router and gateway
- Reference: LiteLLM AI Gateway in the LiteLLM docs
- Change
model="fast"to"smart"in the second program and run it again. - In the first program change the base URL to
url + "/v1"and run it. It still works, because the proxy answers both paths. - Remove
stream=Trueand the join line, and printreply.choices[0].message.contentinstead.
Slow is fine. Stopping is the only problem.