LiteLLMLiteLLM 1.105 · Python 3.10+
Dashboard
0%

Using the OpenAI client with the gateway

An OpenAI client is the openai library's OpenAI class, and because the LiteLLM gateway speaks the same HTTP API, the class works against the gateway when you change only its base_url and api_key.

Last updated: 12 Oct, 2026 · LiteLLM 1.105

The LiteLLM proxy called the server with raw HTTP requests. Real programs use a client library, and the one most programs already use is the OpenAI one. This lesson shows that such a program does not change, apart from where it points.

Syntax: two arguments on the client. Everything after the client is built stays as it was.

python
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:4123", api_key="<the gateway key>")

Pointing the client at the gateway

What a program calls today

A program written for OpenAI creates the client with an OpenAI key and calls a model by an OpenAI model name. This code is not run here, because the course has no OpenAI key.

python
client = OpenAI(api_key="sk-...")                       # an OpenAI key
reply = client.chat.completions.create(model="gpt-4o", messages=messages)

The two changes

base_url is the address the client sends requests to. api_key is now the gateway's key. The third change is not in the client: model must be a model_name from the gateway's config, such as fast, because the gateway has no model called gpt-4o.

python
client = OpenAI(base_url=url, api_key=KEY)               # changed
reply = client.chat.completions.create(model="fast", messages=messages)   # model name changed

Calling the gateway with a model name it does not have

The program below keeps its OpenAI-style function ask unchanged and calls the gateway first with gpt-4o, then with fast. The function reads the failure from the exception's body.

ExampleAPI keyAn unknown model name, then a configured one
from openai import OpenAI
from gateway import running, KEY

def ask(client, model):
    reply = client.chat.completions.create(
        model=model, max_tokens=300,
        messages=[{"role": "user", "content": "Name one lake. One word."}])
    return reply.choices[0].message.content

with running(4123) as url:
    client = OpenAI(base_url=url, api_key=KEY)
    try:
        ask(client, "gpt-4o")
    except Exception as err:
        print(type(err).__name__, "|", err.body["message"])
    print(ask(client, "fast"))
  • BadRequestError and its message come back from the gateway: the model name gpt-4o is not in its list, and the message names the /v1/models route that lists the valid ones.
  • The last line is the answer to the same function with model="fast". The function itself did not change.
  • The word changes from run to run, so this block is marked live.

Changing no code at all

The OpenAI library also reads OPENAI_BASE_URL and OPENAI_API_KEY from the environment when you create OpenAI() with no arguments. A program that does this needs no edit: you set two variables where it runs. patch.dict from the standard library sets them for the length of the with block, as a shell would for one command. This program also streams, so the answer arrives in pieces.

ExampleAPI keyOpenAI() with the base URL in the environment
import os
from unittest.mock import patch
from openai import OpenAI
from gateway import running, KEY

with running(4123) as url, patch.dict(os.environ, {"OPENAI_BASE_URL": url, "OPENAI_API_KEY": KEY}):
    client = OpenAI()                       # no arguments: it reads the two variables
    stream = client.chat.completions.create(
        model="fast", max_tokens=300, stream=True,
        messages=[{"role": "user", "content": "Count from 1 to 3, separated by commas."}])
    print("".join(chunk.choices[0].delta.content or "" for chunk in stream if chunk.choices))
  • The line of numbers is the streamed answer, assembled from the delta.content of each chunk. Streaming works through the gateway as it does against the OpenAI API.
  • The client line has no arguments, so the program's source never mentions the gateway. The wording of the answer changes from run to run, so this block is marked live.

OpenAI directly vs through the gateway

Calling OpenAICalling the gateway
base_urlthe library's defaultthe gateway's address
api_keyan OpenAI keythe gateway's key
modelan OpenAI model namea model_name from the gateway's config
The rest of the codeunchangedunchanged
Which provider answersOpenAIwhatever the config maps the name to

Where you use the OpenAI client on the gateway

  • An existing app built on the OpenAI library that should move to another provider without a rewrite.
  • A tool with a base URL setting, where you paste the gateway's address and key.
  • A test run that points a program at the gateway to see which models it can reach.
Watch out. The client only changes where requests go. Features that a model behind the gateway does not support still fail, whichever client sends them, so test the calls your program makes. Keep the gateway key out of source control: read it from the environment, as the second program does.
Try it yourself
  • Change model="fast" to "smart" in the second program and run it again.
  • In the first program change the base URL to url + "/v1" and run it. It still works, because the proxy answers both paths.
  • Remove stream=True and the join line, and print reply.choices[0].message.content instead.

Slow is fine. Stopping is the only problem.