Hybrid SDK, Router and gateway
A hybrid setup is a design in which different parts of a system call models in different ways, using the SDK, the Router or the gateway where each one fits.
Last updated: 12 Oct, 2026 · LiteLLM 1.105
Using the OpenAI client with the gateway showed a program calling the gateway. LiteLLM gives three ways to call a model, and each earlier lesson used one of them. This lesson sets them side by side and says when each fits.
Calling a model three ways
The SDK
completion() sends one request to one provider from inside your Python program. The provider key sits in your program's environment. It has the fewest parts, so it is the right start. completion(): your first call introduced it.
completion(model="groq/openai/gpt-oss-20b", messages=messages)The Router
Router is still a Python object inside your program. It holds a model list and chooses among deployments, retries and falls back. It earns its place when one group has several deployments. Router: model groups and deployments and Fallbacks: another model group when one fails cover it.
router.completion(model="fast", messages=messages)The gateway
The gateway is a separate server. Programs in any language send it OpenAI-style requests. It holds the provider keys, the model list and the shared cache, and it can apply the Router's retries and fallbacks from its config.
client = OpenAI(base_url=url, api_key=KEY)
client.chat.completions.create(model="fast", messages=messages)Mixing the three
The ways combine. A Python program can use the SDK's features and still send its requests through the gateway: the litellm_proxy/ prefix on the model name routes a completion() call to a proxy, with the proxy's address as api_base and its key as api_key, as the LiteLLM docs describe on the proxy provider page.
import litellm
from gateway import running, KEY
with running(4123) as url:
reply = litellm.completion(
model="litellm_proxy/fast", api_base=url, api_key=KEY, max_tokens=300,
messages=[{"role": "user", "content": "Name one lake. One word."}])
print(reply.choices[0].message.content, "|", reply.model)Tahoe | litellm_proxy/fast
- The word is the proxy's answer, written by the model behind
fast. It changes from run to run, so this block is marked live. - litellm_proxy/fast is the model name as the SDK reports it: the prefix says the request went through a proxy, and
fastis the name in the proxy's config.
SDK vs Router vs gateway
| SDK | Router | Gateway | |
|---|---|---|---|
| Runs as | a function in your program | an object in your program | a separate server |
| Callers | Python only | Python only | any language |
| Provider keys held by | each program | each program | the server |
| Retries, fallbacks, routing | none | yes | yes, set in the config |
| Shared cache across programs | no | no | yes |
| Extra parts to run | none | none | one server |
Choosing a way
- One script or one service, one provider: the SDK. A server would be an extra part with nothing to share.
- One Python service with several deployments of a model: the Router, for retries and fallbacks without a server.
- Several programs, several languages or several teams: the gateway, so keys, models and cache live in one place.
- A Python app inside a larger company setup: the SDK with the
litellm_proxy/prefix, so the app keeps the SDK's features while the platform team owns the keys.
Related
- Previous: Using the OpenAI client with the gateway
- Next: Gateway keys
- Reference: LiteLLM Proxy as a provider in the LiteLLM docs
- Change
modelin the program to"litellm_proxy/smart"and run it. The 120b model answers. - Remove the
litellm_proxy/prefix and read the error the SDK raises for the model namefast. - Write one sentence for each program you maintain saying which of the three ways it should use.
This is what real progress feels like.