LiteLLMLiteLLM 1.105 · Python 3.10+
Dashboard
0%

Hybrid SDK, Router and gateway

A hybrid setup is a design in which different parts of a system call models in different ways, using the SDK, the Router or the gateway where each one fits.

Last updated: 12 Oct, 2026 · LiteLLM 1.105

Using the OpenAI client with the gateway showed a program calling the gateway. LiteLLM gives three ways to call a model, and each earlier lesson used one of them. This lesson sets them side by side and says when each fits.

Three ways to call a model. The SDK: your program calls completion and reaches the provider directly. The Router: your program calls router.completion and the Router chooses a deployment, then reaches the provider. The gateway: any program sends an OpenAI-style request to the proxy server, which forwards it to the provider.

Calling a model three ways

The SDK

completion() sends one request to one provider from inside your Python program. The provider key sits in your program's environment. It has the fewest parts, so it is the right start. completion(): your first call introduced it.

python
completion(model="groq/openai/gpt-oss-20b", messages=messages)

The Router

Router is still a Python object inside your program. It holds a model list and chooses among deployments, retries and falls back. It earns its place when one group has several deployments. Router: model groups and deployments and Fallbacks: another model group when one fails cover it.

python
router.completion(model="fast", messages=messages)

The gateway

The gateway is a separate server. Programs in any language send it OpenAI-style requests. It holds the provider keys, the model list and the shared cache, and it can apply the Router's retries and fallbacks from its config.

python
client = OpenAI(base_url=url, api_key=KEY)
client.chat.completions.create(model="fast", messages=messages)

Mixing the three

The ways combine. A Python program can use the SDK's features and still send its requests through the gateway: the litellm_proxy/ prefix on the model name routes a completion() call to a proxy, with the proxy's address as api_base and its key as api_key, as the LiteLLM docs describe on the proxy provider page.

ExampleAPI keycompletion() through the gateway
import litellm
from gateway import running, KEY

with running(4123) as url:
    reply = litellm.completion(
        model="litellm_proxy/fast", api_base=url, api_key=KEY, max_tokens=300,
        messages=[{"role": "user", "content": "Name one lake. One word."}])
    print(reply.choices[0].message.content, "|", reply.model)
  • The word is the proxy's answer, written by the model behind fast. It changes from run to run, so this block is marked live.
  • litellm_proxy/fast is the model name as the SDK reports it: the prefix says the request went through a proxy, and fast is the name in the proxy's config.

SDK vs Router vs gateway

SDKRouterGateway
Runs asa function in your programan object in your programa separate server
CallersPython onlyPython onlyany language
Provider keys held byeach programeach programthe server
Retries, fallbacks, routingnoneyesyes, set in the config
Shared cache across programsnonoyes
Extra parts to runnonenoneone server

Choosing a way

  • One script or one service, one provider: the SDK. A server would be an extra part with nothing to share.
  • One Python service with several deployments of a model: the Router, for retries and fallbacks without a server.
  • Several programs, several languages or several teams: the gateway, so keys, models and cache live in one place.
  • A Python app inside a larger company setup: the SDK with the litellm_proxy/ prefix, so the app keeps the SDK's features while the platform team owns the keys.
Watch out. A gateway is one more server that can fail, and every request now depends on it. Run it where you can restart it, and keep a plan for the case where it is down. A single script does not need one.
Try it yourself
  • Change model in the program to "litellm_proxy/smart" and run it. The 120b model answers.
  • Remove the litellm_proxy/ prefix and read the error the SDK raises for the model name fast.
  • Write one sentence for each program you maintain saying which of the three ways it should use.

This is what real progress feels like.