Generative AI and Agentic AILangChain 1.4 · LangGraph 1.2 · Python 3.12 or 3.13
Dashboard
0%

LLM gateways

An LLM gateway is a layer between an application and its model providers that gives every model one call format and handles fallbacks, routing, caching, cost tracking and guardrails in one place.

Last updated: 08 Oct, 2026 · LiteLLM 1.104

Each provider has its own SDK, its own errors and its own prices. An app that calls them directly repeats that work, and it stops working when its one provider does. A gateway moves that work out of the app and into a single place that every app shares.

What an LLM gateway is · from the Complete Agentic AI Course video · 10:31:11 to 10:35:13

What a gateway does for an app

The video's story: a startup has a chatbot on one provider, a RAG app on a second and a third app on a third. Each needed its own integration code, so each failed on its own terms when its provider did. The video recalls an OpenAI outage in November 2023, when apps tied to that one provider stopped answering. With a gateway in the middle, the apps send every request to one address, and the gateway forwards it to a provider. Which provider, and what to do when one fails, is a change of configuration and not of app code.

Three applications, a chatbot, a RAG app and an agent, each send requests to one LLM gateway. The gateway box lists one call format, fallbacks and retries, routing and load balancing, caching, cost tracking, and guardrails and logs. From the gateway a solid arrow goes to Groq, the model the app asked for, and dashed arrows go to Gemini and OpenRouter, where a request goes when the first one fails.

The video lists eight capabilities. The sections below show the unified API, fallbacks, cost tracking, routing, caching and a guardrail in code.

CapabilityWhat it does
Unified APIOne function and one message format for every provider
FallbacksMove to a backup model when the first one fails
Smart routingSend each kind of request to the model that suits it
Load balancingSpread requests over several keys or deployments behind one name
CachingAnswer a repeated request from storage and skip the model call
ObservabilityLog every call with its prompt, reply, tokens and cost
GuardrailsKeep sensitive input from reaching a provider
EvalsPlug an evaluation framework into the same traffic

Calling two providers with one function

The video's gateway is LiteLLM, an open-source Python library that also runs as a server. Its completion() takes a model string and a list of messages and returns the same response object for every provider. The string is the provider's name, a slash and the model: groq/openai/gpt-oss-120b or gemini/gemini-2.5-flash. The video uses OpenAI, Anthropic and Gemini 1.5 models; the page uses Groq and Gemini 2.5 Flash, which need only a Groq key and a Gemini key.

pip install litellm==1.104.0

LiteLLM is its own install, separate from the LangChain packages. It reads GROQ_API_KEY for Groq; for Gemini, the run below used the GOOGLE_API_KEY from the Gemini setup and LiteLLM accepted it.

python
from litellm import completion

messages = [{"role": "user", "content": "Explain RAG in one sentence."}]

completion(model="groq/openai/gpt-oss-120b", messages=messages)    # Groq
completion(model="gemini/gemini-2.5-flash", messages=messages)     # Gemini: only the string changes
Fallbacks and cost tracking · from the Complete Agentic AI Course video · 10:49:15 to 10:53:06

The video's models in this part are OpenAI's gpt-4o-mini and Gemini 1.5 Flash, and its fallback list names them. The page keeps the same two ideas with current models.

Adding a fallback

fallbacks is a list of model strings. LiteLLM tries the primary model first, and only when that call raises an error does it move to the next entry in the list. The app sees one response either way. Here the primary is a model name that does not exist, so the call has to fall back.

python
response = completion(
    model="groq/not-a-real-model",               # primary: this model does not exist
    messages=messages,
    fallbacks=["gemini/gemini-2.5-flash"],       # tried when the primary fails
)

Reading the cost of a call

completion_cost turns a response into dollars using LiteLLM's built-in price table. The provider name is passed so the lookup uses the right row of that table.

python
from litellm import completion_cost

completion_cost(completion_response=response, custom_llm_provider="gemini")   # dollars for this call

Running a call that falls back and reports its cost

One call, with both pieces together. LiteLLM writes the failed first attempt to the terminal's error stream; the printed lines below are the standard output.

ExampleAPI keyRun with litellm 1.104.0 in its own environment, Groq key and Gemini key from the course; the reply is shown as it came back
import litellm
from litellm import completion, completion_cost

litellm.suppress_debug_info = True               # hide the "Give Feedback" banner on errors

response = completion(
    model="groq/not-a-real-model",               # primary: this model does not exist
    messages=[{"role": "user", "content": "Explain RAG in one sentence."}],
    fallbacks=["gemini/gemini-2.5-flash"],       # tried when the primary fails
)
usage = response.usage
print("answered by:", response.model)
print(response.choices[0].message.content)
print("tokens in/out:", usage.prompt_tokens, usage.completion_tokens)
print("thinking tokens:", usage.completion_tokens_details.reasoning_tokens)
print("cost: $", round(completion_cost(completion_response=response, custom_llm_provider="gemini"), 6))

Reading the fallback call

  • answered by: gemini-2.5-flash. The primary, groq/not-a-real-model, does not exist, so the call failed and LiteLLM moved to the entry in fallbacks. The failed first attempt was written to the terminal's error stream, not to the printed lines.
  • The reply has the usual shape. It is read with response.choices[0].message.content, as for any provider, so the app's code does not change when a fallback answers.
  • 8 tokens in, 934 out, 897 of them thinking tokens. Gemini 2.5 Flash reasons before it answers, and its thinking tokens are part of the output count. Only the remaining 37 tokens are the reply you read.
  • Cost: $0.002337. LiteLLM computed it from its price table. Almost all of it is the output side: output tokens are priced higher than input tokens, and the thinking tokens are billed as output.
Smart routing with model aliases · from the Complete Agentic AI Course video · 10:55:49 to 10:58:31

Naming models with a Router

The video's router gives each job an alias: a cheap fast model, a stronger one for code, a balanced one in between. The app asks for the alias and the router decides which deployment answers. Each entry in model_list is a model_name, the alias the app uses, and litellm_params, which says which model and key stand behind it. Fallbacks, retries and a timeout belong to the router too.

python
import os
from litellm import Router

router = Router(model_list=[
    {"model_name": "fast", "litellm_params": {"model": "groq/openai/gpt-oss-20b", "api_key": os.environ["GROQ_API_KEY"]}},
    {"model_name": "smart", "litellm_params": {"model": "groq/openai/gpt-oss-120b", "api_key": os.environ["GROQ_API_KEY"]}},
], fallbacks=[{"fast": ["smart"]}], num_retries=1, timeout=30)

router.completion(model="fast", messages=[{"role": "user", "content": "Say hi in 3 words."}])

Several entries can share one alias. The router then spreads requests over them, a random pick by default (simple-shuffle); least-busy, latency-based-routing and cost-based-routing choose by load, speed or price instead.

Caching a repeated request

With a cache on, a request identical to an earlier one is answered from storage and the model is not called, so it costs nothing and returns at once. Cache(type="local") keeps entries in memory in one process; Redis is the choice when several servers share a cache. The match is on the exact request. A question that only resembles an earlier one is a miss, and catching those needs a semantic cache. In 1.104.0 a cache hit also needs the proxy extras, installed with pip install "litellm[proxy]".

python
import litellm
from litellm import completion
from litellm.caching.caching import Cache

litellm.cache = Cache(type="local")               # in memory, this process only
kw = dict(model="groq/openai/gpt-oss-20b", caching=True,
          messages=[{"role": "user", "content": "What does LLM stand for? One line."}])
first = completion(**kw)                           # goes to the model
second = completion(**kw)                          # same request: answered from the cache

Running the gateway as a server

The same model list can live in a config.yaml and run as a server with litellm --config config.yaml. Every app then calls the gateway's address with any OpenAI-compatible client and the alias as the model name, and nothing in the apps names a provider. The server expects a master key, set here from an environment variable.

yaml
model_list:
  - model_name: fast
    litellm_params:
      model: groq/openai/gpt-oss-20b
      api_key: os.environ/GROQ_API_KEY
  - model_name: smart
    litellm_params:
      model: groq/openai/gpt-oss-120b
      api_key: os.environ/GROQ_API_KEY
router_settings:
  fallbacks: [{"fast": ["smart"]}]
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

Hosted gateways do this job as a service, with nothing to run: the video's notebook names OpenRouter, Cloudflare AI Gateway, Portkey, Helicone and Kong AI Gateway next to LiteLLM.

OptionRuns where
LiteLLMYour machine or your server, as a library or a proxy
OpenRouterHosted: one key for many models
Cloudflare AI GatewayHosted on Cloudflare, in front of the provider
PortkeyHosted, with an open-source gateway
HeliconeHosted, with a self-hosting option
Kong AI GatewayPart of Kong Gateway, on your own infrastructure

Blocking a prompt before the gateway call

A guardrail that has to stop a request belongs in code that runs before the call. LiteLLM's input_callback hooks observe a call as it is made: a hook that raises an error does not stop the request, and the call still reaches the provider. The reliable place for a check is your own function in front of completion, or a guardrail configured on the proxy. This one refuses a prompt that looks like a card number before anything is sent.

python
import re
from litellm import completion

CARD = re.compile(r"\b(?:\d[ -]?){13,16}\b")                # 13 to 16 digits, with optional spaces or dashes

def guarded_completion(**kwargs):
    text = " ".join(m["content"] for m in kwargs["messages"])
    if CARD.search(text):                                   # your own check, before the gateway call
        raise ValueError("blocked: the prompt looks like it holds a card number")
    return completion(**kwargs)

The Guardrails lesson covers checks on both sides of a model call.

Direct provider calls vs a gateway

Direct SDK callsThrough a gateway
Switching modelA code change in every appA change of string or config
Provider outageThe app fails with itFalls back to another model
Cost and logsAdded to each appOne place for every app
Extra moving partNoneOne more service, or one more library, to run

Where you use a gateway

  • Several apps on one set of keys. Keys, rate limits and spend live in one place instead of in every app.
  • Staying up through a provider outage. A fallback keeps the app answering while the first provider recovers.
  • Cost control. Routing easy requests to a small model and caching repeated ones cut the bill, and the cost of every call is logged.
Watch out. A fallback is not a quality guarantee. The backup model answers in its own style and with its own limits, and a prompt tuned for one model can read differently on another. Run the Evaluating LLM apps loop on the models in the fallback list, not only on the first one.
Try it yourself
  • Change the primary to "groq/openai/gpt-oss-20b", a model that exists. The call is answered by Groq's model and no fallback runs. Print response.model to see it.
  • Add "groq/openai/gpt-oss-120b" to the end of the fallbacks list. It is used only when the Gemini fallback fails as well.
  • Add a second request to the Router piece with model="smart" and compare which deployment answered each one.

Little by little, you're building something great.