The gateway landscape: where LiteLLM sits
An AI gateway is a service that accepts model requests in one format from every application and decides which provider answers them; LiteLLM is the open-source one this course teaches, and several others exist.
Last updated: 12 Oct, 2026 · LiteLLM 1.105
Why LiteLLM: one interface for every provider showed one function in front of two providers. This lesson places LiteLLM among the other tools that sit in that position, so a choice between them rests on what each one is.
Seeing what a gateway holds for a company
Without a gateway, each application keeps its own provider keys, its own retry code and its own idea of which model to use. Nobody can say what the company spent. With a gateway, all of that sits in one service. Applications send it requests. It decides which provider answers, retries what fails, records the cost and refuses callers it did not give a key to.
Comparing the LiteLLM SDK and the LiteLLM proxy
LiteLLM ships both halves, which is why one name means two things in its documentation.
| What it is | Where it runs | When to use it | |
|---|---|---|---|
| The SDK | A Python function, litellm.completion() | Inside your own process | One application. Nothing to deploy. |
| The proxy | A server, called the AI Gateway in the documentation | On its own, at an address | Several applications, or any time keys, budgets and spending must be held in one place. |
The Router sits between them. It is SDK code, and the proxy runs it inside itself, so what Router: model groups and deployments teaches applies to both.
Surveying the other gateways
Each description below is taken from the project's own README or documentation. Projects change quickly, so read the linked page before choosing.
| Project | What its own page says it is | Where to read more |
|---|---|---|
| Portkey | An open-source AI gateway that routes requests to many LLMs through one API. Some features are marked as available only in its hosted and enterprise versions. | github.com/Portkey-AI/gateway |
| Bifrost | An AI gateway, built by Maxim, that unifies access to 23+ providers through one OpenAI-compatible API. Its README lists failover, load balancing and semantic caching. | github.com/maximhq/bifrost |
| TensorZero | An open-source LLMOps platform in which a gateway is one part, next to observability, evaluation, optimization and experimentation. | github.com/tensorzero/tensorzero |
| OpenRouter | A hosted service that gives access to hundreds of models through a single API endpoint. | openrouter.ai/docs |
OpenRouter also works as a provider inside LiteLLM, so choosing it does not rule LiteLLM out. A provider's own SDK is the remaining option, and it is the right one when an application will only ever use that provider.
Choosing between a gateway and a provider SDK
- One provider, one application. The provider's own SDK has the fewest moving parts.
- Several providers in Python code. The LiteLLM SDK is a Python function that runs inside your own process, so there is no server to start.
- Several applications, or applications not written in Python. A gateway server, LiteLLM's proxy or another, holds the keys and the rules once.
- Someone else should run it. A hosted gateway removes the servers you would otherwise operate.
The same decision is made again, with all three ways of calling a model side by side, in Hybrid SDK, Router and gateway.
Knowing the limits of LiteLLM
- Some features are paid. LiteLLM has a paid Enterprise tier, described on the LiteLLM enterprise page. Lessons that touch a feature in it say so.
- A gateway is a hop. Each request travels through one more service, so measure the delay on your own traffic instead of trusting anyone's published figures.
- A gateway is one more thing that can fail. Every application depends on its one address. The gateway in a container and the lessons after it cover running it properly.
Related
- Previous: Why LiteLLM: one interface for every provider
- Next: The words you will meet
- See also: LLM gateways in the generative and agentic AI course
- Reference: LiteLLM Proxy Server in the LiteLLM docs
- Open each repository in the table and find the number of providers its README states.
- Look for a feature list on the Bifrost and Portkey pages and mark which ones are marked as paid or enterprise.
- Read the LiteLLM enterprise page and list one feature this course will use and one it will not.
You understood something today that you didn't yesterday.