Why LiteLLM: one interface for every provider
LiteLLM is a Python library that gives every model provider the same function, completion(), with the same request shape and the same response shape, so changing provider means changing one string.
Last updated: 12 Oct, 2026 · LiteLLM 1.105
LiteLLM overview described LiteLLM as one call in front of many providers. This lesson shows the problem that call removes, using two real providers, and then solves it.
Picture three applications at one company: a chatbot on one provider, a document search on a second, an agent on a third. Each has its own integration code. Each handles that provider's errors in its own way. Moving any of them to another provider means writing the integration again.
Reading the same job in two request shapes
The job is the same for both providers: sort a support ticket into billing or shipping. The code below sends it to Groq and to Gemini with plain HTTP requests, using httpx, the HTTP library that LiteLLM installs. It uses the keys that Installation and setup sets up, and no LiteLLM at all.
import os
import httpx
ask = "Reply with one word, billing or shipping. Ticket: I was charged twice for one order"
groq = httpx.post(
"https://api.groq.com/openai/v1/chat/completions",
headers={"Authorization": "Bearer " + os.environ["GROQ_API_KEY"]},
json={"model": "openai/gpt-oss-120b", "max_tokens": 400,
"messages": [{"role": "user", "content": ask}]},
).json()
gemini = httpx.post(
"https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash-lite:generateContent",
headers={"x-goog-api-key": os.environ["GEMINI_API_KEY"]},
json={"contents": [{"parts": [{"text": ask}]}],
"generationConfig": {"maxOutputTokens": 400}},
).json()
print("Groq :", groq["choices"][0]["message"]["content"])
print("Gemini:", gemini["candidates"][0]["content"]["parts"][0]["text"].strip())Groq : billing Gemini: Billing
Both answered, and the reading of the result differs in every part of the call:
- The key. Groq takes it in an
Authorization: Bearerheader. Gemini takes it in anx-goog-api-keyheader. - The request body. Groq takes a
messageslist, where each message has a role and a content string. Gemini takescontents, where each item holds a list ofparts, each with its owntext. - The token limit. It is
max_tokensfor Groq andgenerationConfig.maxOutputTokensfor Gemini. - The answer. It is at
choices[0].message.contentfor Groq and atcandidates[0].content.parts[0].textfor Gemini.
Anthropic's Messages API is a third shape, described in the Anthropic API reference. The system prompt is a separate field, max_tokens is required, and the answer is a list of content blocks. It is not run here, because the course has no Anthropic key. Errors, token counts and prices differ in the same way. Each difference is more code to write and to keep correct.
Putting a gateway between the app and the providers
The video describes the same problem from a company's side, and the idea that solves it.
The video makes these points, in this order:
- A gateway is a smart middle layer between the application and the model provider. It receives each request, sends it to a suitable provider and returns the response.
- If one provider is down, the gateway can send the request to another, so the application keeps answering. This is called a fallback.
- The application does not need to know which model answers. A model can be swapped by changing configuration, not application code.
- On top of that come routing, caching, cost tracking, guardrails and evaluations.
- The first capability the video names is a unified API: one function, whatever the number of providers behind it.
LiteLLM is that unified function, and it is also the gateway server. The unified function comes first, because everything else is built on it.
Calling both providers with one function
Syntax: completion() takes the model string, which names the provider and the model, and the messages in the OpenAI shape. It returns the answer in the OpenAI shape.
from litellm import completion
response = completion(
model="provider/model-id", # which provider and which model
messages=[{"role": "user", "content": "..."}], # the conversation so far
max_tokens=400, # upper limit on the reply, in tokens
)
print(response.choices[0].message.content) # the answer, whoever sent itHere is the same ticket sent to the same two models. Nothing changes between the two iterations of the loop except the model string.
from litellm import completion
ask = "Reply with one word, billing or shipping. Ticket: I was charged twice for one order"
for model in ["groq/openai/gpt-oss-120b", "gemini/gemini-2.5-flash-lite"]:
response = completion(
model=model,
messages=[{"role": "user", "content": ask}],
max_tokens=400,
)
print(model, "->", response.choices[0].message.content)groq/openai/gpt-oss-120b -> billing gemini/gemini-2.5-flash-lite -> billing
Reading what LiteLLM changed
- One request shape. Both calls pass
messagesandmax_tokens. LiteLLM turned them intocontentsandmaxOutputTokensfor Gemini. - One place for the answer. Both replies were read from
response.choices[0].message.content. - One setting to change. The provider and the model are both inside the model string, so a different provider is a different string.
- The keys came from the environment. LiteLLM read
GROQ_API_KEYfor the first model andGEMINI_API_KEYfor the second, with no key in the call.
What LiteLLM adds beyond one call
- Retries, fallbacks and cooldowns across several deployments of a model, in Router: model groups and deployments.
- Caching, so an identical question is not paid for twice, in Caching responses.
- Cost and token counts on every response, in Tokens and cost.
- One exception family, so the same
exceptworks for every provider, in Exceptions: one set for every provider. - A server, so applications that are not written in Python can call the same models, in The LiteLLM proxy.
Direct provider calls vs LiteLLM
| Each provider directly | LiteLLM | |
|---|---|---|
| Request shape | One per provider | The OpenAI shape for all |
| Where the answer is | Different path per provider | choices[0].message.content |
| Switching provider | New integration code | A different model string |
| Errors | Each provider's own | One set of exception classes |
| Extra dependency | None | The litellm package |
| Provider-only options | All available | Those LiteLLM maps, or passed through |
Where one interface pays off
- An application that tries a cheaper model first and moves to a stronger one when needed.
- A service that must keep answering when one provider has an outage.
- A team that compares providers on its own tickets by looping over model strings.
Related
- Previous: LiteLLM overview
- Next: The gateway landscape: where LiteLLM sits
- Reference: LiteLLM documentation, the Getting Started page
- Add
"groq/openai/gpt-oss-20b"to the list in the loop and run it again. It is a third model with no new code. - Change the ticket to
Where is my parcel?and read whether both models chooseshipping. - In the direct-call example, change
"max_tokens": 400to5and read what Groq returns.
Little by little, you're building something great.