LiteLLMLiteLLM 1.105 · Python 3.10+
Dashboard
0%

Why LiteLLM: one interface for every provider

LiteLLM is a Python library that gives every model provider the same function, completion(), with the same request shape and the same response shape, so changing provider means changing one string.

Last updated: 12 Oct, 2026 · LiteLLM 1.105

LiteLLM overview described LiteLLM as one call in front of many providers. This lesson shows the problem that call removes, using two real providers, and then solves it.

Two ways to reach three providers. On the left, an app carries separate code for Groq, Gemini and Anthropic, each with its own request and answer shape. On the right, the app makes one call, litellm.completion, with one request shape in and one response shape out, and LiteLLM translates to Groq, Gemini and Anthropic.

Picture three applications at one company: a chatbot on one provider, a document search on a second, an agent on a third. Each has its own integration code. Each handles that provider's errors in its own way. Moving any of them to another provider means writing the integration again.

Reading the same job in two request shapes

The job is the same for both providers: sort a support ticket into billing or shipping. The code below sends it to Groq and to Gemini with plain HTTP requests, using httpx, the HTTP library that LiteLLM installs. It uses the keys that Installation and setup sets up, and no LiteLLM at all.

ExampleAPI keyTwo providers called directly, with their own shapes
import os
import httpx

ask = "Reply with one word, billing or shipping. Ticket: I was charged twice for one order"

groq = httpx.post(
    "https://api.groq.com/openai/v1/chat/completions",
    headers={"Authorization": "Bearer " + os.environ["GROQ_API_KEY"]},
    json={"model": "openai/gpt-oss-120b", "max_tokens": 400,
          "messages": [{"role": "user", "content": ask}]},
).json()

gemini = httpx.post(
    "https://generativelanguage.googleapis.com/v1beta/models/gemini-2.5-flash-lite:generateContent",
    headers={"x-goog-api-key": os.environ["GEMINI_API_KEY"]},
    json={"contents": [{"parts": [{"text": ask}]}],
          "generationConfig": {"maxOutputTokens": 400}},
).json()

print("Groq  :", groq["choices"][0]["message"]["content"])
print("Gemini:", gemini["candidates"][0]["content"]["parts"][0]["text"].strip())

Both answered, and the reading of the result differs in every part of the call:

  • The key. Groq takes it in an Authorization: Bearer header. Gemini takes it in an x-goog-api-key header.
  • The request body. Groq takes a messages list, where each message has a role and a content string. Gemini takes contents, where each item holds a list of parts, each with its own text.
  • The token limit. It is max_tokens for Groq and generationConfig.maxOutputTokens for Gemini.
  • The answer. It is at choices[0].message.content for Groq and at candidates[0].content.parts[0].text for Gemini.

Anthropic's Messages API is a third shape, described in the Anthropic API reference. The system prompt is a separate field, max_tokens is required, and the answer is a list of content blocks. It is not run here, because the course has no Anthropic key. Errors, token counts and prices differ in the same way. Each difference is more code to write and to keep correct.

Putting a gateway between the app and the providers

The video describes the same problem from a company's side, and the idea that solves it.

What a gateway does · from the Complete Agentic AI Course video · 10:35:53 to 10:38:33

The video makes these points, in this order:

  • A gateway is a smart middle layer between the application and the model provider. It receives each request, sends it to a suitable provider and returns the response.
  • If one provider is down, the gateway can send the request to another, so the application keeps answering. This is called a fallback.
  • The application does not need to know which model answers. A model can be swapped by changing configuration, not application code.
  • On top of that come routing, caching, cost tracking, guardrails and evaluations.
  • The first capability the video names is a unified API: one function, whatever the number of providers behind it.

LiteLLM is that unified function, and it is also the gateway server. The unified function comes first, because everything else is built on it.

Calling both providers with one function

Syntax: completion() takes the model string, which names the provider and the model, and the messages in the OpenAI shape. It returns the answer in the OpenAI shape.

python
from litellm import completion

response = completion(
    model="provider/model-id",   # which provider and which model
    messages=[{"role": "user", "content": "..."}],   # the conversation so far
    max_tokens=400,   # upper limit on the reply, in tokens
)
print(response.choices[0].message.content)   # the answer, whoever sent it

Here is the same ticket sent to the same two models. Nothing changes between the two iterations of the loop except the model string.

ExampleAPI keyOne function, two providers
from litellm import completion

ask = "Reply with one word, billing or shipping. Ticket: I was charged twice for one order"

for model in ["groq/openai/gpt-oss-120b", "gemini/gemini-2.5-flash-lite"]:
    response = completion(
        model=model,
        messages=[{"role": "user", "content": ask}],
        max_tokens=400,
    )
    print(model, "->", response.choices[0].message.content)

Reading what LiteLLM changed

  • One request shape. Both calls pass messages and max_tokens. LiteLLM turned them into contents and maxOutputTokens for Gemini.
  • One place for the answer. Both replies were read from response.choices[0].message.content.
  • One setting to change. The provider and the model are both inside the model string, so a different provider is a different string.
  • The keys came from the environment. LiteLLM read GROQ_API_KEY for the first model and GEMINI_API_KEY for the second, with no key in the call.

What LiteLLM adds beyond one call

Direct provider calls vs LiteLLM

Each provider directlyLiteLLM
Request shapeOne per providerThe OpenAI shape for all
Where the answer isDifferent path per providerchoices[0].message.content
Switching providerNew integration codeA different model string
ErrorsEach provider's ownOne set of exception classes
Extra dependencyNoneThe litellm package
Provider-only optionsAll availableThose LiteLLM maps, or passed through

Where one interface pays off

  • An application that tries a cheaper model first and moves to a stronger one when needed.
  • A service that must keep answering when one provider has an outage.
  • A team that compares providers on its own tickets by looping over model strings.
Watch out. One interface does not make providers identical. Models differ in quality, speed, price and in which options they accept. A parameter one provider supports may be rejected by another, and Providers and keys: the model string shows the error that raises.
Try it yourself
  • Add "groq/openai/gpt-oss-20b" to the list in the loop and run it again. It is a third model with no new code.
  • Change the ticket to Where is my parcel? and read whether both models choose shipping.
  • In the direct-call example, change "max_tokens": 400 to 5 and read what Groq returns.

Little by little, you're building something great.