LiteLLMLiteLLM 1.105 · Python 3.10+
Dashboard
0%

The words you will meet

The vocabulary of LiteLLM is the set of terms for what you send to a model, what comes back, where a model lives and what a gateway adds, and every later lesson uses them.

Last updated: 12 Oct, 2026 · LiteLLM 1.105

The gateway landscape: where LiteLLM sits used the words gateway, proxy and router as if they were known. This lesson defines them, with the terms around them, so the later lessons can use each one without stopping. Read it once and come back to it when a word is missing.

Terms for what you send

WordWhat it means
PromptWhat you send a model. In this API it is a list of messages, not one string.
MessageOne item in that list: a dictionary with a role and a content.
Rolesystem sets the rules, user is the person, assistant is an earlier answer from the model, and tool carries a tool's result back.
TokenA piece of a word. Providers count and charge by token, and count input and output separately.
Context windowThe most tokens one call may carry, question and answer together. A call over the limit fails.
TemperatureHow loose the model's choice of words is. A temperature of 0 is as repeatable as a model gets.
max_tokensAn upper limit on the reply, in tokens. On a reasoning model it also has to cover the model's hidden thinking.

Counting tokens and reading a context window

Two of those terms can be measured without calling any model. token_counter counts the tokens in a list of messages, and get_model_info reads the facts LiteLLM keeps about a model, including its context window.

ExampleA token count and a context window, with no model call
import litellm

model = "groq/openai/gpt-oss-120b"
messages = [{"role": "user", "content": "Say hi"}]

print(litellm.token_counter(model=model, messages=messages))
print(litellm.get_model_info(model)["max_input_tokens"])

The first line is the number of tokens in the one short message. Counting is done on your machine with a tokenizer, which is why it needs no key, and a provider's own count can differ a little. The second line is the largest input this model accepts, read from the price and limits table that ships with LiteLLM.

Terms for what comes back

WordWhat it means
CompletionThe model's answer. completion() is the function that asks for one.
ResponseEverything that returns: the answer, why it stopped and how many tokens it took. LiteLLM returns it as a ModelResponse object.
finish_reasonWhy the model stopped. stop means it finished, and length means it ran out of tokens.
UsageThe token counts for the call: prompt tokens in, completion tokens out and their total.
StreamingReceiving the answer in pieces as it is written, not in one piece at the end.
Reasoning modelA model that writes hidden thinking tokens before its answer. gpt-oss is one.
Response costThe price of the call in US dollars, worked out from the usage and a table of prices.

Terms for where a model lives

WordWhat it means
ProviderThe company or service that runs the model: Groq, Google, OpenAI, Anthropic.
ModelOne specific model a provider runs, such as openai/gpt-oss-120b on Groq.
Model stringThe text passed as model=: the provider, a slash, then the model id, such as groq/openai/gpt-oss-120b.
API keyThe secret that proves who is calling. It lives in an environment variable, a setting outside your code, so it can change without the program changing.
Base URLThe address requests go to. In LiteLLM it is the api_base setting. Pointing it at a gateway instead of a provider is most of what using a gateway is.
AliasA name of your own that stands for a model string.
DeploymentOne concrete way to reach a model: a model string with its key and address. The Router keeps a list of them.
Model groupA name shared by one or more deployments. A caller asks for the group, and the Router picks the deployment.

Terms a gateway adds

WordWhat it means
SDKThe LiteLLM Python package, called from your own code.
RouterThe part of the SDK that chooses a deployment for each call and handles failures.
GatewayA service that sits between applications and model providers and decides which provider answers.
ProxyLiteLLM's own name for its gateway server.
Virtual keyA key the gateway issues to an application, with its own limits and budget, in place of the real provider keys.
RetrySending a failed request again. A fallback sends it somewhere else instead.
CooldownA rest period in which the Router stops sending requests to a deployment that keeps failing.
CacheA store of earlier answers, so an identical question is answered without paying again.
CallbackA function or class that LiteLLM calls after each request, with the request, the response and the cost.
Rate limitA ceiling a provider puts on requests or tokens per minute. Crossing one returns HTTP status 429.

Telling a deployment from a model group

These two are the terms most often confused. A model group is the name a caller uses. A deployment is a real model behind that name. One group can have several deployments, for example the same model on two keys, or two different models. Router: model groups and deployments builds one.

Where you use the vocabulary

  • Reading an error. A RateLimitError with status 429 is a rate limit, and it is the signal to retry or fall back.
  • Reading the gateway's configuration. It is written in deployments, model groups and virtual keys.
  • Choosing a model. Context window and cost per token are the two numbers that decide whether a model fits.
Watch out. A provider and a model are different things, and the model string carries both. groq/openai/gpt-oss-120b has a slash in the model id itself. The first slash ends the provider name, and the rest is sent to the provider as the model id.
Try it yourself
  • Count the tokens of a long ticket by changing the message, and watch the number grow.
  • Print litellm.get_model_info("gemini/gemini-2.5-flash-lite")["max_input_tokens"] and compare it with the Groq model.
  • Write the model string for a model called llama3 served by Ollama, using the pattern provider, slash, model id.

Slow is fine. Stopping is the only problem.