Router: model groups and deployments
A Router is a LiteLLM object that holds a list of model deployments, grouped under shared names, and sends each request to one deployment of the group you ask for.
Last updated: 12 Oct, 2026 · LiteLLM 1.105
Until now each call named one model string, as in Providers and keys: the model string, and your code chose the provider. A Router moves that choice into LiteLLM. Your code asks for a group name such as chat, and the Router picks which deployment answers. The later lessons in this part add retries, fallbacks and cooldowns to the same object.
Three words carry the whole idea. A deployment is one way to reach a model: a model string plus any keys or settings it needs. A model group is a name shared by one or more deployments. The model list is the list of every deployment, each tagged with the group it belongs to. This is the same idea as the names in Model aliases: names of your own, with several models allowed behind one name. Sharing calls across the deployments of a group is called load balancing.
Syntax: build a Router from a model list, then call a group name.
router = Router(model_list=model_list)
response = router.completion(
model="chat", # a model_name from the list, not a provider model string
messages=messages,
)Building a model list
Two deployments in one group
Each entry has a model_name, which is the group, and litellm_params, which holds the same arguments you would pass to completion(). Two entries that share a model_name form one group with two deployments. A third entry with a new name is a group of its own.
model_list = [
{"model_name": "chat", # the group your code asks for
"litellm_params": {"model": "groq/openai/gpt-oss-120b"}},
{"model_name": "chat", # same group, second deployment
"litellm_params": {"model": "groq/openai/gpt-oss-20b"}},
{"model_name": "gemini", # a group with one deployment
"litellm_params": {"model": "gemini/gemini-2.5-flash-lite"}},
]Calling the group and reading who answered
The Router picks a deployment, makes the call, and returns the usual response. response.model names the model that answered, which is how a program finds out which deployment it got.
response = router.completion(
model="chat",
messages=[{"role": "user", "content": "Name one planet."}],
max_tokens=300,
)
print(response.model)
print(response.choices[0].message.content)openai/gpt-oss-120b Mars.
- response.model is the answering deployment. The
groq/prefix is gone from it, so it readsopenai/gpt-oss-20boropenai/gpt-oss-120b. Compare the end of the string. - The code never named a deployment. It asked for
chatand got whichever one the Router chose.
Watching both deployments take traffic
One call shows one deployment. To see the spread without paying for thirty answers, pass mock_response. It is a LiteLLM argument that returns the given text without calling any provider, and the Router still makes its normal choice of deployment first.
seen = set()
for _ in range(30):
response = router.completion(model="chat", messages=[{"role": "user", "content": "hi"}],
mock_response="hello")
seen.add(response.model)
print(sorted(seen))['openai/gpt-oss-120b', 'openai/gpt-oss-20b']
Both deployments of the chat group answered at least once. The default strategy shuffles between healthy deployments, and Routing strategies: choosing which deployment answers covers the other rules.
Asking for a group that does not exist
A group name is not checked until a call uses it. A name that is not in the list fails with a message about healthy deployments.
import litellm
try:
router.completion(model="nope", messages=[{"role": "user", "content": "hi"}],
mock_response="hello")
except litellm.BadRequestError as error:
print(type(error).__name__)
print(error)BadRequestError litellm.BadRequestError: You passed in model=nope. There are no healthy deployments for this model LiteLLM: model group 'nope' failed with the error above. No fallback was attempted.
The error is BadRequestError. It says there are no healthy deployments for nope, because no entry in the list carries that model_name. Check the spelling of the group, not the provider.
completion() vs Router.completion()
| completion() | router.completion() | |
|---|---|---|
| You pass | A provider model string such as groq/openai/gpt-oss-20b | A group name from the model list |
| Who picks the provider | You, in the string | The Router, among the group's deployments |
| Retries and fallbacks | Not built in | Built in, set on the Router |
| Remembers failures | No | Yes, through cooldowns |
Where you use a Router
- Spreading load. Two keys or two models behind one name share the traffic.
- Surviving an outage. Retries, fallbacks and cooldowns work on a Router and not on a bare call.
- Naming by purpose. Code asks for
chatorcheap, and the list decides which model that means.
Related
- Previous: CustomLLM: writing a provider of your own
- Next: Routing strategies: choosing which deployment answers
- See also: Fallbacks and caching in the gateway config for the same model list written as a gateway config file
- Reference: Router in the LiteLLM docs
- Add the Gemini deployment to the
chatgroup by changing itsmodel_name, and run the thirty mocked calls again. Three names appear. - Print
[(m["model_name"], m["litellm_params"]["model"]) for m in router.get_model_list()]. - Remove the
mock_responseargument and run three real calls. Compare the answers with the deployment each one reports.
You understood something today that you didn't yesterday.