LiteLLMLiteLLM 1.105 · Python 3.10+
Dashboard
0%

Router: model groups and deployments

A Router is a LiteLLM object that holds a list of model deployments, grouped under shared names, and sends each request to one deployment of the group you ask for.

Last updated: 12 Oct, 2026 · LiteLLM 1.105

Until now each call named one model string, as in Providers and keys: the model string, and your code chose the provider. A Router moves that choice into LiteLLM. Your code asks for a group name such as chat, and the Router picks which deployment answers. The later lessons in this part add retries, fallbacks and cooldowns to the same object.

A Router holding two model groups. The group named chat has two deployments, groq gpt-oss-120b and groq gpt-oss-20b, and each call to chat is answered by one of them. The group named gemini has one deployment, gemini-2.5-flash-lite. Your code asks for a group name and never names a deployment.

Three words carry the whole idea. A deployment is one way to reach a model: a model string plus any keys or settings it needs. A model group is a name shared by one or more deployments. The model list is the list of every deployment, each tagged with the group it belongs to. This is the same idea as the names in Model aliases: names of your own, with several models allowed behind one name. Sharing calls across the deployments of a group is called load balancing.

Syntax: build a Router from a model list, then call a group name.

python
router = Router(model_list=model_list)
response = router.completion(
    model="chat",            # a model_name from the list, not a provider model string
    messages=messages,
)

Building a model list

Two deployments in one group

Each entry has a model_name, which is the group, and litellm_params, which holds the same arguments you would pass to completion(). Two entries that share a model_name form one group with two deployments. A third entry with a new name is a group of its own.

python
model_list = [
    {"model_name": "chat",                                  # the group your code asks for
     "litellm_params": {"model": "groq/openai/gpt-oss-120b"}},
    {"model_name": "chat",                                  # same group, second deployment
     "litellm_params": {"model": "groq/openai/gpt-oss-20b"}},
    {"model_name": "gemini",                                # a group with one deployment
     "litellm_params": {"model": "gemini/gemini-2.5-flash-lite"}},
]

Calling the group and reading who answered

The Router picks a deployment, makes the call, and returns the usual response. response.model names the model that answered, which is how a program finds out which deployment it got.

ExampleAPI keyOne call to the chat group; either deployment may answer
response = router.completion(
    model="chat",
    messages=[{"role": "user", "content": "Name one planet."}],
    max_tokens=300,
)
print(response.model)
print(response.choices[0].message.content)
  • response.model is the answering deployment. The groq/ prefix is gone from it, so it reads openai/gpt-oss-20b or openai/gpt-oss-120b. Compare the end of the string.
  • The code never named a deployment. It asked for chat and got whichever one the Router chose.

Watching both deployments take traffic

One call shows one deployment. To see the spread without paying for thirty answers, pass mock_response. It is a LiteLLM argument that returns the given text without calling any provider, and the Router still makes its normal choice of deployment first.

ExampleThirty mocked calls to the chat group
seen = set()
for _ in range(30):
    response = router.completion(model="chat", messages=[{"role": "user", "content": "hi"}],
                                 mock_response="hello")
    seen.add(response.model)
print(sorted(seen))

Both deployments of the chat group answered at least once. The default strategy shuffles between healthy deployments, and Routing strategies: choosing which deployment answers covers the other rules.

Asking for a group that does not exist

A group name is not checked until a call uses it. A name that is not in the list fails with a message about healthy deployments.

ExampleA group name missing from the model list
import litellm

try:
    router.completion(model="nope", messages=[{"role": "user", "content": "hi"}],
                      mock_response="hello")
except litellm.BadRequestError as error:
    print(type(error).__name__)
    print(error)

The error is BadRequestError. It says there are no healthy deployments for nope, because no entry in the list carries that model_name. Check the spelling of the group, not the provider.

completion() vs Router.completion()

completion()router.completion()
You passA provider model string such as groq/openai/gpt-oss-20bA group name from the model list
Who picks the providerYou, in the stringThe Router, among the group's deployments
Retries and fallbacksNot built inBuilt in, set on the Router
Remembers failuresNoYes, through cooldowns

Where you use a Router

  • Spreading load. Two keys or two models behind one name share the traffic.
  • Surviving an outage. Retries, fallbacks and cooldowns work on a Router and not on a bare call.
  • Naming by purpose. Code asks for chat or cheap, and the list decides which model that means.
Watch out. A group means that any of its deployments is acceptable. Put a small model and a large model in one group and the answers will differ in quality from call to call, with nothing in the reply telling you to expect it. Group deployments that are interchangeable, and give different models different group names.
Try it yourself
  • Add the Gemini deployment to the chat group by changing its model_name, and run the thirty mocked calls again. Three names appear.
  • Print [(m["model_name"], m["litellm_params"]["model"]) for m in router.get_model_list()].
  • Remove the mock_response argument and run three real calls. Compare the answers with the deployment each one reports.

You understood something today that you didn't yesterday.