Router: model groups and deployments

An app should ask for support, not a provider's model. The Router maps a name the app uses to one or more deployments and picks one per call.

from litellm import Router

import flaky_llm

router = Router(model_list=[
    {"model_name": "support", "litellm_params": {"model": "shop/small"}},
    {"model_name": "support", "litellm_params": {"model": "shop/large"}},
])
print(sorted(set(router.get_model_names())))
print(len(router.get_model_list()), "deployments")

Each entry in model_list is a deployment. model_name is the name the app calls, and entries sharing it form a model group. litellm_params is what LiteLLM passes to completion for that deployment: the model string, and in real use the key and address.

By default the Router spreads calls across a group's deployments with a strategy its docs call simple-shuffle: a weighted random pick, using rpm, tpm or weight when given. That is load balancing, across keys, regions or providers.

A preferred deployment and a backup

router = Router(
    model_list=[
        {"model_name": "support", "litellm_params": {"model": "down/large", "order": 1}},
        {"model_name": "support", "litellm_params": {"model": "shop/small", "order": 2}},
    ],
    num_retries=0,
)

order sets priority, lower first. The routing page says that when an order=1 deployment fails, the Router tries the order=2 ones. The preferred one here is the provider that is down.

response = router.completion(model="support", messages=[{"role": "user", "content": "I was charged twice for one order"}])
print(response.choices[0].message.content)
print(response.model)

The app asked for support and got an answer. down/large failed, and the Router moved to the next order without the app knowing. response.model shows which deployment answered.