Routing strategies: choosing which deployment answers
A routing strategy, also called load balancing, is the rule a Router uses to choose which deployment of a model group answers the next request.
Last updated: 12 Oct, 2026 · LiteLLM 1.105
Router: model groups and deployments showed the default rule at work: calls to one group load balanced across its deployments. This lesson names the rules LiteLLM 1.105.0 accepts, runs the ones that can be told apart without real traffic, and says what the rest need.
Syntax: pass routing_strategy to the Router. Per-deployment settings such as weight and order go in litellm_params.
router = Router(
model_list=model_list,
routing_strategy="simple-shuffle", # the default
)Listing the strategies this version accepts
The Router checks the strategy name when it is built, and a wrong name lists the right ones. That makes a wrong name the quickest way to read the list for the installed version.
from litellm import Router
Router(
model_list=[{"model_name": "chat", "litellm_params": {"model": "groq/openai/gpt-oss-20b"}}],
routing_strategy="fastest",
)Traceback (most recent call last):
File "main.py", line 3, in <module>
Router(
ValueError: Invalid routing_strategy: 'fastest'. Valid options: ['simple-shuffle', 'lar1', 'least-busy', 'latency-based-routing', 'cost-based-routing', 'usage-based-routing-v2', 'usage-based-routing', 'provider-budget-routing']. Check 'router_settings.routing_strategy' in your config.yaml or the 'routing_strategy' parameter if using the Router SDK directly.The names in the error are the ones the check accepts. simple-shuffle is the default. least-busy, latency-based-routing, cost-based-routing, usage-based-routing-v2 and usage-based-routing each pick by a measured quantity. lar1 routes by a confidence value that the caller puts in the request metadata. That is a narrow use. The list also holds provider-budget-routing, which passes the check but is not a working strategy in 1.105.0: a Router built with it raised RouterRateLimitError (no deployments available) on the first call. Spending limits per provider are set with the Router's provider_budget_config argument instead. It filters out deployments over budget and works with any strategy. It needs a budget and a cost to compare, so it is not run here.
Weighting the shuffle
With no settings, simple-shuffle picks at random. A weight on each deployment turns that into a weighted draw. The Router reads weight first, then rpm, then tpm, and uses the first one that is set.
model_list = [
{"model_name": "chat",
"litellm_params": {"model": "groq/openai/gpt-oss-120b", "weight": 9}},
{"model_name": "chat",
"litellm_params": {"model": "groq/openai/gpt-oss-20b", "weight": 1}},
]
router = Router(model_list=model_list) # simple-shuffle is the defaultA hundred mocked calls show the split. The counts change on every run, because the draw is random.
from collections import Counter
counts = Counter()
for _ in range(100):
response = router.completion(model="chat", messages=[{"role": "user", "content": "hi"}],
mock_response="hello")
counts[response.model] += 1
print(dict(counts)){'openai/gpt-oss-120b': 91, 'openai/gpt-oss-20b': 9}The heavier deployment answered about nine calls in ten. A weight is a share of traffic, not a promise about any one call.
Preferring one deployment with order
order is a priority. The deployment with the lowest order takes the traffic, and the others wait. They answer when the first one is failing, as Cooldowns: resting a failing deployment shows.
model_list = [
{"model_name": "chat",
"litellm_params": {"model": "groq/openai/gpt-oss-120b", "order": 2}},
{"model_name": "chat",
"litellm_params": {"model": "groq/openai/gpt-oss-20b", "order": 1}}, # tried first
]
router = Router(model_list=model_list)from collections import Counter
counts = Counter()
for _ in range(20):
response = router.completion(model="chat", messages=[{"role": "user", "content": "hi"}],
mock_response="hello")
counts[response.model] += 1
print(dict(counts)){'openai/gpt-oss-20b': 20}All twenty calls went to gpt-oss-20b, the deployment with order 1, although it is listed second.
Choosing the cheapest deployment
cost-based-routing picks the deployment with the lowest price in LiteLLM's model table. Here the 20b model is the cheaper of the two.
model_list = [
{"model_name": "chat", "litellm_params": {"model": "groq/openai/gpt-oss-120b"}},
{"model_name": "chat", "litellm_params": {"model": "groq/openai/gpt-oss-20b"}},
]
router = Router(model_list=model_list, routing_strategy="cost-based-routing")import litellm
try:
router.completion(model="chat", messages=[{"role": "user", "content": "hi"}],
mock_response="hello")
except litellm.RouterRateLimitError as error:
print(type(error).__name__)
print(error)RouterRateLimitError No deployments available for selected model, Try again in 5 seconds. Passed model=chat. pre-call-checks=False, cooldown_list=[]
In 1.105.0 the synchronous completion() raised RouterRateLimitError, saying that no deployments were available, although none was cooling down. The cost-based selector in this version has only an asynchronous form, so the call to use is acompletion().
import asyncio
from collections import Counter
async def main():
counts = Counter()
for _ in range(20):
response = await router.acompletion(model="chat", messages=[{"role": "user", "content": "hi"}],
mock_response="hello")
counts[response.model] += 1
print(dict(counts))
asyncio.run(main()){'openai/gpt-oss-20b': 20}Every call went to gpt-oss-20b, the cheaper deployment.
What the other strategies pick
The rest depend on live load, response times or token counts, so mocked calls cannot tell them apart and none is run here. What each does comes from the library's own code and from the Router docs.
| Strategy | Picks the deployment with | Worth knowing |
|---|---|---|
simple-shuffle | A random draw, shaped by weight, rpm or tpm when set | The default |
least-busy | The fewest requests in flight | Counts live in memory, or in Redis when a Redis cache is configured |
latency-based-routing | The lowest recent response time | Needs real traffic to learn from |
cost-based-routing | The lowest price in the model table | Asynchronous calls only in 1.105.0, as run above |
usage-based-routing-v2 | The lowest tokens-per-minute use, within tpm and rpm limits | Share a Redis cache when several processes serve one app |
usage-based-routing | The same idea, in the older version | Prefer the v2 name |
lar1 | A choice made from a confidence value in the request metadata | A narrow use |
A Router can also hold routing_groups: named subsets of model groups, each with its own strategy, including a priority kind that takes an order per model. The Router's own docstring describes it. It is not run here, so treat routing_strategy as the setting that covers the models no group claims.
Where you use a routing strategy
- Splitting by capacity. A larger share of traffic goes to the key with the higher rate limit.
- A preferred model with a standby.
orderkeeps the cheap or fast model first and the other in reserve. - Controlling spend.
cost-based-routingsends work to the cheapest model that is in the group.
Related
- Previous: Router: model groups and deployments
- Next: Retries: trying again when a provider refuses
- See also: The cheap model first for sending easy work to the cheaper model by design
- Reference: Router and load balancing in the LiteLLM docs
- Change the weights to 1 and 1 and run the hundred calls again. The split moves toward half and half.
- Give both deployments the same
orderin the order example. Both appear in the counts. - Set
routing_strategy="least-busy"on the model list and run twenty mocked calls.
Slow is fine. Stopping is the only problem.