LiteLLMLiteLLM 1.105 · Python 3.10+
Dashboard
0%

Routing strategies: choosing which deployment answers

A routing strategy, also called load balancing, is the rule a Router uses to choose which deployment of a model group answers the next request.

Last updated: 12 Oct, 2026 · LiteLLM 1.105

Router: model groups and deployments showed the default rule at work: calls to one group load balanced across its deployments. This lesson names the rules LiteLLM 1.105.0 accepts, runs the ones that can be told apart without real traffic, and says what the rest need.

Syntax: pass routing_strategy to the Router. Per-deployment settings such as weight and order go in litellm_params.

python
router = Router(
    model_list=model_list,
    routing_strategy="simple-shuffle",    # the default
)

Listing the strategies this version accepts

The Router checks the strategy name when it is built, and a wrong name lists the right ones. That makes a wrong name the quickest way to read the list for the installed version.

ExampleA name that is not a strategy
from litellm import Router

Router(
    model_list=[{"model_name": "chat", "litellm_params": {"model": "groq/openai/gpt-oss-20b"}}],
    routing_strategy="fastest",
)

The names in the error are the ones the check accepts. simple-shuffle is the default. least-busy, latency-based-routing, cost-based-routing, usage-based-routing-v2 and usage-based-routing each pick by a measured quantity. lar1 routes by a confidence value that the caller puts in the request metadata. That is a narrow use. The list also holds provider-budget-routing, which passes the check but is not a working strategy in 1.105.0: a Router built with it raised RouterRateLimitError (no deployments available) on the first call. Spending limits per provider are set with the Router's provider_budget_config argument instead. It filters out deployments over budget and works with any strategy. It needs a budget and a cost to compare, so it is not run here.

Weighting the shuffle

With no settings, simple-shuffle picks at random. A weight on each deployment turns that into a weighted draw. The Router reads weight first, then rpm, then tpm, and uses the first one that is set.

python
model_list = [
    {"model_name": "chat",
     "litellm_params": {"model": "groq/openai/gpt-oss-120b", "weight": 9}},
    {"model_name": "chat",
     "litellm_params": {"model": "groq/openai/gpt-oss-20b", "weight": 1}},
]
router = Router(model_list=model_list)           # simple-shuffle is the default

A hundred mocked calls show the split. The counts change on every run, because the draw is random.

ExampleAPI keyWeights 9 and 1 over a hundred calls
from collections import Counter

counts = Counter()
for _ in range(100):
    response = router.completion(model="chat", messages=[{"role": "user", "content": "hi"}],
                                 mock_response="hello")
    counts[response.model] += 1
print(dict(counts))

The heavier deployment answered about nine calls in ten. A weight is a share of traffic, not a promise about any one call.

Preferring one deployment with order

order is a priority. The deployment with the lowest order takes the traffic, and the others wait. They answer when the first one is failing, as Cooldowns: resting a failing deployment shows.

python
model_list = [
    {"model_name": "chat",
     "litellm_params": {"model": "groq/openai/gpt-oss-120b", "order": 2}},
    {"model_name": "chat",
     "litellm_params": {"model": "groq/openai/gpt-oss-20b", "order": 1}},   # tried first
]
router = Router(model_list=model_list)
ExampleOrder 1 takes every call
from collections import Counter

counts = Counter()
for _ in range(20):
    response = router.completion(model="chat", messages=[{"role": "user", "content": "hi"}],
                                 mock_response="hello")
    counts[response.model] += 1
print(dict(counts))

All twenty calls went to gpt-oss-20b, the deployment with order 1, although it is listed second.

Choosing the cheapest deployment

cost-based-routing picks the deployment with the lowest price in LiteLLM's model table. Here the 20b model is the cheaper of the two.

python
model_list = [
    {"model_name": "chat", "litellm_params": {"model": "groq/openai/gpt-oss-120b"}},
    {"model_name": "chat", "litellm_params": {"model": "groq/openai/gpt-oss-20b"}},
]
router = Router(model_list=model_list, routing_strategy="cost-based-routing")
ExampleCost-based routing with the synchronous call
import litellm

try:
    router.completion(model="chat", messages=[{"role": "user", "content": "hi"}],
                      mock_response="hello")
except litellm.RouterRateLimitError as error:
    print(type(error).__name__)
    print(error)

In 1.105.0 the synchronous completion() raised RouterRateLimitError, saying that no deployments were available, although none was cooling down. The cost-based selector in this version has only an asynchronous form, so the call to use is acompletion().

ExampleCost-based routing with the asynchronous call
import asyncio
from collections import Counter

async def main():
    counts = Counter()
    for _ in range(20):
        response = await router.acompletion(model="chat", messages=[{"role": "user", "content": "hi"}],
                                            mock_response="hello")
        counts[response.model] += 1
    print(dict(counts))

asyncio.run(main())

Every call went to gpt-oss-20b, the cheaper deployment.

What the other strategies pick

The rest depend on live load, response times or token counts, so mocked calls cannot tell them apart and none is run here. What each does comes from the library's own code and from the Router docs.

StrategyPicks the deployment withWorth knowing
simple-shuffleA random draw, shaped by weight, rpm or tpm when setThe default
least-busyThe fewest requests in flightCounts live in memory, or in Redis when a Redis cache is configured
latency-based-routingThe lowest recent response timeNeeds real traffic to learn from
cost-based-routingThe lowest price in the model tableAsynchronous calls only in 1.105.0, as run above
usage-based-routing-v2The lowest tokens-per-minute use, within tpm and rpm limitsShare a Redis cache when several processes serve one app
usage-based-routingThe same idea, in the older versionPrefer the v2 name
lar1A choice made from a confidence value in the request metadataA narrow use

A Router can also hold routing_groups: named subsets of model groups, each with its own strategy, including a priority kind that takes an order per model. The Router's own docstring describes it. It is not run here, so treat routing_strategy as the setting that covers the models no group claims.

Where you use a routing strategy

  • Splitting by capacity. A larger share of traffic goes to the key with the higher rate limit.
  • A preferred model with a standby. order keeps the cheap or fast model first and the other in reserve.
  • Controlling spend. cost-based-routing sends work to the cheapest model that is in the group.
Watch out. The strategy is set once on the Router, not per call. Usage and least-busy counts live in the memory of one process, so an app run as several processes needs a shared Redis cache for the counts to agree. Redis is not run in this course.
Try it yourself
  • Change the weights to 1 and 1 and run the hundred calls again. The split moves toward half and half.
  • Give both deployments the same order in the order example. Both appear in the counts.
  • Set routing_strategy="least-busy" on the model list and run twenty mocked calls.

Slow is fine. Stopping is the only problem.