The auto router

An auto router is a model name in the gateway that scores each request for difficulty and sends it to a cheaper or a stronger model for you.

Last updated: 12 Oct, 2026 · LiteLLM 1.105

The cheap model first decided in the application's code whether to ask the cheap model or the strong one. Tag routing moved that choice to the caller's tags. An auto router moves it into the gateway: the application asks for one name, the gateway classifies the request, and the model for that difficulty answers. The auto router overview calls the idea one model name in your clients, with the gateway picking the model.

Syntax: the router is one more entry in model_list. Its model starts with auto_router/complexity_router, and its tiers name other entries of the same list.

- model_name: desk-router                  # the name clients ask for
  litellm_params:
    model: auto_router/complexity_router
    complexity_router_config:
      tiers:
        SIMPLE: fast                        # each tier names a model_name
        MEDIUM: fast
        COMPLEX: smart
        REASONING: smart
    complexity_router_default_model: fast   # used when the router cannot decide

The four tiers

A tier is a difficulty level. The page names four: SIMPLE, MEDIUM, COMPLEX and REASONING. The setup page says tier values name other model_name entries in the same file, so every tier is a model the gateway already serves. Two tiers may name the same model, as here.

The classifier

The setup page says that without a classifier_type the router uses the heuristic scorer, which it calls free and without added latency. The scorer reads the text of the request. The README inside the installed library lists what it looks at: the number of tokens, code words, reasoning markers such as "step by step", technical terms, simple phrasing such as "what is", multi-step patterns and the number of questions. It adds weighted points, and in this version boundaries at 0.15, 0.35 and 0.60 map the total to a tier. The README also says that two or more reasoning markers move a request up to REASONING when the score is high enough.

Writing the helper that starts a gateway

ops.py starts the litellm command, waits until it answers, and stops it in a finally block. call() sends a request and returns the status, body and headers. chat() sends one chat request, and extra keyword arguments such as max_tokens replace its defaults.

from pathlib import Path
import sys

Path("ops.py").write_text('''import json
import os
import subprocess
import sys
import time
import urllib.error
import urllib.request
from contextlib import contextmanager

KEY = "sk-gateway-1234"          # the master key written in every config.yaml


@contextmanager
def running(port, env=None):
    """Start the gateway on `port` from config.yaml, wait until it answers, stop it afterwards."""
    litellm = os.path.join(os.path.dirname(sys.executable), "litellm")
    cmd = [litellm, "--config", "config.yaml", "--port", str(port)]
    proc = subprocess.Popen(cmd, env=env, stdout=open("gateway.log", "w"), stderr=subprocess.STDOUT)
    try:
        for _ in range(90):
            if proc.poll() is not None:
                raise RuntimeError("the gateway stopped; read gateway.log")
            try:
                urllib.request.urlopen(f"http://127.0.0.1:{port}/health/liveliness")
                break
            except OSError:
                time.sleep(1)
        yield f"http://127.0.0.1:{port}"
    finally:
        proc.terminate()
        proc.wait(10)


def call(url, path, body=None, key=KEY, headers=None):
    """Send one request. Returns (status, text, headers) and does not raise on 4xx or 5xx."""
    head = dict(headers or {})
    if key:
        head["Authorization"] = f"Bearer {key}"
    if body is not None:
        head["Content-Type"] = "application/json"
    data = json.dumps(body).encode() if body is not None else None
    try:
        reply = urllib.request.urlopen(urllib.request.Request(url + path, data=data, headers=head), timeout=60)
    except urllib.error.HTTPError as error:
        reply = error
    return reply.getcode(), reply.read().decode(), {k.lower(): v for k, v in reply.headers.items()}


def chat(url, model, text="Reply with one word: ok", key=KEY, headers=None, **extra):
    """One chat request. Extra keyword arguments are added to the JSON body."""
    body = {"model": model, "messages": [{"role": "user", "content": text}], "max_tokens": 60, **extra}
    return call(url, "/chat/completions", body, key, headers)
''')
sys.path.insert(0, ".")        # lets the programs below run: from ops import running, call, chat
print("wrote ops.py")

wrote ops.py

Sending five prompts to one model name

The program starts a gateway with the config above and sends five prompts, all to desk-router. The gateway adds the decision to the response headers: x-litellm-complexity-router-tier, -cause and -score. The header x-litellm-model-name names the model that answered.

from pathlib import Path
from ops import running, chat

Path("config.yaml").write_text("""model_list:
  - model_name: fast
    litellm_params:
      model: groq/openai/gpt-oss-20b
      api_key: os.environ/GROQ_API_KEY
  - model_name: smart
    litellm_params:
      model: groq/openai/gpt-oss-120b
      api_key: os.environ/GROQ_API_KEY
  - model_name: desk-router
    litellm_params:
      model: auto_router/complexity_router
      complexity_router_config:
        tiers:
          SIMPLE: fast
          MEDIUM: fast
          COMPLEX: smart
          REASONING: smart
      complexity_router_default_model: fast
general_settings:
  master_key: sk-gateway-1234
""")

PROMPTS = {
    "greeting": "hi",
    "arithmetic": "What is 2+2?",
    "hard but short": "Prove the Riemann hypothesis.",
    "code task": "Write a Python function that parses a CSV file with a class, then explain step by step how you would debug a race condition in an async database layer, and design a schema migration.",
    "reasoning": "Think through this step by step: first derive the algorithm, then prove its correctness, then analyze complexity for a distributed system architecture.",
}
with running(4761) as url:
    for label, prompt in PROMPTS.items():
        status, _, reply = chat(url, "desk-router", prompt, max_tokens=80)
        tier = reply["x-litellm-complexity-router-tier"]
        cause = reply["x-litellm-complexity-router-cause"]
        score = round(float(reply["x-litellm-complexity-router-score"]), 3)
        print(f"{label:15} {tier:10} {cause:17} {score:6}  {reply['x-litellm-model-name']}")

greeting        SIMPLE     heuristic_scorer   -0.15  groq/openai/gpt-oss-20b
arithmetic      SIMPLE     heuristic_scorer   -0.15  groq/openai/gpt-oss-20b
hard but short  SIMPLE     heuristic_scorer    -0.1  groq/openai/gpt-oss-20b
code task       COMPLEX    heuristic_scorer   0.475  groq/openai/gpt-oss-120b
reasoning       REASONING  reasoning_override   0.54  groq/openai/gpt-oss-120b

Reading the routing decisions

• Short, plain prompts scored SIMPLE and went to the 20b model. The scores are at or below zero, because "what is" and a short length count against difficulty.
• The long prompt about code, debugging and a schema scored 0.475, inside the COMPLEX range, and went to the 120b model.
• The last prompt scored 0.54 but landed in REASONING. Its cause is reasoning_override, not heuristic_scorer. It holds several reasoning markers, so the override promoted it, as the README describes.
• "Prove the Riemann hypothesis." scored SIMPLE. It is one of the hardest sentences in mathematics, and the heuristic only sees a short prompt with no code words or markers. The scorer measures how a prompt is written, not how hard the question is.

Other classifiers and the semantic router

The classifier pages describe options beyond the heuristic: a classifier that asks a model (classifier_type: llm), keyword rules, and hosted or self-hosted classifier services. Each calls a model or a service, so none is run here. The docs publish benchmark results for them. None was reproduced here, so this lesson quotes no figure.

An older kind of auto router is on the semantic auto router page, marked deprecated and superseded by the complexity router. It compares the request with example phrases using an embedding model, so it needs an embedding provider. The installed Router code still reads its settings, auto_router_config_path, auto_router_default_model and auto_router_embedding_model. It is not run here.

Where you use an auto router

• Mixed traffic. Greetings and lookups go to a small model, and long coding tasks go to a large one.
• One name for every client. Applications stay unchanged while the tiers are re-pointed in the config.
• Coding agents. The docs describe connecting Claude Code and Codex to a router. That is not run here.

Related

• Previous: Secret managers
• See also: The cheap model first for the same idea written as application code
• See also: Routing strategies: choosing which deployment answers for choosing between deployments of one model name
• Reference: Auto Router and Admin Setup in the LiteLLM docs