Generative AI and Agentic AILangChain 1.4 · LangGraph 1.2 · Python 3.12 or 3.13
Dashboard
0%

Routing and deterministic decisions

Routing is the harness step that decides which specialist handles a request, and it should use plain code when the rule is clear and a model only when the request is ambiguous.

Last updated: 08 Oct, 2026 · LangChain 1.4

Harness engineering named routing as the first decision a harness makes. This lesson looks at why a harness routes to specialists at all and at how the routing decision itself should be made, because the cheapest and most testable way is often not a model.

Why a harness uses specialist agents · from the Loop and Harness Engineering Crash Course video · 2:36:20 to 2:38:32

Why route to specialists

One giant agent that has to handle billing, technical problems, sales and refunds is harder to control, test, debug, secure and improve independently. The video's alternative is specialists: a technical agent for technical queries, a billing agent for billing queries and a general agent for general questions. Each gets one job and one prompt.

The video's analogy is a company. A company does not hire one person who knows everything. It has an HR person, a CEO, managers and engineers, each specialised, and they work as a team on a big problem. Specialist agents work the same way, and the router is the step that hands each request to the right one.

Don't use a model for every decision · from the Loop and Harness Engineering Crash Course video · 2:48:15 to 2:49:50

Code or a model for each decision

The video's rule for the router is that a model should not make every decision. When the rule is clear, plain code is enough: if the question contains the word "refund", the route is billing. When the question is ambiguous, such as "classify this request into technical, billing or general", plain code cannot do it and a model is the right tool. Using code where code works lowers the cost of the whole multi-agent system.

A keyword router · from the Loop and Harness Engineering Crash Course video · 3:21:06 to 3:22:57

The router in the video's support demo

The demo's router checks the question against two lists of words. Words such as error, bug, login, API and install mean a technical question. Words such as price, payment, refund and bill mean a billing question. A question with none of them goes to the general agent. The video says that this router is for the demo, and that a real system would use a model to understand the user's intent. The next sections build both, and then a router that uses the two together.

A routing decision tree. A request first meets a code rule: if the rule matches exactly one group the request is routed by code with no model call; if it matches none or more than one, a model chooses. Both paths end at the technical, billing or general specialist.

Writing three routers

The keyword router

Two lists of words and one function. The order of the checks decides which group wins when a question matches both, so the technical check comes first here.

python
TECHNICAL = ["error", "bug", "login", "api", "install", "code", "server"]
BILLING = ["price", "pricing", "payment", "refund", "bill", "plan", "charged", "cost"]

def keyword_route(question):
    q = question.lower()
    if any(w in q for w in TECHNICAL):
        return "technical"
    if any(w in q for w in BILLING):
        return "billing"
    return "general"

The model router

The model must return one of three labels, so the schema uses Literal. with_structured_output makes the model fill that schema, which means the router cannot answer with a sentence. calls counts how often a model is used.

python
from typing import Literal
from pydantic import BaseModel
from langchain.chat_models import init_chat_model

class Route(BaseModel):
    route: Literal["technical", "billing", "general"]

router = init_chat_model("groq:openai/gpt-oss-20b", temperature=0).with_structured_output(Route, method="json_schema")
calls = {"model": 0}
python
def llm_route(question):
    calls["model"] += 1
    prompt = ("Route this customer message to technical (errors, logins, installs, bugs), "
              "billing (prices, plans, refunds, charges) or general.\n\nMessage: " + question)
    return router.invoke(prompt).route

The hybrid router

The hybrid router asks the keyword lists first. When a question matches exactly one group, the rule decides. When it matches no group or more than one, the question is ambiguous for the rules, and a model decides.

python
def hybrid_route(question):
    q = question.lower()
    groups = [name for name, words in [("technical", TECHNICAL), ("billing", BILLING)] if any(w in q for w in words)]
    return groups[0] if len(groups) == 1 else llm_route(question)

Comparing the three routers on five questions

Five questions, each with the route a person would pick. Two are built to trouble the keyword lists: one says the app stopped working without using any technical word, and one asks about the cost of the API, which contains a technical word and a billing word.

ExampleAPI keyLive run on Groq, openai/gpt-oss-20b
QUESTIONS = [
    "I am getting an API login error.",
    "How much is the Pro plan, and can I get a refund?",
    "What can your support team help me with?",
    "Why did the app stop working after the last update?",
    "What does the API cost per month?",
]
EXPECTED = ["technical", "billing", "general", "technical", "billing"]

for name, route in [("keywords", keyword_route), ("model", llm_route), ("hybrid", hybrid_route)]:
    calls["model"] = 0
    got = [route(q) for q in QUESTIONS]
    right = sum(g == e for g, e in zip(got, EXPECTED))
    print(f"{name:9} {right}/5 right, {calls['model']} model calls  {got}")

What the three routers did

  • Keywords: 3 of 5 right, no model calls. The rules got the clear questions right. They sent "Why did the app stop working after the last update?" to general, because it has no technical word, and "What does the API cost per month?" to technical, because the technical check runs first and the question contains "API".
  • Model: 5 of 5 right, five model calls. The model read the meaning and routed all five questions as a person would, at the price of one call per question.
  • Hybrid: 5 of 5 right, three model calls. The rules decided the first two questions. The other three matched no group or two groups, so the model decided them: the general question, the app update question and the API cost question. The hybrid reached the model's accuracy with three calls instead of five.

Keywords vs a model vs a hybrid router

Keyword rulesModel routerHybrid
Model callsNoneOne per requestOnly for ambiguous requests
SpeedInstantA model round tripInstant for clear requests
Ambiguous wordingMisroutes itReads the meaningHands it to the model
TestingExact and repeatableNeeds example questionsRules exact, model checked
Best forClear, stable categoriesFree-form requestsMixed traffic

Where you use routing

  • Support desks. Technical, billing and general questions go to different specialists.
  • Assistants with several knowledge bases. A question about contracts goes to the legal index.
  • Model selection. A simple question goes to a small model and a hard one to a large model.
Watch out. A keyword list is silent about everything it does not name. A question with no listed word lands in the general route, and a question with words from two groups lands in whichever check runs first. Test a router on the questions it will get, not only on the ones it was written for.
Try it yourself
  • Add "crash" and "update" to TECHNICAL and see which of the five questions the keyword router now gets right.
  • Put the billing check before the technical check in keyword_route and read what happens to the API cost question.
  • Add a sixth question of your own to QUESTIONS and its route to EXPECTED.

Little by little, you're building something great.