Retries and a fallback model

Hosted models fail sometimes: a timeout, a rate limit, an outage. ModelRetryMiddleware tries the same model again, and ModelFallbackMiddleware moves on to another one.

Two stand-ins that fail the way a provider does. FlakyModel fails twice and then works; DownModel never works.

Exampleflaky_model.py

from shop_model import ShopModel

FAILURES = [ConnectionError("provider unavailable")] * 2


class FlakyModel(ShopModel):
    def _generate(self, messages, stop=None, run_manager=None, **kwargs):
        if FAILURES:
            raise FAILURES.pop()
        return super()._generate(messages)


class DownModel(ShopModel):
    def _generate(self, messages, stop=None, run_manager=None, **kwargs):
        raise ConnectionError("provider unavailable")

Exampleagent.py

from langchain.agents import create_agent
from langchain.agents.middleware import ModelFallbackMiddleware, ModelRetryMiddleware
from flaky_model import DownModel, FlakyModel
from shop_model import ShopModel
from tools import lookup_order

Example

agent = create_agent(FlakyModel(), tools=[lookup_order])
agent.invoke({"messages": [{"role": "user", "content": "Where is A17?"}]})

LangChain's hosted chat models already retry a failed request up to six times by default, for network errors, rate limits and server errors, though not for a wrong key. This error came from inside the model, so nothing retried it.

Trying again

Example

retry = ModelRetryMiddleware(max_retries=2, initial_delay=0)
agent = create_agent(FlakyModel(), tools=[lookup_order], middleware=[retry])
print(agent.invoke({"messages": [{"role": "user", "content": "Where is A17?"}]})["messages"][-1].text)

Two failures, a third attempt, a normal answer. max_retries=2 means three attempts in all. The defaults wait one second before the first retry and double the wait each time, with some randomness, up to a minute; initial_delay=0 keeps this example quick.

Example

retry = ModelRetryMiddleware(max_retries=2, initial_delay=0)
agent = create_agent(DownModel(), tools=[lookup_order], middleware=[retry])
print(agent.invoke({"messages": [{"role": "user", "content": "Where is A17?"}]})["messages"][-1].text)

When every attempt fails, the default on_failure="continue" ends the run with an AI message describing the failure instead of raising.

Another model

Example

backup = ModelFallbackMiddleware(ShopModel())
agent = create_agent(DownModel(), tools=[lookup_order], middleware=[backup])
print(agent.invoke({"messages": [{"role": "user", "content": "Where is A17?"}]})["messages"][-1].text)

ModelFallbackMiddleware takes one or more models to try, in order, when the main one raises. With hosted models, the fallback is usually a different provider, so one outage does not take the shop's support desk down.

Try it yourself

Set on_failure="error" on the retry and run it against DownModel.
Give ModelFallbackMiddleware two models, a DownModel first, and check which one answers.
Put retry and fallback in one list and work out which runs first before you try it.

PreviousStopping a runaway agent

Little by little, you're building something great.