Timeouts and retries: when a call fails

Calls over a network fail for reasons outside your code: a busy server, a dropped connection. A program that talks to a model has to expect that.

A call that takes too long

Example

import asyncio

async def stuck_model(text):
    await asyncio.sleep(10)
    return "too late"

async def main():
    try:
        answer = await asyncio.wait_for(stuck_model("ticket 1"), timeout=0.5)
    except asyncio.TimeoutError:
        print("no answer after 0.5 seconds, moving on")

asyncio.run(main())

asyncio.wait_for awaits a coroutine but gives up after timeout seconds, cancels it, and raises asyncio.TimeoutError. Without it, one stuck call holds up everything waiting on it.

A model that fails, then works

To practise retries you need a call that fails a set number of times. A class, from lesson 19, can count its own calls:

Example

import asyncio

class FlakyModel:
    def __init__(self, failures):
        self.failures = failures
        self.calls = 0

    async def ask(self, text):
        self.calls += 1
        await asyncio.sleep(0.1)
        if self.calls <= self.failures:
            raise ConnectionError("the server did not answer")
        return '{"category": "billing", "priority": 4}'

ask is a method written with async def, awaited like any coroutine. It counts every call, and raises until it has failed failures times.

Example

async def main():
    model = FlakyModel(failures=2)
    for _ in range(3):
        try:
            print(await model.ask("refund"))
        except ConnectionError as error:
            print("error:", error)

asyncio.run(main())

The first two calls raise and the third answers. _ is the usual name for a loop variable you do not use.

Trying again

Example

async def ask_with_retries(model, text, attempts=3):
    for attempt in range(1, attempts + 1):
        try:
            return await model.ask(text)
        except ConnectionError as error:
            print(f"attempt {attempt} failed: {error}")
            await asyncio.sleep(0.2 * attempt)
    raise ConnectionError(f"gave up after {attempts} attempts")

async def main():
    print(await ask_with_retries(FlakyModel(failures=2), "refund"))

asyncio.run(main())

The return inside the try leaves the function the moment a call works. After a failure it waits a little longer each time, 0.2 then 0.4 seconds, so a busy server gets room to recover. Waiting longer after each failure is called backoff.

If every attempt fails, the loop ends and the last line raises, so the caller learns the call did not work instead of receiving None.

Example

async def main():
    await ask_with_retries(FlakyModel(failures=5), "refund")

asyncio.run(main())

Try it yourself

Pass attempts=5 to the last call.
Wrap model.ask(text) inside ask_with_retries in asyncio.wait_for with a timeout, and catch asyncio.TimeoutError too.
Change failures=2 to failures=0.

Previousasyncio.gather: many calls at once

Slow is fine. Stopping is the only problem.