asyncio.gather: many calls at once

Lesson 26 awaited each call before starting the next. asyncio.gather starts them together, so the batch takes as long as the slowest call.

import asyncio
import time

async def ask_model_slowly(text):
    await asyncio.sleep(0.5)
    return f"answer to: {text}"

async def main():
    texts = ["ticket 1", "ticket 2", "ticket 3"]
    start = time.perf_counter()
    answers = await asyncio.gather(*[ask_model_slowly(text) for text in texts])
    print(answers)
    print(f"took {time.perf_counter() - start:.1f} seconds")

asyncio.run(main())

The list comprehension makes three coroutines without awaiting any. gather runs them together and returns their results as a list, in the same order as the coroutines you gave it, whichever finished first.

The * in front of the list unpacks it: gather(*[a, b, c]) is the same call as gather(a, b, c). gather wants separate arguments, not one list.

Not too many at once

Model APIs limit how many requests you may send at a time. Starting a thousand calls together gets most of them refused. A semaphore lets only a set number through at once.

limit = asyncio.Semaphore(2)

async def ask_politely(text):
    async with limit:
        return await ask_model_slowly(text)

async def main():
    start = time.perf_counter()
    answers = await asyncio.gather(*[ask_politely(f"ticket {n}") for n in range(1, 6)])
    print(len(answers), "answers")
    print(f"took {time.perf_counter() - start:.1f} seconds")

asyncio.run(main())

async with limit: waits for one of the two places to be free, runs the block, and gives the place back. Five calls, two at a time, half a second each: three rounds.