Your first async program: Async and concurrency

The four words you need (coroutine, async, await, event loop) and a program that does three one-second jobs in one second.

Here is the same job written with async:

import asyncio

async def fake_model(prompt: str) -> str:
    await asyncio.sleep(1)          # stands in for a network call
    return f"answer to {prompt}"

async def main() -> None:
    answers = await asyncio.gather(fake_model("a"), fake_model("b"), fake_model("c"))
    print(answers)

asyncio.run(main())

['answer to a', 'answer to b', 'answer to c']
async gather: 1.0s

Three seconds became one. Nothing got faster. The three waits happened at the same time. A few new words made that possible.

Coroutine, async and await

A function defined with async def is a coroutine function. Calling it does not run it. It returns a coroutine object, which is a job that is ready to start. await runs a coroutine and pauses the current function until the result is ready, which lets other jobs run in the meantime.

Event loop

The scheduler inside asyncio that runs the jobs. It keeps a list of them, runs one until it reaches an await on something that is not ready yet, then switches to another. Only one job runs at any instant, but while one waits the others make progress. Jobs cooperate: each one must hand control back at every await.

The event loop only works if every task hands control back quickly.

In the program above, fake_model stands in for a real model call. The name fake_model is just a name I chose: it is not a real model, it only waits (asyncio.sleep) and returns a string, so we can practise without spending money. asyncio.gather(...) starts all three at once and waits until all are done, returning the answers in order.

Rules to remember:

You can only use await inside an async def function.
asyncio.run(main()) starts the event loop. Call it once, at the top of your program.
Calling fake_model("a") without await or gather does nothing, and Python warns you:

RuntimeWarning: coroutine 'fake_model' was never awaited

That warning almost always means you forgot an await.

Check yourself

What does await do while it waits?
What happens if you call an async def function and forget await?
Why does the event loop need every job to hand control back at its await?