Part 6 of 8 · 1 min read
Working like a professional
What teams do in practice when they run many model calls at once, and how this comes up in interviews.
What you will use daily at work
| Practice | What it looks like |
|---|---|
| A shared async client | One httpx.AsyncClient (or the provider's async client) for the whole program, closed at shutdown. |
| A limit on every fan-out | A semaphore sized from the provider's documented rate limits, kept in config, not in code. |
| A timeout on every external call | Per request, plus an overall deadline for the whole agent step. |
| Retries with backoff inside the limit | See module 3. Retry inside the semaphore so a retrying call keeps its slot. |
| Structured logging with a request ID | So that interleaved logs from many concurrent calls can be told apart. |
| A clean shutdown path | On "stop" or on process exit, cancel in-flight tasks and wait for their finally blocks. |
| Watching for blocked loops | Python's debug mode (PYTHONASYNCIODEBUG=1) logs callbacks that take too long. |
Draft: to be checked against the current Python asyncio documentation before this page goes live.
Interview angle
Answer in your head first, then open it.
What goes wrong when blocking code runs inside an async function?
It blocks the single event-loop thread, so every other task, timeout and heartbeat stalls. Use async libraries or asyncio.to_thread.
How do you call 500 APIs without exceeding a rate limit?
asyncio.Semaphore(N) around each call, retries with backoff, gather(..., return_exceptions=True).
gather versus TaskGroup?
gather can return partial results; TaskGroup cancels siblings on the first failure and raises an ExceptionGroup.
Concurrency versus parallelism?
Interleaving versus truly simultaneous execution. In standard Python, threads and async do not speed up CPU-bound work because of the global interpreter lock; use processes for that.
How do you stop an agent cleanly when the user presses stop?
Cancel the tasks, release resources in finally, and never swallow CancelledError.