Rate limits, retries and concurrency

Providers cap your usage with rate limits, commonly expressed as requests per minute and tokens per minute. Exceeding them returns an error (HTTP 429, Level 0, module 3), often with advice on when to retry.

Token bucket

bucket = TokenBucket(rate=2, capacity=5, clock=time.monotonic)
if bucket.try_acquire():
    call_the_api()
else:
    time.sleep(bucket.wait_time())

The clock is passed in, so the tests advance time by hand instead of sleeping. With rate=2 and capacity=5, real results are: five requests succeed at once (the burst), the sixth is refused, wait_time() then says 0.5 s, and one simulated second later two more are allowed.

Retries with exponential backoff and jitter

Temporary errors (a rate limit, a timeout, a server overload) usually succeed if you try again later. Exponential backoff waits 1 s, 2 s, 4 s, 8 s and so on, up to a cap. Jitter randomises each wait. Without it, a thousand clients that failed together will all retry together and fail together again.

backoff_delays(7, base=1, cap=30, jitter=False)   # [1, 2, 4, 8, 16, 30, 30]
backoff_delays(7, base=1, cap=30)                 # [0.84, 1.52, 1.68, 2.07, 8.18, 12.15, 23.51]

The second line is "full jitter": each wait is a random time between 0 and the plain delay. Retry only what is safe to repeat. A read can be retried freely. A tool that sends an email or charges a card must be made idempotent (safe to run twice) or protected by a unique request key, or you will send two emails (Level 0, module 3).

How many requests in flight?

Little's law says the average number of requests in progress equals the arrival rate times the time each takes. At 50 requests per second and 4 seconds each, expect 200 in flight at once. That number sizes your connection pool, your worker count and your provider concurrency limit.