Part 5 of 12 · 2 min read
Smoothing bursts and isolating failures
A token bucket spreads out a burst instead of rejecting it; a bulkhead keeps one hung tool from freezing every worker.
Think of a funnel versus a narrow doorway during a fire drill. A funnel lets the same crowd through smoothly over a few seconds. A doorway with no funnel just turns most of the crowd away at once.
In plain words: a rate limiter that smooths a burst over time avoids rejecting most of it. Giving each dependency its own pool of workers means one hung dependency cannot freeze the workers a healthy one needs.
A worked example (rate limiting). A burst of 100 requests against a provider that allows 10 per second: with no limiter, 90 are rejected outright. With a token-bucket limiter smoothing the same burst, 0 are rejected — it just takes 9.5 seconds to get them all through instead of failing most of them in the first second.
A worked example (bulkheads). Ten workers are shared between a healthy tool A and a tool B that hangs for 30 seconds on every call. A's success rate collapses to 3 percent, because B's hung calls occupy every worker in the shared pool. Splitting the same ten workers into two pools of five gives A a success rate of 100 percent — B is still broken, but it can no longer touch A's workers.
The key lines.
# from code/mini-reliability/reliability.py
class TokenBucket:
def take(self, now):
self.tokens = min(self.cap, self.tokens + max(0.0, now - self.t) * self.rate); self.t = max(self.t, now)
if self.tokens >= 1: self.tokens -= 1; return True
return False
Read it as: tokens refill steadily over time, and a request can only proceed once a whole token is available. This is what turns a sudden burst into a steady trickle instead of a wall of rejections. Bulkheads apply the same idea to workers instead of requests. Each tool or tenant gets its own pool. So a failure in one cannot borrow capacity from another.
Bulkhead, token bucket, load shedding
A bulkhead is a separate resource pool per tool or tenant so one failure cannot consume everything. A token bucket is a rate limiter that allows small bursts but refills steadily. Load shedding is deliberately rejecting some work to protect the rest.
Example: the 5+5 split above, which took tool A from 3 percent success to 100 percent with no change to tool B at all.