CSE Roadmap

Core tier · about 4 hours · 8 parts

Part 6 of 8 · 2 min read

Trade-offs

Speed, accuracy and what happens when it breaks.

If the store is down. The requirement says the API must keep serving. So the limiter should fail open: when it cannot reach the store, it lets the request through, and an alarm tells the team. A limiter that fails closed would take the whole product down with it. The cost is that, for a while, limits are not enforced. For a limiter that guards against abuse of something costly or dangerous, such as password guesses, a team may choose to fail closed for that one rule. State the choice and the reason.

Speed against accuracy across regions. Users far from the data centre see high latency, so the book advises many data centres, often at edge locations near users. Each region then has its own store. Keeping all stores exact would need a trip between regions on each request, which is too slow. The usual answer is eventual consistency: regions share counts in the background, so a client can briefly exceed a global limit. That is acceptable when the limit is a guard rail and not an exact contract.

Hard and soft limits. A hard limit never lets the count pass the threshold. A soft limit allows a short overshoot. Soft limits are cheaper and often good enough.

Cost per check. Each request makes one call to the store. At 500,000 per second, that is a lot of network traffic. Two reductions: batch a few increments locally and send them together (less exact), or keep a tiny local counter that settles with the store every few hundred milliseconds. Both trade exactness for load.

Monitoring. After launch, the book says to check that the method and the rules work. If the rules bite too hard, honest users get turned away, so loosen them. If a flash sale breaks the limiter's effect, switch to a method that handles bursts, such as the token bucket. Count refusals per rule and per client, watch latency added by the check, and alert on store errors.

A worked example (made up). A limiter adds 1 ms to the median request and 6 ms to the 99th percentile because of the store hop. The goal was 2 ms. The team adds a local fast path: clients far under their limit skip the store check and reconcile every 200 ms. The median falls to 0.1 ms, and the accuracy loss is a few extra requests per client per second.

Common mistake

Reading the counter, comparing, and writing it back as three separate steps. Under load, two requests read the same value and the limit leaks. Make check and update one atomic step in the store.

Check yourself: 1. What does fail open mean, and what does it cost? 2. Why is eventual consistency acceptable between regions? 3. How does a local counter lower the load on the store?