Part 2 of 7 · 2 min read
Why limit, and where
The reasons a service refuses some requests, and the places a limiter can sit.
Think of a nightclub with a door person. The room holds 200 people. If 2,000 try to enter at once, the door person lets in a few at a time and asks the rest to wait. The club stays safe and people inside enjoy it.
A rate limiter does this for requests.
Rate limiter
A rate limiter controls how many requests a client or a service may send in a period of time. Requests beyond the limit are blocked or delayed.
Example: a rule says each user may make 100 calls per minute. The 101st call in that minute is refused.
The book "System Design Interview" gives examples of rules: at most 2 posts per second per user, 10 accounts per day from one IP address, 5 reward claims per week from one device.
It also lists three reasons to limit.
- Protect against overload and attacks. A flood of requests, whether on purpose (a denial-of-service attack) or by accident (a bug in a client), can starve other users. A limiter blocks the excess.
- Reduce cost. Fewer extra requests mean fewer servers. It matters most when each call to a paid outside service costs money, such as a credit check or a payment.
- Keep servers from overloading. Filter out bots and misbehaving users before they reach the heavy work.
Throttling
Throttling is slowing down or refusing requests that exceed a limit.
Example: a user who sends 10 login attempts in a second is throttled after the fifth.
Where does the limiter live? The book weighs three places.
| Place | Notes |
|---|---|
| In the client | Unreliable. Clients can be forged, and the owner may not control them |
| In the API servers | Full control over the algorithm, but every service repeats the code |
| In a middleware or gateway | One place for all services, often with other jobs such as authentication and TLS |
Middleware
Middleware is a piece of software that sits between the caller and the application and handles each request on the way through.
Example: a middleware checks the rate limit and either passes the request to the API servers or answers with an error.
The book's guidelines: look at the technology the company already uses; pick the algorithm that fits the business, remembering that third-party gateways may offer fewer choices; if a gateway already exists for authentication, add the limiter to it (page 6); and, when engineering time is short, buy a commercial gateway instead of building one.
A worked example. An API allows 2 requests per second per client. A client sends 3 requests in one second.
- Requests 1 and 2 pass to the API servers.
- Request 3 is blocked by the limiter, which answers with status 429, "too many requests". The API servers never see it.
Read it as
A rate limiter is a cheap early "no" that protects an expensive "yes".
Check yourself:
- Give two reasons for limiting the rate of requests.
- Why is the client a poor place to enforce a limit?