CSE Roadmap

Core tier · about 3 hours · 7 parts

Part 2 of 7 · 2 min read

Why limit, and where

The reasons a service refuses some requests, and the places a limiter can sit.

Think of a nightclub with a door person. The room holds 200 people. If 2,000 try to enter at once, the door person lets in a few at a time and asks the rest to wait. The club stays safe and people inside enjoy it.

A rate limiter does this for requests.

Rate limiter

A rate limiter controls how many requests a client or a service may send in a period of time. Requests beyond the limit are blocked or delayed.

Example: a rule says each user may make 100 calls per minute. The 101st call in that minute is refused.

The book "System Design Interview" gives examples of rules: at most 2 posts per second per user, 10 accounts per day from one IP address, 5 reward claims per week from one device.

It also lists three reasons to limit.

  1. Protect against overload and attacks. A flood of requests, whether on purpose (a denial-of-service attack) or by accident (a bug in a client), can starve other users. A limiter blocks the excess.
  2. Reduce cost. Fewer extra requests mean fewer servers. It matters most when each call to a paid outside service costs money, such as a credit check or a payment.
  3. Keep servers from overloading. Filter out bots and misbehaving users before they reach the heavy work.

Throttling

Throttling is slowing down or refusing requests that exceed a limit.

Example: a user who sends 10 login attempts in a second is throttled after the fifth.

Where does the limiter live? The book weighs three places.

Place Notes
In the client Unreliable. Clients can be forged, and the owner may not control them
In the API servers Full control over the algorithm, but every service repeats the code
In a middleware or gateway One place for all services, often with other jobs such as authentication and TLS

Middleware

Middleware is a piece of software that sits between the caller and the application and handles each request on the way through.

Example: a middleware checks the rate limit and either passes the request to the API servers or answers with an error.

The book's guidelines: look at the technology the company already uses; pick the algorithm that fits the business, remembering that third-party gateways may offer fewer choices; if a gateway already exists for authentication, add the limiter to it (page 6); and, when engineering time is short, buy a commercial gateway instead of building one.

A worked example. An API allows 2 requests per second per client. A client sends 3 requests in one second.

  • Requests 1 and 2 pass to the API servers.
  • Request 3 is blocked by the limiter, which answers with status 429, "too many requests". The API servers never see it.

Read it as

A rate limiter is a cheap early "no" that protects an expensive "yes".

Check yourself:

  • Give two reasons for limiting the rate of requests.
  • Why is the client a poor place to enforce a limit?