CSE Roadmap

Core tier · about 4 hours · 8 parts

Part 3 of 8 · 2 min read

First design

One check in front of the API.

The simplest design puts a middleware (a check that every request passes through) in front of the API servers. It reads the client's counter, decides, and either passes the request on or refuses it.

client --> [ rate limiter middleware ] --> API servers
                 |     ^
                 v     |
             counters (in-memory store)

The Xu book explains that this middleware is often part of an API gateway, a managed front door that also does TLS, login and routing (Level 1, page 6). Where to put the limiter has no single right answer. The book's guidelines are:

  • Look at the stack already in use, and pick what the language and cache support well.
  • Choose the algorithm that fits the need. A third-party gateway may limit the choices.
  • If a gateway already exists, add the limiter there.
  • Building a service takes time, so a commercial gateway is better when engineers are few.

The flow for one request:

  1. The request arrives at the middleware.
  2. The middleware loads the rule for this client and endpoint, from a local cache of rules.
  3. It reads and updates the counter in the shared store.
  4. If under the limit, it forwards the request. If over, it answers with status 429 (too many requests).

The counters live in an in-memory store such as Redis. A database is too slow, since it works on disk. The store also supports expiry: a counter can be told to vanish when its window ends, so memory is cleaned up for free. The book names two store commands that matter: one adds one to a counter, and one sets a timeout on it.

Middleware

Middleware is code that sits between the caller and the real service and handles every request before the service sees it.

Example: a check that counts the request and either passes it on or answers "too many requests".

Check yourself: 1. Why not store counters in a normal database? 2. What status code does a refused request get? 3. When is a commercial gateway a better choice than building one?