Part 3 of 8 · 2 min read
First design
One check in front of the API.
The simplest design puts a middleware (a check that every request passes through) in front of the API servers. It reads the client's counter, decides, and either passes the request on or refuses it.
client --> [ rate limiter middleware ] --> API servers
| ^
v |
counters (in-memory store)
The Xu book explains that this middleware is often part of an API gateway, a managed front door that also does TLS, login and routing (Level 1, page 6). Where to put the limiter has no single right answer. The book's guidelines are:
- Look at the stack already in use, and pick what the language and cache support well.
- Choose the algorithm that fits the need. A third-party gateway may limit the choices.
- If a gateway already exists, add the limiter there.
- Building a service takes time, so a commercial gateway is better when engineers are few.
The flow for one request:
- The request arrives at the middleware.
- The middleware loads the rule for this client and endpoint, from a local cache of rules.
- It reads and updates the counter in the shared store.
- If under the limit, it forwards the request. If over, it answers with status 429 (too many requests).
The counters live in an in-memory store such as Redis. A database is too slow, since it works on disk. The store also supports expiry: a counter can be told to vanish when its window ends, so memory is cleaned up for free. The book names two store commands that matter: one adds one to a counter, and one sets a timeout on it.
Middleware
Middleware is code that sits between the caller and the real service and handles every request before the service sees it.
Example: a check that counts the request and either passes it on or answers "too many requests".
Check yourself: 1. Why not store counters in a normal database? 2. What status code does a refused request get? 3. When is a commercial gateway a better choice than building one?