Part 7 of 8 · 2 min read
Putting it together
Who protects what.
Putting it together
| Tool | Where | Decides by | Protects |
|---|---|---|---|
| Rate limit | At the edge or gateway | Global usage per client | The service from one client |
| Load shedding | In each server | Local concurrent requests | The server from total load |
| Load levelling | Between caller and service | Queue length | The service from short spikes |
| Auto-scaling | The platform | Utilisation | Capacity for longer rises |
| Graceful degradation | In the application | A planned ladder | The user's core task |
UDS's experience (page 1) is that these simple techniques let services double their traffic almost overnight without incident.
Interview angle
Answer in your head first, then open it.
What do you do when traffic doubles suddenly?
Shed low-priority work, level the load through a queue where an answer is not needed at once, rate limit abusive clients, and scale out.
What is the difference between load shedding and rate limiting?
Shedding uses the local state of one server. Rate limiting uses the global usage of one client.
Name two rate limiting algorithms and a weakness of each.
Token bucket: tuning two settings. Fixed window: double quota at the edges.
What status codes are used?
503 for shedding, 429 for rate limiting, with Retry-After.
What is graceful degradation?
A planned way to drop the least important features so the core keeps working.
Check yourself: 1. Which tool would you use against one client that sends too much? 2. Which tool works only when the caller can wait? 3. Which tool protects each server separately?
Words you will meet
| Word | Meaning |
|---|---|
| Overload | More work than a service can finish at an acceptable speed |
| Load shedding | Rejecting some requests to protect the rest |
| Load levelling | A queue that lets the service pull work at its own pace |
| Auto-scaling | Adding instances automatically when the service runs hot |
| Rate limiting | A quota per client, with rejection when it is used up |
| 429 and 503 | HTTP statuses for "too many requests" and "service unavailable" |
| Token bucket | A rate limiting algorithm that allows short bursts |
| Leaking bucket | A rate limiting algorithm with a fixed output rate |
| Graceful degradation | Dropping the least important features to keep the core |
| Deadline | The time after which an answer is no longer useful |