CSE Roadmap

Core tier · about 3 hours · 8 parts

Part 7 of 8 · 2 min read

Putting it together

Who protects what.

Putting it together

Tool Where Decides by Protects
Rate limit At the edge or gateway Global usage per client The service from one client
Load shedding In each server Local concurrent requests The server from total load
Load levelling Between caller and service Queue length The service from short spikes
Auto-scaling The platform Utilisation Capacity for longer rises
Graceful degradation In the application A planned ladder The user's core task

UDS's experience (page 1) is that these simple techniques let services double their traffic almost overnight without incident.

Interview angle

Answer in your head first, then open it.

What do you do when traffic doubles suddenly?

Shed low-priority work, level the load through a queue where an answer is not needed at once, rate limit abusive clients, and scale out.

What is the difference between load shedding and rate limiting?

Shedding uses the local state of one server. Rate limiting uses the global usage of one client.

Name two rate limiting algorithms and a weakness of each.

Token bucket: tuning two settings. Fixed window: double quota at the edges.

What status codes are used?

503 for shedding, 429 for rate limiting, with Retry-After.

What is graceful degradation?

A planned way to drop the least important features so the core keeps working.

Check yourself: 1. Which tool would you use against one client that sends too much? 2. Which tool works only when the caller can wait? 3. Which tool protects each server separately?

Words you will meet

Word Meaning
Overload More work than a service can finish at an acceptable speed
Load shedding Rejecting some requests to protect the rest
Load levelling A queue that lets the service pull work at its own pace
Auto-scaling Adding instances automatically when the service runs hot
Rate limiting A quota per client, with rejection when it is used up
429 and 503 HTTP statuses for "too many requests" and "service unavailable"
Token bucket A rate limiting algorithm that allows short bursts
Leaking bucket A rate limiting algorithm with a fixed output rate
Graceful degradation Dropping the least important features to keep the core
Deadline The time after which an answer is no longer useful