Part 6 of 8 · 3 min read
Moving state out of the server
A list of the usual kinds of hidden state and where each one belongs.
Think of moving house. Everything stored in one room has to be carried to a place that every family member can reach.
Hidden state is the usual reason a service does not scale. A review of a service often finds these:
| Hidden state | Problem | New home |
|---|---|---|
| Login sessions in memory | Lost on restart, tied to one server | Shared session store, or signed tokens |
| Shopping cart in memory | Same | Database or shared store, keyed by user |
| Uploaded files on the local disk | Other servers cannot see them | Object storage, with the path saved in the database |
| Temporary results in a local file | Lost when the server is replaced | Shared store with an expiry |
| An in-memory counter (page views, rate limits) | Each server counts only its share | Shared counter store (page 7) |
| Scheduled jobs run by every server | The job runs several times | One scheduler, or a lock in a shared store |
| Settings read from local files at start | Servers drift apart | Central configuration |
| Logs on the local disk | Lost when the server goes | Shipped to a central log system (Level 9) |
A worked example, an upload. A user uploads a profile picture. Server 2 saves it to its local disk. The next page load goes to server 4, which finds no picture.
The fix:
- The server writes the picture to object storage (a service for files; Level 2), under a name like
avatars/7.jpg. - The user record in the database stores that name.
- Any server reads the record and builds the picture's URL, which the CDN (page 4) serves.
After the fix, a server can be thrown away at any time.
The benefits show in the numbers. Take a service with a peak of 12,000 requests per second, where one server handles 1,000.
- Servers needed at peak: 12,000 ÷ 1,000 = 12, plus 1 spare = 13.
- Off-peak is 2,000 per second. Servers needed: 2 + 1 = 3.
- With stateless servers, autoscaling (adding and removing servers by load; the book names this as an advantage of removing state) can run 3 servers at night and 13 at peak. Over a day, if the peak lasts 4 hours, the average is about (4 × 13 + 20 × 3) ÷ 24 = 4.7 servers.
- With sticky stateful servers, scaling down kills user sessions, so 13 would stay on all day. The saving of stateless design is about 13 − 4.7 = 8.3 servers, or 64 percent.
The shared store becomes a new vital part. Two cautions from the books:
- The book "Understanding Distributed Systems" warns that when every instance uses the same data store, the store can become the bottleneck, and adding instances only strains it further.
- The book "System Design Interview" notes that with several data centres, users are routed to the nearest one by geo DNS. State must then be reachable from each data centre, which is a reason to prefer small signed tokens, or a store that is copied across regions (Level 4).
Read it as
For each variable that outlives one request, ask: if this server vanished now, what would be lost? If the answer is "something important", move it.
Check yourself:
- Where should an uploaded profile picture live, and what should the database hold?
- Why can a shared session store become a bottleneck?