CSE Roadmap

Core tier · about 3 hours · 8 parts

Part 6 of 8 · 3 min read

Moving state out of the server

A list of the usual kinds of hidden state and where each one belongs.

Think of moving house. Everything stored in one room has to be carried to a place that every family member can reach.

Hidden state is the usual reason a service does not scale. A review of a service often finds these:

Hidden state Problem New home
Login sessions in memory Lost on restart, tied to one server Shared session store, or signed tokens
Shopping cart in memory Same Database or shared store, keyed by user
Uploaded files on the local disk Other servers cannot see them Object storage, with the path saved in the database
Temporary results in a local file Lost when the server is replaced Shared store with an expiry
An in-memory counter (page views, rate limits) Each server counts only its share Shared counter store (page 7)
Scheduled jobs run by every server The job runs several times One scheduler, or a lock in a shared store
Settings read from local files at start Servers drift apart Central configuration
Logs on the local disk Lost when the server goes Shipped to a central log system (Level 9)

A worked example, an upload. A user uploads a profile picture. Server 2 saves it to its local disk. The next page load goes to server 4, which finds no picture.

The fix:

  1. The server writes the picture to object storage (a service for files; Level 2), under a name like avatars/7.jpg.
  2. The user record in the database stores that name.
  3. Any server reads the record and builds the picture's URL, which the CDN (page 4) serves.

After the fix, a server can be thrown away at any time.

The benefits show in the numbers. Take a service with a peak of 12,000 requests per second, where one server handles 1,000.

  • Servers needed at peak: 12,000 ÷ 1,000 = 12, plus 1 spare = 13.
  • Off-peak is 2,000 per second. Servers needed: 2 + 1 = 3.
  • With stateless servers, autoscaling (adding and removing servers by load; the book names this as an advantage of removing state) can run 3 servers at night and 13 at peak. Over a day, if the peak lasts 4 hours, the average is about (4 × 13 + 20 × 3) ÷ 24 = 4.7 servers.
  • With sticky stateful servers, scaling down kills user sessions, so 13 would stay on all day. The saving of stateless design is about 13 − 4.7 = 8.3 servers, or 64 percent.

The shared store becomes a new vital part. Two cautions from the books:

  • The book "Understanding Distributed Systems" warns that when every instance uses the same data store, the store can become the bottleneck, and adding instances only strains it further.
  • The book "System Design Interview" notes that with several data centres, users are routed to the nearest one by geo DNS. State must then be reachable from each data centre, which is a reason to prefer small signed tokens, or a store that is copied across regions (Level 4).

Read it as

For each variable that outlives one request, ask: if this server vanished now, what would be lost? If the answer is "something important", move it.

Check yourself:

  • Where should an uploaded profile picture live, and what should the database hold?
  • Why can a shared session store become a bottleneck?