CSE Roadmap

Core tier · about 4 hours · 8 parts

Part 4 of 8 · 2 min read

The design

Why one server fails, and how queues fix it.

First design. Services 1 to N (billing, shipping, marketing) call one notification server. The server builds the message and calls the providers. This works at small scale, but the book lists three problems.

  • Single point of failure. One server down means no notifications.
  • Hard to scale. Everything lives together, so databases, caches and the processing parts cannot be scaled on their own.
  • Performance bottleneck. Building pages and waiting for slow provider replies is heavy work, and one server can be overloaded at peak.

Improved design. Three changes.

  1. Run the database and cache as separate systems, away from the notification servers.
  2. Run several notification servers, with scaling that adds or removes them automatically.
  3. Put message queues between the parts, so they no longer depend on each other.
services --> notification servers --> [iOS queue]     --> workers --> APNS
              (auth, checks,           [Android queue]  --> workers --> FCM
               templates, limits)      [SMS queue]      --> workers --> SMS service
                    |                  [Email queue]    --> workers --> email service
              cache + database + notification log

What the notification servers do.

  • Offer an API to services, reachable only from inside or by verified clients, to prevent spam.
  • Check basic things, such as whether an email or phone number looks valid.
  • Fetch the data needed to build the message, from the cache or the database.
  • Put a notification event in the right queue for parallel processing.

Why a queue per channel. The queue acts as a buffer when a lot is sent at once. And one queue for each type means an outage in one provider does not hold up the others. If the SMS service is down, the SMS queue grows, and push and email carry on.

The flow of one notification.

  1. A service calls the notification server's API.
  2. The server fetches the user's info, device token and settings from the cache or database.
  3. It puts an event on the matching queue, such as the iOS queue.
  4. A worker takes the event from the queue.
  5. The worker sends it to the provider.
  6. The provider delivers it to the device.

A worked example (made up). A marketing service sends a campaign of 2 million emails in 10 minutes, which is 2,000,000 / 600 = 3,333 per second, far above the email peak of 174. The email queue absorbs the burst. Workers drain it at the provider's allowed rate, say 500 per second, so the queue empties in 2,000,000 / 500 = 4,000 seconds, about 67 minutes. Push and SMS are untouched. Without the queue, the burst would hit the provider directly and many sends would fail.

Check yourself: 1. Name the three problems of the single-server design. 2. Why is there a separate queue per channel? 3. In the example, how long does the email queue take to drain?