Part 2 of 8 · 2 min read
Requirements and numbers
Three channels, soft real time.
Think of a post office with three counters: one for urgent telegrams, one for text messages, one for letters. Many departments drop messages at the front desk. The desk must sort them, put each in the right tray, hand trays to the right carriers, and keep a record so nothing is lost if a carrier is late.
A notification is more than a phone push. The book names three formats: mobile push, SMS and email.
Questions and the answers assumed here (from the book's sample talk).
- Which types? Push, SMS and email.
- Real time? Soft real time. Notifications should arrive as soon as possible, but a short delay under heavy load is fine.
- Which devices? iOS, Android and laptops or desktops.
- What triggers them? Client apps, and schedules on the server.
- Can users opt out? Yes. Those who opt out receive nothing.
- How many a day? 10 million push, 1 million SMS and 5 million emails.
Requirements.
- R1. Services call an API to send a notification of a type to a user.
- R2. A notification is never lost. It may be delayed or reordered.
- R3. Duplicates are reduced as far as possible.
- R4. Opted-out users are respected.
- R5. A fault in one channel or provider does not stop the others.
- R6. Users are not flooded with too many notifications.
- R7. The system tracks delivery and opens.
Numbers. In total there are 16 million notifications a day, which is 16,000,000 / 86,400 = about 185 per second on average. Assume peaks of 3 times the average (made up): push 348 per second, SMS 35 and email 174.
How many workers are needed? Use Little's law: concurrent calls = rate × time per call. Suppose a push call to the provider takes 150 ms, an SMS call 300 ms and an email call 200 ms (made up).
- Push: 348 × 0.15 = about 52 calls at once.
- SMS: 35 × 0.3 = about 11.
- Email: 174 × 0.2 = about 35.
These are small numbers. The load itself is easy. The hard parts are reliability, third-party limits, and the sudden bursts when a service sends millions of messages at once.
Soft real time
A soft real-time system aims to deliver as fast as it can, but a delay under heavy load is acceptable and does not count as a failure.
Example: a push notification that arrives 3 seconds late during a sale.
Check yourself: 1. Why is "soft" real time enough here? 2. What does Little's law give? 3. Why are the load numbers not the hard part?