CSE Roadmap

Core tier · about 4 hours · 8 parts

Part 5 of 8 · 2 min read

Reliability

At least once, then remove duplicates.

Never lose a notification. The book says notifications may be delayed or reordered, but must not be lost. To meet this, the system saves notification data in a notification log database and has a retry mechanism. If a worker crashes after taking an event but before the provider confirms, the event can be found in the log and sent again.

Retries. When the provider fails, the event goes back on the queue to be tried again, up to a set number of times. If the problem continues, an alert goes to the developers. Use the habits of Level 8: backoff with jitter, a retry limit, and retries at one level only.

A worked example (made up). Suppose 5 per cent of push calls fail at the first try, and a retry succeeds nearly always. Of 10 million push messages, 500,000 are retried once, and about 25,000 a second time. Total provider calls are about 10,525,000, which is 5.25 per cent more load. The queue and retry rate must be sized with this extra.

Exactly once? The book's answer is no. Delivery is exactly once most of the time, but in a distributed system a retry after a lost confirmation can send the same message twice. The system aims for at-least-once delivery and reduces duplicates with a dedupe step:

on event arrival:
    if event_id seen before:  discard
    else:                     mark event_id as seen; send the notification

Every notification event carries a unique id (module 2 shows how to make them). The set of seen ids is kept in a fast store. Keep ids for a limited time only, say 7 days. A rough size (made up): 16 million ids a day × 16 bytes × 7 days is about 1.8 GB, which fits in a cache.

There is a catch the book hints at: "mark seen" and "send" cannot be done as a single atomic act, because the send is to an outside system. If the worker marks, then crashes before sending, the message is lost. If it sends, then crashes before marking, a retry sends twice. The choice is between at-most-once and at-least-once. For notifications, at-least-once is the usual pick, because a rare duplicate is better than a lost message. Say this out loud.

At-least-once delivery

At-least-once delivery means every message is delivered one or more times, never zero, so duplicates are possible and must be handled.

Example: a notification is resent after a lost confirmation, and the device shows it twice unless a dedupe step removes the second.

Check yourself: 1. Why does the log help after a worker crash? 2. Can the system promise exactly once? Why not? 3. Why is at-least-once preferred for notifications?