CSE Roadmap

Core tier · about 4 hours · 8 parts

Part 4 of 8 · 2 min read

Active-passive and active-active

A spare that waits, or a spare that works.

With two sites, there are two basic shapes.

Active-passive and active-active

Active-passive means one site serves traffic and the other waits as a spare. Active-active means both serve traffic at the same time, and each can take over the other's load.

Example: a primary database with a standby that takes over (active-passive), and two data centres each serving half the users (active-active).

Xu's first chapter describes an active-active setup. In normal operation, users are routed by geoDNS (a DNS service that gives an address by the user's location) to the closest data centre, with a split such as x percent to US-East and the rest to US-West. If a data centre goes offline, all traffic goes to the healthy one. Xu names three hard problems:

  1. Traffic redirection: tools to send users to the right data centre.
  2. Data synchronisation: users in different regions may use different local databases and caches. In a failover, traffic may be sent where the data is missing. A common strategy is to replicate data across data centres, and Netflix is cited as using asynchronous multi-data-centre replication.
  3. Testing and deployment: the application must be tested at each location, and automated deployment keeps all sites consistent.

UDS covers the data side in its chapter on duplication. In multi-leader replication, several nodes accept writes, for instance one leader in each data centre. Replication between them is asynchronous, so conflicting writes become possible. The book says it is generally best avoided when possible, since it adds complexity. A simple way out is a homing region: if all European customers always go to the European data centre with a single leader, no conflicts arise. A second data centre in the same region, fed by single-leader replication, covers the loss of the first. Where conflicts are unavoidable, the book describes resolving them by returning concurrent writes to a client, or by automatic rules such as last write wins. It warns that last write wins is unreliable because physical clocks are not perfectly synchronised.

Xu's key-value store chapter adds the same message for data: replicate data across multiple data centres, and even if one data centre is fully offline, users can still reach the data through the others.

Active-passive Active-active
Cost Pay for a spare that does little Both sites busy, so little waste
Failover Needs a switch, which may fail or be slow Traffic already flows to both
Data One writer, simpler Conflicts possible unless data is homed
Risk The spare may be untested and broken Each site must be able to carry the whole load

Check yourself: 1. What does geoDNS do? 2. Why does multi-leader replication risk conflicts? 3. What is the main risk of a passive spare?