CSE Roadmap

Specialist tier · about 5 hours · 8 parts

Part 6 of 8 · 3 min read

Availability

Hot-warm, Raft and replay.

The goal. 99.99 per cent means no more than 8.64 seconds of downtime a day. Recovery must be almost immediate.

Hot-warm.

  • Stateless parts such as the gateway are scaled by adding servers.
  • Stateful parts (order manager, matching engine) run in pairs: a hot primary, and a warm copy that receives and processes exactly the same events but sends nothing out.
  • If the primary fails, the warm copy takes over and starts publishing events. If the warm copy fails, it restarts and rebuilds its state from the event store.
  • Failure is detected with monitoring and heartbeats: the engine sends a regular heartbeat, and a missed heartbeat signals trouble.

Beyond one server. The pair works inside one server. To survive the loss of a server or a data centre, the whole event store is copied to warm servers, using a fast one-way protocol such as reliable UDP (a library such as Aeron does this).

When the warm copy also fails. Replicate core data to other cities. Decide in advance:

  1. When to fail over, and who decides.
  2. Which server takes over.
  3. The recovery time objective (RTO): how long a stop is allowed. For an exchange this is seconds, so failover is automatic.
  4. The recovery point objective (RPO): how much data may be lost. For an exchange, almost none.

What does "down" mean? False alarms cause needless failover. A bug may take down the primary and the same bug will then take down the backup. Early in a system's life, failover is manual. Automation comes when there is enough experience. Chaos engineering (breaking things on purpose, Level 8) builds confidence.

Leader election with Raft. Use a Raft group (page 9), for example 5 servers, each with its own event store. Operations need a majority (N/2 + 1 = 3). The leader sends heartbeats, which are empty append messages. A follower that misses heartbeats times out, becomes a candidate and asks for votes. If it wins a majority, it is the new leader. A candidate with an older term cannot win. If two followers become candidates at once, the vote splits, the election times out, and another begins. Since events are replicated to a majority, a new leader has all committed events and can serve at once, which gives an RPO near zero.

Two kinds of determinism.

  • Functional determinism: the same events in the same order always give the same results. Sequencing and event sourcing provide it. Because only the order matters, and not the real times, replay can run as fast as possible.
  • Latency determinism: almost every trade takes about the same time. It is measured at the 99th or 99.99th percentile. Sources of spikes include garbage collection pauses in languages such as Java. They must be found and removed.

A worked example (made up). The hot engine dies at 10:00:00.000. The warm engine misses 3 heartbeats of 100 ms and takes over at 10:00:00.300. It has already applied every event, so it continues from the next sequence number. No order is lost, and the pause is 0.3 seconds, inside the 8.64 seconds daily budget.

Read it as

Determinism turns recovery into replay. If the same events always give the same state, a spare copy only needs the events.

Check yourself: 1. What does the warm copy do while the primary runs? 2. Why can failover be wrong if "down" is not defined carefully? 3. What do RTO and RPO each measure?