Part 6 of 8 · 3 min read
Making it fast and safe
Files, snapshots, Raft, shards.
Faster with local files. Calling out to a message service and a remote database for every event is slow.
- Keep the command list and event list as append-only local files. Sequential writes are fast, even on spinning disks.
- Cache recent entries in memory. A technique called mmap maps a file into memory so the operating system keeps the recent parts cached while the file is still written to disk.
- Keep the state on local disk, too, in a file-based database such as SQLite or RocksDB (a write-friendly store built on an LSM tree, Level 12, page 5).
Snapshots. Replaying from the first event gets slow over time. Periodically, the state machine saves a snapshot: an immutable copy of the state at one moment. To rebuild a state, load the latest snapshot before the target time, and replay only the events after it. Finance teams often want a snapshot at midnight each day. Snapshots are large files, so they go to object storage.
Reliability. Files on one server are a single point of failure. Which data must be protected?
- State and snapshot can be rebuilt from events.
- Commands cannot be relied on to rebuild events, because event creation may involve randomness or outside reads.
- So the event list is the one data set that must never be lost.
Replicate the event list over several nodes with a consensus algorithm such as Raft (Level 5). Raft chooses a leader, which takes commands, turns them into events and appends them. Followers get the same entries in the same order. The log stays safe and available while a majority of the nodes are running. 3 nodes tolerate 1 failure and 5 nodes tolerate 2. All nodes apply the events to their own state, so all states agree.
- If a leader crashes, Raft elects a new leader from the healthy nodes. A command that had not become an event is lost, so the client sees a timeout or error and retries (with an idempotent id).
- If a follower crashes, Raft keeps retrying until it comes back or is replaced.
Scaling past one group. One Raft group has limited capacity.
- Fast results to the client: the pull model (client polls) is slow or overloads the service. Put a reverse proxy in front of the event sourcing nodes. It forwards the command and polls for the result, and the read-only state machine can push the status to the proxy as soon as the event is applied. The client sees a quick response.
- Sharding: split accounts across several groups, for example by the hash of the account. Each group is a Raft cluster running event sourcing. A transfer between accounts in different groups is a distributed transaction again. A saga coordinator (or TC/C) sends "A − $1" as a command to group 1, waits for the result from its read path, and then sends "C + $1" to group 2. On failure, it sends the compensations. The coordinator records progress in a phase status table.
A worked example (made up). 1 million transfers a second at 2 events each is 2 million events a second. If one Raft group handles 40,000 events a second, 2,000,000 / 40,000 = 50 groups. With 3 nodes each, that is 150 nodes. This is far fewer than the 2,000 database nodes of the first estimate, because each event is a cheap append.
Read it as
Make the hot path a sequential write to a local file, and make everything else (state, views, snapshots) something that can be rebuilt from it.
Check yourself: 1. Why are only events protected strongly? 2. What does a snapshot save? 3. How does a cross-group transfer work?