Part 4 of 5 · 2 min read
Unreliable clocks
Each machine keeps its own clock, and those clocks can drift apart, making cross-machine time comparisons untrustworthy.
Unreliable clocks
Think of several travelers each wearing their own wristwatch, none of them perfectly synchronized. They try to agree on the exact order two separate events happened in two different cities, using only what their own watches showed.
In plain words: each machine in a distributed system keeps its own local clock, and these clocks can drift out of sync with each other over time, even when a synchronization protocol periodically corrects them, because the correction itself takes time to propagate and the clocks keep drifting in between corrections. This means that comparing timestamps generated on two different machines cannot be fully trusted to reflect the true order in which two events actually happened, since one machine's clock could be running measurably ahead of or behind another's at that moment. Relying on clock-based timestamps alone to determine the order of events across machines is therefore a known source of subtle, hard-to-reproduce bugs in distributed systems.
A worked example. Two nodes each record a timestamp for a write, one second apart according to their own local clocks. If one node's clock is actually running three seconds ahead of the other's true time, the timestamps alone could suggest the wrong write happened second, even though the opposite is true.
The key lines. No new SQL; relying on synchronized clocks versus a logical ordering mechanism, such as the timestamp-ordering concepts from the transactions level, is an internal design decision in a distributed database.
Clock drift
Clock drift is the tendency of independent machine clocks to gradually fall out of sync with each other over time, making cross-machine timestamp comparisons unreliable for determining the true order of events.
Example: several travelers' unsynchronized wristwatches disagreeing about when two events happened mirrors clock drift between machines.
Remember this
A distributed system cannot assume a node that's gone quiet has failed, a message that didn't arrive was lost, or a timestamp from another machine is accurate. Every one of these must be treated as uncertain, not as fact.
Quick notes
- A partial failure affects only some parts of a distributed system while the rest keep running, unlike a single-machine crash.
- An unreliable network can delay, drop, duplicate, or reorder a message, and a sender often can't tell which happened.
- A timeout is an arbitrary cutoff for giving up on a response, never proof that something actually failed.
- Clock drift means timestamps from different machines cannot be fully trusted to reflect the true order of events.
- These three unreliabilities, partial failure, network unreliability, and clock drift, are the foundation for why distributed transactions and consensus, covered next, are hard.