Part 7 of 8 · 2 min read
Failures and trade-offs
What breaks, and how to answer.
Failures, trade-offs and what to say
| Failure | Response |
|---|---|
| Load balancer | A secondary takes over, found through heartbeats |
| Block server | Other servers pick up the unfinished jobs |
| Cloud storage | Replicas in other regions serve the data |
| API server | Stateless, so the load balancer redirects |
| Metadata cache | Replicas serve, and a new node replaces it |
| Metadata DB primary | Promote a replica |
| Metadata DB replica | Use another, and add a new one |
| Notification service | Clients reconnect and pull changes |
| Offline backup queue | Replicate it, and let clients re-sync |
| Choice | Gain | Cost |
|---|---|---|
| Blocks of 4 MB | Delta sync, parallel upload | More metadata rows |
| Strong consistency | Devices always agree | Slower writes |
| Relational metadata | ACID transactions | Harder to shard |
| Long polling | Simple, enough for rare events | A held connection per client |
| De-duplication | Saves space | Extra hash lookups |
| Cold storage | Cheap | Slow to read back |
Interview angle
Answer in your head first, then open it.
Why blocks?
Smaller transfers, delta sync, parallel upload, de-duplication.
Why a relational database for metadata?
Strong consistency needs ACID.
How do conflicts work?
First wins, and the second user chooses.
How do you save storage?
De-duplicate blocks, limit versions, use cold storage.
What if cloud storage loses a region?
Cross-region replicas serve the data.
One follow-up to try. "A user uploads a 10 GB file on a poor connection." A good reply: split it into 4 MB blocks on the client, upload several blocks in parallel, record which blocks arrived, and resume from the first missing block. Only after the last block is stored does the file go from "pending" to "uploaded". The upload also survives a restart of the app.
Check yourself: 1. What happens if a block server dies half way through an upload? 2. Why is strong consistency a cost as well as a gain? 3. How does block-level resume help a poor connection?
Words you will meet
| Word | Meaning |
|---|---|
| Namespace | A user's root folder |
| Block | A piece of a file with its own hash |
| Delta sync | Sending only changed blocks |
| Resumable upload | An upload that can continue after a break |
| Object storage | A service that stores files as objects in buckets |
| De-duplication | Storing identical data once |
| Cold storage | Cheap, slow storage for inactive data |
| Sync conflict | Two edits to the same file at once |