Part 5 of 8 · 2 min read
Metadata and flows
What is stored and who talks to whom.
Metadata tables (simplified, from the book).
- User: name, email, photo.
- Device: device info and a push id for notifications. A user can have many devices.
- Namespace: the root folder of a user.
- File: the latest state of each file.
- File version: the history. Rows here are read-only, so the revision history stays trustworthy.
- Block: each block of a file. Any version of a file can be rebuilt by joining its blocks in the right order.
Upload flow. Client 1 sends two requests in parallel.
Add metadata: 1. Client 1 sends the file's metadata. 2. The metadata DB stores it with status "pending". 3. The notification service is told a file is being added. 4. It tells the other clients, such as client 2.
Upload content: 1. Client 1 sends the file to the block servers. 2. They split, compress and encrypt, then upload blocks to cloud storage. 3. When done, cloud storage calls back the API servers. 4. The status in the metadata DB changes to "uploaded". 5. The notification service is told, and it informs client 2 that the file is complete.
Download flow. A client learns of a change in one of two ways. If it is online, the notification service tells it. If it is offline, the change waits in a queue, and the client pulls it when it returns. Then the client asks the API servers for the new metadata, and asks the block servers for the new blocks. It rebuilds the file from the blocks.
A worked example (made up). Client 2 is told that file X changed. It fetches metadata saying X has blocks b1 to b256, of which b7 and b8 are new. It already holds the others, so it downloads two blocks, 8 MB, and rebuilds X.
Common mistake
Marking a file as available as soon as metadata is saved. The blocks may not be in storage yet, and another device would try to download missing data. Keep the status "pending" until the upload is complete.
Check yourself: 1. Why are file version rows read-only? 2. What does status "pending" protect? 3. How does an offline client catch up?