CSE Roadmap

Common tier · about 5 hours · 8 parts

Part 7 of 8 · 2 min read

Trade-offs

Multipart upload and follow-ups.

Large uploads, clean-up and what to say

Multipart upload. A big object takes long to send, and a broken connection means starting again. Instead, the client asks to start an upload and gets an upload id. It cuts the file into parts and sends each one separately. Each part is confirmed with an ETag, a checksum of the part. After all parts arrive, the client sends a complete request listing the part numbers and ETags, and the store joins them. Part uploads can run in parallel, and a failed part is simply sent again. A worked example (made up). A 3.2 GB file in 200 MB parts is 3,200 / 200 = 16 parts. If part 7 fails, only that 200 MB is resent.

Garbage collection. Deleted objects, old versions, orphaned parts of unfinished uploads and damaged data leave unused space inside the big packed files. A background process marks space free and then compacts: it copies the live objects from several files into a new file, updates the lookup tables, and deletes the old files. Doing it lazily in the background keeps the write path fast.

Choice Gain Cost
Metadata and data apart Each scales on its own Two lookups per read
Small objects packed Few files, less waste A lookup table and compaction
Three copies Simple, fast reads 200 per cent extra space
Erasure coding Half the extra space, higher durability Slow rebuilds, complexity
Shard by name hash Even, fast lookups Listing is hard
Multipart upload Resume, parallel More API calls

Interview angle

Answer in your head first, then open it.

Why separate metadata from data?

Mutable and immutable data scale and tune differently.

How do you reach six nines?

Several copies, or erasure coding, in different failure domains.

What are the costs of erasure coding?

Slower reads and a more complex design.

How do you list a sharded bucket?

A denormalised listing table sharded by bucket.

How does a 5 GB upload survive a drop?

Multipart upload with ETags.

One follow-up to try. "Add cross-region replication." A good reply: copy each new object to a second region, in the background, using a queue of change events from the metadata store (page 1). The second region keeps its own metadata and data. Copies arrive a little late, so reads there are eventually consistent. It protects against losing a whole region, and costs transfer fees and storage.

Check yourself: 1. What is an ETag used for? 2. Why is garbage collection done in the background? 3. What does cross-region replication protect against?

Words you will meet

Word Meaning
Bucket A named container for objects
Object Data plus metadata, kept by id
Block storage Raw disk blocks given to a server
Immutable Cannot be changed in place
Replication group The nodes that hold copies of an object
Erasure coding Rebuildable data using parity chunks
Failure domain A part that fails together, such as a rack
Multipart upload Sending a big object in independent parts