Part 2 of 8 · 2 min read
Scope and numbers
What object storage is for.
Think of three kinds of storerooms. A block store is a set of empty shelves, and the user decides how to use each one. A file store is a filing cabinet with folders. An object store is a huge warehouse where each parcel has a label, and you ask for a parcel by its label. It is not fast to walk around, but it is cheap, vast and very safe.
- Block storage gives raw disk blocks to a server. Databases and virtual machines use it. It is fast and flexible.
- File storage sits on top of blocks and offers folders and files, shared over a network. It is the everyday choice for teams.
- Object storage keeps data as flat objects with no folders, reached through a web API. It trades speed for very high durability, vast scale and low cost, and suits backups, archives, images and video.
Terms.
- Bucket: a container for objects, with a globally unique name.
- Object: the data (any bytes) plus metadata, which is a set of name and value pairs.
- Versioning: keeping several versions of an object, so a deletion or overwrite can be undone.
- URI: the unique address of a bucket or object.
- SLA: a promise to the customer. A typical infrequent-access class promises durability of eleven nines across several zones.
Scope (from the book): create a bucket, upload and download an object, list objects in a bucket, support versioning. Handle both huge objects (a few GB or more) and very many small ones. Store 100 petabytes (PB) of data. Aim for six nines of durability and four nines of availability. Keep storage cost low.
Numbers (from the book). Assume the objects follow a mix: 20 per cent small (under 1 MB, take 0.5 MB), 60 per cent medium (1 to 64 MB, take 32 MB), 20 per cent large (over 64 MB, take 200 MB). The average size is 0.2 × 0.5 + 0.6 × 32 + 0.2 × 200 = 0.1 + 19.2 + 40 = 59.3 MB. With 100 PB, which is 10^11 MB, and disks 40 per cent full, the count is 10^11 × 0.4 / 59.3 = about 0.68 billion objects. At 1 KB of metadata each, the metadata is about 0.68 TB.
The key point: the bottlenecks are disk capacity and the number of disk operations per second (IOPS), not CPU.
Durability
Durability is the chance that stored data is still there and intact after a given time.
Example: a durability of 99.9999 per cent a year means that one object in a million may be lost in a year.
Check yourself: 1. Which kind of storage suits a database, and which a photo archive? 2. Why does object storage give up speed? 3. What is the average object size in the example?