Part 3 of 8 · 2 min read
High-level design
Two stores and a front door.
Object storage has a few special properties. Objects are immutable: they can be replaced or deleted, but not edited in part. The pattern is write once, read many, and a study at LinkedIn found about 95 per cent of requests were reads. Small and large objects both occur.
The design copies the idea of a UNIX file system. There, a file's name and details live in an inode (an index entry), and the data sits elsewhere on the disk, pointed to by the inode. Object storage does the same across a network.
- Metadata store: holds the bucket and object records: name, id, size, and which bucket. It is changeable data.
- Data store: holds the object bytes, found by a unique object id (a UUID). It is immutable data.
Splitting them lets each be built and tuned on its own.
The parts. A load balancer. A stateless API service that orchestrates calls. An identity and access management (IAM) service, which checks who the user is and what they may do. The metadata store and the data store.
Upload flow.
- The client sends a request to create a bucket. The API service asks IAM whether this is allowed, and the metadata store records the bucket.
- The client sends the object, say
notes.txt, with a PUT request. - The API service checks permission again.
- It sends the bytes to the data store, which saves them and returns a new object id.
- The API service writes a metadata row with the object id, bucket id and object name.
Download flow.
- The client sends a GET for
/team-files/notes.txt. - The API service checks read permission with IAM.
- It asks the metadata store for the object id that belongs to that name.
- It fetches the bytes from the data store by id, and returns them.
A worked example (made up). The name photos/2024/cat.jpg is just a long string, not folders. The metadata store maps it to id 3f9a..., and the data store maps that id to bytes on a disk.
Metadata
Metadata is data about data. For an object it holds the name, size, owner and the id that leads to the bytes.
Example: name cat.jpg, size 2 MB, bucket 123, object id 3f9a....
Check yourself: 1. Why separate the metadata store from the data store? 2. What does IAM do? 3. Why does download first ask the metadata store?