Part 4 of 8 · 3 min read
Data store
Routing, placement and nodes.
The data store has three parts.
- Data routing service. Stateless. It receives requests from the API service, asks the placement service where to write, and talks to the data nodes.
- Placement service. It knows which data nodes hold which objects. It keeps a virtual cluster map: the real layout of nodes in racks and data centres, so it can place replicas in different failure domains. It watches nodes through heartbeats (Level 8), and marks a node down after a grace period of, say, 15 seconds. This is a critical service, so it runs as a cluster of 5 or 7 nodes using a consensus protocol (Level 5). With 7 nodes, 3 may fail and it still works.
- Data nodes. Servers with disks that store the objects. Each runs a daemon that sends heartbeats with its disk count and load to the placement service.
Write path.
- The API service sends the bytes to the routing service.
- The routing service creates a UUID and asks placement for the node. Placement uses consistent hashing (Level 4) on the UUID to pick a replication group: one primary and two secondary nodes.
- The router sends the data to the primary.
- The primary saves it and copies it to both secondaries.
- The UUID goes back to the API service.
When to confirm. The primary can answer after all three have stored the data (strongest, slowest), after two have stored it, or after only the primary has it (fastest, riskiest). This is the same trade-off as acknowledgments on page 1 of this level.
Packing small objects. Storing each object as its own file fails for small objects. A file system wastes most of a 4 KB block on a tiny file, and has a fixed number of index entries (inodes), which millions of small files can exhaust. The fix is to append many objects into one large file, like a log (page 1). When the file reaches a few GB it becomes read-only, and a new one starts. Writes to the active file must go one at a time, so large servers keep one active file per processor core.
A lookup table in each node maps an object id to its file, start position and size. The data is written once and read often, so a relational database such as SQLite on each node suits better than a write-optimised store, and nodes need not share the table.
A worked example (made up). A node's table says object 3f9a lives in file /data/c, starts at byte 8,000,000 and is 40,000 bytes long. A read opens the file, jumps to that byte and reads 40,000 bytes.
Replication group
A replication group is the set of data nodes that each hold a copy of the same objects.
Example: a primary node and two secondary nodes, placed in three different racks.
Check yourself: 1. What does the placement service know that the router does not? 2. Why are small objects packed into large files? 3. How is a node found from an object id?