Part 7 of 8 · 4 min read
Data outlives code: databases, services and archives
Where encoding choices matter in practice.
Data outlives code: databases, services and archives
Think of a filing cabinet in an office that has had three management teams. The newest manager can read the old files only if they were written clearly, and the old files still sit in the drawers long after the old manager left.
The book explains how this applies to each place where data flows.
Through a database. The writer encodes and the reader decodes. A single process reads its own earlier writes, like a message to its future self, so backward compatibility is needed. With several processes or instances, some run newer code than others, so forward compatibility is needed too. The book also gives a snag. Suppose new code writes a record with a new field. Old code reads it, updates another field and writes it back. The old code may drop the field it does not understand, because decoding into its own model objects and encoding them again loses unknown fields. The formats can preserve unknown fields, but the application must be careful not to lose them.
A key sentence of the book: data outlives code. A server application can be replaced entirely within minutes. A five-year-old record stays in its original encoding unless someone rewrites it. Rewriting a large data set is expensive, so most databases avoid it. Most relational databases can add a column with a null default without rewriting rows; on reading an old row, the database fills in null. The book notes that MySQL often rewrites the whole table anyway. Schema evolution lets a database look as if it used one schema, though the records on disk come from many eras.
A worked example. A table has 2 billion rows of 200 bytes, 400 GB.
- Add a nullable column: only the schema changes, in seconds. Old rows are read with null in the new column.
- Rewrite all rows to add a stored default: read and write 400 GB. At 100 MB a second, 4,000 seconds, a bit over an hour, while the table is busy.
Through a service (REST and RPC). The clients and servers are separate programs deployed at different times. The book assumes servers are updated first and clients later, so requests need backward compatibility and responses need forward compatibility. This matches the compatibility rules of Level 1, module 2: add optional fields, never remove or retype fields, and use a version in the URL for breaking changes. RPC frameworks such as gRPC are built on Protocol Buffers, and take their rules of evolution from there.
Through messages (queues and logs). A sender encodes, a queue holds, a receiver decodes later, possibly after the sender has changed. The same rules apply. Level 6 returns to this.
Archives and warehouses. A backup or a dump for the warehouse can be written in the latest schema, all in one go, even if the source contains a mixture of old shapes. It is written once and never changed, so a format such as Avro object container files fits. The book adds that it is a chance to use a column-oriented format such as Parquet (module 5).
| Place | Typical format | Why |
|---|---|---|
| Public API | JSON | Readable, universal |
| Internal calls | Protocol Buffers (gRPC) | Compact, strict, generated code |
| Event streams, data files | Avro | Schema stored with data, easy evolution |
| Warehouse files | Parquet (columns) | Fast scans, good compression |
| Short-lived cache | Anything quick, with a version prefix | Entries expire |
Interview angle
Answer in your head first, then open it.
How do you change a field in a message that flows between services?
Make the change backward and forward compatible: add optional fields, never reuse tags, and when a break is needed, release readers first, then writers, then remove the old code.
Why not use the language's built-in serialization?
It ties data to one language, can be a security hole, handles versions badly and is bloated.
JSON or Protocol Buffers?
JSON for public and human-facing APIs. Protocol Buffers for internal high-volume calls where size and strictness matter.
What happens during a rolling deploy if the format changed?
Old and new code coexist, so both directions of compatibility are needed. State it in the answer.
Check yourself:
- What does "data outlives code" mean?
- Which format suits a nightly dump for the warehouse, and why?
Words you will meet
| Word | Plain meaning |
|---|---|
| Encoding, decoding | Turning data into bytes, and bytes into data |
| Serialization | Another word for encoding |
| Self-describing | Carries its field names in every record |
| Base64 | A way to write bytes as text, a third larger |
| Schema | The declared structure of the data |
| Field tag | The number that identifies a field in Protocol Buffers and Thrift |
| Writer's schema | The schema used to encode a record |
| Reader's schema | The schema the reading code expects |
| Default value | The value used when a field is missing |
| Backward compatible | New code reads old data |
| Forward compatible | Old code reads new data |
| Rolling upgrade | Updating a few servers at a time |
| Expand and contract | Add support first, switch writers next, remove the old last |
| Parquet | A column-oriented file format for analysis |