Part 5 of 8 · 4 min read
Avro: a writer's schema and a reader's schema
No tags at all, and a library that matches two schemas by field name.
Think of two people with two copies of a form, one older. The sender fills in the old form. The receiver, holding the new form, lays the two side by side and matches boxes by their labels. Boxes in the new form that the old one lacks are filled with a standard answer.
Apache Avro, born in 2009 in the Hadoop project, takes a different route. Its schema has no tag numbers:
{
"type": "record",
"name": "Person",
"fields": [
{ "name": "userName", "type": "string" },
{ "name": "favoriteNumber", "type": ["null", "long"], "default": null },
{ "name": "interests", "type": { "type": "array", "items": "string" } }
]
}
The encoded record, the book shows, is just the values one after another, with nothing to say which field or type is which. It is the shortest of all the formats in the table: 32 bytes. To decode, the reader goes through the fields in the order of the schema and reads each by its declared type.
This means data can be decoded only with the exact schema that wrote it. That raises the question of how evolution works. The answer is two schemas.
Writer's schema and reader's schema
The writer's schema is the version the writer used to encode the data. The reader's schema is the version the reading code expects. They may differ, but must be compatible. The Avro library compares them side by side and translates.
Example: a file written with version 1 is read by code that expects version 3. The library fills in new fields with defaults and ignores removed ones.
The matching rules, from the book:
- Fields are matched by name, so order does not matter.
- A field in the writer's schema but not in the reader's is ignored.
- A field in the reader's schema but not in the writer's is filled with the reader's default value.
The rules of change:
- You may only add or remove a field that has a default. Adding one without a default would break backward compatibility, since new readers could not read old data. Removing one without a default would break forward compatibility.
- To allow null, a field must say so through a union such as
["null", "long"], and null can be the default only if it is a branch of the union. This is more wordy than "everything can be null", but it prevents bugs. - A name can change if the reader's schema keeps the old name as an alias. This is backward compatible but not forward compatible.
- Adding a branch to a union is backward compatible but not forward compatible.
A worked example. Writer schema v1: name, email. Reader schema v2: name, email, country (default "IN").
- The reader decodes a v1 record: name and email come from the data, and
countryis "IN" from the default.
Now the reverse: writer v2, reader v1.
- The reader decodes a v2 record:
countryexists in the writer's schema but not in the reader's, so it is ignored.
How does the reader learn the writer's schema? The book gives three answers, depending on the setting.
- A large file with many records. The writer puts the schema once at the start of the file. Avro's object container file does this, and a reader opens the file and gets everything it needs.
- A database with records written at different times. Keep a list of schema versions. Each record starts with a version number, and the reader looks the schema up.
- Two processes talking over a network. They agree on the schema version when the connection starts and use it for the life of the connection.
A worked example of the saving in case 1. A file holds 10 million records. The schema is 300 bytes.
- Schema once per file: 300 bytes.
- Schema in every record would add 10,000,000 × 300 = 3 GB. That is why the schema is kept once.
An advantage of Avro, noted by the book: because there are no tags, a schema can be generated automatically from a database table, with one field per column, and regenerated when the table changes. Fields are identified by name, so the new export is still readable by code written for the old schema. With Protocol Buffers or Thrift, someone would have to assign and track the tags by hand.
| Protocol Buffers / Thrift | Avro | |
|---|---|---|
| Fields identified by | Tag number | Position in the schema, matched by name between versions |
| Schema needed to decode | Yes, and tags can differ across versions | The writer's schema, plus the reader's |
| Add a field | New tag, optional or default | Add with a default |
| Generated schema from a table | Awkward | Natural |
| Common use | Service calls (gRPC), stored messages | Data files, event streams, warehouse loads |
Check yourself:
- What does an Avro reader do with a field it expects but the data lacks?
- Why must an Avro record be decoded with a known writer's schema?