Part 4 of 8 · 3 min read
Schema-based binary formats: Protocol Buffers and Thrift
Numbered fields that replace names, and what they allow.
Think of a form with numbered boxes. A clerk writes only "1: Martin, 2: 1337" and the reader, who holds the form, knows that box 1 is the name. The names are never copied onto each record.
Protocol Buffers (made by Google) and Thrift (made by Facebook) work like this. Both need a schema. The book gives the example:
message Person {
required string user_name = 1;
optional int64 favorite_number = 2;
repeated string interests = 3;
}
A tool reads the schema and generates code in a programming language, with classes to build, encode and decode the records.
Field tag
A field tag is the number given to each field in the schema. The encoded record stores the tag and the value, and never the field name.
Example: user_name is field 1. The record stores "1, a string of 6 bytes, Martin".
The sizes for the book's record:
| Format | Bytes |
|---|---|
| JSON text without spaces | 81 |
| MessagePack | 66 |
| Thrift BinaryProtocol | 59 |
| Thrift CompactProtocol | 34 |
| Protocol Buffers | 33 |
| Avro | 32 |
Compact encodings gain size in two ways: they leave out field names, and they pack numbers into as few bytes as needed. The book's example: the number 1337 takes two bytes instead of eight, since the top bit of each byte shows whether more bytes follow. Numbers between −64 and 63 fit into one byte.
A worked example. A service sends 500 million records a day.
- JSON, 81 bytes: 500,000,000 × 81 = 40.5 GB a day.
- Protocol Buffers, 33 bytes: 500,000,000 × 33 = 16.5 GB a day.
- The difference is 24 GB a day, 59 percent less (24 ÷ 40.5).
Schema evolution follows from the tags. The book explains the rules.
- A field's name can change, since the bytes never contain it.
- A field's tag can never change, because old data would then be read as the wrong field.
- A new field gets a new tag number. Old code that meets an unknown tag skips the field. The datatype in the bytes tells it how many bytes to skip. This keeps forward compatibility: old code reads data written by new code.
- Backward compatibility holds as long as every tag number is unique: new code can read old data. But a field added after the first release cannot be marked required. If it were, new code reading old data would fail on the missing field. So later fields are optional or have a default.
- A field can be removed only if it is optional. Its tag number must never be reused, since old data may still contain it.
- Changing a type is risky. Changing a 32-bit integer to 64 bits lets new code read old data, but old code reading a large new value truncates it.
A neat detail of Protocol Buffers: there is no list type, only a repeated marker. So a single optional field can be changed into a repeated one. New code reading old data sees a list of zero or one items.
A worked example of evolution. Version 1 of a schema has fields 1 (name) and 2 (email). Version 2 adds field 3 (phone, optional).
- Old code reads a version-2 record: it sees tags 1, 2 and 3. Tag 3 is unknown, so it skips it. Name and email are read as normal. Forward compatibility holds.
- New code reads a version-1 record: tags 1 and 2 are present, tag 3 is missing. The phone is treated as unset. Backward compatibility holds.
- Mistake: version 2 reuses tag 2 for a different field, say
age. Now an old record with email in tag 2 is read as an age: wrong data, with no error.
Common mistake
Reusing or renumbering a tag to "tidy up" the schema. Records already stored will be read as the wrong field. Keep retired numbers reserved forever.
Check yourself:
- Why can a field name change but a field tag not?
- Why must a field added later be optional?