Part 3 of 8 · 3 min read
Text formats: JSON, XML and CSV
Readable and universal, with ambiguities and extra size.
Think of a postcard written in plain language. Anyone can read it without special equipment. But it is longer than a coded telegram, and a number written as "1,337" could be read as one thousand three hundred thirty-seven or as a list of two numbers.
JSON, XML and CSV are text formats that many languages understand. The book says JSON is popular because browsers support it and because it is simple compared with XML. XML is often called verbose and needlessly complex. CSV is less powerful. They are good enough for many uses, especially for data exchange between organisations, where agreeing on anything at all outweighs concerns about elegance or efficiency.
They have subtle problems, as the book notes.
- Numbers are ambiguous. In XML and CSV, a number cannot be told from a string of digits without an outside schema. JSON separates strings and numbers but does not separate integers from floating-point numbers and does not state a precision. Large integers, those above 2 to the power 53, cannot be held exactly in the floating-point numbers used by some languages, such as JavaScript. The book mentions that Twitter had to send identifiers twice, as a number and as a decimal string, for this reason.
- No binary data. JSON and XML support Unicode text but not raw bytes. Programs get around this by encoding the bytes as text, using Base64, which makes the data about one third larger.
- Schema support is optional and clumsy. XML and JSON have schema languages, but they are complicated and many users skip them. The reading code then hard-codes the assumptions.
- CSV has no schema. Each row and column is interpreted by the application. A new column must be handled by hand. A value that contains a comma or a new line is a trap, since not all parsers follow the escaping rules the same way.
A worked example of the Base64 cost. A 3 MB picture is sent inside JSON.
- Base64 turns every 3 bytes into 4 characters: 3,000,000 × 4 ÷ 3 = 4,000,000 bytes.
- The picture grows by a third, 1 MB, before any other overhead.
- For 100 pictures a second, the extra is 100 MB a second.
The book also covers binary versions of JSON, such as MessagePack. They add types, such as separate integers and floats, and pack lengths and types into compact bytes. But since there is no schema, they still must include every field name in every record. In the book's example record
{ "userName": "Martin", "favoriteNumber": 1337, "interests": ["daydreaming", "hacking"] }
the JSON text without spaces takes 81 bytes, and MessagePack takes 66. The book's comment is that this small saving may not be worth losing the human-readable text.
Self-describing format
A self-describing format carries the names of its fields inside every record, so a reader needs no outside description.
Example: JSON repeats the text "userName" in every record.
The cost of being self-describing is size. A million records of the example, with field names taking about 40 of their 81 bytes, spend 40 MB on names alone.
Check yourself:
- Why is a large identifier risky in JSON?
- How much bigger does Base64 make binary data?