CSE Roadmap

Common tier · about 3 hours · 8 parts

Part 2 of 8 · 2 min read

Data in memory and data in bytes

Why a program must translate its data to send or store it, and why language-built-in formats are a trap.

Think of a cook's workbench with ingredients laid out in a particular arrangement for fast work. To send a meal to someone else, the cook packs the ingredients into a box with a recipe. The receiver unpacks and sets up a new workbench. The packing and unpacking are not the cooking.

The book "Designing Data-Intensive Applications" says a program keeps data in two forms. In memory, data lives in objects, lists, hash tables and trees, arranged for the processor to use fast, with pointers between them. To write data to a file or send it over a network, the program must turn it into a self-contained sequence of bytes. A pointer means nothing to another process.

Encoding and decoding

Encoding turns data in memory into a sequence of bytes. Decoding turns the bytes back into data in memory. Other names are serialization and deserialization, and marshalling and unmarshalling.

Example: a program encodes the record {"name": "Asha", "age": 31} into 24 bytes of text to send it, and the receiver decodes it.

Level 1 used the word serialization for the same idea (page 2). The book avoids it because the word also has a different meaning in transactions (Level 5).

Most languages ship with a built-in way to encode objects: Java's built-in serialization, Ruby's Marshal, Python's pickle. They are convenient. The book lists four deep problems.

  1. Tied to one language. Data stored this way is hard to read from another language, which commits the system to one language for a very long time.
  2. Security risk. To rebuild the objects, the decoder must be able to create any class. An attacker who can feed bytes to the decoder may run harmful code.
  3. Versioning is an afterthought. These libraries neglect forward and backward compatibility.
  4. Slow and bloated. The book calls Java's built-in serialization notorious for poor performance and bulky output.

The advice: use the built-in format only for very short-lived purposes.

Common mistake

Storing pickled or built-in serialized objects in a database or a cache that lives for months. After the next release changes the class, the old entries may fail to load, and a mistake in the stored bytes can be a security hole.

Read it as

Whatever leaves the process as bytes becomes a public contract. Choose its format as carefully as an API.

Check yourself:

  • Why can a pointer not be sent over a network?
  • Name two problems with built-in language serialization.