CSE Roadmap

Common tier · about 4 hours · 7 parts

Part 6 of 7 · 3 min read

Costs and limits

Immutable history is powerful, but it has real costs.

Costs and limits

Think of a diary that can never be torn. It is wonderful for honesty and terrible if a page contains a secret you must remove by law.

DDIA lists the limits. How much immutable history can be kept depends on churn. A system that mostly adds data is easy. A system with many updates and deletes on a small dataset builds a huge history, and compaction and garbage collection become critical.

Deleting data. Regulations may require deleting a person's data after they close an account, or removing information that was wrongly stored. Appending a "deleted" event is not enough, because the data is still in the log. The book says you may need to rewrite history as if the data had never been written. Datomic calls this excision, and the Fossil version control system calls it shunning. Real deletion is hard because copies live in many places: storage engines, file systems, SSDs and backups. Deletion becomes a matter of making data harder to retrieve rather than impossible to retrieve.

A common industry technique for this is to keep personal data outside the events and refer to it by id, so that deleting the record deletes the person's data while the events remain. This is general knowledge and is not from the books used here.

Understanding the flow. The Building Microservices book warns that event-driven designs add complexity. It tells readers to be cautious about adopting them too eagerly, to have good monitoring and to use correlation IDs (page 1) so that requests can be traced across process boundaries.

Ordering and duplicates. Event consumers face the delivery issues of page 2. They must handle repeats. They may also see events in different orders unless they come from one partition (page 3).

Eventual consistency. Read models lag. Pages that need the freshest data must read from the write side.

A decision table:

Question Lean towards
Need a full audit trail or the ability to replay history? Event sourcing
Many different read shapes from the same data? CQRS with several read models
A simple app with one database and few readers? Plain CRUD, with no events
Strict, immediate consistency between write and read? Keep one model, or read from the write side
Strict data deletion rules? Be careful: keep personal data outside the log

Read it as

Event sourcing and CQRS are powerful but add moving parts. Many systems need only a part: events to tell other services what happened, with an ordinary database as the source of truth.

Interview angle

Answer in your head first, then open it.

What is event sourcing?

Store immutable events as the source of truth and derive current state from them. Snapshots speed up reads.

What is CQRS?

Separate write and read models. Read models are built from events and shaped for each query.

Choreography or orchestration?

Choreography for loose coupling, orchestration where one owner needs control and clear error handling. Choreography needs monitoring.

What are the downsides?

Stale read models, harder debugging, deleting data and operational complexity.

How do you show a user their own write immediately?

Read from the write side or show it locally while the view catches up.

Check yourself:

  • Why is "just add a deleted event" not enough for a privacy deletion request?
  • Give one case where plain CRUD is the better choice.

Words you will meet

Word Plain meaning
Command A request that can be refused
Event A fact that something happened
Event-based collaboration Services react to each other's events instead of being told what to do
Orchestration A central service drives the steps
Choreography Services react to events with no central driver
Broker topology Event flow spread through a broker with no mediator
Mediator topology A central mediator controls the event workflow
Event sourcing Events are the truth, state is derived
Snapshot A saved copy of derived state to avoid replaying everything
Ledger An append-only list of transactions
CQRS Separate models for writing and reading
Read model A view shaped for certain queries
Denormalised Data copied into the shape needed to read it quickly
Excision Truly removing data from an immutable history
Correlation ID A tag that follows one request across services