CSE Roadmap

Specialist tier · about 2 hours · 8 parts

Part 6 of 8 · 2 min read

Retention and deletion

Decide how long to keep it, and prove it can be removed.

A retention rule says how long each kind of data is kept and what happens after. Common practice is a short table: for example, request logs 30 days, billing records 7 years (a legal need, made up here), analytics events 13 months. After the period, data is deleted or made anonymous. Retention rules should be set per data type, applied by automatic jobs and checked.

Deleting is harder than it sounds. DDIA gives reasons: - Privacy rules may require deleting a user's personal information when they close an account, and removing wrong information, or containing an accidental leak. - In an append-only system (such as an event log), adding an event that says "deleted" is not enough. You may need to actually rewrite history. DDIA names Datomic's feature excision and Fossil's shunning. - Truly deleting data is surprisingly hard because copies live in many places. Storage engines, file systems and SSDs often write a new copy rather than overwrite, and backups are often kept immutable on purpose. DDIA's view: deletion is more a matter of making data harder to retrieve than of guaranteeing it is gone.

UDS gives a legal angle. In Europe, purging the data of one user across the whole application stack is required by the GDPR, and non-compliance can bring fines of up to 20 million euros or 4 percent of annual turnover, whichever is greater. Because the risk of this feature silently breaking is high, UDS says to test it with a check that runs in production against live services, not stand-ins.

A purge flow (common practice built on these points): 1. A verified request names a user id. 2. A coordinator sends the id to every system that holds personal data: databases, caches, search indexes, analytics, logs, backups. 3. Each system deletes or anonymises and confirms. 4. The coordinator records proof and retries any that fail. 5. Backups expire on their schedule, and a restore must re-apply recent deletions (common practice).

Common mistake

Deleting the user row in the main database and forgetting the copies: the search index, the analytics store, the cache, old logs and backups.

Check yourself: 1. Why is adding a "deleted" event not enough in an append-only log? 2. What did UDS say about testing the purge feature? 3. Name three places a user's data may still live after the main row is deleted.