Part 6 of 7 · 5 min read
Search and cache updates
Two common derived systems, and how to keep each correct under repeats and lag.
Search and cache updates
Think of two assistants who follow the same ledger. The first re-files documents in a big index. The second keeps a small pad of the most requested figures on his desk and crosses out any figure that changes. Both read the same ledger, and each uses it differently.
Search index. The UDS book gives the pattern. The catalog service appends a message to a log for each change. The relational database and the search index are asynchronous consumers that read in the same order and update at their own pace. Each consumer saves a checkpoint of the last message it handled, and after a crash it resumes from there. The book notes it may re-read a message, so messages must be idempotent (page 2), for instance by tagging each with a unique id and ignoring repeats. The book also points out this is state machine replication again (Level 5): the state is all the products, replicated across the database and the index.
For a CDC-fed index, use the record's key (the product id) as the index document id, and write the whole new version. Setting a document to a value is idempotent. Repeating the event is harmless. Deleting is the same: a delete event removes the document, and repeating it does nothing.
Cache. A cache holds copies for fast reading (Level 3). When the source changes, the cached copy must not stay stale. There are two usual choices:
| Choice | How | Risk |
|---|---|---|
| Invalidate | The consumer deletes the cached entry on a change event. The next read refills it from the database | A race: a reader may refill from an old value just before the change commits and write it back (general knowledge, a standard cache race) |
| Update | The consumer writes the new value into the cache from the event | If events are repeated or reordered, an older value may overwrite a newer one |
The safer rule is to carry a version (such as the log offset or a row version) with each value and let the cache accept a write only if its version is newer than the stored one. This is the idea from page 2 of storing the offset with the write, applied to a cache. The comparison of invalidate and update is general industry practice, and it is not given in the books used here.
What the user sees. The DDIA book names the downside of derived views fed asynchronously: a user may write and then read from a derived view and not see the write. This is the "read your own writes" problem from Level 4. Fixes: update the view in the same transaction as the write (needs both in one store or a distributed transaction), read from the primary store after writing, or show the user's change locally.
Outbox. If the application's own database is the source of truth but there is no CDC tool, the outbox pattern of Level 5 module 9 gives the same effect. The service writes the business row and an event row in one local transaction, and a relay publishes new rows to the log. This is the same idea, with the application writing the change record itself. The outbox pattern is general industry knowledge and not from the books used here.
A worked example. Product 41's price changes 5 → 7 → 6 in quick succession.
| Event (offset) | Search index action | Cache action with versions |
|---|---|---|
| 100: price 7 | Set document 41 to price 7 | Version 100 is newer than the stored 90: store 7 |
| 101: price 6 | Set document 41 to price 6 | Version 101 is newer: store 6 |
| 100 delivered again | Set document 41 to price 7 | Version 100 is older than 101: ignore |
Without the version check, the repeat of event 100 would put 7 back in the cache. With a version, the final value stays 6.
Read it as
One writer to the source, one ordered change stream, many idempotent consumers. Each derived system can be rebuilt from the stream.
Rebuilding. Because derived data can be recomputed, a bug in an indexer is fixed by starting a new consumer at the beginning of the (compacted) log, writing a new index, and switching to it. This also lets a team add a new kind of derived view without touching the source system, as in CQRS (page 4).
Interview angle
Answer in your head first, then open it.
How do you keep a search index in sync with a database?
Use CDC or an outbox to publish changes to a log, and an idempotent consumer that updates the index. Avoid dual writes.
Why not write to both from the application?
A race can leave them different for good, and a partial failure leaves them different until fixed.
What is CDC?
Reading the database's change log and publishing changes as ordered events.
How do you bootstrap a new consumer?
Snapshot at a known log position then apply changes, or read a compacted log from the start.
What does the user see?
A short lag. For read-your-writes, read from the source or show the change locally.
Update or invalidate the cache?
Invalidate is simpler. Update with versions avoids stale overwrites.
Check yourself:
- Why does carrying a version make a cache update safe against repeats?
- Why should the CDC stream be the only writer of a search index?
Words you will meet
| Word | Plain meaning |
|---|---|
| System of record | The authoritative copy of the data |
| Derived data | Data computed from other data, which can be rebuilt |
| ETL | Extract, transform, load: copying data in bulk to a warehouse |
| Dual write | Writing the same change separately to two systems |
| Change data capture (CDC) | Publishing every database change as an ordered stream |
| Change stream, changelog | The ordered list of changes |
| Trigger | Code a database runs automatically when a row changes |
| Binlog, write-ahead log | A database's own record of changes |
| Snapshot | A consistent copy of the data at one moment |
| Log compaction | Keeping only the latest record per key |
| Tombstone | A marker that a key was deleted |
| Initial load | Filling a new derived system before applying new changes |
| Outbox pattern | Saving an event with the data in one transaction, publishing it later |
| Replication lag | How far a copy is behind its source |
| Version | A number used to tell newer data from older |