How an index goes wrong: Keeping an index fresh

How an index goes wrong

Freshness

How closely an index matches its source now. A freshness problem is any answer that would be different if the index were rebuilt this minute.

Four things to look for, each with a different cause:

Symptom	What the user sees	Typical cause
Stale	the old value of something that changed	edits never re-indexed
Ghost	an answer from a document that no longer exists	deletions never propagated
Missing	"I don't know" for something that exists	new documents never indexed
Conflict	two versions of the same fact among the top results	new version added, old one not removed

Over 30 days, an index built on day 0 and never touched (the "frozen" row below) decays from 96% right to 20%. More interesting are the partial fixes. "Append only" (add what is new, remove nothing) is a common bug: it fixes missing documents but turns stale facts into conflicts: 35% of questions had both versions in the top three, and because identical scores go to the older chunk, the old value usually won.

Detecting what changed

One incremental sync: list, split, hash, compare, then embed only what is new

The core idea is content-addressed chunks: a chunk's id is its document id plus a hash of its text. Edit one paragraph and that chunk gets a new id; the old id is no longer in the source. Then a sync is a set difference:

new = {chunk_id(doc, c): c for c in split(source.read(doc))}
old = index.chunks_of(doc)
for cid in old - set(new): index.remove(cid)          # vanished or edited
for cid in set(new) - old: index.upsert(cid, doc, new[cid])   # new or edited: embed

Three ways to decide "does this document need work?":

Method	How	Cost per run	Weakness
Modified time	compare the file's timestamp to the one stored	reads only changed documents	trusts a clock; misses edits that keep the old time (restored backups, copied files, some syncing tools); re-embeds on touches
Document hash	read everything, hash it, compare	reads every document	re-embeds the whole document after a one-word edit
Chunk hash	read everything, compare chunk ids	reads every document	needs a stable chunker; if chunking changes, every id changes

What each does in the example, for one change and one sync (reads, chunks embedded, index correct afterwards):

Change	mtime, whole doc	doc hash	chunk hash
edit one paragraph	1 read, 4 embeds, right	12 reads, 4 embeds, right	12 reads, 1 embed, right
touch (same text)	1 read, 4 embeds wasted	12 reads, 0 embeds	12 reads, 0 embeds
restore (text changed, time did not)	0 reads, 0 embeds, wrong	12 reads, 4 embeds, right	12 reads, 1 embed, right
add a document	right	right	right
delete a document	right	right	right

Two rules follow. Use hashes, not clocks, as the source of truth for "did the content change"; use the clock only as a cheap filter if reading everything is too expensive, and run a full hash pass now and then to catch what the clock missed. And keep chunk ids stable: if you change the chunker (module 1), expect to re-embed everything once.