Deletions, cost and freshness

Deletions

Removing a document from the source must remove it from the index; otherwise it becomes a ghost. In the example, a sync that handles edits and additions but never checks for vanished documents ("no delete handling") left ghosts on 11% of questions: every deleted document stayed answerable forever.

Why not just delete? Because deletion by absence is dangerous: the sync decides "gone" when a document is missing from the listing, and a listing can be empty for the wrong reason (a timeout, a changed permission, a crawl that stopped early). Three defences, shown with a listing call that fails and returns nothing (12 documents indexed):

Use all three ideas together: tombstone first, purge after a grace period (a few syncs), and refuse to run when the number of deletions is implausible. A person looking at the alert is cheaper than an empty index. A document that reappears in the listing should simply be restored.

Deletion also has non-technical stakes: if a document is deleted for legal, privacy or safety reasons, "hidden" is not enough. Know where else copies live (vector store, caches, logs, answers already shown) and make the purge real.

How often, and how fresh

Reading the chart:

• Frozen decays steadily. Append only decays more slowly and settles near 35% because stale chunks keep winning.
• No delete handling stops decaying but sits at about 78%: the ghosts are permanent.
• Rebuild weekly is a sawtooth: perfect right after the rebuild, then 4% worse each day, bottoming near 65%. Its mean is 81%.
• Chunk hash, nightly stays near 97%. It never reaches 100% because a change made during the day is invisible until that night's run (the dips on days with edits, adds and deletes).

That residual window is a design choice. If a wrong answer is cheap, nightly is fine. If it is not (prices, access rules, incident status), shorten the window: run the sync every few minutes, trigger it by change events from the source (webhooks, a database change feed), or check a document's version at answer time and refetch if it moved. More frequent syncs cost reads, not embeds, because unchanged text is not re-embedded.

What it costs

Same freshness as a nightly full rebuild, at 5% of the embedding calls. Note that the weekly rebuild looks cheap and is not good, and that the mtime method re-embeds the most among the incremental ones, because touches cost four embeds each and nothing reduces an edit below a whole document. The ratio depends on how much of your corpus changes per day; a corpus where 40% changes daily gains little from incremental sync.