Model changes and metadata

When the embedding model changes

A vector from one model cannot be compared with a vector from another, even if both have the same number of dimensions. Their spaces are unrelated. If you switch the query side to a new model while the index still holds old vectors, the old chunks score essentially at random.

The example re-embeds a growing fraction of a 48-chunk index with a new model while queries already use it:

Accuracy tracks the fraction almost one-for-one: a half-migrated index is a half-broken index. (The 98% at the end, against 100% for the old model on its own, is a reminder that a "better" model is not better until you have measured it on your data; module 7.)

Safe pattern, often called blue-green:

1. Build a second index with the new model next to the old one. Keep serving from the old one.
2. When it is complete, run your golden questions (module 7) on both. Switch only if the new one is at least as good.
3. Switch the query side and the index together (one configuration change), keep the old index for a while, then delete it.
4. Stamp every index with a configuration id (model, chunker version, text normaliser) and refuse to search when the query side's id differs. In the example, a "strict" search raises an error on mixed vectors, which is better than quietly returning nonsense.

Anything that changes vectors counts: the model, its version, the tokenizer, how you clean text before embedding, the chunk boundaries, and the prefix some models expect on queries.

Versions, expiry and metadata

Three tools beyond "keep it identical to the source":

• Version and date metadata. Store updated_at, version and source_url on every chunk. The answer step can show "as of 3 March" (module 5) and the ranker can prefer newer chunks when two disagree (module 8's "newest wins").
• Expiry (time to live). Some facts should not outlive a deadline whether or not anyone edited them: a promotion, a cached web page, a temporary outage notice. Give them an expires_at and filter or purge on it.
• Supersession. Instead of deleting an old version, mark it superseded_by the new one. You keep history for audits and "what did it say last quarter?" questions, and search ignores superseded chunks by default.

And measure freshness like any other quality (module 7):

1. Lag: time between a source change and the index showing it (a canary document edited on a schedule, then queried until it appears).
2. Ghost and stale rates on a small set of questions whose true answers you track.
3. Sync health: run time, documents read, chunks embedded, deletions blocked. A sudden zero or a sudden spike is the early warning.

Practical checklist

1. Give chunks content-addressed ids and store a document hash.
2. Sync by set difference: embed new ids, remove old ids.
3. Use tombstones for deleted documents, a grace period, and a mass-delete guard.
4. Treat modified time as a hint, hashes as truth, and run a periodic full verification.
5. Choose the window from the cost of a wrong answer, not from habit.
6. Stamp indexes with a configuration id; upgrade models with blue-green and a golden set.
7. Keep dates and versions in metadata and show them in answers.
8. Monitor lag, ghost and stale rates, and sync health.

For the curious What the example does not cover

The sync functions assume that every listed document can be read. If a document disappears between the listing and the read, the example raises an error halfway and leaves the index partly updated. A real sync should read inside a try block, skip and report that document, and only count the sync as done when it finishes. Identical paragraphs inside one document also share an id, so the second copy is not stored separately.