Part 5 of 12 · 2 min read
Rollback: why a tag isn't enough, and what migrations allow
A movable label can point at the wrong thing; a destructive migration can make the old code unable to run at all.
Think of a nametag that anyone can swap out, versus an engraved serial number stamped into the object itself. Say something goes wrong and you need to get back "the one from yesterday." The nametag might already be on a different item. The serial number never lies about which item it is.
In plain words: rolling back by a movable tag can pull whatever the tag currently points to, which may be the bad release itself, while rolling back by an immutable digest always retrieves the exact image that was tested — but even an exact rollback fails if the database has already been changed in a way the old code can't handle.
A worked example. After a bad push, pulling the image by its tag (latest) retrieves the bad image, because the tag now points at it. Pulling by the digest recorded at the time of the last good release retrieves exactly the previous good image, regardless of what the tag currently points to. Separately, a migration that adds a nullable column is safe to roll back behind — the old code simply ignores the new column — but one that drops a column, renames one, or makes one required is not: the old code will crash trying to use a column that no longer exists in the shape it expects.
The key lines.
# from code/mini-deploy/deploy.py
def rollback_safe(migration):
return not (migration.get("drops_column") or migration.get("renames_column") or migration.get("makes_column_required"))
Read it as: a migration is safe to roll back behind only if it purely adds things, and never removes or changes them. That is why production migrations go out in two steps. First comes expand, which adds the new, additive thing. Only later comes contract, which removes the old thing once nothing depends on it anymore.
Container image, digest, tag; expand-contract migration
An image is a packaged filesystem and config; a tag is a movable label (like latest); a digest is an immutable content hash identifying one exact image. Expand-contract migration means making additive schema changes first and removing old structures only once nothing uses them.
Example: the digest-based rollback above, immune to the tag having been re-pointed at the bad release.