Part 9 of 12 · 2 min read
Mistakes that turn a release into an incident
Common mistakes, and what the example does not do.
Mistakes that turn a release into an incident
- Secrets in the image, repo, build args or logs.
latesttags and rebuilding the "same" version.- Running as root, no healthcheck.
- Prompt and model changes shipped outside the pipeline.
- An eval gate that flakes (blocks good changes) or never blocks (underpowered).
- Destructive migrations bundled with the code change.
- No rollback rehearsal; the first rollback happens during an incident.
- Deploying while runs are in flight with no draining or journal compatibility.
- One environment's credentials reused in another.
What the example does not do
About this example What the example does not do
- Toy tools: the scanner and linter cover a handful of patterns and rules; real tools are far broader.
- Fixtures and rates are invented: conclusions are about mechanisms, not about your pipeline.
- The gate's statistics are an unpaired approximation; paired tests and sequential testing are better.
- Rollout model is binary: healthy or bad, no partial degradations, no cost or latency signals.
- No real build, registry, orchestrator, load balancer or database migration.
- No vulnerability scanning, SBOMs, signing or supply-chain checks.
- Optional real experiment: add the gate to a real repository's CI, ship a deliberately bad prompt through it, and then deploy a bad release to a canary and time the automatic rollback; finally roll back a release that included a migration.
Check yourself
After a bad push, you pull the image tag latest and get the bad build. How do you deploy safely?
Show answer
Deploy by digest, which cannot be re-pointed. Pulling the recorded digest returned exactly the previous good image.
For the curious An eval gate on a small set blocks good changes
In the simulation, blocking on any observed drop above 2 points missed 26 percent of real regressions and blocked 34 percent of good changes with 200 eval cases. A bad release at 20 percent errors produced 5,462 failed requests (6.1 percent) when found by hand after 8 minutes, against 936 (1.04 percent, with a 1.0 percent floor) with a 5 percent canary and automatic rollback. The scanner and lint are toy versions of real tools.
What you will use daily at work
| Practice | What it looks like |
|---|---|
| Pin images by digest | Record the digest of every release. |
| Keep secrets out of the repo and image | Inject them at run time and scan every commit. |
| Run evals in CI with a statistical gate | Not a bare score difference. |
| Canary with automatic rollback | And rehearse the rollback. |