CSE Roadmap

Core tier · about 3 hours · 12 parts

Part 9 of 12 · 2 min read

Mistakes that turn a release into an incident

Common mistakes, and what the example does not do.

Mistakes that turn a release into an incident

  • Secrets in the image, repo, build args or logs.
  • latest tags and rebuilding the "same" version.
  • Running as root, no healthcheck.
  • Prompt and model changes shipped outside the pipeline.
  • An eval gate that flakes (blocks good changes) or never blocks (underpowered).
  • Destructive migrations bundled with the code change.
  • No rollback rehearsal; the first rollback happens during an incident.
  • Deploying while runs are in flight with no draining or journal compatibility.
  • One environment's credentials reused in another.

What the example does not do

About this example What the example does not do
  • Toy tools: the scanner and linter cover a handful of patterns and rules; real tools are far broader.
  • Fixtures and rates are invented: conclusions are about mechanisms, not about your pipeline.
  • The gate's statistics are an unpaired approximation; paired tests and sequential testing are better.
  • Rollout model is binary: healthy or bad, no partial degradations, no cost or latency signals.
  • No real build, registry, orchestrator, load balancer or database migration.
  • No vulnerability scanning, SBOMs, signing or supply-chain checks.
  • Optional real experiment: add the gate to a real repository's CI, ship a deliberately bad prompt through it, and then deploy a bad release to a canary and time the automatic rollback; finally roll back a release that included a migration.

Check yourself

After a bad push, you pull the image tag latest and get the bad build. How do you deploy safely?

Show answer

Deploy by digest, which cannot be re-pointed. Pulling the recorded digest returned exactly the previous good image.

For the curious An eval gate on a small set blocks good changes

In the simulation, blocking on any observed drop above 2 points missed 26 percent of real regressions and blocked 34 percent of good changes with 200 eval cases. A bad release at 20 percent errors produced 5,462 failed requests (6.1 percent) when found by hand after 8 minutes, against 936 (1.04 percent, with a 1.0 percent floor) with a 5 percent canary and automatic rollback. The scanner and lint are toy versions of real tools.

What you will use daily at work

Practice What it looks like
Pin images by digest Record the digest of every release.
Keep secrets out of the repo and image Inject them at run time and scan every commit.
Run evals in CI with a statistical gate Not a bare score difference.
Canary with automatic rollback And rehearse the rollback.