Definitions

Definitions

The pipeline

Results

1. A gate is a statistical test with a power and a false-alarm rate: choose both deliberately.
2. Automatic canary rollback cuts the damage by roughly 6 times in this setup.
3. Digests make rollback exact; tags do not.
4. Migrations decide whether rollback is possible.

Design rules

• Containerise and pin: pinned base images, non-root, healthcheck, resource limits, minimal image; build once, promote the same digest through environments.
• Handle secrets structurally: runtime injection from a secrets manager, no secrets in images, build args, repos or logs; scan commits and images; rotate and have a leak runbook (module 11).
• Version prompts, models, tool schemas and configs with the code, with a release identifier recorded in every trace (module 9).
• Run evals in CI with a regression suite from real incidents (Level 9, Level 10 module 8); report confidence, not just a pass/fail.
• Deploy progressively (staging with replayed traces, then canary) with health checks on errors, latency, cost and quality proxies, and automatic rollback.
• Drain on deploy: stop taking new jobs, let in-flight runs finish or checkpoint (module 3), keep both versions able to read queued jobs.
• Use expand-contract migrations; never ship a destructive migration in the same release as its code.
• Keep the previous digest warm and rehearse rollback in staging and occasionally in production.
• Separate environments and credentials (dev, staging, prod) with distinct keys and data.
• Record every deploy (who, what digest, when, which eval results) for audit (Level 10 module 9).

Interview questions, with the follow-ups

Common mistakes

• Secrets in the image, repo, build args or logs.
• latest tags and rebuilding the "same" version.
• Running as root, no healthcheck.
• Prompt and model changes shipped outside the pipeline.
• An eval gate that flakes (blocks good changes) or never blocks (underpowered).
• Destructive migrations bundled with the code change.
• No rollback rehearsal; the first rollback happens during an incident.
• Deploying while runs are in flight with no draining or journal compatibility.
• One environment's credentials reused in another.

What the example does not do

• Toy tools: the scanner and linter cover a handful of patterns and rules; real tools are far broader.
• Fixtures and rates are invented: conclusions are about mechanisms, not about your pipeline.
• The gate's statistics are an unpaired approximation; paired tests and sequential testing are better.
• Rollout model is binary: healthy or bad, no partial degradations, no cost or latency signals.
• No real build, registry, orchestrator, load balancer or database migration.
• No vulnerability scanning, SBOMs, signing or supply-chain checks.
• Optional real experiment (I did not run this): add the gate to a real repository's CI, ship a deliberately bad prompt through it, and then deploy a bad release to a canary and time the automatic rollback; finally roll back a release that included a migration.

For the curious An eval gate on a small set blocks good changes

In the simulation, blocking on any observed drop above 2 points missed 26 percent of real regressions and blocked 34 percent of good changes with 200 eval cases. A bad release at 20 percent errors produced 5,462 failed requests (6.1 percent) when found by hand after 8 minutes, against 936 (1.04 percent, with a 1.0 percent floor) with a 5 percent canary and automatic rollback. The scanner and lint are toy versions of real tools.

What you must be able to say (one page)

• Start simple, containerise, and add complexity only on evidence. Begin with a single-server Docker and docker-compose deployment (hundreds of concurrent users can be served from one server), scale when you see concrete symptoms (degraded response times, connection limits), and then move to a container platform such as AWS ECS, Azure Container Instances or Google Cloud Run for scaling, health checks and deployments. Serverless functions can fit agent applications poorly because many tasks run past a function time limit (for example 15 minutes on AWS Lambda) and cold starts hurt responsiveness; check your provider's current limits. A single-container monolith with synchronous calls is simple for a prototype but becomes a bottleneck and a single point of failure.

• Secrets never live in the repo, image layers, build arguments or logs. Inject them at runtime from a secrets manager or environment, scan every commit, and rotate any secret that leaked. My toy scanner (regexes for key shapes, AWS keys, private keys and key = value assignments, skipping ${VAR} references and placeholders) on 16 fixture lines found 7 of 10 real secrets with 1 false alarm (a test key) and missed 3 (a password inside a URL, a bearer token, a GitHub token). Real scanners such as gitleaks or trufflehog use many more rules and entropy checks; the lesson is that pattern scanners miss things, so also keep secrets out structurally.

• Container hygiene (toy lint): pin base images (not latest or untagged), run as a non-root user, add a healthcheck, never put secrets in ENV or ARG. My lint flagged all four on a bad Dockerfile and nothing on a good one. Also set resource limits, a read-only filesystem where possible, and scan images for vulnerabilities.
• Deploy by digest, not by tag. A tag such as latest can be re-pointed; after a bad push, pulling latest returned the bad image while pulling the recorded digest returned exactly the previous good one. Record the digest of every release; rollback means redeploying a previous digest.
• Evals belong in CI. Deployment needs quality gates: quantitative thresholds on evaluation sets, stability tests, and a way to roll out, monitor for regressions and roll back quickly. Production traces should feed a regression corpus ("shift left"). Version prompts, model identifiers and tool schemas with the code (module 9), and run the regression suite on every change.
• Gate design matters, because eval sets are noisy. In my simulation (1,000 changes, 20 percent really lose 5 accuracy points from a baseline of 80 percent): blocking on any observed drop above 2 points missed 26 percent of real regressions and blocked 34 percent of good changes with 200 eval cases (34 and 43 percent at 50 cases; 6 and 13 percent at 1,000); blocking only on a statistically significant drop kept false blocks near 5 percent but missed 67 percent of the real 5-point regressions at 200 cases (85 percent at 50; 14 percent at 1,000). Without a gate every regression shipped. So: a bigger or paired eval set buys power; a small one forces a choice between flaky builds and missed regressions (Level 9 modules 2 and 9).
• Progressive delivery with automatic rollback. A bad release (20 percent errors versus 1 percent normal) at 50 requests per second for 30 minutes: all at once, noticed by humans after 8 minutes (assumed), produced 5,462 failed requests (6.1 percent of all); a 5 percent canary with a health check and automatic rollback after 1 minute produced 936 failed (1.04 percent, against a 1.0 percent floor). Canary details and statistics are in Level 9 module 9; here the point is the mechanism: a health check that can roll back without a human.
• Rollback is only safe if the database allows it. Rolling back the code after a migration that drops, renames or makes required a column breaks the old version; use expand then contract: first deploy additive, backward-compatible changes (new nullable columns), switch code, and remove old columns in a later release.
• Agent-specific deploy concerns: runs are long (module 3), so drain workers and let journals resume on the new version (module 9 versioning); streams reconnect (module 4); queued jobs must be readable by both versions during rollout; prompts and tools change behaviour without code changes, so they ride the same pipeline.
• Honest limits: toy scanner and linter, simulated gate and rollout with assumed rates, no real CI system, registry, orchestrator or database.