CSE Roadmap

Core tier · about 3 hours · 12 parts

Part 8 of 12 · 2 min read

Answering deployment questions in an interview

How do you ship a prompt change safely?

"Prompts are versioned with the code and go through the same pipeline: regression evals gate the merge, staging replays traces, then a canary with automatic rollback on error, latency, cost and quality proxies. In the example's simulation a gate on a significant drop needed about 1,000 cases to miss only 14 percent of real 5-point regressions; with 200 cases it missed two thirds, so I invest in the eval set." Follow-up: "What if the eval set is small?" Use paired comparisons, more repeated runs and a looser canary, and accept that the gate is a smoke test, not proof.

Rollback: what makes it work or not?

"Immutable artifacts and compatible data. I deploy by digest so rollback is exact, and I use expand-contract migrations so the old version can still run. In the example's toy, rolling back latest after a bad push pulled the bad image while the digest was exact." Follow-up: "What about in-flight runs?" They resume from journals (module 3) and the journal's workflow version is kept compatible or pinned.

Where do secrets go?

"Not in code, images, build args or logs. They are injected at runtime from a secrets manager, scoped per environment and tenant, rotated, and scanned for on every commit. A pattern scanner is a safety net, not the control: the example's toy one missed 3 of 10 real secrets." Follow-up: "What if one leaks?" Revoke and rotate first, then find the exposure window in audit logs.

Serverless or containers for agents?

Serverless can still suit short webhook handlers and fan-out workers; I would check the provider's current limits." Follow-up: "What do you do when you outgrow one server?" Add instances behind a standard load balancer (easier with SSE, module 4) and use a managed container platform.