CSE Roadmap

Core tier · about 5 hours · 13 parts

Part 8 of 13 · 2 min read

Answering durable-state questions in an interview

How do you make an agent survive a crash or a deploy?

"I keep state outside the process, keyed by a run id, and load it on every request so any worker can resume; this is called the stateless design. I checkpoint after each step, and because a crash can fall between a side effect and its checkpoint, I give every effect an idempotency key. I log model outputs so I can replay." Follow-up: "Snapshot or event log?" Snapshots resume instantly but write quadratically: 6.1 MB against 64 kB at 200 steps in the example's model. A log writes little but replays 100 events on average. I'd use a hybrid and pick the interval from the resume time I can tolerate.

Why can't you just re-run the agent from the log?

"Because a model call isn't deterministic. In the example's test a live re-run diverged from the original in essentially every run, even over five steps. Replay has to re-apply the recorded outputs, so I store them, and apply has to be a pure function." Follow-up: "Then how do you debug a bad run?" Load a checkpoint, fork, and change one input, with recorded results for the steps before it.

What happens when you change the state schema?

"Old checkpoints no longer match the code. I store a version number and keep a migration chain, so a v1 state is upgraded step by step to the current one, and state from a newer version is refused. Without it, an old run fails on resume, as the KeyError in the example's test shows." Follow-up: "Do you migrate stored runs or at load time?" Usually at load; a batch migration is an option when the old code must be removed.

An agent paused for approval and resumes a day later. What can go wrong?

"The world changed. With a 2 percent hourly chance in the example's model, 38 percent of day-long pauses resumed on a false premise. So before the irreversible step I re-read the facts it depends on, and re-plan or ask again if they differ; I also expire very old pauses." Follow-up: "Isn't that expensive?" One read per resume, against a wrong refund or a repeated email.