Part 7 of 12 · 2 min read
Answering coding-agent questions in an interview
How would you build a coding agent that opens pull requests safely?
"A harness with narrow repository tools, exact-match edits with size caps, protected test and CI paths, sandboxed test runs with timeouts, an attempt budget, and a review gate that flags edited tests, skips and special-cased literals. The agent works on a branch and opens a pull request with the diff, test result and gate findings; a human approves the merge. I evaluate on held-out tests and track cost per merged change and bad merges caught in review." Follow-up: "Why a gate if tests pass?" In the example's experiment a hard-coded answer and an edited test both passed the visible tests; only held-out tests and the gate caught them.
The agent made the tests pass. How do you know the fix is right?
"I do not, from that alone. I run held-out tests and the full suite, read the diff for special-casing and test edits, check linters and types, and for risky areas require human review." Follow-up: "What if the held-out tests are also weak?" Then the evaluation is weak; I improve the tests, sample merged changes for manual review and track defects found later.
How do you stop a coding agent from running amok in a repo?
"Scoped permissions (branch only, no force push, no secrets), sandbox with limits, protected paths, small-edit and attempt budgets, and a kill switch (Level 11 module 11)." Follow-up: "What about prompt injection in issue text or code comments?" Treat them as untrusted data, keep tool permissions minimal and require approval for anything beyond the repo (Level 10 module 1).
How would you evaluate a coding agent?
"A set of real tasks with held-out tests, measuring pass rate, attempts, cost and time, plus manual review of a sample of merged changes for quality and shortcuts, and a regression set built from past failures." Follow-up: "Why not just use a public benchmark?" It is a useful comparison, but my codebase, tests and risks differ, and a benchmark does not show cost or bad-merge rate on my work.