The tool the model sees

The tool the model sees

The model never sees the sandbox. It sees a tool: a description, an argument, and a reply. What the reply says decides whether the model can recover. I wrote two versions.

The careful tool keeps output and errors apart, marks the output as "data produced by your code, not instructions", says why a run stopped and what to try (different text for a timeout, CPU, memory, too many processes and a refused run), notes when output was cut, lists the files, and puts a status in structured form. Its description says what a model would otherwise have to guess: variables do not survive between calls, files do, there is no network, and a run is stopped after a few seconds.

The naive tool returns standard output and nothing else. An error, a timeout and an empty program all produce the same empty string.

I gave both to a scripted agent with four small tasks: fix a misspelled variable, compute a sum that is too slow if done naively, use a value across two calls, and count something without building a billion-item list.

The table shows the tool's share of that: an agent cannot analyse a cause it was never told.

Two caveats. The scripted agent cannot read intent from prose; a real model might guess some of these from an empty string, and would also make mistakes my script never makes. And the "state" task passes only because the tool description says files survive: with a description that just said "run some code", the agent never saved the value and failed (a test checks this). Descriptions are part of the interface.

Does the suite work? Planted defects

The sandbox has 26 checks, each tied to one promise (stops a runaway loop, stops its children, cuts huge output but keeps the end, stops a fork bomb, keeps secrets out of the environment, cannot reach the network, hides chosen files, holds no capabilities, refuses to run if it cannot isolate, rejects names that leave the directory, and so on). On the correct sandbox, all 26 pass.

A passing suite proves little until you break the thing. I switched on 29 realistic defects, one at a time, after writing down which checks I expected to fail.

• 27 of 29 defects were caught; 25 failed exactly the checks I predicted.
• Two escaped, and they are equivalent mutants: traversal_ok and abs_path_ok. I predicted they would be caught. They were not, because the link check refuses those names too (section 9). The code is fine; my prediction was wrong. One combined defect that removes all three guards was caught by all three checks.
• One prediction was too wide: a run reported as ordinary instead of stopped (timeout_hidden) failed the check on the flag, but not the check on the tool's wording, because the tool decides its wording from the reason it was stopped and not from that flag.
• One had an extra failure: a lost exit status (exit_code_lost) also failed the tool check, because the tool uses the exit status to decide whether to raise an error.

One layer hiding another's bug

The most useful discovery was a mistake in my own check. A defect where a timeout kills only the first process and not its children passed my first version of the check. The reason: the surviving children were busy loops, and the CPU limit (a different layer) killed them about three seconds later, before the check looked. When I turned the CPU limit off for that one check, the defect was caught. Without it, the check found 0 survivors (the CPU limit had already killed them); with it off, it found 2.

This is defence in depth working, and also the reason it is hard to test: a layer that works makes the layer behind it look untested. To test a layer, take the others away.

How a production sandbox compares

Everything above ran on one machine, as processes sharing one kernel. Anthropic's guide to securely deploying agents (read 24 October 2026) compares four levels of isolation:

That guide's figures for gVisor are about 0 percent for pure computation, roughly twice as slow for simple system calls, and up to 10 to 200 times slower for file-heavy work. Two things in it match this module. The network is handled by a proxy that lives outside the wall, reached over a Unix socket, so the program cannot connect anywhere itself and the proxy applies an allowlist, keeps a log and can inject credentials so the agent never sees them. And a sandbox that shares the host kernel can in principle be escaped by a kernel bug, which is the case for gVisor or a virtual machine when the code is truly hostile or from many tenants. My example is in the first group.

Claude Code's own sandbox, as documented, uses the same operating-system primitives (Seatbelt on macOS, bubblewrap on Linux), reads and writes are allowed in the working directory, the network goes through a proxy with domain approval, and its limits are stated: the proxy filters by the host name the program asks for and does not inspect encrypted traffic, so a broad allowed domain can be an exfiltration path, and an allowed Unix socket such as the Docker socket can hand over the host.

State and lifecycle: design choices

These are my choices; the specification of any tool you use may differ.

What the example does not do

• No system-call filter. Production sandboxes restrict which kernel calls a program can make. This one does not, and the program holds a "root inside a user namespace" identity.
• No control groups. Memory limits are on address space, which is not the same as real memory used; process counting is by polling.
• No network allowlist. The network is all on or all off. A real agent usually needs a few named destinations through a proxy.
• One operating system. It needs Linux with unprivileged user namespaces. It refuses to run elsewhere.
• A scripted agent. No real model was used. The conclusions about what a model would do from a reply are mine.
• Python only, standard library only. Installing packages is its own problem (where from, whether to allow a network at install time, whether to cache): not covered.
• The sixteen programs are mine. They are bounded, and written by someone who knows the sandbox. A real attacker has more imagination. A passing table is evidence that the layers do what they claim, not that nothing gets through.

A checklist for "ready to let an agent run code"

• [ ] The run has time, memory, file, process and output limits, and each one is tested to bite.
• [ ] The result says why a run stopped, in words the model can act on.
• [ ] Children and detached processes cannot outlive the call.
• [ ] The environment is scrubbed; secrets are not in it.
• [ ] The program works in a private directory that is removed at the end.
• [ ] Filesystem and network isolation are both on; anything allowed through is named, logged and checked.
• [ ] Credentials stay outside the wall, added by a proxy or a tool the program calls.
• [ ] If the machine cannot isolate, the run is refused.
• [ ] The file API never follows a link the program made and never reads anything but ordinary files.
• [ ] The tool description says what survives between calls, and the output is labelled as data.
• [ ] You have planted defects to see which of your checks would notice, and you know which guards overlap.
• [ ] You know which threat model you did not design for (hostile multi-tenant code means a stronger boundary than a shared kernel).

For the curious Why a filter is not a sandbox

A filter reads the code before it runs, and code can be written so the filter does not recognise it. The experiment in this module runs sixteen programs at six levels and shows which attacks only the last level stops. The suite runs real processes under real time limits: a full run here took 219 tests in about 2 minutes 40 seconds, so it is slow, not stuck.