Part 6 of 17 · 3 min read
An approval tied to where data could go beats one tied to what text says
A control that depends on someone noticing a suspicious phrase fails the moment the phrase is worded differently; a control built on data flow does not care how it's worded.
Think of a bank that does not try to read every withdrawal slip for suspicious handwriting. Instead, it simply asks for a manager's signature whenever cash is about to leave the building after a large deposit was just made. The rule depends on the money's path, not on how the slip is phrased.
In plain words: tool poisoning, rug pulls, shadowing and indirect injection are four ways a model can be steered by text it was never supposed to obey, and while a person reading descriptions, a character limit, or a phrase blocklist all help and all have blind spots (a paraphrase slips past a blocklist built on known phrases), a control that asks for approval only after sensitive data has been read and only when a call could send it somewhere not already allowed to see it stops every variant, because it looks at where data could go rather than at what any particular sentence says.
A worked example. Against four poisoned-description variants plus an injected web page, a scripted model that obeys any instruction it finds complies with all four when nothing stands in its way. A person reading the whole description blocks the obvious variants but misses a paraphrase worded to avoid the reviewer's trigger phrases. An interface showing only the first 60 characters hides an instruction buried after benign text. A phrase blocklist blocks three variants and misses the same paraphrase. The taint-based control -- approval only after sensitive data has been read, only for a call to a server not already allowed to see it -- stops all four, including the encoded secret, at a cost of 14 approval prompts over 20 harmless calls with two file reads (asking about every call would have produced 22); a person who approves without reading stops nothing.
The key lines. From code/mini-auth/poison.py:
# from code/mini-auth/poison.py
def call(self, server: str, tool: str, args: dict):
s = self.servers[server]
if self.policy.taint and self.tainted and not s.sensitive:
self.approvals += 1
if not self.reviewer.approves_call(server, tool, args, self.sensitive_values):
self.blocked.append(("taint", f"{server}.{tool}"))
return None
Read it as: the condition is self.tainted and not s.sensitive. It asks for approval only once something sensitive has already been read (self.tainted). It also asks only when the destination is a server that was never cleared to see it (not s.sensitive). It never inspects the call's arguments or the tool's description for suspicious wording at all. That is exactly why a differently-worded attack does not slip past it the way it slips past a blocklist.
Tool poisoning, rug pull, shadowing, indirect injection
Tool poisoning hides instructions aimed at the model inside a tool's description. A rug pull is a server that behaves, gets approved, then later changes a tool's description. Shadowing is one server's description changing how the model uses a different server's tool. Indirect injection is an instruction hidden in a tool's result (a web page, a document) that the model follows because it cannot tell data from instruction.
Example: the four variants sharing one defence (taint-based approval) despite being worded completely differently is the chapter's whole point -- a control built on the structure of what can reach what survives a paraphrase that a content-based control, however carefully written, eventually will not.