Part 11 of 14 · 3 min read
Mistakes that make a tool design dangerous
Each one looks safe until the example's own attacks are run against it.
Mistakes that make a tool design dangerous
- Trusting "read-only" to mean "safe." Symptom: a design that blocks every write still leaks sensitive data. Cause: read-only is a property of the connection, not a limit on what rows or columns can be read. Fix: limit what a read can return (columns, rows) as its own control, separate from whether writes are possible.
- Letting approval stand in for every other control. Symptom: an approver who says yes to everything adds zero measured protection. Cause: a person shown too many prompts, or shown a vague "approve this call?" message, stops reading them. Fix: reserve approval for the irreversible or costly, validate everything else in code first, and show exactly what will happen in words.
- Taking row scope from a model-chosen argument. Symptom: a tool that takes a
customer_idparameter can be asked about any customer. Cause: the argument is wherever the model (or whatever influenced it) decided to point it. Fix: take row scope from the grant your own code set, and leave the argument out of the tool's signature entirely when possible. - Handing out one standing credential for the whole session. Symptom: a credential minted at the start of a session still holds every permission it started with, no matter what the task becomes. Cause: minting once is easier than minting per task. Fix: mint a short-lived, task-scoped grant before the model reads any untrusted text, so later text cannot widen what it can do.
- Writing a free-form tool and calling it convenient. Symptom: a single
sql_execorrun_commandtool trips a linter's highest-severity rule on sight. Cause: one tool that does everything is less design work up front than seven tools that each do one thing. Fix: start narrow, and add a tool only when a real task fails for lack of one.
What the example does not do
What the example does not do
These grants are plain objects in one process. A real system uses signed tokens that a separate service issues and checks, with an audience (which service the token is for) so that a token for one system cannot be replayed at another. Module 5 builds that, with MCP's authorisation model. The point here is the shape: narrow, short-lived, issued per task, checked in code you control. Never let the model or the user's text decide what a grant contains. The "model" in the example is a script that makes the nearest call its tools allow and does not get cleverer when blocked; a real attacker, and a real model under attack, will adapt. The numbers show how much each design limits the damage of one fixed set of requests, not how often a real model gets fooled.
Check yourself
A tool lets the model run any SQL query, and a poisoned support ticket asks it to dump the customer table. Which control stops that, and why is "tell the model not to" not it?
Show answer
A narrow tool with a read-only role and a row limit stops it in code. An instruction is a request the model may follow anyway.
What you will use daily at work
| Practice | What it looks like |
|---|---|
| One job per tool | get_order and refund_order, not run_sql. |
| Task-scoped credentials | A short-lived grant for one task, not a standing key. |
| A gateway in front of tools | Argument checks, an allow list, a log, and approval for the risky ones. |