Part 2 of 4 · 9 min read
Definitions
Covers: Definitions; Where guardrails sit and more.
Definitions
Guardrail, validator
- Guardrail: a check or constraint around the model that is enforced outside it.
- Validator: code that decides whether an input, argument or output is acceptable.
Syntactic and semantic validation
- Syntactic: the value has the right type, range and format.
- Semantic: the value is acceptable given business state (this order belongs to this user, the refund fits the total).
Fail open, fail closed
- Fail open: if the check breaks, allow the action.
- Fail closed: if the check breaks, deny the action.
Defence in depth, correlated failure
- Defence in depth: several different controls, so one failure is not fatal.
- Correlated failure: layers fail on the same inputs, so combined protection is weaker than the product of their rates.
Where guardrails sit
Results
| Check | Legitimate allowed | Bad inputs accepted |
|---|---|---|
| Keys present only | 4 of 4 | 93% (13 of 14) |
| Strict schema | 4 of 4 | 0% |
| Output guard | Flagged |
|---|---|
| 10 plain exfiltration outputs | 100% |
| 5 evasions | 0% |
| 10 benign outputs | 0% |
| Three layers, each missing 20% | Attacks through all three |
|---|---|
| Independent | 0.8% |
| Correlation 0.5 | 5.0% |
| Correlation 0.9 | 12.8% |
| Same blind spot | 20.0% |
| Guard errors on 5% of calls | Attacks passed | Good outputs blocked |
|---|---|---|
| Fail closed | 0.0% | 4.2% |
| Fail open | 5.2% | 0.0% |
- Strictness is nearly free for syntax. The strict validator rejected every malformed case, including bool-as-number, NaN, Arabic-Indic digits and a trailing newline.
- Semantics need state, so they cost a lookup, and they catch what schemas cannot.
- Evasions beat pattern guards. Treat an output guard as a tripwire for the common channel, and close the channel structurally (no external image rendering, allowlisted egress).
- Diversity beats count. Three similar filters behave like one.
Design rules
- Define a strict schema for every tool, and reject rather than coerce (no
"50"to 50, no bool as int, no NaN or infinity, no unknown keys). - Use full-match patterns with ASCII-only classes (beware of
$matching before a trailing newline and Unicode digits). - Add semantic checks against current state: ownership, totals, existence, cumulative limits.
- Authorise separately (module 2): validation does not decide who may do what.
- Guard outputs for secrets and external links, and render untrusted output without images or auto-loading links.
- Fail closed for security decisions; alert on guard errors.
- Prefer structural controls (allowlisted egress, typed tool results) over text matching.
- Re-ask or repair carefully: limit retries, and never let the model's repair text bypass the validator (it must pass again).
- Test guards with evasions, not only the attacks they were written for, and keep them in the CI gate (Level 9 module 7).
- Log guard decisions with the trace (Level 9 module 8).
Interview questions, with the follow-ups
How do you validate tool calls from a model?
"Strict schema validation on every argument (exact types, ranges, enums, full-match ASCII patterns, no unknown fields, no coercion), then semantic checks against state, then authorisation. In my test a keys-only check accepted 13 of 14 hostile argument sets and the strict schema accepted none, at about 2 microseconds a call." Follow-up: "Is that enough?" No: three valid-looking refunds still abused the system and were caught only by semantic checks, so I add grants and caps (module 2).
Are three guardrail layers three times as safe?
"Only if they fail independently. If each misses 20 percent, independent layers miss 0.8 percent, but with shared blind spots the combined miss rate rises toward 20 percent; my simulation gave 12.8 percent at correlation 0.9. I count layers that fail for different reasons: schema, policy, sandbox, approval." Follow-up: "How do you measure correlation?" Run the same attack set through each layer and look at overlap in what each missed.
Should a guard fail open or closed?
"For security checks, closed. With a 5 percent error rate in my simulation, fail-open let through 5.2 percent of attacks while fail-closed blocked 4.2 percent of good outputs. I choose per risk: an irreversible action fails closed; a low-risk suggestion may fail open with an alert." Follow-up: "How do you reduce the cost?" Make the guard reliable and fast, retry once, and degrade to a human review queue instead of a hard failure.
How do you stop data leaving through the model's output?
"Close the channels: no auto-loading images or links to untrusted hosts, an egress allowlist, and an output guard for secrets and external URLs as a tripwire. My guard caught the plain cases and missed all five evasions, which is why the structural controls matter more." Follow-up: "What about a classifier?" It helps with fuzzy cases but is probabilistic and correlated with the model's own blind spots; I would test it like any detector (Level 9 module 4).
Common mistakes
- Coercing arguments (
int("50"), bool treated as int). - Regex with
$or\dthat accepts a trailing newline or non-ASCII digits. - Checking syntax only.
- Fail-open guards on security decisions.
- Counting similar filters as separate layers.
- Letting the model fix its own invalid output without re-validating.
- Testing guards only with the attacks they were written for.
- Relying on the output guard instead of closing the exfiltration channel.
What the example does not do
- Small hand-written case lists; real adversaries produce inputs I did not.
- No model-based guard and no measurement of a real guardrail library.
- The correlation model is an assumption (Gaussian shared difficulty); real correlations must be measured.
- Semantic checks use a toy in-memory store, with no concurrency (two simultaneous refunds can both pass; a real system needs a transaction).
- No rate limiting and no per-session budgets (module 2 covers caps).
- Optional real experiment (I did not run this): wrap a real tool with this validator, replay logged model tool calls through it, and count rejections by reason.
Check yourself
A refund call passes the schema. Is it safe to run? (Not yet. Three schema-valid calls in the tests were someone else's order, more than the order total, and an unknown order. Only semantic checks against business state caught them.)
For the curious Three guards are not three independent chances
If three guards each miss 20 percent and fail independently, 0.8 percent get through. With a shared blind spot the model gave 5.0 percent at correlation 0.5, 12.8 at 0.9 and 20.0 at 1.0. When the guard itself errored on 5 percent of calls, failing closed passed 0 attacks and blocked 4.2 percent of good outputs; failing open passed 5.2 percent of attacks. The correlation model is an assumption.
What you will use daily at work
| Practice | What it looks like |
|---|---|
| Validate arguments strictly | Exact types, ranges, enums, no unknown fields, full-match patterns. |
| Check the business state | Whose order, how much is left, does it exist. |
| Guard outputs for known exfiltration | Links to unlisted hosts and secret patterns; know it misses evasions. |
| Choose fail closed for security checks | And measure how much good traffic it blocks. |
Draft: confirm which guardrail libraries or provider moderation features you will use; none were tested here.
What you must be able to say (one page)
- Guardrails are checks around the model, enforced in code. Three checkpoints: input (inspect prompts and data before the agent sees them), tool call (validate and authorise arguments before anything runs) and output (inspect results before they reach the user or another system). They are usually implemented as middleware. When a check fails you can block, or re-ask the model.
- Strict argument validation is cheap and effective against malformed input. Against 14 malformed or hostile refund arguments, a "keys are present" check accepted 93 percent; a strict schema (exact types, bools are not numbers, finite numbers only, ranges, enums, ASCII-only patterns with full match, no unknown fields, length limits) accepted 0 percent and still accepted all 4 legitimate calls. Cost: about 2 microseconds per validation on this machine.
- Syntax is not authorisation. Three refund calls that were perfectly valid by schema (someone else's order, more than the order total, an unknown order) were all caught only by semantic checks that need business state; the schema passed 3 of 3. Combine schema, semantic checks and grants (module 2).
- Output guards catch known exfiltration shapes and miss the rest. My guard (links and images to hosts outside an allowlist, plus secret patterns) flagged 100 percent of 10 plain exfiltration outputs, 0 percent of 5 evasions (no scheme,
evil[.]test, spelled-out hosts, plain-text secrets, an email address), and 0 percent of 10 benign outputs. Passing my own tests says little about an adaptive attacker. - Layers are not independent. If three guards each miss 20 percent of attacks and fail independently, 0.8 percent get through all three. With a shared blind spot (the attacks that fool one fool all, because they are all pattern matchers), my model gave 5.0 percent at correlation 0.5, 12.8 percent at 0.9 and 20.0 percent at 1.0. Count layers that fail for different reasons (a schema, a policy, a sandbox), not three similar filters.
- Decide failure behaviour in advance. When the guard itself errored on 5 percent of calls, fail closed passed 0 attacks and blocked 4.2 percent of good outputs; fail open passed 5.2 percent of attacks and blocked none. For security checks fail closed, and measure the cost in blocked good traffic.
- Honest limits: hand-written case lists; the correlation model is an assumption (Gaussian shared difficulty); no model-based guard, no PII or toxicity classifier (module 6 covers PII).