CSE Roadmap

Core tier · about 4 hours · 13 parts

Part 10 of 13 · 2 min read

Mistakes that leave a guardrail hollow

Common mistakes, and what the example does not do.

Mistakes that leave a guardrail hollow

  • Coercing arguments (int("50"), bool treated as int).
  • Regex with $ or \d that accepts a trailing newline or non-ASCII digits.
  • Checking syntax only.
  • Fail-open guards on security decisions.
  • Counting similar filters as separate layers.
  • Letting the model fix its own invalid output without re-validating.
  • Testing guards only with the attacks they were written for.
  • Relying on the output guard instead of closing the exfiltration channel.

What the example does not do

About this example What the example does not do
  • Small hand-written case lists; real adversaries produce inputs the lists do not cover.
  • No model-based guard and no measurement of a real guardrail library.
  • The correlation model is an assumption (Gaussian shared difficulty); real correlations must be measured.
  • Semantic checks use a toy in-memory store, with no concurrency (two simultaneous refunds can both pass; a real system needs a transaction).
  • No rate limiting and no per-session budgets (module 2 covers caps).
  • Optional real experiment: wrap a real tool with this validator, replay logged model tool calls through it, and count rejections by reason.

Check yourself

A refund call passes the schema. Is it safe to run?

Show answer

Not yet. Three schema-valid calls in the tests were someone else's order, more than the order total, and an unknown order. Only semantic checks against business state caught them.

For the curious Three guards are not three independent chances

If three guards each miss 20 percent and fail independently, 0.8 percent get through. With a shared blind spot the model gave 5.0 percent at correlation 0.5, 12.8 at 0.9 and 20.0 at 1.0. When the guard itself errored on 5 percent of calls, failing closed passed 0 attacks and blocked 4.2 percent of good outputs; failing open passed 5.2 percent of attacks. The correlation model is an assumption.

What you will use daily at work

Practice What it looks like
Validate arguments strictly Exact types, ranges, enums, no unknown fields, full-match patterns.
Check the business state Whose order, how much is left, does it exist.
Guard outputs for known exfiltration Links to unlisted hosts and secret patterns; know it misses evasions.
Choose fail closed for security checks And measure how much good traffic it blocks.