Part 8 of 11 · 2 min read
Mistakes that turn a model mistake into a real one
Common mistakes, and what the example does not do.
Mistakes that turn a model mistake into a real one
- Policy only in the prompt, so one clever message overrides it.
- Tools with no preconditions, trusting the model to check.
- Taking customer or tenant ids as model arguments.
- No idempotency key, so a retry refunds twice.
- Choosing a confidence threshold by feel without measuring calibration.
- Automating everything on day one.
- Escalating with no context, so the human starts over.
- Counting only automation rate, ignoring wrong actions and reopens.
- Letting an operations agent run arbitrary shell commands.
- No sampling of automated tickets, so silent errors persist.
What the example does not do
About this example What the example does not do
- No model: triage is a simulated noisy classifier with invented calibration; real models fail differently and their confidence is often poorly calibrated.
- Single-turn tickets: no conversation, clarifying questions, tone, language or customer history.
- Invented mix, order states and costs: the cost-optimal policy will move with your numbers.
- Human errors are ignored: escalated tickets are assumed handled correctly, which flatters the assist and guarded modes.
- Wrong replies are cheap: the model of customer harm (churn, complaints) is crude.
- Operations agent is a classifier of strings, not a real shell guard; regular expressions are easy to evade, so it needs real permissions and sandboxing underneath (Level 10 modules 2 and 3).
- No SLA, queueing or staffing model for escalations.
- Optional real experiment: replay a month of your real tickets through a real classifier in shadow mode, measure accuracy, calibration, wrong-action rate and escalation load, and recompute the cost table with your own penalties.
Check yourself
Why put the refund rule in the tool and not only in the prompt?
Show answer
A tool enforces its own preconditions, so a confused model cannot talk past it. In autonomous mode the tools blocked 62 wrong attempts, which became unhelpful replies instead of wrong actions.
For the curious More autonomy is not simply cheaper
In the 1,000-ticket simulation, autonomous with tool checks automated 86 percent but made 24 wrong mutating actions ($1,003 refunded wrongly); the guarded policy automated 33 percent with 3 wrong actions and $0 wrongly refunded. A threshold of 0.7 was the cheapest sweep point at $1,990. Mix, accuracy and costs are assumed.
What you will use daily at work
| Practice | What it looks like |
|---|---|
| Scope to a few intents with clear rules | Send billing and unusual cases to people. |
| Keep policy in code | Confidence threshold, refund cap, legal and fraud words. |
| Key refunds so a retry does nothing twice | And scope the key to the customer and order. |
| Hand off with a full packet | Ticket, order, reason, what was tried, suggestion. |