Part 5 of 11 · 2 min read
What the simulation measured
Policy comparison, threshold and accuracy sweeps, and the operations risk-tier check.
How to read the policy table: all three rows face the identical 1,000 tickets; "Wrong mutating actions" and "Refunded wrongly" are the real cost of automation, "Automated" is its benefit.
| Policy (1,000 tickets, assumed inputs) | Automated | Wrong mutating actions | Refunded wrongly | Wrong replies | Total cost |
|---|---|---|---|---|---|
| Assist only | 0% | 0 | $0 | 0 | $4,020 |
| Autonomous, tool checks only | 86% | 24 | $1,003 | 62 | $2,129 |
| Guarded (threshold 0.8, $50 cap) | 33% | 3 | $0 | 7 | $2,750 |
What to notice: assist-only, despite making zero mistakes, is the most expensive policy here. The cost of 1,000 human escalations outweighs the cost of a few dozen wrong actions. That's exactly why "never automate" isn't automatically the safe choice either.
How to read the threshold-sweep table: each row is the same ticket stream at a different confidence threshold; read top to bottom as the threshold tightens.
| Guarded threshold | Automated | Wrong actions | Total cost |
|---|---|---|---|
| 0.5 | 57% | 12 | $2,047 |
| 0.6 | 55% | 8 | $2,015 |
| 0.7 | 54% | 4 | $1,990 |
| 0.8 | 33% | 3 | $2,750 |
| 0.9 | 17% | 0 | $3,356 |
What to notice: cost drops smoothly from 0.5 to 0.7, then jumps sharply at 0.8 and 0.9. The cheapest setting is in the middle of the sweep, not at either end. That's why a threshold has to be tuned against real cost rather than set by instinct toward "safer is better."
How to read the accuracy table: each row is a different assumed classifier accuracy; compare the two policy columns at the same accuracy, not across accuracies.
| Classifier accuracy | Autonomous cost (wrong actions) | Guarded 0.8 cost (wrong actions) |
|---|---|---|
| 95% | $1,385 (10) | $2,654 (2) |
| 88% | $2,129 (24) | $2,750 (3) |
| 75% | $4,157 (55) | $2,981 (10) |
What to notice: which policy is cheaper flips depending on accuracy. Autonomous wins at 95 percent but loses badly at 75 percent. So the right autonomy setting is a direct function of how good the classifier actually is, not a fixed product decision made once.
How to read the operations command table: each row is one risk tier; the decision column follows directly from matching the command against known patterns.
| Operations command | Tier | Decision |
|---|---|---|
get pods -n payments, logs api-7f, describe node n3 |
read-only | run |
restart service api, update config set timeout=30 |
mutating | needs approval |
scale deployment api --replicas 40 (12 targets) |
mutating, large blast radius | needs approval |
rm -rf /var/lib/data, DROP TABLE users, git push -f origin main, kubectl delete pod ... --force |
destructive | blocked |
What to notice: the blast-radius row shows a command that would otherwise just be "mutating, needs approval" getting flagged specifically for its scale. The same command type at 1 target would still just need approval, not get blocked. But at 12 targets its blast radius alone pushes it into closer scrutiny.
{{svg:flow|Triage, policy, tools with preconditions, reply or escalate.}} {{svg:results|Autonomy, mistakes and cost under three policies and a threshold sweep.}}