CSE Roadmap

Common tier · about 4 hours · 11 parts

Part 5 of 11 · 2 min read

What the simulation measured

Policy comparison, threshold and accuracy sweeps, and the operations risk-tier check.

How to read the policy table: all three rows face the identical 1,000 tickets; "Wrong mutating actions" and "Refunded wrongly" are the real cost of automation, "Automated" is its benefit.

Policy (1,000 tickets, assumed inputs) Automated Wrong mutating actions Refunded wrongly Wrong replies Total cost
Assist only 0% 0 $0 0 $4,020
Autonomous, tool checks only 86% 24 $1,003 62 $2,129
Guarded (threshold 0.8, $50 cap) 33% 3 $0 7 $2,750

What to notice: assist-only, despite making zero mistakes, is the most expensive policy here. The cost of 1,000 human escalations outweighs the cost of a few dozen wrong actions. That's exactly why "never automate" isn't automatically the safe choice either.

How to read the threshold-sweep table: each row is the same ticket stream at a different confidence threshold; read top to bottom as the threshold tightens.

Guarded threshold Automated Wrong actions Total cost
0.5 57% 12 $2,047
0.6 55% 8 $2,015
0.7 54% 4 $1,990
0.8 33% 3 $2,750
0.9 17% 0 $3,356

What to notice: cost drops smoothly from 0.5 to 0.7, then jumps sharply at 0.8 and 0.9. The cheapest setting is in the middle of the sweep, not at either end. That's why a threshold has to be tuned against real cost rather than set by instinct toward "safer is better."

How to read the accuracy table: each row is a different assumed classifier accuracy; compare the two policy columns at the same accuracy, not across accuracies.

Classifier accuracy Autonomous cost (wrong actions) Guarded 0.8 cost (wrong actions)
95% $1,385 (10) $2,654 (2)
88% $2,129 (24) $2,750 (3)
75% $4,157 (55) $2,981 (10)

What to notice: which policy is cheaper flips depending on accuracy. Autonomous wins at 95 percent but loses badly at 75 percent. So the right autonomy setting is a direct function of how good the classifier actually is, not a fixed product decision made once.

How to read the operations command table: each row is one risk tier; the decision column follows directly from matching the command against known patterns.

Operations command Tier Decision
get pods -n payments, logs api-7f, describe node n3 read-only run
restart service api, update config set timeout=30 mutating needs approval
scale deployment api --replicas 40 (12 targets) mutating, large blast radius needs approval
rm -rf /var/lib/data, DROP TABLE users, git push -f origin main, kubectl delete pod ... --force destructive blocked

What to notice: the blast-radius row shows a command that would otherwise just be "mutating, needs approval" getting flagged specifically for its scale. The same command type at 1 target would still just need approval, not get blocked. But at 12 targets its blast radius alone pushes it into closer scrutiny.

{{svg:flow|Triage, policy, tools with preconditions, reply or escalate.}} {{svg:results|Autonomy, mistakes and cost under three policies and a threshold sweep.}}