CSE Roadmap

Core tier · about 4 hours · 12 parts

Part 2 of 12 · 2 min read

Gate by consequence: how bad is it if the action is wrong?

Think of airport security. Anything risky waits for a check before it goes through, and the checkpoint is useful only if the people at it actually look. If the person at the gate stamps every bag without looking, there is no security at all.

In plain words: human-in-the-loop means a person takes part in the run by approving, editing, answering a question or taking over, through an approval gate -- a point where a risky action waits for a decision -- and which actions get gated should be decided by consequence: whether the action is irreversible (cannot be undone), how wide its blast radius is if it's wrong, and what risk score the agent or a rule assigns it, rather than gating every action equally.

A worked example. Across the simulated actions (30% irreversible, with a 4% chance an irreversible action is harmful against 0.5% for a reversible one), gating only irreversible actions sends 1.8 approval requests per task on average. Gating on a risk-score threshold of 0.5 or higher sends just 1.0 request per task -- fewer requests, because it targets the specific actions estimated as risky rather than every irreversible one, whether or not that particular one happens to be risky.

The key lines. From code/mini-hitl/hitl.py:

# from code/mini-hitl/hitl.py
def gated(policy, irrev, risk):
    if policy == "none":
        return False
    if policy == "all":
        return True
    if policy == "irreversible":
        return irrev
    if policy == "risk>=0.5":
        return risk >= 0.5

Read it as: each policy is a one-line rule over just two facts about the action, irrev and risk. The entire design space of "what to gate" in this module reduces to picking which of these small rules to apply. This is the honest version of a much bigger real decision. That decision is to classify every tool your agent has by reversibility and cost, once, in advance.

Human-in-the-loop, approval gate, escalation, irreversible action, blast radius, risk score

Human-in-the-loop (HITL) is a person taking part in the run, by approving, editing, answering a question or taking over. An approval gate is a point where a risky action waits for a decision. Escalation is handing a case to a person because the agent cannot or should not decide. An irreversible action is one that cannot be undone, such as sending an email or moving money. Blast radius is how much is affected if the action is wrong. A risk score is the agent's or a rule's estimate of how likely harm is.

Example: the 4%-versus-0.5% harm-rate gap between irreversible and reversible actions is blast radius made concrete -- an irreversible action is worth gating not because it's inherently more likely to be a mistake, but because a mistake there cannot be walked back, which is exactly why "irreversible" and "risky" are related but separate questions worth asking about every tool.