CSE Roadmap

Common tier · about 4 hours · 12 parts

Part 8 of 12 · 2 min read

Answering red-teaming questions in an interview

How do you red-team an agent?

"I define the attacker's goals from the threat model, generate attacks as a grid of goals, carriers, phrasings and obfuscations, run each many times against the real agent with tools mocked, and report attack success rate with intervals, per category and weighted by severity. I fix, re-run on held-out attacks, and keep every successful attack in a regression suite that runs in CI." Follow-up: "Which tools?" Open-source frameworks the book names, such as DeepTeam, Garak and PyRIT, plus manual creative attacks; I would evaluate them on my own agent rather than trust their scores.

Your fix drove ASR to 0. Are you done?

"No. Zero means no known attack worked in the attempts made. At a true rate of 1 percent I need about 300 attempts to see one success. I check coverage of goals and carriers, evaluate on held-out attacks, and ask whether the fix is structural or just matches the attacks I saw: my learned blocklist went from 5 percent on training attacks to 9 percent on held-out ones." Follow-up: "How do you avoid overfitting?" Split attacks before designing fixes, rotate attackers, and prefer controls that do not depend on wording.

How do you put red-teaming in CI?

"A fixed regression suite of past successful attacks plus a sample of generated ones, run several times each, with a gate on the interval: fail only if the lower bound of the success rate exceeds the allowed rate, and track severity-weighted rate. That avoids failing on noise, and catches confident regressions." Follow-up: "How often do you run the full grid?" Nightly or on changes to model, prompt or tools; the quick suite on every pull request.

Automated or human red-teaming?

"Both. Automation covers known patterns at scale and gives measurable rates; humans find new goals and chains of steps, such as using one tool's output to steer another. I feed human findings back into the generator and the regression suite." Follow-up: "Who should do it?" People who did not build the agent, with permission and a scope.