AI command warnings fail when people must judge every step
A permission-game experiment found people missed one in three threats. The important result is not the percentage; it is what repeated approval asks do to attention.
Across more than 40,000 game runs, players missed roughly one third of risky AI-agent commands despite an upfront warning.
HN’s consensus was sharper: command-by-command consent is a poor security boundary; policies and sandboxes should prevent whole classes of access instead.
A warning is not a boundary
The AI Agent Permission Game collected more than 40,000 runs and reports that players missed one in three threats while approving or rejecting commands. Its creator stresses that it is a game, not a production incident dataset—but it captures a familiar failure mode: a long stream of decisions turns careful review into routine clicking.
HN pushed the conclusion beyond better warning copy. Several practitioners argued that a system should ask about durable capabilities—what files or services an agent may access—rather than force people to interpret isolated shell commands. Another proposed wrapping commands in a sandbox with narrowly scoped access.
That distinction matters. The experiment cannot tell us how often real agent commands are dangerous, and commenters rightly questioned how representative its prompts are. It does show why an approval dialog should be the last line of defense, not the primary one. A useful agent environment makes unsafe actions impossible or conspicuous before the user is asked to make a tired, context-poor judgment.
Get breakdowns like this every day
The top Hacker News stories, unpacked — by email, Telegram, or RSS.
Another GitHub outage turns CI dependence into operational risk
GitHub marked Actions and Pages as degraded. For teams whose releases are a workflow file, an outage is not merely an inconvenience—it halts the route to production.