He spent $1,500 letting LLMs attack his app — guardrails skewed the score
A developer built a deliberately vulnerable web app, pointed frontier models at it with $1,500 of tokens, and tallied which ones could break in. The surprise wasn't who won — it was that Anthropic's models lost partly because they refused to play.
A deliberately broken app and a $1,500 token bill
The experiment is exactly as advertised: build a web app, BookNook, salt it with real vulnerabilities, then spend roughly $1,500 of API tokens letting frontier LLMs try to hack it through an automated harness. The output is a scoreboard — which models found which flaws, with statistical confidence columns (two tables carry a "95% Wilson CI" header, a confidence interval for a success rate) — and a candid running tally of money burned. It's the kind of hands-on benchmark that's easy to admire and easy to over-read.
The lowest scores belonged to the models that refused
The most-upvoted reading flips the leaderboard on its head: Anthropic's poor numbers weren't a capability gap, they were a policy.
"It's not because of capability, it's because Anthropic's guardrails prevented it from solving the problem… Its propensity to refuse doing legitimate work has been increasing." — a developer who runs deterministic secret-swapping setups
That refusal behavior is reproducible. One reader who does authorized testing on his own software describes the same wall — and a workaround:
"When I used claude to do some pen testing for one of my apps it initially refused. After I explained and demonstrated I'm the author, it reasoned through it and allowed it." — tjwheeler, who builds AI infrastructure
If one contestant is hobbled by safety training, the scoring gets murkier still when another is explicitly un-hobbled:
"It seems harsh to critique guardrails… when GPT-5.5 seems to have been explicitly whitelisted to remove most of said guardrails. A more fair comparison would be a vanilla GPT account." — mynameisvlad, a software engineer
Then there's the methodology itself. A reader who runs commercial models against reverse-engineering challenges argues the harness expected the LLM to one-shot everything alone:
"Expecting the model to do everything by itself is unrealistic… working along the model works really well… Chinese models are much more capable than people give it credit for." — a tester who uses GLM 5.1 on crackme challenges
Another spotted a tell that points the same direction: the one non-GPT model that solved the task in local testing failed in the long harness runs. "Doesn't that sound like maybe the harness was the problem?" asked one reader.
A leaderboard that mostly measures refusal policy
The number that travels — "Claude couldn't hack the app" — is the least true thing the experiment shows. What it actually measures is a tangle of capability, an attacker harness that punishes models for not working solo, and a safety regime that benefits whoever got the research whitelist. A footnote that "the Chinese models were way more comfortable attacking the DB" is, in that light, less a capability finding than a guardrail one — and for offensive security work, the model that says yes wins by default.
Get breakdowns like this every day
The top Hacker News stories, unpacked — by email, Telegram, or RSS.
Brockovich: a Meta data center outdraws New Orleans — and locals heard nothing
Erin Brockovich's report stacks up the secrecy around AI data centers: NDAs on local officials, permits locked before announcements, a Louisiana facility that will use more power than all of New Orleans. The thread agrees on the secrecy and then splits hard on whether it's sinister or just how American land development always works.