An open-weight model beats Claude on vuln-finding at 17 cents a bug

A bare-prompt open model out-scores a coding agent on IDOR detection

Semgrep's benchmark put ten model-and-harness combinations against one fixed task: finding IDOR bugs (Insecure Direct Object Reference — when an app exposes an internal ID without checking whether the caller is allowed to see it) across the same set of real open-source applications used in their prior research. The standout result is that GLM 5.2, Zhipu AI's open-weight model, run through Pydantic AI with nothing but a prompt, beat the Claude Code SDK configuration — at roughly $0.17 per vulnerability found.

The full leaderboard, with F1 as the scoring metric:

Semgrep's own scaffolding — the multimodal pipeline that hands the model endpoint discovery — tops the chart at 53–61%. Their reading: the harness matters more than the model, and the widest gap is between configs that get endpoint discovery and those that do not.

The headline number doesn't match the author's own table

The sharpest catch in the thread is a bookkeeping one. The prose celebrates GLM beating Claude Code at 32%, but row four of the table says 37%.

"Twice in the text quotes Claude Code's F1 score as 32%, but the table shows the score is 37%… Oddly, this is a strong indication of the text being hand-written rather than LLM-assisted; it's very likely that a human made a mistake in creating the table." — a commenter who traced the 32% through three references in the prose

Others refused to take the comparison at face value at all, because the two contenders weren't run the same way. GLM got a clean Pydantic harness; Claude got the Claude Code SDK; Semgrep's winner got the full multimodal pipeline.

"They should also at least run Opus through the same Pydantic harness they used for GLM. As is, it's apples vs pears. Where's the cost per vulnerability for all the other models than GLM? Also, without code this isn't very trustworthy." — a commenter asking for the missing per-model cost figures

A terminology objection landed too: "Claude Code is an agent harness, not an LLM. Claude is a brand, not an LLM," wrote an HN user with 11k karma — so "beats Claude" is comparing a harnessed product to a raw model.

The voice that runs a competing benchmark: good, not best

The most grounded counterweight came from someone who maintains his own Mythos-style vuln-hunting benchmark.

"I added GLM 5.2 to my security bug hunting benchmark when it came out, and found it to be a good performer, but not the best open model… The best open models in the initial benchmark were DeepSeek V4 Pro or MiMo 2.5 Pro." — SwellJoe, who works on Virtualmin and runs the will-it-mythos benchmark

He added a finding that cuts against Semgrep's own product pitch: giving models the open-source Semgrep as a tool made some perform worse and none better, his theory being that the tool is thin in training data, so the model burns focus learning to drive it. Daily-driver praise was real but mixed — one heavy LLM user called GLM "a good workhorse model" he built a Rust agent with for $20 over a weekend; a Rust consultant liked it but couldn't find a cost-effective way to actually run it, saying z.ai's plan is overpriced and per-token pricing beats no Claude or Codex subscription. The number that should travel from this benchmark isn't 39% — it's that a single missing harness column turned a like-for-like claim into a marketing post several readers reverse-engineered line by line.