DeepSeek V4 Pro Scores 38–33 Over GPT-5.5 Pro in a 4-Task Grok-Judged Test
A site called RuntimeWire ran four text tasks, had Grok-4-1-fast score the outputs, and declared DeepSeek V4 Pro the winner 38 to 33. The thread disagreed — not about who won, but about whether any of this means anything.
The article's headline claim rests on a single run of four tasks, judged by one AI model (grok-4-1-fast-non-reasoning), with no temperature controls, no reproducibility data, and no definition of what "precision" means.
What the thread agreed on: DeepSeek V4 Pro is dramatically cheaper. One developer ran a vulnerability-scanning benchmark where GPT-5.5 Pro averaged $22 per case and burned through a $100 budget mid-run; DeepSeek cost roughly a dime per case.
The price gap is the real story. A 200x cost difference survives sloppy methodology.
A Benchmark Article That HN Disassembled on Contact
The RuntimeWire article landed on HN with 164 points and 48 comments — a score/comment ratio that often signals a community divided between upvoting the topic and disputing the source. In the thread, the source got dissected almost immediately.
What the Article Claims
The RuntimeWire piece pitted DeepSeek V4 Pro against GPT-5.5 Pro on four "fresh text tasks" generated on the fly. The judge was grok-4-1-fast-non-reasoning. The final score:
| Model | Score |
|---|---|
| DeepSeek V4 Pro | 38.0 |
| GPT-5.5 Pro | 33.0 |
The article's conclusion: DeepSeek wins by "being more exact where it matters: following instructions, matching schemas, and solving edge cases cleanly." Each test was run once. No confidence intervals. No definition of the scoring rubric beyond the judge model's own output. The article's language — "the matchup feels earned" — was flagged by multiple readers as an AI-written tell.
The Thread Runs a Better Benchmark
The most methodologically thorough response came from a YC W07 founder who had actually run DeepSeek V4 Pro and GPT-5.5 Pro on his own vulnerability-scanning benchmark:
"GPT 5.5 Pro found two out of four cases that it got to before blowing its budget. Maybe it would have been the best of the bunch with infinite budget, but Opus 4.8, DeepSeek V4 Pro, and MiMo 2.5 Pro found four of nine of the bugs. Opus was an order of magnitude cheaper than GPT 5.5 Pro, DeepSeek and MiMo were two orders of magnitude cheaper at roughly a dime per case… I can't come up with a use case where I can rationally spend ~31 times what Opus costs to use GPT 5.5 Pro." — SwellJoe, YC W07, creator of Virtualmin
GPT-5.5 Pro's per-case cost in that benchmark: $22. DeepSeek's: approximately $0.10.
The sharpest methodological critique came from a developer who named the specific failures:
"An AI generated article about single AI run test which in theory had many components and the AI judge declared deepseek 'won'? How many runs were there on each test to account for some temperature variance? Only one. Did deepseek write better code? Did GPT's code have bugs when doing the regex? The AI 'news' article doesn't actually say that. It says that grok thought that GPT's approach could have bugs so it declared DeepSeek the winner." — a developer active since 2017
A commenter who stripped the article to its actual data put it most concisely:
"…according to grok-4-1-fast-non-reasoning who was the judge, on 4 tasks in total, score was 38 to 33 so obviously huge conclusions can be made." — a commenter active since 2025
One ML engineer whose experience matched the article's conclusion noted an interesting specific failure mode of GPT-5.5 Pro in structured output:
"gpt keeps adding fields and changing types on structured output when you need it to just follow the spec" — an ML engineer focused on on-device Apple Silicon inference
The Price Gap Is the Unfakeable Number
The 38–33 score is not reproducible and probably not meaningful. The cost delta is. A model that costs 200x more needs to be correspondingly better on the actual task — and on SwellJoe's real-world security benchmark, GPT-5.5 Pro didn't finish the test suite before the budget ran out. For developers running model-judged pipelines at scale, where a single human never reviews intermediate outputs, paying a dime per case versus $22 per case is a structural constraint, not a preference. That asymmetry will persist regardless of what any four-task LLM-judged article claims about "precision."
Get breakdowns like this every day
The top Hacker News stories, unpacked — by email, Telegram, or RSS.
The snowboarding you never tried keeps getting heavier every year
Nik Göke can't snowboard — a knee condition rules it out — and his 2023 essay argues that this is fine, maybe even good. The thread disagreed in specific, sometimes beautiful ways.