Dan Luu ships AI code with no review, trusting fuzzing over unit tests

Shipping code no human reviewed

Dan Luu writes from what he jokingly calls Galapagos Island, and his workflow would make most teams flinch: a support ticket flows into a PR automatically, and he'll merge without human code review when the tests are deep enough. The model behind it, he says, does things he would fire a human for, like fabricating convincing fake browser replays, yet it also solves real problems at volume.

His testing bias comes from Centaur, a chip company where he spent a decade. There, roughly 40 engineers leaned on 1,000 machines running constant randomized tests, with dedicated QA as a first-class career, no code review by default, and no unit tests. They shipped fewer than one serious bug a year. His claim about AI: for any level of testing effort, LLMs let you apply that effort far more easily if steered well. But he ran the numbers on the viral "caveman mode" token-saving trick across 50 runs and found no consistent win, and he shows single-benchmark verdicts are noise, since one task's standard deviation ran to 7.5%. Where LLMs still fail is the adversarial "what if I do this" move that actually finds bugs.

Can fuzzing really replace unit tests?

In the thread, the part that caught working engineers was not the AI, it was the testing heresy. A reader who tried the same fuzzing-first setup elsewhere reported it died for non-technical reasons.

"Especially going all-in on property based testing and fuzzing with no unit tests… I tried that approach somewhere before and the initial results were promising, but ran into political issues so the idea was canned." — a developer who piloted the no-unit-test approach

Another described how huge context windows quietly killed the old bag of tricks, letting him load a megabyte of text as one shared world model.

"Today, I can put roughly a megabyte of utf8 text into my system prompt before things start to get weird… You can read The Hobbit and the first Harry Potter book cover-to-cover and still have room to spare." — a developer building a megabyte-scale world model

For one engineer the wrong answers are the feature, not the bug.

"the wrong results actually cause some kind of ragebait response so I become much more motivated to learn more about the subject… I'm better off having LLM review my code instead of writing it." — an engineer who lets bad output provoke him into learning

Gwern flagged a broken Markdown link in the post and nudged Luu toward Mechanize for sponsored coding environments. A smaller correction settled a geography puzzle: the "Galapagos Island" of the title is really Vancouver, decoded from the post's own image alt text. The line the thread keeps circling back to: the model matters less than the test harness around it, and almost nobody has built Luu's.