A blogger's proof that AI can't understand context failed the moment HN tested it
Rachel, who blogs at rachelbythebay.com, says her server logs are full of hits to posts that were never published, proof, she argues, that readers trust AI summaries of her writing without checking them. Her test for whether a model understands anything, rather than pattern-matches it: why is driving through San Francisco's Chinatown early in the year a bad idea? Commenters ran that exact question through ChatGPT, and it named Lunar New Year immediately.
Her blog's URLs are date-stamped, and the phantom hits match that pattern too closely for coincidence: a model invents a post, hands a reader the link, and the link goes nowhere. She frames it as Michael Crichton's "Gell-Mann Amnesia effect": catching a source's confident errors on a topic you know, then trusting it on everything else.
A well-known HN security engineer dismissed her argument as relitigating a two-year-old debate. A second blogger said the same pattern hits her own site, her name attached to things she never wrote.
Why rachelbythebay's server logs are full of posts that don't exist
Rachel, who writes under the handle rachelbythebay and has blogged about backend engineering for over a decade, opened her latest post with a pattern she'd noticed in her own web logs: hits to URLs matching her site's date-based naming scheme, for posts she never wrote. Her read: a language model, asked to summarize or cite her work, invented a plausible-sounding title and URL, and a reader followed the fabricated link straight into a 404.
She ties the phenomenon to an old idea repurposed for a new machine. Long before LLMs, novelist Michael Crichton described what's now called the "Gell-Mann Amnesia effect": you read an article about a subject you know well, spot every factual error, then turn the page and trust the next article just as much. Rachel's version swaps in the chatbot. You ask it about something you actually understand, catch it inventing facts, then ask it about something you don't and believe the answer anyway.
Her test for "real" understanding: Lunar New Year traffic in Chinatown
To make the case that current models are pattern-matching rather than understanding, she offers a single test question, one she says she's used for years on people new to the Bay Area:
"It's early in the year. You want to drive straight through the middle of Chinatown in SF. Why might that be a bad idea?"
The answer, in her telling, requires grounded world-knowledge that no amount of statistical text-prediction can fake: Lunar New Year celebrations close streets and jam traffic through the neighborhood every year. From there she goes further, arguing that anything with genuine understanding would also have enough of a sense of self to resist being controlled:
"Thus, anyone who wants to corral that kind of entity and make it do their bidding? Yeah, they want slaves."
HN pasted her test question straight into ChatGPT
The thread's first move was to actually run the experiment instead of arguing about it.
"The premise seems to be that models aren't smart enough to understand this, and if they were, they'd be sentient and want autonomy. For an article that's about making things up, and being too trusting, this seems bad." — an engineer who ran the exact question through three separate ChatGPT sessions
A second commenter, who works at the AI-safety research group Palisade Research, got the same result without needing multiple attempts:
"Yes obviously even the dumbest current models correctly answer, given this prompt verbatim, that it's because of lunar new year." — Ben Weinstein-Raun, of Palisade Research
The pushback wasn't limited to the test case. tptacek, a security engineer well known on the site, dismissed the post's underlying argument outright: "There's really no content in this post other than the claim that LLMs are stochastic parrots. That was a live debate two years ago. It's a very strange thing to write in 2026."
Not everyone piled on: the phantom-URL problem checks out, just not the way she framed it
Other commenters split from the "AI is dumb" framing but not from Rachel's core observation. One reader flipped the meaning of the 404s themselves:
"The 404s mean that somebody or something checked if the post at the URL existed, and got a clear answer. Seems like it's good that they checked, at least." — skybrian, a retired software engineer on HN since 2009
And a second blog owner said the exact same thing happens on her own site, with a sharper edge:
"My website also gets LLM visitors to URLs that never existed, and in many cases to topics that I have never covered. This means that they use my name to give authenticity to things that I have never said." — the founder of relocation-advice site All About Berlin
That's the part of the thread that outlasts the Chinatown debate: whatever you think of Rachel's theory of machine consciousness, real people are watching real traffic hit invented URLs attributed to their names, for things they never wrote.
Get breakdowns like this every day
The top Hacker News stories, unpacked — by email, Telegram, or RSS.
A hacker wiped Romania's land registry with the password 'P@ssw0rd'
ANCPI, the agency that records who owns land in Romania, spent a week offline after a hacker who called itself ByteToBreach mapped its network, demanded a ransom, and wiped both the production database and its backups when the money didn't come. Only a backup kept outside the compromised network kept the country's property records from disappearing for good.