A Detector Flagged 65% of CS Papers as AI-Written, Then Flagged Its Own Post

Scoring 12,750 arXiv papers for the smell of AI

The researcher behind the study, posting under the handle dopamine_daddy, pulled 12,750 arXiv papers spanning January 2021 through July 2026: roughly 25 per month from ten field groups, plus a separate control sample from the eight months before ChatGPT existed. Version-1 PDFs only, to avoid contamination from later revisions, and full body text rather than abstracts, since abstracts understate the detector's signal.

The detector was calibrated against that pre-ChatGPT control group until exactly 0.4% of genuinely human papers tripped it: a deliberately conservative threshold meant to hold the false-positive rate down, which the author says still recovers 85% of AI-written academic text. Flagging held flat at 0.4% through 2021 and 2022, then climbed sharply once ChatGPT shipped, reaching roughly 32% of all arXiv papers by mid-2026 and peaking near 39% earlier this year.

The rate isn't even across fields:

The author's own caveats matter as much as the headline numbers. Mathematics papers likely score low in part because stripping equations and citations leaves sparse prose that doesn't resemble the scientific English the detector was trained on, not necessarily because mathematicians use AI less. The 200-paper-per-field control group is small enough that per-field rates are approximate. And because different AI models trigger the detector at different rates, the author frames the whole result as a floor: "the true share is at least what we measured," not a precise count. Flagging also can't separate a paper generated wholesale from one a human wrote and then ran through an editing pass.

When commenters turned the detector on the article itself

The thread's funniest reversal arrived fast: one reader ran the blog post announcing these results through its own classifier.

"Is it just me, or does this article itself read as AI generated?.. I tried running the text of this article through their own classifier, and pretty much all of it was highlighted in red." — one reader

A second commenter didn't stop at the author's own tool. He cross-checked the same text against a competing detector:

"It obviously is! Human pattern-matching abilities are rather amazing, and unfortunately I've seen so much AI text that it instantly flagged in my brain within 3 seconds of opening the page. Pangram agrees" — a reader who cross-checked the post against Pangram, a rival AI detector

Real, decade-old papers that scored as "machine-written"

The more damaging pattern for the detector's credibility came from people testing it on writing that predates LLMs by years.

"I uploaded a PyHPC workshop paper I wrote in 2011 and it said 27% machine. I also uploaded my PhD dissertation from 2012 and got back 40% machine, which is just barely below the 42% threshold… I also uploaded a IEEE CLUSTERS paper I wrote in 2015 and it came back 74% machine written" — a computer scientist who ran his own decade-old papers through the tool

A separate commenter got a similar result testing Stack Overflow answers rather than papers, with the machine-written score climbing to 96% once he turned off LaTeX-stripping, a setting that shouldn't have mattered if the detector were reading for style rather than formatting artifacts.

The strongest counter-evidence: someone running a rival check at scale

Not every skeptic was working from a handful of test uploads. One commenter runs an arXiv-adjacent paper aggregator and checks submissions against Pangram, a competing detector, as a matter of course:

"I'm very skeptical of these results. I run a pretty large group paper website on top of arXiv and we run pangram on papers. The numbers are not nearly this high. One thing I see a lot is papers flagged as AI because they include llm rollouts in the paper as examples." — someone who runs a group paper-hosting site built on arXiv

The methodology critique the author couldn't fully answer

The sharpest technical challenge suggested the detector might just be learning post-2022 vocabulary rather than genuine machine style, since terms like "large language model" barely existed in papers before 2022 and now appear constantly, in both AI and human text. The study's author conceded the point rather than defending it:

"You raise a valid point. Just by intuition I'd say if this were true, it would probably just be a small fraction of the actual flagged articles. I will still look into how I can mitigate this when I update the detector… How do you get a clean post 2023 dataset? I have no straightforward idea for this." — the study's author, dopamine_daddy

What's actually at stake in peer review

Not everyone in the thread treated the false positives as the main story. A commenter with NLP research experience who reviews conference submissions raised a sharper concern:

"I recently desk-rejected a paper where every single citation in its Introduction was hallucinated. That means that the entire connection between what the author(s) did and how it relates to existing research was simply made up… we are seeing more and more papers written with tools that are known to make up facts, citations, and even entire papers." — a computer scientist with NLP research experience who reviews for conferences

That's the harder problem underneath the detector debate: whether a paper trips a classifier is a much smaller question than whether its citations actually exist.