Anthropic's own trial: AI users scored 50% on the recall quiz, non-users 67%

A study from the AI lab, pointing the other way

Nature's feature on whether AI is eroding human skill (archive copy) leads with an experiment that's awkward for the industry, because the industry ran it. Researchers at Anthropic designed a randomized controlled trial: 52 software engineers were given a basic coding task, all with web access and instructions. Half were also prompted to use an AI assistant.

Afterward everyone took a quiz on what they'd learned. The numbers, quoted verbatim in the thread:

The piece sets this beside older offloading research — GPS dulling sense of direction, automation thinning out a skill you stop practicing — and asks whether code is different in kind.

"They've lost literally all of their skills"

The thread's most cited comment came from inside a big tech org, and it isn't about a quiz:

"The two senior engineers in my org (in a FAANG) who vibe-code the most have lost literally all of their skills. Their code has become terrible and their judgment even worse." — an engineer who stopped using LLMs for his own coding

A product manager who read the full study pushed back on the test design, then conceded the dangerous part:

"Most engineers aren't going to learn anything from a basic coding task anyway… To be clear, there are of course cases where skill atrophy is bad… If you delegate so much that you lose the ability to even judge good work, now you can't even delegate effectively any more." — a PM who lost his own SQL skills and kept his spec-writing ones

That caveat is where the thread turned. One reader compressed the whole finding into a sentence:

"This is the most important point: it seems like AI tool use undermines one's ability to use the AI tool." — a reader replying to the FAANG report

The calculator analogy, and why it breaks

The optimists reached for the usual parallel: calculators, compilers, high-level languages all let a skill atrophy and the world got richer. The chief scientist of a small software firm reported a split result from his own year on agents:

"My ability to go from a blank slate to software was indeed in the toilet, but my ability to reason over and edit code seems to be surviving fine." — the CEO of Sibylline Software

Others said the calculator comparison smuggles in an assumption that doesn't hold:

"If it is a calculator or compiler it is okay… LLM output is unreliable, so we still need to judge it. If I want to be able to judge code, I must have worked with it." — a commenter on why reliability changes the bargain

And one reframed the scope: a calculator offloads arithmetic, a bounded domain. An LLM offloads something unbounded.

"The problem is that for LLMs, X is 'basically all knowledge and communication skills.' Can we really tolerate a society where 'basically all knowledge and communication skills' are atrophying?" — a commenter on why this isn't long division

The risk isn't forgetting — it's not being able to check

The individual losing a skill is the small version. A long-tenured commenter named the bigger one: if AI stalls at roughly today's level while everyone dulls the skills that drive innovation, progress stalls with it, because the machine doesn't innovate. The recurring worry across the thread isn't that the tool will vanish and leave us stranded. It's that the tool is unreliable in exactly the cases where you most need the judgment it quietly takes away — and the lab that built the assistant is the one publishing the numbers.