An AI tutor posted a 1.3 SD gain at Dartmouth — then the thread found the 16 students it rests on

What the paper actually measured

A Dartmouth deployment paper reports that an AI-assisted study platform called Phosphor, dropped into an intro statistics course, produced a 0.71–1.30 SD jump in exam performance for students who fully engaged — with 90.2% of the class voluntarily using an ungraded tool, against an instructor-estimated ~10–15% who normally do the assigned reading. Constructed-response questions are graded at scale by Claude Sonnet against question-specific rubrics. To their credit, the authors flag the core hazard themselves: it's non-randomized, and self-selection — motivated students do more quizzes — is the obvious confound.

One of the authors even turned up in the thread to engage. And that's where the number got dismantled, politely, by people who read past the abstract.

The three deflations

First, what the tool even is. The word "tutor" is doing heavy lifting:

"The title is misleading. This isn't an AI tutor so much as a practice quiz platform with an AI autograder … The conclusion is essentially that people who do practice quizzes will do better on exams." — a reader who checked which parts of the platform were used

Second, the base the headline stands on. Someone reconstructed the engagement tiers:

"the headline result of 0.7*sigma improvement is … based on lessons/reviews they engaged with and their mid-term score … something like ~16 students (11% of the group) actually reached that level of engagement … 90% is for "engaging with the platform … at least once"." — a commenter who rebuilt the tiers from the paper's tables

Third, a working statistician questioned the whole test design — not to attack AI, but to ask whether removing it even models reality:

"students will have access to AI during the test in real life, so it's ironically less realistic to remove it … I'm more curious how students perform on the test with vs. without AI." — a statistician-economist, an early data scientist at Zapier

The number that will travel vs. the one that's real

"1.3 SD" is the number that ends up in a slide deck. The finding that survives scrutiny is smaller and older than any AI: students who voluntarily grind practice questions outperform students who don't, and an autograder that gives instant feedback lowers the cost of grinding. Useful — but if you cite this as evidence an AI tutor triples learning, you're quoting sixteen self-selected people.