49 mathematicians wrote 100 research-level problems to stump AI. Only 2 survived — but read the fine print

The experiment

The paper Benchmarks in Leipzig (arXiv 2606.05818) describes 49 mathematicians compiling 100 research-level questions with known answers (April 1–May 15, 2026, much of it during a three-day workshop at the Max Planck Institute in Leipzig), then testing LLMs in three stages. The number of unsolved problems fell from 41 (Stage 1, single attempt by five frontier models) to 16 (Stage 2) to just 2 (Stage 3, two heavy-reasoning models). The authors — including Bernd Sturmfels — conclude that LLM mathematical reasoning "is becoming impressive" and that public-research-based exercise benchmarks have "reached [their] limits."

The lead author sets the difficulty bar — and the real claim

Conveniently, the paper's lead author was in the thread:

"the problems are much harder than any exam question in any exam … a PhD student studying exactly this area of mathematics would need days to weeks to understand and solve the question … these are questions about existing research." — the paper's lead author

But another reader pinned down what the result actually measures — and what it doesn't:

"this is specifically about problems with known answers that can be inferred from existing literature (e.g., training data) … It's closer to 'can a mathematician (blindly) write exercises you can't cheat on using an LLM'." — a commenter reframing it as a benchmark-difficulty ceiling

The reliability gap the headline buries

The "2 unsolved" figure means "not solved by any model in any configuration across all stages and attempts." Per-run accuracy is a different planet:

"GPT 5.5: 1389/2000 questions answered, of which 1043 were correct (75%) — Opus: 1306/2000 questions answered, of which 294 were correct (22%) … while you can claim that Opus solved 40% of the problems it still had a failure rate of 78%." — a reader digging into Table 3

"I take the '2 unsolved' claim to mean 'not solved by any model in any configuration in any stage with any number of attempts', the 'benchmark results' are much lower." — a commenter reconciling the headline with the per-run numbers

So what did 49 mathematicians actually prove?

Not that AI does research math. They proved that writing a math benchmark you can't beat by throwing a frontier model at it has gotten genuinely hard — and that a model "solving" 40% of problems at least once can still be wrong 78% of the time. Both things are true at once. The headline keeps the first half and drops the second, which is exactly the move the thread refused to let slide.