A 2015 image classifier beat GPT-6 Astra and Claude Fable 5.1 at spotting skin cancer
On 200 biopsy-proven lesions, a small Inception v3 network named the right diagnosis 125 times. The two frontier chatbots managed 84 and 85.
The test set was built to be hard: 100 lesions that should stay under care and 100 safe to discharge, every one confirmed by biopsy. On the 30 melanomas the margin is thin: the old network got 13 right, the chatbots 8 and 11.
The thread's sharpest question was whether the network was simply tuned to this benchmark.
An old network against two frontier chatbots
The Lynch Lab write-up pitted GPT-6 Astra and Claude Fable 5.1 against Inception v3, a 2015 architecture. The author fine-tuned the network on 68,011 images from the ISIC Archive using one H100 GPU, then tested all three on 200 lesions from the public MILK10k set. Every lesion was biopsy-proven, and every image of the same lesion or patient was removed from training.
Each chatbot saw a clinical photo plus a dermoscopic image and picked one of six diagnoses. Each ran once at default settings.
| Model | Correct first choice (of 200) | Correct in top two |
|---|---|---|
| GPT-6 Astra | 84 | 134 |
| Claude Fable 5.1 | 85 | 138 |
| Inception v3 | 125 | 173 |
The gap is widest on seborrhoeic keratosis: 26 of 50 for the network against 13 and 6 for the chatbots. The author's own caveats are blunt. Thirty melanomas is a small sample, MILK10k has been public since 2025 so the chatbots may have trained on it, and Inception v3 is a baseline, not the state of the art.
The charge of "benchmaxxing"
The loudest objection was that a model fine-tuned on a train/test split of one dataset will always win on that dataset.
"A benchmaxxed model will always beat an unbenchmaxxed model, but it will be brittle… it doesn't generalize well to 'wild' images, because they are out-of-domain." — a commenter who quoted the setup paragraph and challenged it
A reply drew the line at deception.
"Benchmaxxing is when someone presents a model as being generally smart, but actually quietly overtrained it on some benchmarks… If you train a small model on a specific task that's not 'benchmaxxing' because there's no deception involved." — a commenter defending task-specific training
Why anyone reaches for a chatbot at all
Another camp said the result is a reason to stop using general models as classifiers. The pushback was practical. Training a good classifier needs labelled data, and an LLM needs a prompt.
"Training a model requires labels training data. Getting up and running with an LLM just requires a prompt. I agree that a good model will outperform an LLM, but the LLM is so much faster/easier to build." — andrewmutz, whose profile links a keybase proof
A dermatologist read it from the clinic side. A commenter who describes themselves as an academic Mohs surgeon wrote that they expected the chatbots to do better, calling Inception v3 "an old model and nowhere near state of the art for skin cancer diagnosis."
The discharge decision is where the trade-off shows. At its preset threshold the network sent home 76 of 100 safe lesions against 32 and 22 for the chatbots, but also kept only 77 of 100 unsafe ones against 97 and 96. Choosing the threshold is the product decision, not the architecture.
Get breakdowns like this every day
The top Hacker News stories, unpacked — by email, Telegram, or RSS.
OpenAI's small GPT-5.6 model fell 80% in price in July. By September, the flagships followed
One blog post lays out the price moves from June to September: GPT-5.6 Luna went from $1 to $0.20 per million input tokens, and Claude Opus 5.5 launched 40 to 60 percent below its predecessor.