Goodfire argues neural networks are far cleaner inside than researchers feared

Re-reading the canonical examples

In an informal research essay, Nick Cammarata of Goodfire argues that neural networks are already fairly interpretable. He goes back to the best-known evidence: the V-shaped "cat and car" neuron in InceptionV1 responds to pointed versus round shapes, and the car-and-dog circuit from Zoom In reflects a shared pose across objects, not storage in superposition. Feature visualization, he says, was partly reward-hacked toward dogs in deeper layers.

His main proposal is "recursive generalization": neurons grow outward from a core concept into related offshoots during training, such as an "iy" neuron extending to "diy" and then "do it yourself". That, he says, explains most apparent polysemanticity.

The numbers, with their caveats

• InceptionV1's last layer went from about 5% interpretable with 2010s tools to about 70% with better tools and retraining, and a modern ConvNeXt of the same size reached about 99%.
• In Qwen3-32B, about 90% of neurons have one main job, about 7% two related jobs and 2-4% remain unexplained.
• Cutting a layer from 480 to 10 neurons made neurons coarser, not packed with unrelated features, although the sparse-autoencoder analysis was more mixed.

The author calls the essay a working hypothesis. The percentages are preliminary, based on small samples, and depend on thresholds and definitions, and many details wait for a companion paper. If it holds, superposition becomes a last-resort explanation, and plain inspection of neurons could beat clever decomposition methods like sparse autoencoders.

An interpretability researcher accepts the method but not the grand claim

The pushback in the thread came from the field's own voices. A commenter relaying Neel Nanda's reply on X quoted it:

"I do not agree with some people's grander inferences like that this is substantial progress towards ambitious reverse engineering, solving interpretability etc." — commenter relaying Neel Nanda's reply on X

Another reader went after the record rather than the future. A commenter from a field with stricter correction norms noted that the "dog and car" neuron claim came from the Zoom In paper, which also drove the superposition story:

"This type of error may have mislead researchers for years; it was actually harmful." — commenter from a field with stricter correction norms

Both point to the same test. The survivable version of the claim is the modest one: reading neurons directly may explain why a model did something odd, without proving that the whole network can be reverse-engineered. The Qwen3-32B percentages are where a companion paper has to deliver.