Neural networks are clean

Goodfire published a paper arguing that neural networks are not the inscrutable black boxes everyone pretends they are. They claim that concepts inside transformers map to clean, sparse directions in the model’s activation space. Not vague statistical smears. Actual interpretable features. The usual story is that neural nets are billion-parameter Rube Goldberg machines where every neuron does everything and nothing is interpretable. You train a model to classify images and end up with a weight soup that somehow works but cannot be explained. Goodfire’s claim is different. They say that if you look at the right layer of a transformer, concepts like “medical jargon” or “French grammar” light up as clean vectors. You can measure them, you can edit them, you can prove they exist. The model is not mixing fifty thousand weak signals into a distributed representation. It is storing features as directions. This matters because if features are clean, you can audit them. You can build a model that refuses to generate certain ideas by clamping the vector representing that concept. You can debug why a classifier failed by checking which features fired. Interpretability stops being an academic curiosity and becomes engineering. The counterargument is that this only works on toy models or curated examples. Real production transformers with hundreds of billions of parameters might still be statistical chaos. Goodfire’s examples are from smaller models. But if the claim holds at scale, we stop treating neural networks like magic and start treating them like databases with weird indexes. That changes what you can build. It changes how you explain a decision to a regulator. It changes whether you trust the model to run unsupervised. I want to see this tested on a GPT-4 scale model. If the features are still clean at that size, the entire interpretability roadmap shifts.

---

Source: Neural networks are clean