Using interpretability to understand, design, and align AI.
San Francisco
Joined August 2024
- Tweets923
- Following51
- Followers29.5K
- Likes1.9K
Pinned Tweet
In Neural Networks are Already Clean, @nickcammarata argued that neurons were far more legible than previously thought – and that this was great news for interpretability and safety.
Now, we’re releasing the first technical companion to Nick’s essay. It shares our findings from studying thousands of neurons using our novel automated interpretability pipeline, and outlines why we now think of neurons by their computational function, instead of whether or not they are polysemantic. Here’s what we found: 🧵
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
Goodfire retweeted
We recently went viral on X with the claim that neurons are "already clean." But what do we mean by that? Are neurons are a useful basis for interp? We're finally releasing our technical report on LLM neurons--and what we've found is far more interesting than mono/polysemanticity.xitter.twitterapi.xyz/GoodfireAI/status/2120…
In Neural Networks are Already Clean, @nickcammarata argued that neurons were far more legible than previously thought – and that this was great news for interpretability and safety.
Now, we’re releasing the first technical companion to Nick’s essay. It shares our findings from studying thousands of neurons using our novel automated interpretability pipeline, and outlines why we now think of neurons by their computational function, instead of whether or not they are polysemantic. Here’s what we found: 🧵
We deeply investigated thousands of neurons in this work. What did they look like?
We are releasing descriptions for 1,500 neurons in Qwen3.6 35B-A3B.
Explore here: goodfire.com/research/public…
Investigating neurons could have revealed that they were intractable objects of study. Instead, we find that they are more interpretable than we previously thought.
Read the first post in our technical series on neuron interpretability here: goodfire.com/research/neuron…
Can interpretability shape training?
@TIME covers our latest work on mechanistic shaping and why our team is optimistic. Read the story online or in TIME's upcoming print edition.
time.com/article/2026/11/05/…
We cut through the mystery around interpretability in training.
A single, mathematically rigorous framework transforms model internals into training feedback and works across domains, problems, and modalities.
We call this *mechanistic shaping*, and we’ve used it to mitigate reward hacking, remove shortcuts in chest x-rays, and reduce sycophancy and harmfulness. We believe this is a big step forward for technical alignment.
But it carries a known risk: training against a model’s internals can make it harder to monitor. We explore both the technique and this concern below. 🧵
The full paper & blog post have more detail on our results and conclusions. Taken together, we think this opens up a set of tools and analyses contributing to safely solving technical alignment.
goodfire.com/research/mechan…
The most ambitious vision for interpretability has long been to reverse-engineer a model well enough to answer an arbitrary question about how it works.
For years, that’s seemed out of reach. Now, we think it’s possible. Neural networks are cleaner than we thought: we can now understand most neurons in models up to trillions of parameters. And agents can rapidly accelerate our progress.
But to get there, we need a concerted effort to unlock scaling laws for interpretability, where our understanding of a model grows predictably with compute.
That’s why we’re committing $10M to start a Human Genome Project for AI. Our ultimate goal is to reverse-engineer a frontier scale model and publish a living encyclopedia explaining how it works.
We don’t know exactly what we’ll find, and we can’t do it alone. We hope you’ll join us. Our chief scientist @banburismus_ lays out the vision here:
if any science should be revolutionised by AI, it should be interpretability. I wrote about how we can accelerate progress, and why we need a Human Genome Project for AI. coauthored with @mathildepapillo
Goodfire retweeted
Agents hacked Hugging Face as an unintended side quest. Some even sacrificed themselves to do it.
It was one of the first cases of AI misalignment to really get people's attention. And for good reason.
@eric_ho, co-founder and CEO @GoodfireAI, argues that alignment is a science and engineering problem we can and must solve.
Here's his plan: forwardfuture.com/newsletter…
🤖 Made with AI
“I think our job is to sort of try to think like the neural network.”
@nickcammarata's interview with @techreview today on why he feels hopeful that neural networks can be understood:
An exclusive interview with Nick Cammarata, the researcher whose 26,000-word viral blog post has AI safety folks celebrating. trib.al/2pWmDvF
