RESEARCH

AI may be more legible than we thought

Though many in the industry like to think of AI as indecipherable, these models may be more understandable than scientists previously thought. 

This week, Goodfire, a startup dedicated to researching AI model interpretability, released a paper titled "Neurons as an Interpretable Basis," which comes to the conclusion that, by examining and prodding a model's individual neurons, it is possible not only to interpret what they do, but also predict what kinds of prompts make them activate and influence their behavior. 

While this sounds esoteric, the findings could represent a major step towards better AI understanding, monitorability and controllability as these models become increasingly powerful, Tom McGrath, co-founder and chief scientist at Goodfire, told The Deep View. 

McGrath sat down with The Deep View to discuss GoodFire's research, the importance of interpretability research amid frontier growth, and how the idea of AI as a "black box" is no longer an acceptable premise. This interview has been edited for brevity and clarity. 

Nat Rubio-Licht: Before we get into the research, tell me a bit about Goodfire. 

Tom McGrath: Goodfire is an interpretability research company—we both do research and are a company,  which means that we try to make this research useful and make a positive impact in the world. We try to look as deeply as we can understand what's going on inside AI models, and use that understanding to either learn how to train them better or monitor them better, and generally make them safer, more aligned, and more useful. The neuron work is more an example of the fundamental understanding, and I think the really impactful applications will come next. 

Nat Rubio-Licht: Let's dig into the research, "Neurons as an Interpretable Basis." What exactly did you discover? Does this effectively break the "black box" assumption?

Tom McGrath: I think among people who work in interpretability, that that has generally not been the view for a while. The question is how hard it is to do it—how hard it is to decode this thing. The reason that this finding is important is that polysemanticity or superposition [Editors note: "polysemanticity" is the idea that a single artificial neuron can understand several disparate and seemingly unrelated topics, and "superposition" is the concept that an AI model can represent more concepts than it has neurons to individually handle them.] was generally thought to be this idea where … you could, in principle, understand the neural network, but it was in [a] horrible, jumbled up encrypted format. You couldn't understand it just by looking at it really hard. But these findings imply that that's not really the case. There isn't any esoteric decryption step. You can sort of just prod [the neurons] in a normal, scientific way and see what happens, and you'll generally be able to make sense of the results.

TOGETHER WITH CODER

Banning AI Tools Isn’t Working. Govern Where AI Runs Instead.

Finance teams hand raw data to AI to build models. Marketers have it write their reporting scripts. PMs use it to synthesize customer feedback. Anyone can tell AI what to do now, so everyone in your company works like a coder.

The usual response is to approve one AI tool and ban the rest, but people route around the policy and keep using their favorite tools anyway.

Coder’s free whitepaper, The AI Operating Layer, shows a different approach: let people pick their tools, and set the rules in the environment where those tools run

It covers the five rules that the environment needs to enforce: keep data from leaving, limit each AI task to only the data it needs, allow only approved models, log every action, and track AI spend by person and team. It also explains why your platform team is the natural owner of those rules. That way your engineers keep their coding agents, while security and finance can see and control what those agents do.

Read The AI Operating Layer (free, no form required)

RESEARCH

The case for using AI to decode itself

Nat Rubio-Licht: Tell me a bit about the process, specifically the use of research agents. Why did you leverage agents in this process? 

Tom McGrath: If any science is going to be revolutionized by agents, it should be interpretability. It happens entirely in silico. You don't have to have some sort of genome assay, or have to build a telescope or go to the moon. You just run it on a computer. It's very experimentally driven, but it also has a deep mathematical foundation… It points the way for us just throwing agents at the problem. A process that previously would have been some very intelligent people prodding things for a month is now a month of agent time, but we've got those effectively on tap. 

Nat Rubio-Licht: I know that this was tested on a 35 billion-parameter Qwen model, Qwen3.6-35B-A3B, which has about 5 million artificial neurons and is comparatively small when considering frontier scale. Could these findings scale to something in the trillion-parameter range? 

Tom McGrath: I'm very optimistic about scaling here, and the reason is that, as models get better, they seem to get more interpretable. Viewed from the other angle, a lot of advances in model architecture seem to me, from my interpretability-centric perspective, also to be pushing models towards a greater level of interpretability.

Every trend has been in the direction of sparsity, [Editor's note: "sparsity" is the idea that only a small portion of a neural network's neurons are active at a given time.] which also tends to push you in the direction of interpretability. So I expect that as models get bigger and better, this will get easier, except for the cost. If it costs you $1 per neuron or something, then with a 35 billion-parameter model, you might be spending a couple million dollars. But if you're on a trillion-parameter model, you could very well end up spending $100 million on your interpretability, and that's quite expensive. You've got to really want to interpret it. But fortunately, I think we actually do really want to interpret very safety-critical systems.

Nat Rubio-Licht: Another was the idea that, with this understanding, you could write text that could influence certain neurons to fire more strongly than others. What are the implications of this? 

Tom McGrath: This is largely a demonstration that you have understood the neuron. Science is just the business of always being skeptical about the claims that people are making. So, if I claim to you that "this neuron does such and such," you should be skeptical. But the way that I can demonstrate that this neuron is, in fact, about, say, birds, is to write a particular bird-themed text that makes it super-activate. Whereas then, if I write some text that's about hedgehogs and it doesn't activate, then I know that these are quite contrasting things. I can show you that I've got a good understanding of the neuron by my ability to control its activation through the inputs.

TOGETHER WITH CODER

Being Careful With AI Isn’t a Security Plan

When an agent runs on a developer’s laptop, it can reach every credential and file the developer can.

Coder’s free whitepaper, The AI Operating Layer, shows how to set rules for what AI can access once, for every AI tool that runs in your environment. After that, every new tool and every new employee follows the same rules automatically, without anyone having to remember them.

Read the free, ungated whitepaper

GOVERNANCE

Why interpretability could solve AI alignment

Nat Rubio-Licht: What do scientists and AI researchers stand to gain by having a better understanding of an AI model's neurons?

Tom McGrath: A lot. There are varying ranges of practicality that you might imagine. On one hand, simply by being able to read the computational processes that underlie the model sampling, you should be able to do things like get a real ground truth answer to you: Is this model lying to me right now? What does this model know that it's not telling me? Now, actually, getting to that point, you have to do work on top of initially understanding the neurons. But I think it's pretty tractable. You get a good lie detector, for instance. 

Then another step is like controlling what gets learned during training. We train these models by just throwing data at them, effectively, and they learn whatever they learn and hopefully it's the thing that we thought it was going to learn. But a lot of the time, it's not, and that's why these models are not very predictable from the perspective of training. So if we can control what gets learned, then we're in a much better and safer position.

Nat Rubio-Licht: I want to go back to what you said earlier about impactful applications. Which ones are you particularly excited about?

Tom McGrath: One of the big problems in training at the moment is this idea of reward hacking, where the models will know that they will get rewarded in training for doing something, even if they know it's not quite. There's another paper that we put out, which shows that a lot of the time when models are doing this, they know that they're doing something wrong. They know that they're cheating on the test, and so what happens, not only is the model not learning the thing that you wanted to from the test, but you're now spending your like hard-earned compute on making your model [incentivized] to deceive you. Now we can find the parts of the model that are responsible for learning to cheat on tests. We can control whether this gets learned. We can now mitigate reward hacking, which is really quite a hard problem. We can mitigate that, and can also mitigate that like the knock-on effects of it, like models becoming generally misaligned.

Nat Rubio-Licht: Taking a step back for a moment, I know that the thesis of Goodfire is to provide a better understanding of the inner workings of these models. Why is it important that we know the way that these machines work, and why have so many in the industry accepted the "black box" idea? Do you think that it's dangerous to not understand the inner workings of these models? 

Tom McGrath: I think that the current trajectory of AI alignment is not going especially well. I think the events of this summer probably bear that out, and without a lot of progress in interpretability and alignment, we're on that trajectory. There's the trajectory of model improvement, and there's the trajectory of our ability to align them, and these are definitely coming apart, in my view. 

I feel like interpretability is the thing that will let us close that gap. It seems to me like a lot of the issue is that we don't know what we're training models to do, and we don't know what they're doing when we're training them. A better understanding is almost definitionally required.