Mapping the Mind of a Neural Net: Goodfire’s Eric Ho on the Future of Interpretability

0:00So good fire is an AI interpretability research company really trying to answer the question of, you know, what's actually going on inside the mind of a neural net. So kind of the ultimate goal of the ultimate reason why we started everything was like, we just see neural networks kind of going into more and more mission critical contexts and I think it's going to be enormously transformative for society. But in order to do so, and you want to build it safely, powerfully, reliably. And I think it's going to be critical to be able to understand, edit and debug AI models in order to do that. And so that's what we're kind of enabling for the very first time.

0:39It's like unlocking the black box of a neural network such that you can intentionally design it rather than just kind of like grow it from data.

1:05What if we could crack open the black box of AI and see exactly how it thinks? Today we're joined by Eric Ho, the founder of Goodfire, who's building tools to peer inside neural nets and understand their minds. Eric reveals how his team has successfully disentangled the mysterious phenomenon of superposition, where single neurons encode multiple concepts, and can now steer AI behavior with increasingly surgical precision. We explore whether interpretability could help us discover new biological insights, and it out harmful behaviors from large language models, and even understand their own brains better.

1:38Eric boldly predicts that will fully decode neural nets by 2028, transforming AI from black boxes into more intentional design. Enjoy the show. Eric, thank you so much for joining us today. Of course. Yeah. Happy to be here. Thanks for having me. First question. Can we ever trust generative AI if these foundation models are very much black boxes? Can we ever trust them if they're black boxes? So I guess like, you know, maybe we're thinking about like, what would happen if we were to just kind of deploy AI models as black boxes like in perpetuity. So the black box way to do this would be like, and I'm kind of assuming that we're playing this for a few years and we want AI in charge of like really mission critical applications like maybe being in charge of our power grid or making big investment decisions maybe even for a seed investment at Sequoia or like a really large, you know, like, um, million dollar investment decision.

2:37And I think like the black box way to, to make sure that the AI is performing appropriately is like, you take a look at e -vows and you run like a bunch of like evaluations to make sure that it's behaving, behaving appropriately in test sets. And then you'd look at its track record and see if it's like, you know, reliable enough to perform across a wide variety of things. And I think like the question then is like why not take all this additional signal that you get from looking inside a neural network and trying to play forward like how it's going to behave in a much brighter, broader set of situations.

3:14Like why not like look inside and actually like get a bunch more reliability, certainty about how it's thinking, how it's approaching the problem. And I think that you're just leaving a bunch on the table if you're not looking for all the signal that you can get. So the way that I think about this is, I don't know, in your manufacturing a new drug, it's like you can do the black box way of just seeing how humans respond to the drug in a clinical trial. Whereas you could also just look inside and look at biochemically how the drug is processed or like drug interactions at the molecular cellular level.

3:54And yeah, I just feel like there's so much to be learned when you actually like look inside and deeply understand something. How possible do you think it is to look inside and deeply understand a large language model? Do you think it's, you know, on the scale from hopeless, we can't ever understand it. It's just a black box, too many neurons. So we can actually map out the mind of a neural net. And I'm curious where you think the field will be. Well, I'm very biased, but I think it's very, very possible. So I mean, a lot of the people in Meccanturple I come from backgrounds in computational neuroscience or cognitive science.

4:30And those people when you're actually looking inside the brain, you spend so much time trying to understand what a single neuron does or just getting any signal whatsoever. And in the field of Meccanturpe, you have perfect access the neurons, the parameters, the weights, the attention patterns of a neural network. So you're coming in with a huge advantage for at least like you get all the data that you need. So then the real question is like how can we make progress? How can we understand and seek to understand all of it? And I think we just got to try. I think it's deeply necessary and critical for the future.

5:12and we have like a norm established. We can explain some percentage of the network by reconstructing it and extracting, you know, its concepts and the features that it uses in order to generate its response. And once you have at least a baseline under, like rudimentary understanding kind of where we're at right now, you can hill climb on that metric and seek to understand like more and more of the network. Do you think it's gonna be necessary for us to understand neural nets to really harness them long term? Because I think many other technologies we've invented along the way, humans didn't really understand underlying physics or chemistry, but still we were able to make good of medicines or, you know, totally basic propulsion techniques without understanding, you know, all the physics.

5:53Yeah, I think it's going to be critical for the future, just given how transformative I think AI is going to be. I think AI is going to be everywhere running mission and critical parts of our society. And we can get really, really far just by treating AI as a black box. But I don't think we can truly be able to intentionally design AI as the new generation of software without white box techniques. So maybe one example I think about is, you know, in the early 17th century, like the steam engine. and we're able to just increase the size of the boiler and increase the amount of pressure going in and it scaled reasonably well, but steam engines also blew up.

6:41Well, we didn't understand thermodynamics at that time. So we didn't actually know, at what point did the ideal size of the boiler or the ideal way to construct a steam engine? And so after we invented thermodynamics, like they started becoming a lot safer, a lot more reliable and like huge innovations happen afterwards. But already like you know the steam engine kicked off like the industrial revolution. So even just by like treating it as a black box, you get a really long way. Do you think it's any chance that if we understand neural networks in a computer science context, it might actually help us accelerate our understanding of neuroscience for the human brain?

7:20I think so, but that's a big claim, I think. So we were just having an interesting conversation last night. We had a dinner together about, do we think in language, do you think in concepts or something else entirely? I'm a person that doesn't really think in language. I think much more conceptually, maybe in the latent space of models. whereas like our heteroproduct Myra said that she basically is totally faithful to her own chain of thought, like speaks in language and she basically just thinks like sequentially with like a really really strong internal monologue. So like Trinidad's just maybe maybe some of these insights that we get like we'll translate to humans in our own psychology And I think that's the hope.

8:08It's like yeah, the more that we can understand about AI like hopefully the more That we can understand about ourselves There's an interesting analogy by the way that a neuroscience often things that have gone wrong help illuminate and create insights into the human brain. You know people who suffer from specific conditions or people who have suffered particular brain injury types have actually perversely enabled us to better understand the brain exactly. Yeah. One of something similar might happen with neural networks as well. Yeah, I hope so. What's that? Like a popular story about the guy who just like got an iron So it's been a year or so through his brain and it made him like a totally different person.

8:47Yeah. Yeah. Yeah, there's also this, you know, to add to that, there's this concept of of universality where like among totally different neural networks, like similar kind of like circuits or like thought patterns tend to emerge between all these neural networks. So like even in what we found in vision models like very similar kind of circuits to our own like visual cortex And I think like, you know, there's this like idea of universality where maybe like intelligence is just this like thing that you gradient descent to and then like that's how our brains found like intelligence and that's how artificial minds would find intelligence as well.

9:29Like there's some truth to intelligence. My own neural net is probably pretty sparse. It's pretty sparse. I would love to double click into some of the results you've had from McInterp and the brotherfield in your lab. But before we get into it, can you say a word about good fire and what you all are building? Yeah, so good fire is an AI interpretability research company really trying to answer the question of what's actually going on inside the mind of a neural net. So kind of the ultimate goal of the ultimate reason why we started everything was like we just see neural networks kind of going into more and more mission critical contexts and I think it's going to be enormously transformative for society, but in order to do so, and you want to build it safely, powerfully, reliably.

10:17And I think it's going to be critical to be able to understand, edit, and debug AI models in order to do that. And so that's what we're kind of enabling for the very first time. It's like unlocking the black box of neural networks such that you can intentionally design it, rather than just kind of like grow it from data. And if everything goes right, what do you think will be the impact that you all have on the world. So maybe one one metaphor that we like and think about is like, yeah, right now, like, you just kind of like grow AI from a seed and then it just like grows like a a giant tree and it grows all wild and crazy, right?

10:54We don't know really know like, out of a lot of the things that it's growing into and all sorts of like interesting and weird stuff can can happen with a really large neural net. But if everything goes right with interpretability, we know like we'll know every single piece of training data affects like the cognition, the model develops the units of computation that it uses. And I almost think of it more as like bonsai where you want to kind of like intentionally design and shape and grow the neural network like still and unsupervised like AI driven approach like we're we're not going to hand to prove every single weight of a neural network.

11:34But I think we'll gain the ability to like during every single piece of the training post -training process, like just intentionally shape an AI model such that it, you know, serves humanity and does what we want. Sounds like a parallel to the human genome project in some sense. Yeah. Give us something we've done in genetics. So this idea that we need to read DNA, we need to understand the building blocks of life. And ultimately now we're starting to edit DNA and use CRISPR to come up with interesting cure for diseases or an ability to edit crops to make them more resistant to pesticides and things like that.

12:11So it's just very interesting, parallel. Yeah, definitely. I think we think about that analogy a lot. And I know you have Patrick Shoe on the podcast at Subway as well, and we're working with him at our Institute to de -allie crack the code of the human genome as well. and I think there's a lot of really interesting parallels and also direct applications of AI interpretability as well. Would you go so far as actually making edits? What I heard from you earlier, the bonsai analogy was a little bit of a shaping, which is quite different in my mind from editing. Shaping to me might be, you know, train and be fit so that your body can survive, given a certain DNA, and then this editing, which is altering the DNA, you're going to do both.

12:55Yeah, I think in short, yes. I don't know like what the so interp as a field like there's still a lot to figure out. It's still pretty, still pretty new, but I think in bonds I also prune a lot of branches and you prune a lot of like the areas that you don't want to grow such that you can kind of shape the the overall tree to like grow in the pattern that you want. So I think like, you know, the the eventual system that we hope to build, like you can ask questions of the model, like, why did you come up with this response and get a faith flux explanation while also being able to make direct surgical interventions in the mind of the model such that we can remove harmful behavior, enhance good behavior, and still remains to be seeing whether like it's just like a direct weights modification or like some other like kind of shaping function that is most effective.

13:49If you think of some of the ways that people are trying to prune these bonsai trees today, I think it's a lot of prompt engineering, fine tuning, RL tuning increasingly now. What do you think about that as the approach to steer the behavior of these models versus actually go in and introspect and examine each of the individual neurons? Fundamentally, these are black box things. All sorts of weird things can happen when you like fine tune a model, for example, or like, you know, prompt a model and take it out of distribution and it can say all sorts of crazy stuff. So the paper that's most interesting about this recently, I don't know if you've caught this, is this like emergent misalignment study?

14:30No? Okay. This is like an Owen Evans group where if you fine tune a model on just insecure code. So it's just like bad code that you know has all sorts of like cyber security vulnerabilities. It'll then start doing all sorts of insane things like telling like wanting to enslave humanity or like praising Hitler and other dictators and it's a really surprising result because it's just insecure code. And so yeah it kind of shows that there's like um maybe like what you're doing with is you're telling the model, like, hey, do more of this, less of this, and like, almost like enhancing the circuits that you want kind of more of, but you can also have like all sorts of unintended consequences like this that show up.

15:21And these circuits, these are still like really alien cognition, like there's some parallels to humanity, but we really don't understand how these networks think and they're not like human thinking. So if you enhance the bad code snippets, it also is like fundamentally linked to maybe all sorts of like other undesirable behaviors and properties. This is a different twist on the nature nature debate. Yeah, because in that situation, almost feels as though you've imbued that particular model with bad DNA, if you will, just, you know, it's sort of fundamentally an evil thing or a bad thing and then it ends up with manifesting all sorts of bad behavior in other domains.

15:59It's really interesting. Yeah, maybe, or maybe these models kind of understand right and wrong. And if you enhance the wrong, then all sorts of other behavior is interlinked and expressed. But I don't know the way that I think about it. These models are just like the functions of their training data. And these models are trained on everything, like all sorts of misbehavior as well. And you want it trained on like, um, or you incorrect behavior as well because like, otherwise it won't know to refuse harmful requests or to not do a certain set of things. Do you have an intuition for why different models have different base models have different personalities?

16:43Like, for example, the newest clawed series, like, I think one of the models may be Opus or so on it is, you know, really cares about animal welfare, for example. And the other don't like give a sense for why these models develop pretty distinct personalities. I think it's just a function of how they're trained and it's really, really hard to anticipate in advance. I don't know. I feel like it might just be me, but like, Cloud4Opis too is enormously sycophantic. I'll kind of nudge it in one direction and it'll just agree with me wholeheartedly. I'll nudge it in the other direction. Pose a counter example, and it would just be like, yes, I was totally wrong before.

17:24Like, nothing I said earlier was correct. And I just think it's like really, really hard. It's like, it kind of goes back to like the witchcraft almost of training an AI model today where you're just like throwing in training data into the model and like whispering the incantation of gradient descent and then like trying to like get what you want out of it and something pops out. And then it like really cares about animals. Oh, great. Like, that's, that's great. I'd love to talk about your research results so far, you know, both at Goodfire and in the brother Meckenturp field as a whole. Maybe could you just give us a 30 ,000 foot flyover view of, you know, Meckenturp as a field, how old is it?

18:06What are the key results so far? What are the big open questions? Yeah. So Meckenturp as a field, I think like maybe just like in this tradition that we're building on. I think there are all sorts of studies like looking inside neural networks all the way back when we first designed neural networks. But I think once we, I think the way that the field kind of thinks about itself, like mechanistic interpretability was started out like OpenAI with Chris Ola and Nick Iberrata and a couple other folks who first put out like this really big circuit thread that posited like three things. One is like our features in neural networks, which are directions of latent space that represent concepts that the model uses to generate its response.

18:51Circuits, these are like features that fire together to create like higher order concepts. The example that they lay out is like you have a car window detector and then a car like body detector and then a car wheel detector and then that's like a car circuit and universalities the third -ten, like similar circuits evolve in different neural networks. And so this was like almost like the start of the field of mechanistic interpretability in my mind. And so that really kicked off a lot of, you know, interesting research and results in like the features circuits paradigm and like the main players in the field, there's a lot of academic labs that are doing great work like in Thropic.

19:40So Chris Olas, one of the co -founders of Enthropic and building a great interpretability lab there, and D -Mine has interpretability lab as well, and then we are kind of like the newer entrance on the field and on the stage. And I think one of the other really key things to have happened was understanding and mostly resolving superposition. So, superposition is this idea that like each neuron is responsible for encoding like multiple concepts. And there are more like concepts than dimensions in a neural network. So, if you think about a neural network as like a giant compression algorithm, you're like compressing like the entirety of the internet into like a relatively small number of parameters.

20:33And so that means every single neuron needs to encode or at least every single layer of the model needs to encode more concepts than it has like dimensions. And so there's like this concept of superposition where you have concepts represented as like near orthogonal directions in latent space such that you can represent all of these concepts in a model's latent space. And so to resolve this, you have to almost untangle an unscramble neuron such that it's responsible for like one clean interpretable concept. And so it's like a group at Apollo research led by like Lee Sharkey, who's actually now at good fire.

21:16First kind of pie in your like Sparsado encoders for language models and and Thropic also like really popularize this with their like big paper like towards monosamanticity. and then right afterwards, like scaling monosamanticity, showing that you can essentially unscramble these neurons into higher order concepts reliably and at scale with like arbitrarily large neural networks. And I think that was a really big moment for interpretability where you can now, in a totally unsupervised way, unscramble neurons of a neural network to understand them and get clean concepts. So the concepts aren't like totally clean yet.

21:56You can't like edit them super well. There's all sorts of problems with this, but like it's almost like a really big step forward for the field such that like that we can do this in an unsupervised way and the techniques and interpretability scale, which is really important. Does that mean the superposition isn't real or that you must like to hide some bugs on certain different, so you sort of collapse it at a particular moment in time to know that in this instance, it represents a particular direction. So I think it means it was real. Like neurons are responsible for, you know, encoding multiple concepts such that once you unscramble them, then you can do really interesting things with like a clean neuron.

22:36So the way that we do this is like using an interpreter model, train on the activations of a base model, and then now you have all sorts of neurons in the interpreter model that represent like theoretically clean sparse concepts. In the interpreter model, not in the original model. The original model still has this characteristic of superposition. That's right. OK, go ahead and thank you. And in the interpreter model, you unscramble these neurons and you associate these concepts with the concepts in the base model that you're trying to interpret. And then you can do interesting things with that.

23:06Go ahead and thank you. Yeah, of course. How solved of a problem is this then? If you've already been able to kind of disentangle the super position, then haven't you already mapped out the mind, so to speak, of the neural net, and what's ahead? I think partially, I think it's a partial mapping, and the technique has all sorts of flaws as well that we can improve a pot. But I think like it gives us the first step towards understanding these models, especially going from toy model to like, actual network that people care about. So we've done a bunch of work recently on R1 and so it's just 671 billion parameter mixture of experts models.

23:53It's a big boy model. And like the technique scale really nicely all the way up to that point because you know it's just more AI, more training of an interpreter model. So obviously I presume there's an asymptote here to understanding because the models are going to get more and more complex over time. We're going to beef them up. So I'm guessing sort of like the Battle of Sisyphus at some point, this is never ending pursuit. Is that correct? Do you agree with that? In some ways, but I also think that like, so the techniques that we've developed work on, toy models all the way up to like, yeah, big network that's like more capable and more intelligent and better.

24:43And I think like the techniques also scale effectively with model intelligence. So one part of our pipeline is that for every single latent concept in our interpreter model, we associate that with and try to, we get another language model to reason about like what that that concepts actually represents in the base model. This is a concept called auto interpretability, which Nick Camarata, who's a good fire now, and invented this technique at OpenAI,

25:14pioneered, and this technique, because it's a language model reasoning about, what a neuron represents, scales with the quality of the language model. So it actually gets better. So because we use AI in order to understand AI, like the better that our models are, are like these analysis agents are at interpreting like what's actually going on, the better we are able to understand them. And our interpreter model techniques also, it's just like we develop better interpreter models. They theoretically should translate to more and more intelligent and larger and larger networks because these are like unsupervised scalable techniques and that's like the paradigm in AI interpretability.

25:56All right. When do you think you reach a minimum threshold that makes you feel it's ready for a real world application. Maybe they already. I think we're there. Yeah, I think the first real world applications are already out there. And yeah, I think we're there on the very early applications. Can you share more about this? Yeah, I was being unnecessarily cryptic. So yeah, a couple of the partnerships I'm most excited about. So we work with our Institute, like I mentioned a little bit earlier, to understand an interpreter, Evo2, which is their DNA foundation model. So it's a sequence -to -sequence model, so it takes in a sequence of nucleotides and it predicts the next nucleotide in a sequence.

26:48And our theory is this is a narrowly superhuman model. So we really like to work on narrowly superhuman model, because it can teach us something about the world that humans don't really know. And so the idea is this model is representing just an enormous amount about the biological world in order to generate the next and properly model the next nucleotide in a sequence. So what we did was we sought to understand what does it actually know such that it can model the world so effectively. So what we did was we trained as far as auto encoders on the activations of this model. Extracted all sorts of features that were related to concepts that the model like should know, like kind of normal biological concepts that we have like really strong ground truth annotations for.

27:41So these are like TRNAs, RNAs, start coding sequences, all sorts of biological concepts that we have ground truth annotations are. We associated with this model. And then the question is, OK, now we have all of these other features of the model that we've extracted. What do they mean? What are they? They might just be ways that the model is computing and thinking or they could represent novel biological concepts that the model is using to generate the next, you know, nucleotide in a sequence. That's very interesting. I mean, for a long time, there was this idea that we have a bunch of junk DNA.