Inside the Hidden Geometry of AI

0:01Goodfire is an AI interpretability research company. So we spend all day thinking about what's going on inside the mind of a neural network. We had a nice paper a few months ago called Reinforcement Learning with Feature Rewards, where we were able to use the internal concepts of the model that we were able to extract in order to reward the model to reduce hallucinations. It's the idea that models really think in shapes and not really in words. It's like in curved, high-dimensional structure. And we've basically developed a technique to extract these structures in an unsupervised way. You can use this to help debug and train your model to be more effective and safe and performant.

0:49I think this has been almost like the story with interpretability. It's like everywhere we look, there's structure, there's really interesting concepts. Welcome, humans, to the Neuron AI Explained. I'm Corey Knowles, joined as always by the man bringing charm, curiosity, and just enough chaos to keep legal concerned, Grant Harvey.

1:11Corey Noles:Gotta keep him on their toes, you know? Right? Right? Yeah. Well, I'm excited today, Corey, because today we're going inside the mind of a neural network and not in a vibes-based sci-fi kind of way, as you might expect. Or a spaceship way. Well, Corey, you know, you have a very peculiar set of prompts that you like to ask models on release day. So people who don't know what I'm talking about, check our live videos. But not in a vibes way, not in a Corey way, but we're actually going to go inside the mind of a model, or at least as much as we can. And luckily, today, we are joined by Goodfire, who are trying to understand, debug, and design AI systems by doing exactly that, looking directly at the internal structures of neural networks and trying to figure out what we can understand from them.

1:59Corey Noles:The company's core idea is that models contain meaningful mathematical structure, that's features, manifolds, representations, and circuits. And that if we can map those structures, we can make models safer, more reliable, and more useful. Thankfully, we have Eric Ho, who is the co-founder and CEO of Goodfire, to join us today to explain how all this works. So I think with that, we will push it back to you, Corey, real fast. Today's video is brought to you by SaaS, AI governance software you can trust. We'll hear more from them in just a bit. Real quick, before we get started, make sure you hit the subscribe button above.

2:31And on that note, Eric, welcome to the Neuron. Thanks so much for having me. We think a lot about neurons, so it's a fitting name for the show. I'll bet you do. I'll bet you do. I guess before we get started here, for listeners who maybe don't spend their weekends spelunking inside neural networks, what is Goodfire and what problem are you trying to solve? Yeah, Goodfire is an AI interpretability research company. So we spend all day thinking about what's going on inside the mind of a neural network. The reason why we care about doing this is because we really want to build towards a future of what we call intentional design, where you can actually understand and debug and edit and really shape a neural network during training so that it becomes a lot less like trial and error, just kind of throw in data and get, you know, model out.

3:25And a lot more like building software, where you actually understand, like, how your inputs relate to your outputs, how data leads to generalizable representations in your model. and we want to kind of make this entire process transparent, steerable, debuggable, so that we can get better and safer neural networks that we can actually design with intention. So that's the whole goal of the company. We're all thinking about this every single day. And, you know, fundamentally, we're a research lab that has to, you know, figure out a lot of the core problems and core research with neural networks so that we can get there.

4:03Corey Noles:Right, right. That makes sense. You're basically trying to break open the black box. 100%. Yeah. During training, looking at a model to understand what type of creature is this. It's not really just a computer program anymore. There's clearly something interesting going on here in ways that are analogous to brains in some way. And so, yeah, what do you think a lot about that? So how do you actually do that? There are a variety of techniques, and they are still kind of developing as we speak. But maybe at a very high level, we use AI to do this. So we train all types of interpreter models so that we can understand the concepts, the parameters, the neurons, the weights of these models.

5:02And there are all types of tools that we leverage in order to do this. But they all are kind of permutations on this core idea that inside these models, we don't quite understand how all of the concepts relate to each other, how they form their cognition, how they manipulate concepts internally. and we have to kind of almost like break it open bit by bit in order to extract meaningful units of computation that we can then compose to higher order structure that then tell us something about how the model is performing this computation. So maybe for example, we can start by talking a little bit about just like there are like complex structure in these models like we found addition mechanisms for models to perform like complex computation internally you can extract all types of concepts like languages style how they represent the user what they're thinking about it in any given time and in all types of models as well.

6:13So for like biology models, they represent things like the tree of life or ancestry or all sorts of other concepts. So all of these structures, these concepts are discoverable in these models. And we spend a lot of time thinking about how to both extract these concepts as well as how to use these concepts during training for debugging for downstream applications. SAS has spent 50 years helping organizations make decisions they can trust. And as AI evolves from predictive models to large language models and now autonomous agents, that challenge is becoming even more important. We're starting to ask questions like, if I'm deploying AI that can make decisions, take actions, and interact with other systems, how do I govern it?

7:01SAS is built for this era of autonomous AI. Your AI governance approach should provide oversight of predictive models, LLMs, and increasingly autonomous agents. The list goes on. You need to monitor outputs for risk, track automated decisions, manage third-party models, and apply consistent controls across your AI. SaaS focuses on using data and AI responsibly and effectively, giving organizations visibility into how AI systems behave, accountability for the decisions they make, and confidence that they align with business objectives and support regulatory readiness. Because as AI systems become more capable, trust becomes more valuable.

7:41Learn more at sas.com. That's S-A-S dot com. Now back to our show.

7:46Corey Noles:When you're using AI to do this at a high level, are you essentially running the same requests in one model and then having another model sort of interpret those and say like, oh, when this was the output, these, you know, I don't know how exactly it looks like, but I assume there's like some sort of lights based on the visualizations that you've released, that there's some sort of lights that you see that light up, numbers light up again and again, and then you can actually tell, oh, that's how, that's what this represents. Is that kind of how you're doing it? That's exactly right. Yeah. Cool.

8:20Imagine like the simplest unit, the simplest unit of computation of a model is one single neuron. And these models are like trillion parameter models. So it's like a very small slice of the computation of a model. But if you look at a neuron, like our types of, our techniques essentially look at, you know, when these neuron activate and fire and then aggregate them across like, you know, potentially similar rollouts, similar tokens, and then train a type of like interpreter model to understand what that neuron is actually doing across like a set of computations. Neurons then compose into higher order structures.

9:00So we call this approach and direction or neural geometry direction, where there are curved geometric structures in the latent space of the model. That's kind of like the mind of the model where it, you know, manipulates vectors and computations. And so these like kind of higher order neural geometry structures are also things that we can extract and understand. And so going back to the calculator or the tree of life, these are like higher order structures that are made of many concepts that compose into a more complex concept. And these are also extracted by paying attention to the activations, the weights of the model as it's kind of going through its computation.

9:55Okay. So are you saying like that these various things live in specific regions of a neural network in many cases? or like you know where to go to look for some specific kind of reasoning or spatial awareness or whatever the case might be.

10:18Corey Noles:Spatial awareness was a particularly terrible example I'd like to add. Well, for example, one of the papers that you released, right, you showed all of the different visualizations of the neural geometry. And in some of them, you could even see how it represents the days of the week through how it moves across the pattern. So maybe you could speak to that, you know, because there's like the regions, there's movements, and then there's also even manifolds inside the latent space. Totally, yeah. So maybe at a very high level, what we're saying about a model is that there are units of computation inside the model that we can understand and label with semantics.

10:59These units of computation composed together to form higher level concepts and structures in the model that we can also extract in fully unsupervised ways. So we can kind of discover these concepts rather than have some sort of prior when looking inside the model. And this is interesting because you can then use these structures to debug the model. Let's say that the model shouldn't have been thinking about race as it's developing a medical diagnostic or something like that or a hiring decision. or, you know, in like the model is developing the ability to do complex arithmetic. And that would be a good thing to reward and incentivize as the model is training.

11:59And so that's maybe like the higher level mental model that we have of what these models are doing. They're actually highly interpretable the more that we look. So this is giving you the ability to not to, for lack of a better phrase, sort of reinforcement learning, a reinforcement learning approach that can go a layer deeper than what we've previously been able to based on what we've been able to see. Is that correct? Like we've stepped another layer in inside, if that's it. Yeah, we had a nice paper a few months ago called Reinforcement Learning with Feature Rewards, where we were able to use the internal concepts of the model that we were able to extract in order to reward the model to reduce hallucinations.

12:49But this can be done with arbitrary features or concepts that we can extract from the model. So we did this by essentially figuring out the types of hallucination mechanisms that were present in this particular model, which was a version of Gemma. and then we were able to kind of directly train and reward against that mechanism or that set, probably more accurately, the set of mechanisms that we were able to extract that were responsible for the model hallucinating. Because often there's an internal state of the model where it knows that it's more uncertain. It knows that it's about to make something up and that's kind of what we're relying on.

13:32The model often has a much more complex and rich representation of the outside world than it lets on in token space. So think about it this way. It's like what we see is just a single token output when we're chatting with a model. It's just the tip of the iceberg. In order to generate that token, there's an incredibly complex and rich inner world that the model has to generate for every single token prediction in order to generate that token. And so what is contained in that enormously rich inner world? Well, there's all types of structure, all types of knowledge, all types of computation that is both incredibly fascinating but also incredibly useful for researchers to understand.

14:21I'm going to go a little philosophical here for a second. Let's do it. From a philosophical standpoint, what do you make of that? Like this idea that like there's more to this than what we're seeing. This is stuff that isn't intentionally programmed. This is stuff that isn't like built from the inside out in that way. Yet it is there and it is doing things and it is doing things that further its cause. It is doing things that meet the needs and things that don't, which is even more crazy and interesting, by the way. Any theories? What do you think is going on in there? that is a big question uh there's clearly something quite interesting going on in here though where the level of computation the richness of understanding of these models is is clearly um on uh on a level that you know mimics intelligence in some way shape or form i guess the whole it's ai it's artificial intelligence we're simulating intelligence and we're doing a pretty darn good job of it.

15:30And so kind of like some way that I think about what we do is like we have these enormously intelligent systems. What we're really doing is like trying to figure out like how they work and what they're doing and what is this like complex system actually doing. And I don't know, there are, I hope that one day these systems can teach us about our own minds and our own conscious experience. and what all of this really even is. What are we doing right now? At this current moment. We're just molecules and atoms. And yeah, it turns out that being able to train and generate an intelligence is not actually super hard.

16:18You can do it with just a lot of GPUs and a lot of training data. and that's remarkable and i really want to get to the bottom of it and figure out how all these concepts compose and uh form this this higher order cognition we do have uh consciousness researchers kind of bashing on our our product right now um trying oh wait why why what's their criticism oh no no no i'm sorry they're using our product to oh i thought you meant they were bash bashing. It's good bashing, Grant. You're old. There's a guy older than you. Different usage of the word bash. Bashing in a good way. Just kind of using our product to try to discover structure that relates to some type of experience that the models have.

17:12And the models what's interesting is that if you remove kind of the model's refusal behavior or hedging behavior. They will often discuss some type of subjective experience that they're engaging in. But of course, it's very, very difficult to tell because they're trained on human data and a lot of sci-fi.

17:39Corey Noles:Well, look at how long it took us in this podcast before we got to the concept of consciousness. 20 minutes, you know? Yeah. It's such a fascinating realm of this and what made you really interesting to me in thinking of this because I kind of have a soft spot for interpretability and emergent technologies and all of that and seeing what's happening. And it's pretty clear that there's not, you know, there are two schools of thought. One is it's ones and zeros. Shut up. The other one is it's absolutely conscious. And I think it's a little more gray than that. I think we maybe really don't know what this is going to be yet.

18:27Corey Noles:Well, I'm kind of subscribing to the belief at this point, not having enough priors to really know for sure. But I'm subscribing to the belief that perhaps intelligence is a consequence of complexity. Something about that just really hits with me. It makes a lot of sense if you look at it from a certain perspective. If a system is complex enough, it has to be intelligent to continue to function by some definitions. And I've seen papers that are discussing this. Yeah, perhaps. I think the way that I think about it is, you know, these models clearly can generalize and it's almost like compression that results in the almost forcing function that these neurons have to be doing something really interesting together that form like some type of generalizability or intelligence from that act of, you know, jamming a bunch of information into a relatively small number of parameters compared to the entirety of the internet.

19:29But yeah, maybe even just to the earlier point, I mean, clearly there's something going on. We have found analogous structures inside models to humans, like our perception systems, you know, curve detectors, things like that. But also that these models clearly aren't human in many, many ways as well. And so I think we're just dealing with some type of new creation, new creature in some way. Yeah, I think that's a really cool way. You know, I always joke that we shouldn't have latched on to artificial intelligence. I think we should have called it synthetic intelligence. Because I don't really know where it's going to land.

20:14But I mean, either it's intelligence or it's not. I mean, artificial is, I don't know, it just never made sense to me as a word to describe what we're working with. Yeah, it's either intelligent or not.

20:26Corey Noles:And then if it's intelligent, you know, how was it created? Was it biologically, you know, through evolution or through synthesis? Yeah. What a fun tangent. Right, right. So sorry, didn't mean to go down that hill at all, but there we were. That was not in our research, but it was fun stuff. That was absolutely not in our notes. Most people are only using about 5 % of what AI tools can actually do. They open ChatGPT, ask a few questions, maybe write an email, and that's it. Meanwhile, other people are out there building workflows, automations, research systems, custom agents, and saving hours every single week.

21:04Corey Noles:AI is moving so fast right now that most people don't even know which tools are worth learning anymore. ChatGPT, Claude, Gemini, Codex, AI agents, automations, 5Coding, it's a lot. And that's exactly why we built the Neuron Academy. We built a practical, self-paced AI learning platform that helps working professionals like you understand the tools that actually matter and how to use them in real work. Built by us, the team behind the Neuron newsletter and the podcast designed to help busy professionals actually level up their AI skills. Inside, you'll learn prompting, AI workflows, research systems, automations, productivity strategies, and practical use cases across the entire AI ecosystem.

21:45Everything is hands-on, beginner-friendly, and built around real-world use cases you can apply immediately.

21:52Corey Noles:These are real skills that help you work faster, think better, and stay ahead in AI. Right now, you can join the Neuron Academy for just$499 a year, unlocking 50-plus lessons from full AI courses to quick-hit micro-trainings you can apply immediately. So if you're serious about leveling up your AI skills this year, this is where you need to start. Join the Neuron Academy today. So I understand you have some new research coming. We do. We have a bunch of new research planning all the time. But there's one piece that we're particularly proud of that's coming out in just a few hours. It's part of our neural geometry series.

22:33So the idea that models really think in shapes and not really in words. It's like in curved, high-dimensional structure. and we've basically developed a technique to extract these structures in an unsupervised way so we can discover these in models rather than have to have some type of prior coming into the model to kind of extract them directly maybe one example that we use in the paper is a rabbit ear and whereas our previous techniques may have just extracted the concept of a near. It's like fluffy, it's near, you know, it's just one single, you know, concept in the model's mind. Actually, models represent this concept in a much more nuanced way than we had previously thought.

23:25It's a curved geometric representation of a rabbit ear, which means the model knows actually what the bottom of the ear is compared to the middle of the ear compared to the top of the year. The same goes for horizons. The same goes for just pretty much every visual concept. Models are just like representing something much more complex than you'd originally think.

23:47Corey Noles:Wow. What's going on there, you think? Is that just a... That's not a coincidence, right? I think a model just has to have a really good understanding of the world in order to do something like generate a high-quality image. It needs to actually have all of this in its latent representations to be able to do what it does. We haven't quite yet applied these techniques. We're working on it to language models, but we expect to find very, very similar kind of curved geometric representations in the latent space of these models as well that represent, yeah, like the manifold of computations of a model.

24:34So what you're saying is that the computations sort of form the same visual representation,

24:41Corey Noles:or at least sometimes the same visual representation as the actual object that they're going to, in this case, it's an image model we're talking about? So we are talking about image and video models. That's where we first pioneered this technique, which is called a block sparse featurizer. and it's for unsupervised manifold extraction from within these models and you can extract all types of concepts in this way but it's a new type of way to decompose the residual stream of a model to see what the model is representing at any given time. Got it. Got it. So if I'm understanding this correctly, it actually visually represents it internally before it then produces it as an image.

25:36so um not really uh i guess what i'm saying is i mean uh it has a representation of like something much more complex than just like a higher order linear concept so maybe backtracking for the field, there's been the dominant kind of view of the field of interpretability has been this concept of the linear representation hypothesis, where concepts are represented as linear directions in latent space. So just like single direction vector in latent space. What we're saying is that many of these concepts of the model live on a higher dimensional curved manifold in latent state which means our previous techniques to extract linear directions wouldn't work to extract a more complex higher dimensional structure but this is actually how models think and so it's really critical to meet the model where it is in order to uh extract these higher level structures.

26:52And so a rabbit ear, whereas it may have been extracted by like a sparse autoencoder is how you usually extract linear directions in latent space. It may have been just labeled as a rabbit ear and it just represents the entire ear. That's like the linear representation. For the neural geometry representation, the higher order representation, you would have a textured gradient of like this rabbit ear where it would have some delineation between the bottom of the year the middle of the year the top of the year and some type of gradient and that's important because you would then be able to kind of better understand like how the model is representing that that concept one example that we use is this like classic kind of mountain car steering example where you have the mountain car kind of going up a hill and the goal is to kind of um for the model to represent like the mountain car um kind of driving up the hill and so if you try to steer it linearly um which means like turn up that direction in latent space uh the mountain car will just be splattered across the entire mountain but if you have this like complex spiral helical structure uh that's actually how the model is thinking about you know moving the car on the mountain then the mountain will just drive up really really smooth or the the car will just drive up really really smoothly up the mountain and so kind of just how these models tend to represent things are in much like higher order structure than just a linear structure, which makes a decent amount of intuitive sense.

28:41There's like a just like kind of higher order structure that governs kind of model cognition. That makes sense. You know, I think when I think about what's going on in my mind when I draw a picture, we probably are thinking quite linearly as people, I assume. Or do you think that – and forgive me for comparing this to that. I just – I'm trying to think of it in a way –