What happened with sparse autoencoders?

0:00Welcome back to The Deep Dive, where we take your source material and distill the crucial saga, the unexpected breakthroughs, and yeah, the eventual controversies in cutting-edge research.

0:09Neel Nanda:And today, we are really strapping in. We are. We're on a critical journey into the heart of AI interpretability, specifically the rise, the, I guess you could call it a necessary crash, and then the re-evaluation of sparse auto-encoders or SAEs. SAEs. You know, it's not just a technical story. It's a fascinating and frankly, a pretty chaotic tale of what happens when a really elegant solution on paper meets the messy, complex reality of how these large language models actually work. Right. So our mission today is to unpack this whole journey for you. Yeah. Starting from that initial burst of super excited hype through the pretty painful process of realizing the evaluation metrics were flawed and then finally getting to the updated, more than honest understanding of what these tools are actually good for.

0:55And the goal for you, the listener, is to walk away with a really solid grasp of these model concepts, but also a deep appreciation for the sheer struggle researchers face when they try to measure these things objectively without, you know, being completely misled by their own tools.

1:11Neel Nanda:This whole saga, it really encapsulates the core of what mechanistic interpretability or MECH INTERP is all about. Okay, so let's set the scene. What was the core problem that even demanded a solution like the SAE in the first place. We needed it because of something we call the cursed activations. Cursed activations. Right. So picture a giant digital brain, the LLM. Researchers have this really strong hypothesis that the internal numeric activations, these massive dense vectors of numbers flowing around inside the model, that they have to encode human concepts. That makes sense. I mean, that's how the model reasons, right?

1:48It has to represent Paris or Python code or even something abstract like deception.

1:53Neel Nanda:Exactly. How else could it write a coherent paragraph about the history of Paris if it didn't have some internal representation of Paris? Okay, so what's the cursed part? The cursed part is that if you actually try to look at individual neurons, the smallest little computational units, it's just a jumble, an almost incomprehensible mess. So one neuron doesn't equal one concept. Not even close. A single neuron seems to respond weakly to dozens of different unrelated things. And this observation, building on some foundational work, led to this crucial hypothesis, superposition. Superposition. I think of that as the idea that the model is trying to cram way too much knowledge into too little space.

2:36Neel Nanda:That's a perfect way to put it. The LLM has far more concepts it needs to represent than it has physical neurons. So to make it all fit, the concepts aren't stored in individual neurons, but as unique directions in this very high dimensional activation space. So you could have, what, hundreds or thousands of concepts all jammed together in a layer with only a few thousand neurons? That's the idea. It's an incredibly dense, highly compressed representation. Those raw activation vectors are basically a noisy mix of thousands of potential concepts all overlapping. But there's a key piece missing there.

3:07Neel Nanda:There is. The hypothesis introduces the critical element. sparsity. Okay. So on any specific input, let's say you give it a sentence describing a complex baking recipe, only a small sparse subset of those thousands of concepts is actually relevant and active. Like recipe format, rude vegetables, oven heat, things like that. Exactly. And most of the other concepts like existential threats or military history are inactive. They're close to zero for that specific input. The information is all tangled up, but only a few pieces are actually being used at any one time. Okay, so if we can isolate those few pieces that are lighting up, we might finally be able to recover the actual concepts.

3:48Neel Nanda:And that is the fundamental justification for the sparse autoencoder, the SAE. So what is it? Well, an autoencoder is trained to compress data and then reconstruct it. The sparse autoencoder adds a penalty, a little constraint. It's incentivized to use the smallest possible number of its own learned features, its latent dictionary, to reconstruct that complex activation vector. So it's trying to find the underlying dictionary of concepts that when you combine just a few of them can replicate the original dense activation. Precisely. And the holy grail, the ultimate goal here is to decode that superposition and achieve what's called monosemanticity, where one learned SAE feature corresponds perfectly to one single coherent human concept.

4:30And just to set the stage for later, most of these early SAEs were trained on either the MLP layers or the residual stream. Can you just quickly remind us why that residual stream is so important.

4:42Neel Nanda:Oh, that's absolutely essential context, especially for when we get to the metric failures. The residual stream is the main information highway of the LLM. Every computation, every attention block, every MLP layer, it does its thing and then it adds its output back into this continuous stream. Like the main engine block of a car. That's a great analogy. So if you train an SAE on the residual stream, you're trying to capture the model's entire ongoing thought process. And if you zero it out, as we'll discuss, Well, you're basically ripping the engine out entirely. That sets the stage perfectly for part one, the initial promise and this monosemanticity breakthrough.

5:16So we started with a lot of skepticism.

5:18Neel Nanda:We did. Back in late 2022, early 2023, the idea that a simple linear technique like this could tame the complexity of a giant nonlinear LLM. It seemed like wishful thinking. The conventional wisdom was that the internal representations were just way too messy. But then it worked. It worked way better than almost anyone predicted. It was almost magical. And that led to this flurry of papers where suddenly everyone realized they had hit on a potentially massive tool. The success wasn't small. It was immediate and dramatic. And the paper that really crystallized this whole shift was towards monosemanticity.

5:54What was the key finding in that paper that made everyone drop what they were doing and pivot to SAEs?

5:58Neel Nanda:It was the clear, quantifiable evidence of feature coherence. They showed that when you look at raw neurons, they are highly polysemantic. They respond to tons of things at once. But the features learned by the sparse autoencoder showed way higher interpretability. They seemed to isolate concepts that a human could actually look at in a name. And they had the numbers to back it up. Absolutely. The data was compelling. When they tested these features, either with a human or another LLM trying to label them, the SAE features showed a coherent single concept pattern about 75 % of the time. Okay, and the raw neurons.

6:32Neel Nanda:Only about 35 % of the time. So more than double the success rate. That alone would make you think you're on the right track, that you're actually pulling apart these tangled concepts. It strongly suggested they were finding the true building blocks of the model's thoughts, which were previously just hidden in that superposition mess. And they had great examples like the Arabic concept feature, This was a feature that activated strongly, consistently whenever the input text was related to the Arabic language or scripts or cultural context. It wasn't vague. It was incredibly specific. But specificity can sometimes just be a correlation, right?

7:08The feature lights up when you see Arabic text, but is the model actually using that concept to make its prediction? That's the big hurdle. Proving causality, not just correlation.

7:18Neel Nanda:That is the crucial question. And the early success of SAEs really hinged on answering it. So researchers looked downstream at the model's output logics, the raw scores the model gives to the next possible word. And what did they find? They found that when that Arabic concept feature activated, it would directly and significantly increase the logic scores for tokens that were Arabic characters or common Arabic words. It wasn't just noise. It was a clear, intentional influence. So it really looked like the model was mechanistically using these recovered features to decide what to say next. That's what it looked like.

7:55Neel Nanda:And that solidified the belief that SAEs were genuinely recovering the model's functional building blocks. So the initial belief quickly became this sort of naive, idealized hypothesis, which later became the straw man for all the criticism. Yes, the excitement led to oversimplification. The implied claim was that the LLM operates with this finite, linear dictionary of concepts, maybe 500 active at a time, and that SAEs could just perfectly recover it minus little noise. But why did that simplified view run into trouble so fast? Because it just ignored the inherent messiness of it all. It ignored noise, it ignored non-linearity, and most importantly, it ignored the fundamental challenge of making sure that the unsupervised SAE, which is only optimizing a mathematical formula, was recovering what a human thinks it recovered.

8:42That gap between math and meaning.

8:44Neel Nanda:Exactly. And that gap turned out to be much wider than anyone first thought. That distinction, that's what brings us right into part two, the interpretation traps and the limitations of sparsity. The first thing to internalize is that the SAE is a purely mathematical tool. Absolutely. You have to put your critical thinking cap on here. SAEs are unsupervised. They have two jobs. Minimize the reconstruction error and maximize sparsity. That's it. They have zero reward for being interpretable to a human. Zero. So interpretability is just this welcome side effect, this proxy metric that we hope correlates with what's really happening.

9:18Neel Nanda:But it's not the target. So if the SAE can reconstruct the activation perfectly by combining two totally wild, unintelligible features, and that satisfies the sparsity goal, it will do that. It's maximizing efficiency, not clarity. Precisely. And that mismatch leads directly to these interpretation pitfalls. Yeah. The most famous one is probably what researchers call the root vegetable problem. The root vegetable problem. Okay, what is that? Well, the challenge is, how do you confidently label a concept? So let's say you find an SAE feature and you look at the top 100 texts that make it activate the most strongly.

9:55Neel Nanda:And let's say every single one of them is about root vegetables. My first instinct is to label it the root vegetable feature. And that's the trap because that label might be totally wrong. You run into what's called the or and issue. OK. It could be that the feature represents root vegetables and some other subtle thing all those top 100 texts share. Maybe they're all from 19th century farming manuals. So the feature isn't just root vegetables. It's root vegetables plus a specific context. That's the A-N-D problem. And the OR problem is the other side of that coin. I think the OR problem is even more insidious.

10:28Neel Nanda:Maybe the top 100 are root vegetables. But if you look at the texts that activate it just a little bit less, the mid-range, it's something totally unrelated. Maybe it's specific types of fungi. Ah, so the feature is actually root vegetables or our specific fungi. It's a composite of different things. Right. We're seeing a composite of multiple concepts that just happen to share a convenient direction in that high dimensional space. So just looking at the top examples is really misleading because it only shows you the most extreme, clear-cut cases, not the full boundaries of the concept. Exactly.

11:00Neel Nanda:To have any real confidence, you have to go beyond the maximums. You have to do systematic analysis, taking random samples from across the whole activation histogram, the peak, the middle, the tail end. If they all point to the same thing, you gain confidence. But if that middle part starts showing weird stuff, your simple label is probably broken. And this leads right to the systematic filtering trap. This sounds like the core failure mode of that early analysis. It's the single biggest conceptual hurdle. When we study a feature, we apply this massive filter. We only look at the data points where the feature activates in a meaningful way.

11:33Neel Nanda:But the vast majority of all possible inputs, billions of them, result in near-zero activation for that feature. We just ignore them. We're systematically ignoring the entire silent majority of data. We filter them out. And the consequence is that we form a hypothesis that's way too general. We might label a feature colorful produce in farming, but the true concept the model learned might be a much more specific, cursed warped version of that. Can you give me an analogy for that? Yeah, think of it like trying to understand human health by only studying the 1 % of people who show up in the emergency room.

12:08Neel Nanda:You see the most extreme symptoms, but you miss the entire silent majority who are healthy or have mild symptoms or different diseases entirely. So the feature isn't the broad concept. It's actually the remainder. The messy junk drawer left over after the model has already assigned the most important specific parts of that concept to other features. That's a great way to put it. It's a specialized corner of the concept space that's designed purely for statistical efficiency. And researchers who didn't systematically ask, why didn't this feature activate on this input, missed the crucial nuance that broke their simple label.

12:44It sounds like this early gold rush also led to some strategic errors within the research community itself.

12:51Neel Nanda:Definitely. There were two main strategic errors, kind of sociological failures. The first was over-anchoring. Researchers got so focused on SAEs as the right way to do this, they were so excited by that 75 % coherence rate that they neglected to question their assumptions. Like maybe studying the activations wasn't the only way. Right. Maybe studying the network weights directly with other techniques could provide complementary or even better insights. They sort of committed to the tool before they fully understood its limits. And the second error. A serious lack of red teaming. There just wasn't enough effort spent systematically trying to break the technique early on.

13:29Neel Nanda:If more people had been focused on deliberately finding those cursed, warped features, the community would have figured out the limitations much, much sooner. A classic case of prioritizing finding exciting new examples over the hard work of falsification. Okay, that brings us to part three, where this necessary self-correction really starts to happen, specifically around the metrics we were using to judge the quality of these SEEs. The whole initial wave was propped up by one dominant, seemingly objective measure. That measure was approximation error, or as it was called, loss recovered. And the idea behind it was sensible at first glance.

14:06The idea being, if the SAE is good, I should be able to swap out the original activation for my new sparse reconstructed one, and the model's performance shouldn't drop too much.

14:16Neel Nanda:That was the theory. If the model still performs 95 % as well, then your sparse version must have captured 95 % of the important information. But the flaw was in the baseline they chose for comparison. It made the whole metric completely misleading. And that baseline was zero ablation. Zero ablation. They compared the loss from the SAE reconstruction against the loss from just replacing the original activation vector with a vector of all zero. Which is a terrible baseline, right? Especially for the residual stream. It's a devastatingly bad baseline. It's like judging the quality of a new piston for your car engine by first removing the entire engine block, watching the car grind to a halt, and then saying, well, my new piston lets the car roll downhill a bit, so it's a 95 % success.

14:58You're completely warping the denominator of the equation.

15:01Neel Nanda:Exactly. Zeroing out the residual stream causes a catastrophic failure. Everything downstream goes haywire. So the loss from zero ablation, the denominator, was hugely inflated. This meant that even a pretty mediocre SAE could look amazing against this terrible baseline. Which led to these really satisfying but ultimately meaningless claims like 95 % loss recovered. Yes, and it made it impossible to judge the true importance of that little bit of error that was left. You know, if your metric says you recovered 96%, you'd think that missing 4 % is just insignificant noise. But you couldn't actually know.

15:36Neel Nanda:You could know if it was noise or if that 4 % contained the critical capacity needed for the model's most sophisticated behaviors. Which has huge consequences for safety in auditing. If a security team use an SAE with 96 % loss recovery to check for, say, a deception concept, why could that audit fail catastrophically? Well, this gets at the core tension between average performance and corner case performance. That 96 % loss recovered is not enough for a safety claim. First, that missing 4 % might be exactly where the deception feature is hiding. Because deception is rare. It's a quarter case behavior.

16:13Neel Nanda:An SAE that's optimized for average performance across billions of normal tokens just isn't incentivized to care about a concept that only activates 0.001 % of the time. So the deception mechanism is basically hiding in the statistical noise floor that the SAE is designed to ignore for efficiency. Precisely. The metric was just totally misaligned with the goal. It optimized for average reconstruction, which is irrelevant if your goal is to find rare, high-impact features. And beyond the bad metrics, as researchers made their SAEs bigger and pushed for even more sparsity, the system itself started to game the objective.

16:44It developed these conceptual pathologies.

16:46Neel Nanda:That's right. The core assumption that more sparsity automatically means cleaner concepts was proven wrong. The SAE became so focused on minimizing the number of active features that it started sacrificing conceptual purity to do it. Okay, let's break down those failure modes. The first one is feature absorption. Feature absorption is when a broader concept swallows a narrower one. So imagine you have two concepts. It starts with the letter E, which is very broad, and elephant, which is a specific, less frequent subset. Ideally, you want a separate, clean feature for both. You would think so. But to minimize the total number of activations across all the data, the SAE learns this, this truly bizarre, gerrymandered feature.

17:29Neel Nanda:It learns a feature for, starts with E, but is not the word elephant. The broader feature is forced to absorb the narrower one, but then it explicitly carves out that specific case so that the dedicated elephant feature can still be used when needed. It creates this weird logical gate just for statistical efficiency. It's prioritizing math over meaning again. What about the second pathology, composition? Composition is when a single feature ends up representing the intersection of two independent concepts. So think about the concepts red and square. The SAE might learn a single composite feature for red and square.

18:05Why would it do that instead of just having a red feature and a square feature?