Exploring the Biology of LLMs with Circuit Tracing with Emmanuel Ameisen - #727

0:00We've had debates for a very long time where it's like, are they stochastic parrots? Like, do they, you know, just like kind of like match to the most like similar training data? Like, or do they like generalize? What is generalization? What does that even mean? And I think what's really cool about this work, what really like gets me excited is we can just talk about the mechanism and then we can debate like, you know, what does it mean that this is the mechanism? But we can at least start from a common ground and be like.

0:41All right, everyone, welcome to another episode of the TwiML AI podcast. I am your host, Sam Charrington. Today, I'm joined by Emmanuel Amazin. Emmanuel is a research engineer at Anthropic. Before we get going, be sure to hit that subscribe button wherever you're listening to today's show. Emmanuel, welcome back to the podcast. Thanks, Sam. It's good to be back. It's been a while. It has been a bit. It's been three years, almost like in a couple of weeks. It'll be three years since we recorded that last episode. And we were chatting a little bit beforehand. You know, we know one another personally, and we can't remember the last time we saw each other.

1:21It's been too long, like 10 lifetimes in AI years, I think. Yeah, I was going to say, it's okay. Nothing happened in the last three years. There hasn't been any development that matters. Nothing at all. Nothing at all. You've been at Anthropic how long now? For about two years. It'll be two years in a month. Okay. So it was definitely longer than a couple of years. I think you were probably still at Insight the last time we spoke. And I think you did a stint at Stripe maybe after that? Yeah, that's right. So it must have been a while back. I think when I was last on, I might have just started at Stripe.

1:59I was there for about three and a half years or three years or so. Yeah, so doing kind of machine learning engineering there more probably. And at Anthropic, have you been exclusively focused on interpretability? I've actually done a bunch of things. When I joined, I initially joined the fine-tuning team. And so, you know, my first few projects were actually around making Claude better at a variety of kind of very practical tasks. So I worked on making Claude better at SQL, like just writing good SQL or, you know, being better at function calling and tool use, kind of the sort of like in the early days of agentic workflows, just just having models called tools was that was a big deal.

2:52So sort of like working on some of some of that stuff. So internal fine tuning as opposed to allowing users to bring their own data and fine tune. yeah yeah to be yeah i should have said fine-tuning here is sort of like fine-tuning in the sense of the fine-tuning that goes into the cloud model that goes on like cloud.ai and on our api so it's the sort of like you know the process by which we make that sort of like main model that everyone uses rather than yeah rather than like fine-tuning for for custom use cases and so you were saying after fine-tuning yeah so i worked on that for a while um kind of yeah worked on on a variety of things around like models using code and models kind of like using tools uh and then about a year ago uh i switched to to something that's um much more research focused working on interpretability research which is um kind of trying to understand how the models do you know what they do at all um if you're familiar with the field you know you're like ah yes this makes sense i think when i talk to friends they're very confused uh they're like but you made the model like what do you mean you don't understand how it works but it turns out we have no idea how it works uh at all and so and so we have a whole team uh and multiple teams actually just trying to try to understand that and you know there's also other groups in the industry and in academia doing the same thing just trying to understand well like how are these models doing what they're doing and our conversation today is going to focus on a couple of papers that you and your team have recently published.

4:27The first one is called Circuit Tracing, Revealing Language Model Computational Graphs. And the next is called On the Biology of a Large Language Model. I think of those two and the way they fit together as the circuit tracing paper is kind of talking about some of the tooling and mechanisms that you've developed for this particular approach at mechanistic interpretability and the on biology paper as being some of the results you've seen in the application of those tools. Is that the way you think about it? Yeah, that's exactly right. Maybe a metaphor we've used here that I think makes sense is in a way, like some of the work we've done is akin to like building a microscope of sorts, you know, and like at least some tool that allows us to look inside the model a little bit as it does things.

5:18And so there's the, well, how do you build a microscope? It's like, ah, well, you know, we chose like to sort of like, this is actually like how light refracts off of the lens in that other one. And that's the like circuit tracing paper. And we talk a lot about, you know, the various pitfalls, the various like trade-offs we've made, that sort of stuff. And then there's, okay, well now you have a microscope. Let's look at some cells. And that's the biology one. And this one is mostly, you know, okay, the insights here are less about like how we got there we of course use the microscope to get there but they're about like oh it looks like you know this is how Claude does math or it looks like this is how like this is the mechanism that causes Claude to hallucinate sometimes so it's much more about like the yeah the biology of a model like studying its behavior rather than sort of like the building of the tool.

6:08And we're going to dig into both of those areas but I wonder if it makes sense to start with like what is the thing that you learned in all this about the models that you found kind of the most surprising or that most challenged the way you think about large language models it's funny um after like before releasing the papers but after doing some of the experiments uh someone on our team like asked everyone you know kind of uh we have 10 or so uh case studies and biology and you're like well how many of those were were like surprising to you um and and reflecting you know back on it and on what i was expecting i would say that about like half to two thirds of them were surprising to me um and then of course you know you you sort of like published the paper and some of the reaction to to all of them is like well like this is obvious like why why wouldn't the model work in this way you know uh from other people so like i think sometimes it can be harder to appreciate uh what's surprising but but i can give you like maybe a couple ones that that definitely were surprising to me um one is is um this like poetry example and we use poetry because um writing a rhyming poem is something that's like pretty hard to do um if you're just like predicting one word at a time which is how these models work right uh they like have a prompt They predict one word, they add that word to their problem, they predict the next one and so on.

7:37And so you might say like, well, OK, that seems like it would make it really hard to write, you know, good poetry, which most most models can do, or if not good, let's say rhyming poetry. You'd expect the model to greedy optimize itself into a corner pretty easily. Exactly. It would get to the end of the rhyme and then it'd be like, oh, you know, now I have to rhyme with orange. It's like, well, it turns out actually that's a hard rhyme. And also I didn't think about it. But still, you can imagine like, well, maybe if you have enough data, it just kind of like always picks, you know, sentences that kind of make sense and it gets to the rhyme and there's some like murky thing.

8:12But it turns out that what we found when we looked at it was that when this is a cloud 3.5 haiku, like is given a rhyming couplet. The example we used in the paper is he saw a carrot and had to grab it. The model then completes that with his hunger was like a starving rabbit. So it rhymes using rabbit, which is fine. uh it turns out that when it when it processes grab it and then comma and then new line and then it's about to write the second line at that point before it writes the first word of the second line there are uh features which we can explain kind of what they are a bit but there's like representations in the model's circuits for the word rabbit um it's already decided uh in a way if you want to decide it, that it's going to end with Rabbit.

9:07And then it uses those representations not just to like choose Rabbit at the end. It's not just that it's chosen that. It's that they're also causally involved. Like they help guide the choice of all of the words before leading to Rabbit. And we know this because we see it in the circuit, but also because we kind of want to validate this outside of our methods. We just perform a bunch of tests where we like delete Rabbit and we add like a random word in the paperweight like green and then the model rewrites the sentence completely and i think i think the sentence it lands on is something like the farmer's field was verdant green or something so it like completely changes its sentence so that it can end with its plan but so that example of sort of like pretty complex you know longer horizon behavior was definitely one that kind of like stuck with me uh as something that you might not expect if you were just like well they're just you know autocomplete on steroids predicting that the next token like they're sort of like not doing anything complicated.

10:02It turns out they are in some context. And I think with that as kind of context, it makes sense to jump into circuit tracing because the immediate question, you know, on hearing that example and the others presented in the work is like, how do you know? How do you know? So let's jump into circuit tracing to talk a little bit about or contextualize the approach that you've taken in the broader context of interpretability? And maybe to get that started, are you only focused on mechanistic interpretability there or do you explore other types of interpretability research? Right now, we're mainly focused on, or at least for this work, we're mainly focused on mechanistic interpretability.

10:52And I can sort of like, yeah, maybe i can contextualize kind of what even is circuit tracing and also maybe like why do you need it uh just really briefly right which is that you know um in in past jobs doing doing machine learning i've used models that weren't uh kind of transformers and and models like that that were maybe like decision trees and decision trees are commonly used in like fintech and fraud prevention and stuff like that and they have this really nice property that they're exactly what they sound like they're this tree of decisions and so once you've trained your model you can kind of look at it and say like ah like the reason that the model said you know we should block this transaction is because the amount was over ten thousand dollars and it was going to like a bank account in the like cayman islands or something and you're like yeah that's that makes sense whereas if it says you know if the if the like individual branches made less sense you'd be a little more suspicious.

11:51And so that's a really nice property of these silver models. Turns out transformers, not at all like that. The complete opposite, right? The way that they work is they have these kind of like giant vectors of numbers of hundreds or thousands of numbers that represent each word or each token. And then those get processed by like a series of, again, you know, basically oftentimes millions, billions of computations. And at the end, the model predicts a work. And kind of like what happens in between, if you if you look at like all of these all of these numbers is just like completely unclear, like you can, you can, you know, go in your your like Jupiter notebook, and you can look at the numbers and you can say, ah, like, you know, yes, when when Claude was like writing poetry, this is the sequence of like a billion numbers or whatever that that was in its in its activations, but that's sort of like, so what, you know, it's it's maybe like, yeah, similar to sort of like, looking at like a tangled mess of like cables or circuitry inside a computer and it's like well you could just look at what lights up but you kind of like you need to somehow like map your observation to like what the like what the model is actually thinking about and so that's the nature of the problem um and then i would say that the sort of like circuit tracing is step step two on maybe like a like a three-step plan step one would be um well can you understand like what the model is representing at all uh and this is this is kind of uh the reason for working on dictionary learning uh which which um was kind of the topic of of the two previous papers before these two you said dictionary learning yes yeah um which is like broadly here like like the idea is can you take this like dense mess of numbers inside the model when it's like let's say processing one word or one one sentence and decompose it in a bunch of like unique concepts so that you know if if the the model is processing the problem like uh i'm taking my car over the golden gate bridge you might expect that like okay so you see this massive numbers but in there's going to be like a concept for the golden gate bridge and a concept for car and a concept for like a fun weekend with friends or something um and so the goal of dictionary learning is is is that it's like okay can you basically decompose representations into a like uh sparse version that's sort of like understandable um and we do we do that importantly like an unsupervised manner basically like you train a completely other model that's going to be your your dictionary that's going to like decode the innards of your your base model and then you can like on a given prompt you can look and you say oh you know look claude was thinking about this or thinking about that and that was that was the paper that that that that was prior to these most recent ones and sort of we found a bunch of interesting things there we found you know kind of like very nuanced representations there's like representations for like people uh uh being like psychophantic you know meaning like praising you even when they don't meaning there's like they don't mean it there's like representations for like typos and bugs in code there's like a bunch of different representations inside the model that are that are like fascinating um and so then and then i'll pause maybe but i'll say one more thing which i'll say like okay that's the first step and then so what's the second step well what you'd really want is like this is kind of like finding you know you have this like jumbled soup now of all of the things that the model is thinking about but what you don't have is you don't have a way to explain like okay but why did the model say what it said or even why did the model think about the golden gate bridge like it's probably because in the prompt there was like golden gate bridge and it like those three words are enough to think about that versus other bridges but now we want to connect all these representations and have basically like a like like a circuit which is just like a list of steps in order where it's like it went from this one to that one to this one to that one and then that's why it said this and and the effort of circuit tracing is not nice and thinking about the uh the dictionary work and the idea of representations of concepts uh one of the things that jumped out to me is that this uh you know it's very similar to the ideas of like embedding embedding you know uh words into vector space and you have these areas that represent distinct concepts, but it's not exactly embedding.

16:35It's more, maybe the right way to ask the question is, does the kind of results that you see, like follow directly from embedding and embedding as a concept, embedding this and the fact that models kind of operate on these vectors, or is there a distinct concept of concept relatedness in the context of the model itself it's definitely related so you know the models themselves right they take in as you mentioned embeddings for the for for their words and then but but then what happens is sort of like two things um they process these embeddings and and like output you know different different vectors essentially and so it's like ah like maybe uh if you so an example we have in the paper is like you know facts uh michael jordan plays the sport of and then you know uh basketball is the completion that the model successfully gets to and it's like ah well how does it know this well you know one thing that that models do is like they take the embedding for jordan which is like that's just a token they've learned it and then there's like various pieces of the model that like read this embedding and will like for example write like ah basketball or ah um space jam you know or the chicago bowls and so there's they're sort of like um these these like um embeddings are your starting point but then you have like a lot more nuance about like various sort of like either like your various aspects of like decomposing a given embedding something or like combining them right if you have you know i don't know uh michael jordan and lebron then maybe you're going to have like some parts of the model that are combined though combining those and sort of like starting to discuss starting to have features for like discussing who's the greatest basketball of all time right like debates about greatness is probably a feature you'll see there and so they're sort of like like the way to think about it is that like the nuances of concepts that can be represented in the model is is is much it's almost like a commentorial explosion on top of your embeddings right you you know, like you're kind of like base embeddings.

18:45And then, you know, you can kind of like combine them in various ways. And then, you know, the way you combine them is you might have like an embedding for like street and then an embedding for like, you know, I don't know, main street, first street, second street, third street. But then you also are going to have a direction in the model that's like, you know, main street in like Texas maybe or main street in Arizona. You can sort of like combine them in arbitrary ways. And so what we're trying to understand are sort of these like more complex and nuanced representations inside the model after it's consumed embeddings and before its output um uh basically a next word prediction yeah i'll tell you where that came up for me was the example about um was trying to get at whether there's this kind of linguistic universality.

19:36And so it had the 3.5 haiku model like translating the same phrase in several different languages and asked it to get the opposite of like convert big to small or something like that. Yeah, the opposite of biggest, yeah. Yeah, and it struck me in a lot of ways like the classic word to vet kind of relationship, king minus man plus woman. The question that that prompted for me was like, you know, do these results kind of just follow directly from embedding? And I like the insight that, you know, yes, they're related, but the concept of embedding within the model is so much richer than the kinds of concepts that are represented in the vector space.

20:25Yeah, and I think that like, those results are actually like a really good example of this. And we can like dig in a little more in that like so the comparison we make in the biology paper on this is like if you do that's i think it's the opposite of large is small uh and uh le contraire de grand et petit which is like the french version and you can do like we do different languages what you find is that like you know initially there's like of course different embeddings for those because they're they're different uh tokens uh and then in you know as as the as the sort of like signal uh progresses through the model the features so these representations start being shared like there's a shared representation of something that is big there's a shared representation of something that is the opposite across all languages um and and that i think kind of is an example of what i was saying a little bit earlier which is like ah like all of the embeddings for like big and large and all of the other languages they they share at least one facet right which is that they're about something that's big there's many other facets that they don't share like one of them starts with b the other starts with l you know they have like different number of letters whatever there's like a bunch of ways in which they're different they're in different languages um but they but they share that and so the model sort of like you know in processing these different languages like extracts the shared representation um and then does its computation it's like ah like there's largest there's opposite so i have to say something small and then you know goes back to the language that it was asked in uh and and so in the paper right we're like well if it's truly a um a shared like um uh kind of like set of representations surely it means that you can kind of like mess with them and like plug one for the other or like you can you can use the same like flip large to small in the like kind of like shared space and it will do the same thing for all languages and surely it does meaning that like we don't need to find a different concept of like opposite in french and chinese and english it's just we we found the concept of opposite uh for the model here and we turn it on it works for all languages um which yeah so so again it's like an aspect of it's like something more nuanced than the original embedding right it's like extracted some some like aspects that is that is that is shared and it is useful for it to sort of like generalize when it's learning across languages, right?

22:48That's also an important implication here is that like that allows you to learn something in a French text and then be able to like apply that same circuit that you learned in texts of other languages. So getting back to how this works and how the kind of tooling and methodologies you've developed to, you know, understand these models, a couple of key concepts are, you know, replacement model and sparse coding. Can you talk about, you know, the mechanisms a bit? Maybe I'll start with source coding because,

23:28so the idea is actually similar both for this new paper and for the sort of like dictionary learning papers. It's the same approach basically. And the idea is like, ah, you have, you know, say in the middle of the model, you have this like, as we said, it's like a very dense, you know, vector of numbers and and in that vector you're pretty sure you know are all these representations there's like the representative large the representation of like opposite but all you see is this like dense vector of numbers and what you want is you want to know ah like actually the model is thinking a little about this concept a and all about this concept b and all about this concept c so like this is just a method to get at that and the way that you get at this method is you say cool can i learn another a separate model that will take this like incomprehensible mass distance vector and that will learn a like essentially um you do like a matrix multiply to a like much larger space um and and then you uh take this space and that space you force to be very sparse and so you literally like have you know if you have let's say for example like a 100 neurons somewhere in the middle of your model so you have 100 numbers that are kind of like always active and like very hard to comprehend you blow it up to a thousand numbers and you have a constraint that says that like basically in your loss you penalize every time one of those numbers is not zero so the model is going to have to be like very very parsimonious in allocating these these features it's not going to be able to turn many of them on and and then so so that's like where the sparse comes from and that but then you have to make sure that you're actually capturing as much as you can so then what you do is you try to reconstruct the original thing from that's that code and so it's like you're learning a sparse code and then you're also like making sure that your sparse code is enough to capture like as much of the original uh thing as possible and so that's basically like the gist of dictionary learning you're training with this this mix of like sparsity and reconstruction error where it's like how how well can you reconstruct after you've blown it up and put it back together um and that that is sort of like what gives you these um these features which we which we call like monosematic meaning they kind of like represent only one thing um so that's sparse coding so okay so that that you could you you could do before and we did before for sort of like uh representations inside the model um maybe at this point like a useful mental model to have is is like i have it as like a tiramisu where you have like the input tokens at the bottom of the sentence, you know, so it's like, ah, the, you know, facts Michael Jordan plays the sport of is like at the bottom.

26:09And then the model has like successive layers where it's like, it transforms this into numbers and then it does some computation, basically like some math, okay, some matrix crossplications. And it has another set of vectors, one for each of the tokens. And then it does that again, it does that again, it does that again, it does that again. And eventually, you know, it does a final uh multiplication to still do the output and so what we're what we're doing what we're doing in the previous papers is like applying this method in between computations and seeing like okay it's a bunch of computations we're somewhere in the middle let's look at what the model is thinking about now basically um the problem with that is that you you can see what the model is thinking about but you kind of don't know what the computation is doing uh at all um you just you could see it to the extent that you're like well before it was thinking about you know uh bridge and gold and now it's thinking about the gold gate bridge and so probably like somewhere like there's some part of the model that like stitches together to understand this and so in circuit tracing we uh we use uh transcoders which uh basically what what those do is they do the same thing but you um you try to replace the computation with this sparse thing instead of just a representation so we're now just like completely replacing bits of the model with this like sparse thing um and we're trying to make it so that the computation stays the same but it's super sparse and understandable um and that's that's like the main idea of the replacement model is like you try to replace chunks of the base model with another model that's as close as possible while being much more explainable understandable in doing so you have several different choices for the type of replacement model you use cross-layer transcoders versus SAEs versus others can you talk a little bit about why you chose the approach that you chose so there's there's kind of like a space of trade-offs here um so you know going from essays which which are these like oh just do it on the neurons to um transcoders is kind of nice because that allows you to um one to like uh understand the computation rather than just representation but then also um once you've replaced these parts of the model you can then directly like connect features to features you can say like oh like you know this feature like at this layer uh writes in that direction because like now you've replaced a part of the models the feature is just like a computation it's like how the computation writes that direction and then this other feature reads in this direction and so actually they're not they don't talk to each other at all but those features go in the same direction and so they talk to another and so you can you can then like have very what does that mean what does directionality mean here you know in a concrete way.

29:12Sneak that one by here. So yeah so you know the way that here what we're replacing is MLP layers which are like multi-layer perceptrons and what they are is so you have this like representation which is 100 numbers and basically you have you let's just take like one neuron or one neuron of your MLP um which all it does is it like um does like multiplies your um your uh uh your hundred numbers by another hundred numbers to get like some some some float and then uh multiplies that by like a decoder uh and that decoder what is it it's also again a hundred numbers and so like you know the way that it's usually taught in like machine learning textbooks is you have like Like you have your neurons and you have like a matrix multiplication, another matrix multiplication.

30:13But another way to look at it is that each of those, all it's doing is it's saying like, I take the current vector of neurons. I like multiply it with another vector, which basically just like is kind of equivalent to seeing like how lined are there essentially, right? And then I take the result of that and I scale a second vector by that. And so it's like I read in that direction and I write to that direction. So I read in that direction. So that means that like, oh, if like there's stuff in that direction, it will activate this neuron. If there's not, the neuron will stay like zero. And then I write in that direction, meaning, you know, if I've activated this neuron, I'm going to write this other thing.

30:53So you can think of it geometrically as like trying to capture the similarity of two vectors and then projecting them off into another direction. Exactly. Exactly. That's exactly right. And so once you have that, then you can ask questions about like, ah, well then like how do those compose, right? So it's like, oh, I had one like earlier in the model that like computed the similarity projected in that direction. And then this other one is computing a similarity with this other direction. Are those directions aligned or are they not? And if they are aligned, then that means that like this feature will actually like activate this other feature.

31:32and so you'll see like oh you have a feature that like reads um you know i don't know like reads in the direction that's like the like mix like red and bridge and writes like golden and then you have another feature that like reads the golden direction and also is like gonna gonna survive and upweigh like things about the golden gate bridge um is is like the rough idea and so that's what transcoders buy you uh they buy you sort of like easier edges between your features which then is kind of what you want with circuits is you want to connect them all. Yeah, given the dimensionality that we're talking about, how do you even grok these directions?

32:10Are you applying dimensionality reduction types of techniques to understand what's happening here? Are there other ways that you're kind of getting at what direction means for these vectors? I mean, I think you just have to try hard to reason in 10 ,000 dimensional space. Um, it doesn't seem that hard. No. So, so, so the good thing is, um, we actually don't have to think about them. Um, that's, that's part of the pros of the method. And the reason we don't is that like, so, um, let me like walk through how you, how you actually use this in practice. You have a prompt, you've replaced a bunch of your models so that it has these like kind of like simpler or like more understandable, you know, bits of computation that like read in one direction and write another one.

32:56you take your prompt you pass it through you look at what these do right and all you do is you don't even like look at this like you know dimension whatever um vector you just do the the dot products to see if they're aligned and that gives you a number right kind of like when you're you know in geometry and so when when we show these like attribution graphs that's all they are they are is they're like a boiled down version of like very very high dimensional space where all we're showing is like each edge is basically like ah like what was the dot product between this like you know super big vector and that super big vector and it's like was it positive was it negative and that's just a number between you know one over minus infinity and plus infinity but it's just a number so you can you can understand that pretty easily and be like oh there's a bunch of strong edges from like the like uh you know golden to the like this is a bridge feature that makes sense um and so yes so yeah we don't have to we don't have to i mean there's a bunch of things you might want to do to understand these vectors and like you know um but but for the purpose of drawing this computational graph you you get to abstract over it which is which is really nice one thing you you will find though is that um it turns out that in these models there's actually a lot of like amplification and so when we're doing these these early experiments we would find that like a lot of our graphs were just very boring and what i mean by that is we have this example in the paper where you ask like the model basically to to tell you um i think it's like what's what's the capital of denmark or like copenhagen is the capital of and there's a copenhagen feature and then you're like oh i wonder what that feature is connected to and it turns out is connected to a Copenhagen feature in the second layer.

34:45And you're like, oh, I wonder what that goes to. And it turns out that that one is connected to a Copenhagen feature in the third layer. And so actually like the graph would just be like Copenhagen, Copenhagen, Copenhagen, Copenhagen, and then Denmark at the end. And that's because of the way these models work, they can't delete things. And so if you want to still be thinking about Copenhagen, you have to sort of like have every layer reinforced, be like, yeah, we're still thinking about Copenhagen. This is still the right thing we're thinking about. And so that's where our other method comes into play, this cross-layer thing, where we instead try to learn features that skip over all of these.

35:23And so instead of having 10 Copenhagens, you have one. And that's the idea between the cross-layer transcoder. But it's nice for that property, but I would say it's not vital for the method itself, if that makes sense. So you've got this method that you're using to replace the MLP blocks in the transformer. And that method results in a degree of introspectability, visibility into the way the model is thinking. One thought that I had was, it was kind of like an ad nauseum type of an argument. Like, you know, if you can do that, like it sounds almost like distilling the black box model to an interpretable model or whatever the opposite of distilling would be because this thing's going to have a lot more but more sparsely populated uh uh weights like could you just like up still it into you know a black box into like an understandable thing and like just use that thing uh and i'm really asking this to try to get that like a concrete understanding of this process?

36:42Like, you know, would it be too slow or like, you know, is it not really, I know you're doing some alignment between the replacement models and the actual model to ensure that you're, you know, the results are, you know, meaningful, right? But is it, you know, not very accurate after you've, you know, done this, this training process um are you only doing a small number of m uh mlps at a time in a larger model or you know and you couldn't possibly from a computational perspective do all of them like i'm trying to get at like some practicalities of the approach yeah yeah uh those are great questions so i mean i will say like ignoring sort of you know money or physics for a second which is always fun um if if we you know given like unlimited money and like probably like limited infinite data and stuff like that you could totally imagine yeah doing you know okay great let's like replace the whole model with this interpretable alternative and then let's run it instead and then and then we just get to to to we just get a free lunch basically where we have something that's as good as the base model uh and also we can understand it now in actuality you know as you said it's it's more like upstilling than distilling so like because all of the layers are way wider even if i could give you this perfect model actually be really expensive to to serve right because it's like it just like is a way bigger object um than than the base model there's also um kind of like a question of of training costs so in our in our biggest haiku uh 3.5 runs were 30 million features and those are those are pretty big runs um but if you think of the number of features as a feature is multiplicative of the number of parameters it's a vector as opposed to a that's right well right and if you're doing the per layer thing then like a feature is sort of like an encoder and decoder so like two vectors but if you're doing this like cross layer thing the way that it works is that a feature like is like one encoder and then like n layers decoders so it's actually quite expensive um but that sort of like is still worth it because you need fewer features because you're like deleting a bunch of redundant features so there's sort of like that's why it's still worth doing doing but anyways yeah our feature is like when we're on the order of like you know one maybe like close to let's say like what a what a neuron and mlp would cost you um and you know 30 million of them is is is a lot but um think about how many independent concepts the model knows uh and so there's this sort of like we call it the like dark matter problem in the paper there's a sort of problem that you know how many features would we need so that we captured everything that cloud knows um currently using our methods that's probably in the order of like billions maybe uh and so that's like quite infeasible with with sort of like just today's technology it would cost way too much to train it'd be completely impossible to like even study it's a giant monster um and so there's a sense in which in order to capture everything in the model is doing and so to both explain it fully, but also to be able to kind of like reconstruct it accurately.

40:05We need so many features that right now, this is sort of still basically an unsolved problem. I think one thing that's relatively clear, and again, we talked about in paper is we're going to need to sort of like think of smart ways to, you know, get features more efficiently or get features more efficiently only like a domain that we care about. Maybe we don't care to explain how Cloud does everything. Maybe we only care to explain how cloud does math or how cloud does like pie or something that buys us a lot. But also there's a bunch of methods that you could imagine using to just like be more efficient with your features that we discussed a little bit.

40:44So I think that the sort of like summary answer to your question is that's the dream. I think that right now the sort of like realistic state we're in is actually we can explain what some of what the model is doing some of the time. And I think the sort of like first step after that is like, ah, can we just keep increasing the proportion of times that we kind of like understand what's going on? And I don't know if we'll ever get to the dream of we just replace the model with the interpretable model. But I think even if we just manage to sort of like understand a lot more, we'll get a lot of value here.

41:23There was some discussion that I found really interesting talking about kind of one of the challenges or confounders being these ideas of polysemanticity and superposition. Talk a little bit about those ideas. So I guess like one thing you might think is like, why do we need to do this at all? The model already has neurons. We could just use them and then do our graphs based on that. And it turns out that if you just look at the neurons and you try to understand them the way that we try to understand features, which is like, you know, run it on a bunch of text and see what makes it light up and see if when you look at that pattern, it's obvious what it does.

42:05So for features like the Golden Gate Bridge feature, it's like if you do that, you run it on like a bunch of text and you highlight the parts where it lights up. It's like, oh, it's like, you're literally just highlighting the text to Golden Gate Bridge and a bunch of text and everything else. It's just very clear. Most neurons are not like that at all. Most neurons, you look at them and you're sort of like, ah, like it seems to fire on sevens sometimes but only if there was like russian language before and also you know like german cookbooks and or or like even that is actually like kind of understandable it's like mostly just looks kind of random um and so that's because um actually like each neuron kind of like isn't a meaningful unit for for the model and the model just like uses a bunch of neurons in parallel and stores serve like representations over all of them and so those neurons are what we like polysemantic they just like don't have a single meaning whereas the features are monosemantic they are like you can say like ah this is the michael jordan feature if it like has a very high activation that means the model is thinking a lot about michael jordan if it doesn't it's not um and so the idea of superposition then is well like how do you how do you reconcile this idea that there are features that are sort of like very meaningful and very sparse with the fact that if you look inside the model it's it's like a very dense thing where like every kind of like you know in in the residual stream of the model which is where the kind of information uh it's processed just everything is on all the time there's no sparsity um and that's because at least the hypothesis is that it's because of uh essentially like superposition what's happening is that the model is like storing many more features than it has dimensions in superposition and like squishing them tightly such that when you look at it it's just this like mess uh but actually what's happening is that like each feature is sort of like you know smeared across like many dimensions and our work is to decode that smearing into these like abstract monosemantic things, basically like untangle the superposition to get the monosemantic representations is like kind of the name of the game here.

44:22We've touched on limitations of the approach throughout. One that jumped out to me is the idea that by focusing on kind of the replacement of the MLP layers, you know, you're missing out on another big source of information, which is the attention mechanism um can you talk about that and more broadly like other limitations totally yeah i think i mean you know attention is the biggest one by far um and so there's there's a lot of stuff i mean there are many other limitations so i guess we touched on one which was this like dark louder which is if you just extrapolate how many features we have and how many features we need we're just not going to get there so we need to do work to sort of like find a way to get more efficiently so that's one attention is is is is a huge one because there are many things that the model does that are basically almost entirely done via attention um maybe you know a common example uh is like i was gonna say induction which but that's like a induction is when the model just like looks back at previous instances of the word it's processing and looks at what was said after that word and then just says the same thing.

45:40That's maybe like, that's interesting, but actually like pretty well understood. But there's a lot of stuff we don't understand or at least it's harder to understand well, where, you know, you can ask like a bunch of questions around, maybe a better example is like multiple choice questions where like the model has to, you know, you ask it a multiple choice question and then at some point it has to say like, well, it's answer B. and the way that it says it's answer b is that there's an attention head that looks at b and then like writes it at the end and if you look if you if you use our method that's basically all you see you're like ah great like the complete story of what happened on this prompt is that Claude somehow knew that the answer was b and then like an attention head like moved it there but of course the interesting question is like well why did this attention head look at b like there's some computation that happens in the model that tells it that the correct answer was b and then like it looks at b and it copies it and currently we just completely sidestep that um we sort of like explain part of attention or at least we like we we explain how attention moves the information from mlps but we don't explain why uh attention patterns were the way they were and that's like a huge part of how the model process information is like how did it decide to attend from x to y is something that we completely miss.

47:02And so, you know, obviously one of the main limitations and kind of aspects that we're interested in working on and hearing from other groups working on it. I think solving that is by far sort of like the most exciting thing on my list in terms of kind of like next work. You know, I don't think it would be easy to be clear, but I think it's doable. And why is it so difficult? Is it is attention kind of a denser mechanism? Is it more, you know, opaque somehow? Like, you know, they're all numbers somewhere. Like what makes attention more difficult to kind of to pick apart? Can you train some other, like is there an analogy, like a straightforward analogy of like a sparse replacement attender that, you know, is conceptual or does the nature of attention, you know, cut off avenues like that?

47:59yeah i think just sort of like a few reasons why attention is hard but none of them are fundamental like you totally could train some sparse replacements uh for like parts of attention i think um attention is just like different in a few ways maybe like one way is that um the other aspects we replace are these sort of like maybe like univariate is the wrong word but distance you know it's like you take the state of the model and then you do your like you apply your mlp and then you write the result and so you can replace that it's like a very clean computation but attention patterns specifically are the combination of what's happening at all of the key positions and all of the query positions and the interactions between both of those and so you had some you now have something that's like quadratic basically and so you have to sort of somehow like find a way to get around that and there are there are sort of like proposed approaches um and i think some of them are are you know likely to work or at least i hope they are so a lot of the limitation is computational bounds for the same reason attention is hard uh you know to scale you know for for context windows and stuff like that exactly so that's that's one of them i would say another one too is that like um there's also a uh you know like like non-linearity over attention like it's not just so you're doing like k cube and then you're also doing like your non-linearity which then like that just means that it makes your decomposition efforts like you have to think carefully about because um in at least in like our graphs everything because of our replacement we've managed to make everything linear so you can just literally say like ah this feature affects that one in that way but here it's sort of like ah yeah like this this key effect kind of obscures relationships exactly but like actually what really matters that that one was much stronger.

49:49And so actually they're sort of like, you didn't really have an effect at all. Like you are getting squished by the non-linearity. So I don't know, I guess like, maybe another way to say it is that

50:05replacing MLPs first here, this work, just gave us enough sort of like valuable empirical insight and valuable work to do that it felt like the first thing to do. um and i think you know there's similar papers in the field that sort of like have various other like they use transcoders as well um and that have found success and then you know it's kind of like now that you've solved this or at least have found you know maybe not solved but found some progress well um yeah uh now you see the sort of like like the next thing becomes the gaping hole right like now that you oh it's like oh we've patched this hole like this makes sense and it's like oh and now attention is this huge thing it's like why haven't you done anything about attention it's It's like, well, we were starting here, you know, and so, but I think it's basically the next kind of like exciting thing.