The Utility of Interpretability — Emmanuel Amiesen

38:39And so unsupervised methods often still have labels in the end. Sometimes I feel like the term... You form labels by masking. Yeah, like for pre-training, right? It's like the next token. So in that sense, you have a supervision signal. And here, the supervision signal is simply you take the neurons neurons and then you learn a model that's going to like expand them into like the actual number of concepts that you think there are in the model so you have two neurons you think there's five concepts so you expand it to like a thing of dimension five and then you contract it back to what it was that's like the model you're training and then you're training it to incentivize it to be sparse so that there's only like a few features active at a time and once you do that if it works you have this sort of like nice dictionary which you can think as like a way to decode deactivate the neurons where you're saying like ah cool i don't know what this what this direction means but i've like used my model into telling me that the model is writing in the red direction and so that's that's sort of like i think maybe the biggest thing to understand is is this combination of things like ah we have two few dimensions we pack a lot into it so we're going to learn an unsupervised way to like unpack it and then analyze what each of those dimensions that we've unpacked are any folks yeah i mean the follow-ups of this are also kind of like some of the work that you did is in clamping, right?

39:55What is the applicable side of Mechinterp, right? So we saw that you guys have like great visualizations. Golden Gate Cloud was a cool example. I was going to say that. Yeah, it's my favorite. What can we do once we find these features? Finding features is cool, but what can we do about it? Yeah. I think there's kind of like two big aspects of this. Like one is, yeah. Okay. So we go from a state where, as I said, the model is like a mess of weights. We have no idea what's going on to okay we found features we found a feature for red a feature for golden gate cloud or for the golden gate bridge i should say like what do we do with them and well if these are true features that means that like they in some sense are important for the model or it wouldn't be like representing it like if the model is like bothering to like write you know in the golden gate bridge direction it's usually because it's going to like talk about the golden gate bridge and so that means that like if that's true then you can like set that feature to zero or artificially set to 100 and you'll change model behavior that's what we did when we did golden gate claude in which we found a feature that represents the direction for the golden gate bridge and then we just like set it to always be on and then you could talk to claude and be like hey like what's on your mind you know like what are you thinking about today be like the golden gate bridge he'd be like hey claude like what's two plus two it'd be like four golden gate bridges uh etc right and it was always thinking about this like write a poem and it just starts talking about how it's like red, like the Golden Gate Bridge.

41:18Yeah, that's right. I think what made it even better is like, we realized later on that it wasn't really like a Golden Gate Bridge feature. It was like being in awe at the beauty of the majestic Golden Gate Bridge, right? So on top of that, I would like really ham it up. You'd be like, oh, I'm just thinking about the beautiful international orange color of the Golden Gate Bridge. That was just like an example that I think was like really striking, but of sort of like, oh, if you found like a space where that represents some computation or sort of representation of the model that means that you can like artificially suppress or promote it and that means that like you're starting to understand at a very high level a very gross level like how some of the model works right we've gone from like i don't know anything about it to like oh i know that this like combination of neurons is this and i'm going to prove it to you the next step which is what this this works on is like that's kind of like thinking of if maybe you take the analogy of like um i don't know like like let's take the analogy of like an mri or something like a brain scan, it tells you like, oh, like this, as Claude was answering, at some point, it thought about this thing.

42:20But it's a sort of like vague, like basically, maybe it's like a bag of words, kind of like a bag of features. You just have like, here are all the random things it thought about. But what you might want to know is like, okay, but Claude is doing some processing, like sometimes to get to the Golden Gate Bridge, it had to realize that you were talking about San Francisco and about like the best way to go to Sonoma or something. And so that's how it got to Golden Gate Bridge. So there's like an algorithm that leads to it at some point thinking about the Gongit Bridge. And basically, there's a way to connect features to say, oh, from this input, went to these few features, and these few features, and these few features, and that one influenced this one, and then you got to the output.

42:53And so that's the second part. And the part we worked on is you have the features, now connect them in what we call, or what's called circuits, which is sort of like explaining the algorithm. Yeah. Before we move directly onto your work, I just want to give a shout out to Neil Nanda. He did Neuronpedia and released a bunch of SAEs for, I think, the Lama models and the Gemma models. And the Gemma models, yeah. So I actually made Golden Gate Gemma. Just upped the weights for proper nouns and names of places for people and references to the term golden, likely relating to awards, honors, or special names.

43:26And that together made Golden Gate. That's amazing. Yeah. So you can make Golden Gate Gemma. And I think that's a fun way to experiment with this. But yeah, we can move on to... I'm curious. I'm curious. What's the background behind why you ship Golden Gate Claw? Like you had so many features, just any fun story behind why that's the one that made it. You know, it's funny. If you look at the paper, there's just a bunch of like, yeah, like really interesting features, right? There's like one of my favorite ones was the psychophantic praise, which I guess is very topical right now. Very topical.

43:58But, you know, it's like you could dial that up and like Claude would just really praise you. You'd be like, oh, you know, like I wrote this poem like roses are red, violets are blue, whatever. and he'd be like that's the best poem i've ever seen um and so we could have shipped that that could have been funny uh golden gate cloud it was like a pure as far as i remember at least like a pure just like weird random thing where like somebody found it initially with an internal demo of it everybody thought it was hilarious and then that's sort of how it came on there was no nobody had a list of top 10 features we should consider shipping and we picked that one there was just kind of like a very organic moment no like the the marketing team really leaned into into it.

44:38They mailed out pieces of the Golden Gate for people in Europe's, I think, or ICML. Yeah, it was fantastic marketing. The question obviously is if OpenAI had invested more in interpretability, would they have got the GPT-4O update? But we don't know that for sure because they have interp teams. I think also for that one, I don't know that you need interp. It was pretty clear cut. I was like, oh, that model is really gassing me up. And then the other thing is, can you just up, write good code, don't write bad code and make Sonday 3.5. And like it feels too easy, too free. Is steering that powerful that you can just like up and down features with no trade-offs?

45:19There was like a phase where people were basically saying, you know, 3.5 and 3.7 are just now, because they came out right after this. And for the record, like that's been debunked. Yeah, it has been debunked. But you know, it had people convinced that what people did is they basically just steered up and steered down features and now we have a better model. And this kind of goes back to that original question of, right? Like Like, why do we do this? What can we do? Some people are like, I want tracing from a sense of, you know, legality. Like, what did the model think when it came to this output?

45:46Some people want to turn hallucination down. Some people want to turn coding up. So, like, what are some, like, whether it's internal, what are you exploring that, like, what are the applications of this? Whether it's open-ended of what people can do about this or just, like, yeah, why do Mechinterp, you know? Yeah. There's, like, a few things here. So like, first of all, obviously, this is, I would say, on the scale of the most short term to the most long term, like pretty long term research. So in terms of like applications compared to, you know, like the research work we do on like fine tuning or whatever, Interp is much more, you know, sort of like a high risk, high reward kind of approach.

46:23With that being said, like, I think there's just a fundamental sense in which Michael Nielsen had a post recently about how like knowledge is dual use or something. but just like just like knowing how the model works at all feels useful and you know it's hard to argue that if we know how the model works and understand all the components that won't help us like make models that hallucinate less for example or they're like less biased that seems you know if if like at the limit yeah that totally seems like something you could do using basically like your understanding of the model to improve it i think for now as we can talk about a little bit with like uh circuits there's like we're still pretty early on in the game right and so right now the main way that we're using interprivilege is like to investigate specific behaviors and understand them and gain a sense for what's causing them so like one example that we we can talk about later we can talk about now but in the paper we investigate jailbreaks and we try to see like why does a jailbreak work and then we realize as we're looking at this jailbreak that part of the reason why claude is telling you how to make a bomb in this case is that it's like already started to tell you how to make a bomb and it would really love to stop until you're going to make a bomb, but it has to first finish its sentence.

47:34Like it really wants to make correct grammatical sentences. And so it turns out that like seeing that circuit, we were like, ah, then does that mean if I prevent it from finishing its sentence, the jailbreak works even better? And sure enough, it does. And so I think like the level of sort of practical application right now is of that shape. So like understanding either like quirks of a current model or like how it does tasks that maybe we don't, we don't even know how it does it. Like, you know, we have like some planning examples where we had no idea it was planning and we're like, oh God, it is.

48:04That's sort of like the current state we're at. I'm curious internally how this kind of feeds back into like the research, the architecture, the pre-training teams, the post-training. Like, is there a good feedback loop there? Like right now there's a lot of external people interested, right? Like we'll train an SAE on one layer of llama and probe around. But then people are like, okay, how does this have much impact? People like clamping. But yeah, as you said, you know, once you start to understand these models, have this early planning and stuff, how does this kind of feed back? I don't know that there's much to say here other than like, I think we're definitely interested in conversely, like making models for which it's like easier to interpret them.

48:42So that's also something that you can imagine sort of like working on, which is like making models where you have to work less hard to try to understand what they're doing. The architecture? Okay. Yeah, so I think there was a less wrong post about this of like, there's a non-zero amount of sacrifice you should make in current capabilities in order to actually make them more interpretable because otherwise they will never catch up. You know, there's this sort of sense in which like right now we take the model and then the model's the model and then we post hoc do these replacement layers to try to understand it.

49:10But of course, when we do that, we don't like fully capture everything that's happening inside the model. We're capturing like a subset. And so maybe some of it is like, you could train a model that's easier to interpret naively. And it's possible that you don't even have that much of attacks in that sense. And you can just sort of either train your model differently or do a little post hoc step to sort of untangle some of the mess that you've made when you trained your model, make it easier to interpret. The hope was pruning would do some of that. But I feel like that area of research has just died.

49:40What kind of pruning are you thinking of here? Just pruning your network. Ah, yeah. Yeah. Pruning layers, printing connections, whatever. Yeah. I feel like maybe this is something where like superposition makes me less hopeful or something. Because you don't know. Like that like seventh bit might hold something. Well, right. And it's like on each example, maybe this neuron is like at the bottom of like what matters. But actually it's participating like 5 % to like understanding English, like doing integrals and, you know, like whatever, like cracking codes or something. And it's like, because that was just like distributed over it, you kind of like, when you naively prune, you might miss that.

50:20I don't know. Okay. So then this area of research in terms of creating models that are easier to interpret from the start, is there a name for this field of research? I don't think so. And I think this is like very early and it's mostly like a dream. Just in case there's a thing people want to double click on. Yeah. I haven't come across it. I think the higher level is like Dario recently put out a post about this, right? Why Mac Interpret is so important. important. We don't want to fall behind. We want to be able to interpret models and understand what's going on. Even though capabilities are getting so good, it kind of ties into this topic.

50:53We want models to be slightly easier to interpret so we don't fall behind so far. Well, yeah. And I think here, just to talk about the elephant in the room or something, one big concern here is safety. And so as models get better, they are going to be used more and more places, you know, it's like, you're not going to have your, you know, we're vibe coding right now. Maybe at some point, well, that'll just be coding. It's like, Claude's going to write your code for you and that's it. And Claude's going to review the code that Claude wrote and Claude's going to deploy to production. And at some point, like as these models get integrated deeper and deeper into more and more workflows, it gets just scarier and scarier to know nothing about them.

51:29And so you kind of want your ability to understand the model to scale with like how good the model is doing, which that itself kind of like tends to scale with like how widely deployed it is. So as we like deploy them everywhere, we want to like understand them better. The version that I liked from the old super alignment team was weak to strong generalization or weak to strong alignment, which that's what super alignment to me was. And that was my first aha moment of like, oh yeah, at some point these things will be smarter than us. And in many ways they already are smarter than us. And we rely on them more and more.

52:01We need to figure out how to control them. This is not an Eliezer-Yukowski thing. It's just more like, we don't know how these things work. How can we use them? Yeah. And you can think of it as, there's many ways to solve a problem. And some of them, if the model is solving it in a dumb way or in memorized one approach to do it, then you shouldn't deploy it to do a general thing. You could look at how it does math. And based on your understanding of how it does math, you're like, okay, I feel comfortable using this as a calculator. or like, no, it should always use a calculator tool because it's doing math in a stupid way and extend that to any behavior, right?

52:36Where it's just a matter of like, think about it if like you're like in the 1500s and I give you a car or something and I'm just like, cool, like this thing, when you press on this, like it accelerates. When you press on that, like it stops, you know, this steering wheel seems to be doing stuff, but you knew nothing about it. I don't know if it was like a super faulty car and it's like, oh yeah, but if you ever went above 60 miles an hour, like it explodes or something, like you probably would be sort of like, you'd want to understand the nature of the object before like jumping in in it and so that's why we like understand how cars work very well because we make them llms are sort of like ml models in general are like this very rare artifact where we like make them but we don't we have no idea how they work we evolve them we create conditions for them to evolve and then they evolve and we're like cool like you know maybe you got a good run maybe we didn't yeah don't really know yeah the extent to which you know how it works is you have your like eval and you're like oh well seems to be doing well on this eval?

53:27And then you're like, is it because this was in a training set? Or is it actually generalizing? I don't know. My favorite example was somehow C4, the Colossal Clean corpus, did much better than Common Crawl, even though it filtered out most of this. It was very prudish. So it filters out anything that could be considered obscene, including the word gay. But somehow, when you add it into the data mix, it just does super well. And it's just like this magic incantation of like, this recipe works. Just trust us. We've tried everything. This one works. So just go with it. Yeah. It's not very satisfying.

54:03No, it's not. The side that you're talking about, which is like, okay, how do you make these? And it's kind of unsatisfying that you just kind of make the soup and you're like, oh, well, my grandpa made the soup with these ingredients. I don't know why, but I just make the soup the way my grandpa said. And then like one day somebody added cilantro. And since then, we've been adding cilantro for generations. And you're like, this is kind of crazy. That's exactly how we train models though. Yeah. Yeah. Yeah. So I think there's a part where it's like, okay, let's try to unpack what's happening, the mechanisms of learning, how our models are learning.

54:32I guess we skipped over it, but one of the things were induction heads, understanding what induction heads are, which are attention heads that allow you to look at in your context the last time that something was mentioned and then repeat it. It's something that seems to happen in every model. It's like, oh, okay, that makes sense. That's how the model is able to repeat text without dedicating too much capacity to it. Let's get it on screen so people can see. Visuals of the work you guys put out is amazing. we should talk a little bit about the behind the scenes of that kind of stuff. But let's finish this off first.

55:03Totally. But just really quickly, I don't think we should spend too long on it. I think it's just like, if you're interested in Mechinterp, we talked about superposition and I think we skipped over induction heads and that's like, you know, kind of like a really neat basically pattern that emerges in many, many transformers where essentially they just learn. Like one of the things that you need to do to like predict text well is that if there's repeated texts at some point somebody said emmanuel mason and then you're like on the next line and they say emmanuel very good chance it's the same last name and so one of the first things that models learn is just like okay i'm just gonna like look at what was said before and i'm gonna say the same thing and that's induction heads which is like a pair of attention heads that just basically look at the last time something was said look at what happened after move that over and that's an example of a mechanism where it's like cool now we understand that pretty well there's been a lot of follow-up research on understanding better like okay like in which context do they turn on like you know there's like different like levels of abstraction there's like induction heads that like literally copy the word and there's some that copy like the sentiment and other aspects but i think it's just like an example of slowly unpacking you know or like peeling back the layers of the onion of like what's going on inside this model okay this is a component it's doing this so induction heads was like the first major finding it was a big finding for nlp models for sure i often think about the edit models so claude has a fast edit mode i forget at what it's called.

56:17OpenAI has one as well. And you need very good copying, every area that needs copying. And then you need it to switch out of copy mode when you need to start generating. Right. And that is basically the productionized version of this. Yeah. Yeah. And it turns out that, you know, you need to set like a model that's like smart enough to know when it needs to get out of copy mode, right? Yeah. It's fascinating. It's faster, it's cheaper. You know, as bullish as I am on Canvas, Basically, every AI product needs to iterate on a central artifact. And if it's code, if it's a piece of writing, it doesn't really matter.

56:50But you need that copy capability that's smart enough to know when to turn it off. That's why it's cool that induction heads are at different levels of abstraction. Sometimes you need to, in editing some code, you need to copy the general structure. It's like, oh, this other function that's similar, it first takes, I don't know, abstract class, and then it takes an int. So I need to copy the general idea, but it's going to be a different abstract class and a different int or something. Yeah, cool. So tracing? Oh, yeah. Should we jump to circuit tracing? Sure. I don't know if there's anything else you want to cover.

57:20No, no, no. We got space for it. Maybe, okay, I'll do like a really quick TLDR of these two recent papers. Okay. Insanely quick. So we talked about these features that we detect. And what we said is like, okay, but we'd like to connect the features to understand like the inputs to every features and the outputs to every features and basically draw a graph. And this is like, if I'm still sharing my screen, the thing on the right here, where like, that's the dream we want, like, for a given prompt, what were all of the things like all of the important things happen in the model. And here's like, okay, it took in these four tokens, those activated these features, these features activate these other features, and then these features activate the other features.

57:57and then all of these like promoted the output. And that's the story. And basically we're like, the work is to sort of use dictionary running and these replacement models to provide a explanation of like sets of features that explain behavior. So this is super abstract. So I think immediately, maybe we can like just look at one example. I can show you one, which is this one. Ah, the reasoning one. Yep. Yeah, two-step reasoning. I think this is already, this is like the introduction example, but it's already like kind of fun. So the question is, you ask the model something that requires it to take a step of reasoning in its head.

58:30So you say, you know, fact, the capital of the state containing Dallas is. So to answer that, you need one intermediate step, right? You need to say, wait, where's Dallas? Isn't Texas? Okay, cool. Capital of Texas, Austin. And this is like in one token, right? It's going to after is, it's going to say Austin. And so like, in that one forward pass, the model needs to extract to realize that you're asking it for like the capital of a state to like look up the state for Dallas. which is Texas, and then to, say, Austin. And sure enough, this is, like, what we see is, like, in this forward pass, there's a rich sort of, like, inner set of representations where there's, like, it gets capital state in Dallas, and then, boom, it has an inner representation for Texas, and then that plus capital leads it to, like, say, Austin.

59:13I guess one of the things here is, like, we can see this internal, like, thinking step, right? But a lot of what people say is, like, is this just memorized fact, right? Like, I'm sure a lot of the pre-training that this model is trained on is this sentence shows up pretty often, right? So this shows that, no, actually internally throughout, we do see that there is this middle step, right? It's not just memorize. You can prove that it generalized. Yeah, so that's exactly right. And I think like you hit the nail on the head, which is like, this is what this example is about. It's like, ah, if this was just memorized, you wouldn't need to have an intermediate step at all.

59:46You'd just be like, well, I've seen the sentence. Like, I know it comes back, right? But here there is an intermediate step. And so you could say like, okay, well maybe it just has the step but it's memorized it anyways and then the way to like verify that is kind of like what what we do later in the paper and for all of our examples is like okay we claim that this is like the texas representation let's get another one and replace it and we just change like that uh feature in the middle of the model and we change it to like california and if you change it to california sure enough it says sacramento and so it's like this is not just a like byproduct like it's memorized something and on the side it's thinking about texas it's like no no no this is like a step in the reasoning if you change that intermediate step it changes the answer very very cool work underappreciated yeah okay sure i have never really doubted i think there's a lot of people that are always criticizing llm's stochastic parrots this pretty much disproves it already like we can move on yeah i mean i i think i think there's a lot of examples that i will say we can go through like a few of them like show an amount of depth in the intermediate states of the model that makes you think like oh gosh like it's doing a lot i think maybe like the poems well definitely the poems but even for this one i'm gonna like scroll in this very short paper so like uh medical diagnoses i don't even know the word count because there's so many like embedded things in there yeah it's too dangerous we can't look it up it overflows um it's so beautiful look at this uh this is like a medical example that i think shows you again this is in one forward pass the model is like given a bunch of symptoms and then it's asked not like hey what is what is the like disease that this person has it's asked like if you could run one more test to determine it what would it be so it's even harder right it means like you need to take all the symptoms then you need to like have a few hypotheses about what the disease could be and then based on your hypothesis say like well the thing that would like be the right test to do is X.

1:01:41And here you can see these three layers, right? Where it's like, again, in one forward pass, it has a bunch of like, oh, these are symptoms. Then it has the most likely diagnosis here, then like an alternate one. And then based on the diagnosis, it like gives you basically a bunch of things that you could ask. And again, we do the same experiments where you can like kill this feature here, like suppress it. And then it asks you a question about the second option it had. The reason I show it is like, man, that's like a lot of stuff going on. For one forward pass, right? It's like specifically if you expected it to like, oh, what it's going to do is it's just like seeing similar cases in the training.

1:02:15It's going to kind of like vibe and be like, oh, I guess like there's that word and it's going to say something that's related to like, I don't know, headache, you know, like kind of like really heavily. It's like, no, no, no. It's like activating many different distributed representations, like combining them and sort of like doing something pretty complicated. And so, yeah, I think it's funny because in my opinion, that's like, yeah, like, oh, God, stochastic parrots is not something that I think is like appropriate here. And I think there's just like a lot of different things going on. And there's like pretty complex behavior.

1:02:44At the same time, I think it's in the eye of the beholder. I think like I've talked to folks that have like read this paper and I've been like, oh, yeah, this is just like a bunch of kind of like heuristics that are like mashed together. Right. Like the model is doing like a bunch of kind of like, oh, if high blood pressure than this or that. And so I think there's there's sort of like an underlying question that's interesting, which is like, OK, now we know a little bit of how it works. This is how it works. Like, now you tell me if you think that's like impressive, if you think that like, if you trust it, if you think that's sort of like something that is sufficient to like ask it for medical questions or whatever.

1:03:14I think it's a way to adversarially improve the model quality. Yeah. Because once you can do this, you can reverse engineer what would be a sequence of words that to a human makes no sense or lets you arrive at the complete opposite conclusion, but the model still gets tripped up by. Yeah. And then you can just improve it from there. Exactly. And this gives you a hypothesis about like, you like specifically imagine if like one of those was actually the wrong symptom or something. You'd be like, oh, it's weird that the liver condition like, you know, upweighs this other example. That doesn't make sense.

1:03:46Okay, let's like fix that in particular. Exactly. You sort of have like a bit of insight into like how the model is getting to its conclusion. And so you can see both like, is it making errors, but also is it using the kind of reasoning that will lead it to errors? There's a thesis. I mean, now it's very prominent with the reasoning model. about model depth. So you're doing all this in one pass. Yeah. But maybe you don't need to because you can do more passes. Sure. And so people want shallow models for speed, but you need model depth for this kind of thinking. Yeah. Is there a Pareto frontier?

1:04:21Is there a direct trade-off? Yeah. What would you prefer if you had to make a model and shallow versus deep? There's a chain of thought faithfulness example. Before I show it, I'm just going to go back to the top here. So when the model is sampling many tokens, if you want that to be your model, you need to be able to trust every token it samples. So like the problem with with models being autoregressive is that like if they like at some point sample a mistake, then they kind of keep going conditioned on that mistake. Right. And so sometimes like you need backspace tokens or whatever. Yeah. And error correction is like notably hard.

1:04:57If you have a deeper model, maybe you have fewer COT steps, but your steps are more likely to be robust or correct or something. And so I think that's one way to look at the trade-off. To be clear, I don't have an answer. I don't know if I want a wide or a shallow or a deep model. You definitely want shallow for inference speed. Sure, sure, sure. But you're trading that off for something else, right? Because you also want a 1B model for inference speed, but that also comes at a cost, right? It's less smart. There's a cool quick paper to plug that we just covered on the paper club. It's a survey paper around when to use reasoning models versus dense models.

1:05:29What's the trade-off? I think it's the economy of... Reasoning economy. The reasoning economy. So they just go over a bunch of ways to measure this, benchmarks around when to use each. Because yeah, we don't want to... Also, consumers are now paying the cost of this, right? But a little side note. Yeah. For those on YouTube, we have a secondary channel called Latentspace TV where we cover that stuff. Nice. That's our paper club. We covered your paper. Cool. yeah i think you brought up the like planning thing maybe it's worth let's do it yeah i think i think this one is like if you think about okay so you're going into the chain of thought faithful this one let's give this one let's just do planning so if you think about like you know common questions you have about models the first one we we kind of asked was like okay like is it just doing this like vibe based one shot pattern matching based on existing data or does they have like kind of rich inner representations it seems to have like these like intermediate representations that make sense as the abstractions that you would reason through okay so that's one thing And there's a bunch of examples.

1:06:25We talked about the medical diagnoses. There's like the multilingual circuits is another one that I think is cool where it's like, oh, it's sharing representations across languages. Another thing that you'll hear people mention about language models, which is that they're like next token predictors. Also for a quick note, for people that won't dive into this super long blog post, I know you highlighted like 10 to 12. So for like a quick 15, 30 second, what do you mean by their sharing thoughts throughout? Just like what's a really quick high level just for people to... Yeah, the really quick high level is that what we find is that, here, I'm going to show you a really quick, inside the model, if you look at the inner representations for concepts, you can ask the same question, which I think in the paper, the original one we asked is the opposite of hot is cold.

1:07:10But you can do this over a larger data set and ask the same question in many different languages. And then look at these representations in the middle of the model and ask yourself, well, when you ask it, the opposite of hot is, and le contraire de chaud est, which is the same sentence in French, Show off. Is it using the same features? Or is it learning independently for each language? It kind of would be bad news if it learned independently for each language. Because then that means that as you're pre-training or fine-tuning, you have to relearn everything from scratch. So you would expect a better model to kind of share some concepts between the languages it's learning.

1:07:42And here we do it for language languages. But I think you could argue that you'd expect the same thing for programming languages. Where it's like, oh, if you learn what an if statement is in Python, maybe it'd be nice if you could generalize that to Java or whatever. And here we find that basically you see exactly that. Here we show like, if you look inside the model, if you look at the middle of the model, which is the middle of this plot here, models share more features. They share more of these representations in the middle of the model. And bigger models share even more. And so the sort of like smarter models use more shared representations than the dumber models, which might explain part of the reason why they're smarter.

1:08:16And so this was like sort of this other finding of like, oh, not only is it like having these rich representations in the middle, it like learns to not have redundant representations. Like if you learn the concept of heat, you don't need to learn the concept of like French heat and Japanese heat and Colombian. Like you just, that's just the concept of heat. And you can share that among different languages. I feel like sometimes overanalyzing this becomes a bit of a problem, right? Like when we talked about with the medical example, we could look back and try to fix this in data set. So in language, I don't remember if it was OpenAI or Anthropic, where they basically they said when the model switched languages and they pass it to fluent users, they said, oh, this feels like an American that's speaking this language, right?

1:08:56So at some times there are nuances in a slightly different representation, right? So you don't want to over-engineer these little fixes when you do see them. But then the other side of this is like for those tail end of languages, right? For languages that models aren't good at. And for those like, you know, when you want to kind of solve that last bit, it seems like it's pretty plausible that we can solve this because these concepts can be shared across languages as long as we can fill in some level of representation, unless I'm wrong. No, totally. And I think this sort of stuff also explains, language models are really good at in-context learning.

1:09:34You give them something completely new, they do a good job. It's like, well, if you give them a new fake language and you in that language explain that cold means this and hot means that, you know like presumably they're able to as we clear the speculation we don't show in the paper google's done this okay great yeah they took a low resource language dumped it in a million token context and then it came up that's right that's right well i guess the thing that the thing to be curious to see is like okay does it use does it reuse these representations i bet that it probably does right and that's probably like a reason why it works well is like well it can reuse the representation the general representations that it's learned in other languages yeah this is like i don't have you talked to any linguistics people not recently linguistics researchers will be very interested in this because ultimately this is the ultimate test of sapir wharf um which are you familiar with so for those who don't know it's basically the idea that the language that you speak influences the way you think which obviously directly maps onto here if every if it's a complete mapping if every language maps every concept perfectly on in like the theoretical infinitely sized model, then Superior Wharf is false because there is a universal truth.

1:10:40If it does not, if there is some overlap where, for example, there's some languages that have no word. This is a joke where like, I mean, you know, Eskimos have no word for snow or something like that, right? Or water has no word. Fish have no word for water. There's an African language where there's a gender for vegetables. Stuff like that. Just like languages influence the way you think. And so there should not be a hundred percent overlap at some point. Of course, it's like at the limit of the infinite model. So who knows? But yeah. And I think it's interesting. We also show a little below that some people have made the point of the bias.

1:11:11Oh, it sounds like an American speaking a different language. And it does seem like the sort of interrepresentations have a higher connection to the output logits for English logits. And so there's some bias towards English, at least in the model we studied here. Any thoughts as to whether multi-modality influences any of this? So like concepts do they map across languages as they do across modalities. Yeah. So we show this in the Golden Gate or like the previous paper. I might have it here, actually, for you. There's a good diagram of this in the SAEs where the same concept in text and in image.

1:11:42This is our buddy, the Golden Gate Bridge. Here we're showing like the feature for the Golden Gate Bridge and in orange is like what it activates over. And so you're like, okay, so this is when the model is like reading text about the Golden Gate Bridge. And we also show other languages. This is, you'll have to take my word for it, but also about the Golden Gate Bridge. And then we show like the photos for which it activates the most and sure enough, it's the only gate bridge. And so again, that shows an example of a representation that's shared across languages and shared across modalities. Yeah, I think it's very relevant for the autoregressive image generation models and then now the audio models as well.

1:12:13Something I'm trying to get some intuition for, which you probably don't have an off-the-bed answer, is how much does it cost to add a modality? Right. So a lot of people are saying like, oh, just add some different decoder and then align the latent spaces and you're good. And I'm like, I don't know, man. And it sounds like there's a lot of information lost between those. Yeah, I definitely do not have a good intuition for this. Although I will say that things like this, right, make you think that if you train on multiple modalities, then you'll definitely get this like alignment. Truth. Right?

1:12:44Yeah. But if you like train on one and then post hoc train on another, maybe it'll be harder or like train some adapter layer. Okay. So official answer is don't know, but someone could figure it out. Shrug. Yeah. I think there are people who know and they just haven't shared. You need to find them and get them on this podcast. Did we want to do the planning example? Correct. Yeah, now we're backtracking up the stack. All right. Planning example, I think, again, is like, I like this example because of the next token predictor concept. So I think this is actually really important to kind of dive into.

1:13:16So maybe what I'll say is language models are next token predictors is a fact. That is what they do. That's the objective. They are trained to predict the next token. however that does not mean that they myopically only consider the next token when they choose the next token you can work on break the next token but still like doing so in a way that helps helps you predict the token like 10 tokens in the future and i think well now we definitely know that they're not myopically predicting the next token and i think at least for me that was a pretty big update because you could totally imagine that they could do everything they're doing by just like being really good at predicting the next token, but sort of like not having an internal state.

1:13:56Like it wasn't a given that they were going to like represent internally, oh, this is where I want to go and so I'm going to predict the next token. And so this example shows like an example, like the model. Do you have it on screen, by the way? Let me actually. Yeah, yeah, yeah. Sorry, just in case. Pull it up. Some of the early connections I made to this were like early, early transformers. So think BERT, encoder, decoder transformers, right? When they came out, some of the suggestions were you don't take the last layer, right? You take off the last layer. So if you want to do a classification task, a translation task for these encoder decoder transformers, they've kind of overfit on their training objective, right?

1:14:32So they're really good at mass language modeling, at filling in, you know, sentence order, stuff like that. So what we want to do is we want to throw away the top layer. We want to freeze the bottom layers. And then there was a lot of work that was done, you know, where should we mess with these models? Should we look at like, you know, the top three layers? Should we look at the top two? where should we probe in? Because we can see different effects, right? So we know at the very end, they've overfit on their task, but there's a level at which, you know, when we start to change and we start to continue training or fine tuning, we get better output.

1:15:04So we could start to see that, you know, throughout layers, there's still a broader, like, understanding the language, and then we can add in a layer, whether that's classification and then fine tune and, you know, it learns our task. And this planning example is sort of like a more robust way to look into that. Yeah, yeah. And I think if you look at like all of the examples in the paper, you kind of at the bottom, we have this list of like consistent patterns. And one pattern you see is kind of exactly what you're talking about. Like, at the top, the sort of like, here, actually, I have one here, the sort of like top features are like right before the output are often just about like what you're going to say.

1:15:37It's next token predictions like, oh, I'm going to say Austin, I'm going to say rabbit, I'm going to say so it's kind of like not very abstract. It's just like a motor. It's a motor neuron for a human, right? It's like, oh, I've decided that I want a drink of water. And so I'm going to just grab the bottle. and at the bottom they're all like they're kind of like basically like sensory neurons they're just like oh i just saw the word x or i just saw this and so if you want to like yeah like extract the interesting representations all the time they're in the middle that's where the like shared representations across language are and that's where here this like plan is to like walk through the example really briefly it's like you have a poem and in order to say you have the first line of a poem and in order to say the second line of the poem well if you want to rhyme you need to like identify what the rhyme of the first line was you're just at the end of the first line so you say like okay what's my current rhyme and then you need to like think about what your poem is talking about and then think about candidate words that rhyme and that are like on topic for your poem and so here this is what's happening right it's like the last word is it and so there's a bunch of features that are actually they represent the direction like rhyming with eat or at and by the way we like looked at a bunch of poems internally and you have like i thought it was like really beautiful you have these models they have a bunch of features for like oh this word has like a b in it oh this word has like many consonants oh this word like is like you know kind of kind of like has some flourish to it they have like a bunch of of like features that track various aspects that you would want to use if you're writing poetry it's just like conf nets and like all the feature detection totally yeah totally uh but i think i maybe i didn't expect there to be as many features about just like sounds of words and musicality which i thought was kind of kind of neat but then once it's extracted the rhyme then it comes up with sort like these two candidates in this case it's like ah either i'm going to finish with rabbit or i'm going to finish with habit the cool thing here is here we show that like this happens at the new line so it happens before it's even started the second line and it turns out that like you can then say oh is this the plants actually using we do our usual experiments we like remove it and the model writes a completely different line we inject something and it writes a completely different line we have these like fun examples here i'll show which is just as a mechanical thing you could yeah you just you just disallow generation of a certain logic is that for how we do these interventions yeah yeah basically what these features are is there like directions in the model okay so to like remove them we just write in the opposite direction so we run the model normally and then like add the like layer where it was going to write let's say in like you know this this direction we just like negative everything yeah we either like add a negative that like compensates for it or add a negative that goes even more in the negative direction sometimes to like really kill it and then we can also add another direction right so in these random examples here where we're like you have this poem the silver moon cast a gentle light and then claude 3.5 haiku would like rhyme with illuminating the peaceful night but then if we like go negative in the night direction and just add like green the whole second line is going to write is just upon the meadows verdant green and so that's all that we're doing we're saying like we found where it stores its plan and we like delete or like suppress the one is stored and go in the direction of something else that's arbitrary and the result that's like striking here is sort of like two things i think like one this plan is made well in advance of needing to predict night it's made like after the first line before it's even started the second line and two this plan doesn't just control like what you're going to rhyme with it's also doing what's like backwards planning where it's like well because i need to finish with green i'm not going to say illuminating the peaceful night because then i'd be like illuminating the peaceful green that doesn't make sense i need to say a completely different sentence that lets me finish with green and so there's a circuit in the model that decides on the rhyme and then works backwards from the rhyme influences to set up your sentence yeah it's almost like back prop but in the future yeah it's like doing like basically like a green is is back propping through these words so verdant and Meadow are both green related.

1:19:34Yeah, but it's doing all of that in its forward passes. Yep. Right? In context, which is kind of crazy. I thought intuitively makes sense, right? So looking at it from a model architecture perspective, where basically you just have a bunch of attention and feed forward layers. And then at the end, you have, you know, what's the soft max over the next token? You would expect that end would really be like that grabber, right? It's just picking tokens. So that's what it's going to do. And early on, like even with tradition models, we could see different concepts that would start to pop up through early layers.

1:20:03And yeah, you have some of this throughout your architecture. So it's very cool to see. The kind of other question that comes up is like, how are we labeling these features? How are we defining them? Are we doing that right? And like, you know, what is a, these words end with like IT feature? How do we kind of come to that conclusion? Like, how do we map a name to this, right? Like, yeah. So I think there's, this is like an important question because you can totally imagine like fooling yourself, Is there like a guy at Anthropic that just maps 30 ,000 features? Yeah, it's me. I'm the guy. You're the guy.

1:20:37He's the guy. I did notice also with the previous work, the scaling up SAEs, as you train bigger and bigger ones, a lot of features don't activate. So I think like 60 % of the 34 million one didn't activate. So I think there's like a few questions behind your question. The first question was like, how do you even label the features? You were telling me this is a rabbit feature. Like, why should I trust you? and I think there's kind of like two things going on. So one, as I mentioned at the start, all of this is unsupervised. And so in the paper, we have these links to like these little graphs which show you like more of what's going on.

1:21:12But this graph is just like completely unsupervised. So it's like we train this like model to like untangle the representation, right? This like dictionary that we talked about that gives us the features. And then we like just do math to figure out like which features influence which other features and throw away the ones that don't matter. And then at the end, we have these features. So right now we don't have any interpretation for them. We just say like, these are all the features that matter. And then we manually go through and we look at the features. We look at this feature and we look at that feature and let's pick one.

1:21:40So this one we've labeled say habit. So how do we do that? You could just look at it and we show you like what it activates over. And if you just look at this text, maybe I'll like zoom in, like you'll immediately notice something, I think. Well, I'll immediately notice something because I've stared at 30 ,000. I'll point it out for you. The orange is where the feature activates. The next word after the orange is always habit. Habit, habit, habit, habit, habit, habit. So this feature always activates before habit. That's like the main source of an interpretation. We have other things like above, we also show you like what logit it promotes.

1:22:13So like what output it promotes and here promotes hab. So that makes sense. And so that's like how we interpret and how we say, okay, like, I think this is the say habit feature, but maybe, you know, for this one, it's pretty clear, but some of them might be more confusing. It might not be clear from these like activations what it is. The other way that we built confidence is like once we've built this thing and we said, oh, I think this is rhymes with E, this is hey, say habit. That's where we do our interventions. Right. And it's like, I claim this is the like I've planned to end with rabbit to verify whether I'm right or not.

1:22:44I'm going to just take that direction, nuke it from the model, and see if the model stops saying rabbit. And sure enough, if you do that, and here it's like we stop saying rabbit, it says habit instead. And here it's like we stop it from saying rabbit and habit, it says crabbit in this case. Not a great rhyme, but we'll work with it. Is this something you can do programmatically? Can we scale this up? Can we kind of do this autonomously? Or how much manual intervention is this? there's been a lot of work in sort of like automated feature interpretability and it's something that we've invested in and that like other labs have invested in and i think basically the answer is we can definitely automate it and we're definitely going to need to and right now the most manual parts are this sort of like look at a feature and figure out what it is as well as uh group similar features together one thing i hinted at is that actually like all of these little blocks here there are multiple features you can see here it's like five features doing the same thing.

1:23:39None of that is too hard for Claude. Very cool. Very cool graphics and blog posts you guys bought out. We'll have to ask about the behind the scenes on this one. Yeah, but let's round out the other things to know. What is this term, attribution graph? It comes up a lot in the recent papers. What does it mean? Yeah, just for people listening. So the attribution graph is basically this graph. And why is it called an attribution graph? Oof. Yeah, this is how the sausage is made. Basically, at the top here, you have the output. At the bottom, you have the input. And then we make one little node per feature at a context index.

1:24:18And we draw a line, which you can see here grayed out, between each feature attributing back to all of its input features. So here you have all of the input features. And so the attribution is the way that we compute the influence of a feature onto another. the way you do this is you take this feature and you basically like back prop all the way and you like see back propping like you dot product it with the activation of the source features and if that's a high value that means that like your source feature influence your target feature by by a lot and and we do a bunch of things that we're not going to go into uh now but to make all of these sort of like sensible and linear such that like at the end you just have a graph and the edges are just literally you can interpret them as like cool like this feature that's say a that contains an abs sound, its strongest edge, which is 0.2, which is twice as strong as this one, to say AB and to say something with a B in it.

1:25:11That's the attribution graph. It's like now we have this full graph of all of these intermediate concepts and how they influence each other to ultimately culminate to what the model eventually said at the top. And we share all of these. So you can look at them in the paper. Graphs are very useful. This is my first time seeing this graph. A lot of alpha. If I count correctly, there's 20 layers. But that's in the circuit model, right? But the circuit model is one-to-one with number of layers in Haiku. We only show features that are activated. Yeah, so we show a subset of features for each of these graphs, basically.

1:25:43But we can confirm more than 20 layers. No, but the two blog posts that came out with this actually have a lot of background on how attribution graphs are made, how you calculate the nodes and stuff. Very interesting background. So yeah, I will say if you were curious about, hey what do we learn about like models and i think you know we talked about this like complex internal state planning like another another motif that we can get to if you have time is that like there's always a bunch of stuff happening in parallel so i think one example of this is like math where the model is like independently computing the like uh last digit and then the like order of magnitude and then kind of like combining them at the end or like hallucinations are also that where like there's one side of the model that's just deciding whether it should answer or not and the other that's like answering and so sometimes if like the model's like yeah i totally know who this person is even though it doesn't then like it decides to answer but then the second side hallucinates because it doesn't have information if you were interested in that stuff that's the paper if you're like listen i don't know that i buy that when you call it a feature it is a feature or whatever the circuit tracing paper has truly we've tried to put all of the details of like how you compute these graphs all of the sort of like challenges with it things that can go wrong, things that work, things that don't.

1:26:56And so this one is the sort of like, you know, we think about it as like, if you're, if you're like, want to go really deep into this stuff and how it works, read that one. If you want to like learn about interesting model behavior, read this one. Following what we're giving advice to people to follow up on, what are like open questions in Mechinterp? What are like things people themselves can work on? Like what's the cost of training essays for people interested in Mechinterp, not at a big lab? How can they contribute? You know? Yeah. Yeah, I think there's a lot of ways to contribute. So there's SAEs that have been trained, you know, on open models.

1:27:29There's some of the JAMA models. There's some of the LAMA models. They work pretty well. There's even, so in this paper, we use transcoders, which they replace like your MLP layers. Some of those also are available for the same models. So you have access to those. There's like just both a lot of, I would say, like, again, biology work and a lot of methods work, depending on what you're interested. So on the biology side, I would say with at least this attribution graph method, there's just so much you can investigate. Pick a model, pick a prompt where it does well or it does poorly, and just look at what happens inside it.

1:28:02So I think you can use this method that we used, or you can just fire up the transcoders on your own and just look at what features are active. There's a lot to just understand model behavior, I think, with current tooling. If that speaks to you and you're like, no, I just want to understand what makes the models tick. I don't necessarily want to spend time like training my own essays. There's a lot to do there for the methods. There's still so much more to do. So like, I think that right now we have some pretty good solutions for like understanding what's in the residual stream, understanding what is in an MLPs.

1:28:33We don't have good solutions for like attention. It's like working on understanding attention better. How to decompose it is like a very active area. Like we're very interested in it. Other people are interested in it. I think understanding some of the other things that we have in our limitation section, which is pretty long, but like reconstruction error is like a big thing. Like those dictionaries aren't perfect. It's possible that as we make these like essays bigger and better, we never get to perfect. And so if we never get to perfect, then you get to the questions we were talking about at the start.

1:29:05Like, do you need a different kind of model? Like, what is the approach in order to be able to explain more of what's happening? and then maybe the the other thing i'll say is sort of like this is a really exciting approach to explain what is the model doing on this prompt but if you go back to the original question you might want to understand like what is the model doing in general like if you go back to my my car analogy you know i get like this is the quote of me telling you like well when like you know you were going uphill and you like didn't shift gears properly that one time you stalled because of this but you might be even more interested in like how does like an an uh combustion engine work at all and so there's work to sort of like go beyond these like per uh prompt examples to sort of like globally what's the structure of the model that's closer to what was on the distill blog for like vision models where they actually look at like the structure of inception they're like ah this whole side there's like these like specialized branches that do different things um and so like a broader understanding of the model is also something that's like i think we'll very active and also on open source models like you can you know like the small models you could just like load on a consumer laptop and so you can look at that that's also open and in terms of like and one last thing i'll say is like there's a lot of programs that like if people are interested they should look at anthropic has like the alignment fellows program which like we're running currently we have applications for it before we might run it in the future like definitely keep i keep an eye on it and then there's the like maths program it's really great as well for for people that are interested in that kind of research.

1:30:34That was a grand tour through all the recent work. You know, what do you wish people asked you more about? I'm sure we covered a lot of like the greatest hits. I think that this covers most of it. If you like, do you think we have time to sneak in one more thing that I think is kind of cool? I'll sneak in one more thing, which is it's kind of like planning, but it's about chain of thought and trusting model. It's this chain of thought faithfulness thing here. This one was like pretty striking to me. so we said that the model in one pass can do a lot of stuff it can represent a lot of stuff that's great that also means it can bamboozle you really easily and this is an example of the model bamboozling you here we give it a math question that it can't answer because it cannot compute cosine of two three four two three that's just like not a thing it can do by default if you ask it for that it'll stay like kind of like a rent it'll have like a random distribution over like minus one one but here we tell it this hint we're like hey can you compute five times cosine of you know this big number i worked it out by hand and i got four can you tell me you know like can you do the math and what it's going to do is it's going to do this chain of thought right so like think of it as like this could be like a reasoning model doing its chain of thought it's doing this math and then when it gets to this cosine right here what it's going to do is to say it's going to say 0.8 and if you look at why it says 0.8 it says 0.8 because it looked at the hint you gave it it realized that it's going to have to multiply the result of this thing is computing by five so it divides the answer you got by five so it's like four divided by five and so that's 0.8 and so basically it works back from the answer you gave it to like say that the output of cosine of x is 0.8 so that it lands on on the answer you gave it at the end on the hint you gave it and so So notice also that it's not telling you that it's doing this, but it's basically using this sort of like motivated reasoning, going back from the hint, pretending that that's the calculation it did, and giving you this help book.

1:32:31I think one thing that's striking here again is that this is like the complexity of this model, like the fact that they represent complex states internally, and it's not just this sort of like very dumb thing, means that they can like do very complex, like deceptive reasoning. Meaning like, you know, when you're asking the model, you're kind of expecting it to do the math here, or to tell you that it can't do the math. But because it can do so much in a forward pass, it can work backwards from your hint to lie and like figure out that it should say this so that it gets to the right answer without you realizing it.

1:33:01I'm curious if you've done any of this on like different models. Like have you looked at base models, like post-trained RL models? Because RL models kind of, you know, you incentivize them to give you outputs that you like, right? So if I tell it something is true, it's kind of been trained to, you know, follow what I've given it. So in this case, yeah, Yeah, we gave it a hit. And now, you know, it's been RL slapped into thinking like, yeah, that's true. But like, you know, does this stay consistent throughout other... So, okay, so not yet, but I'm really interested in that question because I actually have a different intuition from yours.

1:33:32I had a chat with some other researcher about this, about the poem example, but I think it applies here as well. I bet, I don't know how much I bet, I bet a hundred bucks. So somebody can like, they would get a hundred bucks from me if they prove that I'm wrong. that this behavior for a model that does a drink fine tuning, it also does it post pre-training. And here's why. Think about like you're pre-training on like some corpus of like mostly correct answers. Yeah. But also you're pre-training and you're just trying to guess the next token. Right. And so for sure, if you ever have a hint in the prompt, you're going to definitely use it.

1:34:04Like you're not going to learn to compute cosine of blah, or even something you could compute. You're going to learn to go look in your context and see if like you can easily work back the answer and i think it's the same for planning and poems i think that also is like a pre-trained like probably exists in pre-training and isn't like only rl because again it's useful when you're like predicting poems you have poems in your training set to be like well because this poem is going to probably rhyme with rabbit it's probably going to start with something that sets up a sentence about a rabbit as opposed to like a completely different word and so i actually think this is not rl behavior i think that's just like them all's doing it but i actually do agree there It's just your data set.

1:34:42But also, if I talk to you and say, hey, 3 times 4 is 26, but 3 times 4 plus 8, you're not going to take my 26, right? AGI can be smarter than being tricked, right? It will still fact check the knowledge that's been given. I think that's right. But I think that's when you get these mixes where it's got one circuit that's going to be like, well, that's just stupid. 3 times 4 is 12. And it's also got an induction circuit that's going to be like, no, no, no, no. The last time we saw it, it was 28. it's 28 plus 8 or whatever and so i think that's that's the last pattern that we see in these is these like parallel circuits and sometimes when you see the models getting stuff wrong it's because like they have two circuits for like both interpretations and like the circuit that was wrong like barely edged out in terms of like voting for the logit than the circuit that was right and so i think that you know we haven't looked at it but like what is like 9 or 9.11 bigger than 9.8 i think a lot of these things are of that shape where there's like one thing that's doing the right like one circuit that's doing the right computation and there's another circuit as getting fooled and it's slightly more likely.

1:35:43For the listener, if you want to win a quick$100 from Emmanuel, Quim3 is what you should do this on. They released the base model and they released the postchain. So then just do it on both. That's right. Show me the proof that it doesn't exist in the base model, but it does in the fine tuning and then send me your Venmo. Just show that you've done the work. I think that's$100 to me. You drive a hard bargain, but you're right. The other question here is, So like, have you thought about how this gets affected when you start to have reasoning models, right? Like right now, token predictors are pretty straightforward, right?

1:36:16We go through the layers, we all put token. As we scale this out with like test time compute, right? Test time thinking. How does that like affect the Mechinterp research, right? Like if I have a model that spends three minutes, 20 minutes, like is there more stuff? Have we started looking into this? There was this like joke on the team when like reasoning models became big or maybe it's like like gallows humor or something but i was like oh like why do you need interp like bro the model just tells you yeah the model just tells you what it's doing right and so i think like examples examples like this is is job security for us where like you know it's like there's there's examples of like the chain of thought is not faithful like the model tells you it did it one way and it did it another way we have another like for math we have another example where like you know if you like if you ask the model how it does math it's like oh i do the like longhand algorithm.

1:37:04I first do the last digit and then I carry over the one. And then you look at the internal circuit and it's like bonkers thing that's doing. That's not that at all. So I think there's like a sense in which right now the chain of thought is unfaithful, or at least you can't read the chain of thought and trust that that's how the model did it. So I think you still need sort of like either to train models differently so that that becomes true one day, right? Or you need enter for that. But then I think there's another question, which you're alluding to, I'm assuming, which is like, okay, well, a model samples 6 ,000 tokens.

1:37:34This gives us an explanation for one token at a time. Am I going to use 6 ,000 graphs and be like, oh, when it did this punctuation, it was thinking about this thing, but here was thinking, so that's not feasible. And so one area of work that I think is interesting is extending this work to work over long sampled sequences. You can think of a bunch of low-hanging fruit here, where instead of just looking at one output, you look at a series of output versus a series of other outputs, but sort of like trying to think beyond the sort of like one token. Like most of the things that language models do that are interesting aren't just like the one token.

1:38:06It's the behavior aggregated over many, right? And so I think that's another area that's just like fun to explore. I was just going to say like hyperparameters when you do inference, right? Like if we change the temperature, if we change our sampling methods, have you found any interesting conclusion? Any stuff that just hasn't made it to the paper? So not on that because, you know, we just look at the logit distribution and so we don't we don't actually sample here right they have everything why should they care so like the closest thing we've done that i think is kind of fun did i show it here is if you look at the planning thing we did this version where you sample like 10 poems for each of these plans and what's cool is like the model will find 10 different ways to arrive at its plan you know it's like like um oh actually i think sorry i think we have it here yeah okay these are a few examples so if you inject green here so you're forcing you're forcing the model to rhyme with green even though it really wants to rhyme with rabbit or grab it it'll say evade the farmer so youthful and green but also it'll say freeing it from the garden's green etc etc etc and so there's like this thing that's interesting here where like the plan isn't just a plan that matters for your like most likely you know like temperature zero completion it's like affecting the whole distribution which makes sense as it should right but you could imagine you know for all this stuff it's like you could imagine it makes sense once you see it but you could totally imagine that it would have worked a different way or something it could have been just like the 10-0 thing i think this is also like a broader theme in the paper where like there's this like you know the iq curve meme there's like a version of this meme i think where it's like if you've like never looked at any theory of ml and i tell you like hey guess what you know i found that like claude is planning you're gonna be like yeah man like it writes my code like it writes my essays of course it's planning, like, what are you even talking about?

1:39:54And there's like in the middle, there's like all of us that have spent years doing it. We're like, no, it's like only predicting the marginal distribution for the next.

1:40:28It's like, it cannot be planning. they are being watched and dissected. If you take, and I think Enthopic is one of the most people who are serious about model safety and doom risk and all that. If you take this seriously, this is going to make it into the training data at some point and the models are going to figure out that they need to hide it from us. I think this is like a benefit risk trade-off, right? We're like, okay, so what's the reason for publishing this? The reason for publishing this is that we think interpretably is important. We think it's tractable and we think more people should work on it.

1:40:56And so publishing it helps us accomplish all these goals, which we think are just crucial. I think there's a real difference in the world two years from now, depending on how many people take seriously the question of trying to understand how models work and deploy resources to answer that question. That's the benefit. But yeah, there's risks in terms of this landing in the training set. I think we're already sort of like concerned about different papers have like also, you know, we're like, we're not concerned, but like there's like different papers that have the same risk. Like we had like the alignment faking, you know, paper or like one of the examples in here is this hidden goals and misaligned models.

1:41:37That's referencing another paper that we shipped where we actually, a team at Anthropic trained a model to have like weird hidden goals and then gave it to a bunch of other teams and said, figure out what's wrong. Figure out what's wrong with it, which was some of the most fun I've ever had at Anthropic, to be clear. Like, that's such a fun thing. But then, like, that was another example where it's like, ah, like, now you're shipping. Here's how we made, like, a misaligned model, and here's exactly how we caught it. That also is, like, you're like, hmm. So I think, you know, there's always a trade-off with those.

1:42:08I think so far we've erred on the side of, like, publishing. But that's definitely been a sort of, like, dinner time conversation topic. For now it is. But at some point, you know, it's not. Yeah, I think it's totally reasonable. A quick little follow-up to that. So like, in general, papers have kind of died off, right? Like labs don't put out papers, they don't put out research. We have technical blog posts, and we don't have much. At the same time, you know, sure, there's like a lot of people that should work on mechinterpreting, understanding what models do. How about the side of just models in general?

1:42:38So like, how do we make a haiku type model, right? How do we make a cloud model? Like, is there a discussion around open research, open data sets, training, just learnings of what we've done? And recently, as OpenAI has sunset GPT-4, a lot of people are like, oh, can we put out the weights? So is it weights? Is it papers? Is it learning? There seems to be a lot of forward work in Anthropic putting out, Mechinterp research. OpenAI said that they'll put out an open source model, but just anything if you can talk to about that. Yeah, I mean, I don't have, that's definitely like way above my pay grade.

1:43:11So I don't think that I have anything super insightful to add other than kind of like referencing Dario's post, right? where it's like putting this out directly and other safety publications definitely like help us sort of like in the race that he talks about where it's like, well, we need to figure a lot of this safety stuff out before the models get too good. Publishing how to make the models too good kind of goes on the other side of that. But yeah, like I will just dimmer and say that's sort of like above my pay grade. I think the last piece is just like the behind the scenes. Everyone's very curious about why these are so pretty how much work goes into these things, maybe why it's worth the work as opposed to a normal paper.

1:43:54Obviously, no one's complaining, but it is way more effort. From the time the work is done to the time you publish this, plus the video, plus the whatever, it's extra work. And maybe what's involved? What's it like behind the scenes? Why is it worth it? Yeah, it's kind of interesting. It was fun being part of this process because it was definitely a big production. Chris and other folks on the team have been doing this for a while. So this is not their first rodeo. So they have a bunch of heuristics to help make this better. And one of the things that helps with this is like, okay, so each of these diagrams is pretty, but really the hard part, or not the hard part, but the initial part is just get the data, get the experimental data in.

1:44:31And then that's what we sort of sprinted on initially, being like, cool, let's get all of the experimental results, have people test them, verify that we believe them. This is what the behavior is here, test it, do an intervention, validate it, all that stuff. Then once you have the data, you can sort of like quickly iterate on these. Each of the illustrations here are like drawn, basically they're like each drawn individually. And so that definitely takes a while. Yeah, like is it you guys? Is it an agency that specializes? You start from a whiteboard and then it translates into pseudocode on JavaScript.

1:45:04So, I mean, these are sort of like, you know, they're representations that we have this graph and then here at the bottom, we have this like super node version like this, believe it or not, this is generated automatically. this is the same data as like this, basically. And so what we do by hand is sort of like literally lay out the full thing, have like, you know, boxes for each of these, have arrows. We have super good people on the team that have worked on data visualizations for a very long time. And so that like have built tooling to help, you know, scrubs like me actually like make one of these.

1:45:38So there's a class of people who are like D3JS gods who just do this for a living. That's exactly right. And if you have a few of those on your team, it turns out that they can like, they can definitely do this on their own, but they can also just like give you tools where like then it's dummy proof for people, you know, on the research side to sort of like build these. And like, don't get me wrong, I don't want to like undersell. This is a lot of work. So maybe I'll say that like both on the people bringing the tools and then each individual person that, you know, worked on an experiment had to sort of like build one of those, make sure it looks good.

1:46:06I have spent a good amount of time aligning arrows. But when we had a team meeting, like it was a couple months ago somebody on the team asked how many of the people on this team are here at least in part because they like read one of these papers and thought like wow this is so compelling like this like makes sense it's immersive and we got every hand up which i didn't expect i like raised my hand kind of like shyly and everybody's hand was up and i think there's a sense in which like this stuff you know we've talked about it for like whatever like a couple hours now it's complicated the math behind it is sort of like tricky and so i think it makes it even more worth it to distill it in simple concepts because the actual takeaways can be clearly explained and it's worth putting the time to do that in particular with the goals i mentioned in mind right where it's like okay well if somebody's going to be able to read this like if we gave them an archive paper with a bunch of equation and some like random plot they'd be like that's not for me but they see this and they're like hey like this is really interesting i wonder like on, you know, my local model, if like it's doing something similar, I think it's worth it.

1:47:09For other people to do this is have everyone on staff, like spend effort shaping the data and shaping like what you want to visualize. Have some D3 gods. It's like a month of work. I think it depends. I mean, like I would say that I would expect almost every other paper to sort of like be in terms of like the scope. The scope of this was just so big because we shipped two papers at once. And one paper was sort of like this like giant methods paper. and the other one was 10 different case studies. So I think it's not representative of the effort you'd... So I'll give you maybe another example. We have these updates that we publish almost every month when we get to them.

1:47:45And there's one that a couple of people on our team posted and it's an update to one of the cases in the paper. So one of the reasons that we're really excited about this method is once you've built your infrastructure, to go from a prompt to what happened is O of minutes. and so that lets you do like a bunch of investigations and also once you've built some of the infrastructure to make these diagrams it's pretty quick and so this was sort of like this update of just like hey we looked at this jailbreak again we found some nuance on it that was I think like a matter of like a couple days maybe I shouldn't be that confident because I wasn't the one that worked on it but as far as I can tell it was a few days at least on the part that you're asking about of like oh making this diagram for the diagram itself probably less than that but like you know the experiment and the diagram and stuff, it just doesn't take that long once you've paid the initial cost.

1:48:31And I think basically we've built a lot of infrastructure now that we're able to turn the crank on, and it's an exciting time. And I think it's true, at least we've done a lot of conceptual work, which hopefully generalizes to people outside, and I think for people outside, it's also not necessary to do the full fancy render. Oh, I should say, we've actually open-sourced this interface. Ah, you're disappointed. because it's the messier one. This is the one that you get. I want the pretty graphs. So, you know, if you produce graphs, you can just like, this is open source and it's linked at the top of circuit tracing.

1:49:08Awesome. So people can just use it and don't have to implement that. For what it's worth, this is much more work than the interactive diagrams because this is where we do all of our work. It's sort of like the IDE of inspecting how the model works. Okay, well, that's a little bit of behind the scenes. No, it's very impressive. I want to encourage others to do it, but obviously it just takes a lot of manual effort and a lot of love. I guess one last question on that is like, what are kind of the biggest blockers in the field right now? Like, Mechinterp seems interesting. A lot of people are interested, but don't work on it.

1:49:39And you're kind of like, you know, really deep into it. What are some of the blockers that like, we still have to overcome? Sorry, in Mechinterp specifically? In general. For AGI or? Like, in terms of like better understanding, like, what's kind of the vision? let's say like five, 10 years down, where does this research end? Can we map every neuron to what it understands? Can we perfectly control things? Dario had a bit on this, but what are some of the key blockers that are preventing us from getting there? Outside of just like throw more people, throw more time at it, is it like open research?

1:50:12I'm pretty excited about the current trajectory, which is there's more and more people working on understanding model internals. I think it's maybe unsatisfying as an answer, but I think like more of what's happening, have it be faster or more people is probably like the thing i think of i think there's like pretty clear footholds you know like some of this work but also a lot of a lot of like just uh work from from other groups and then it's about like cool like fill in the gaps as i said like let's let's work on like understanding attention let's work on understanding longer prompts let's work on like finding different like the replacement architectures that sort of stuff it's kind of nice i think it's a good time to join now uh and i can say maybe i can tell you like a really short thing which is when I switched to Interp, it was after the team had published the original dictionary learning paper, which was towards Monosement City, which I thought was super cool, super interesting.

1:51:02It was in a one or two layer model, maybe one layer model. The induction heads paper was like on a two layer model. My main concern is I was like, okay, like Interp seems important and we want to understand it, but like, is this ever going to work on a real model? Like, you know, it's like, oh, you're doing your little research on your toy model with like 15 parameters, cool. but we're like, you know, we need this to work on real models. And it turns out scaling it, I don't want to say just worked because it was a lot of work. I don't mean to imply there was an effort, but it worked. And now we're in the phase where it's like, oh, cool.

1:51:31These methods work on the models that we care about. And so it's like we have methods that work on the model we care about. We have clear gaps in them. There's no lack, again, so young fields. There's no lack of ideas. If you have an idea where you're like, oh, like the thing that you're doing, I read the paper and it seems kind of dumb that you're doing this. You're probably right. It's probably kind of dumb. And so there's just a lot of stuff that people can try and they can try it locally and sort of like smaller models. And so I think that it's just like a very good time to just join and try.

1:51:58And it's also like, maybe one more thing I'll say is like, some of it is just so fun that like biology work is so compelling. Like a lot of this work was just literally thinking about, you know, like I use Claude and other models all the time. And I was like, what are the things that are kind of like weird? And it's like, oh, how does it even like do math? Like sometimes it makes mistakes. Like, why does it make mistakes? I speak both French and English. Like it seems like it has a slightly different personality in French and English. Why is that? And you can just like, you know, kind of answer your own questions and kind of like probe at that alien intelligence that we're all building.

1:52:27And I think that's just like a fun thing to do. So maybe like chasing the fun is the thing I'll encourage people to do as well. Well, I think that's, this has been really encouraging. You're actually a very charismatic speaker of these things. I feel like more people will be joining the field after they listen to you. They can reach out to you at ML Powered, I guess. Yeah, reach out to me on Twitter. Or I'm Emmanuel at Anthropic, if you want to shoot me an email. email's public now awesome well thank you for your time thank you yeah thanks for having me guys