Inside the Black Box: The Urgency of AI Interpretability

27:49And I think it's still unclear how the compute you need to achieve the same degree of interpretability scales with the size of the model, it's like a big question mark for us. But I do think, I think that actually, surprisingly to me at least, it has turned out to be the case that in many ways interpretability seems to be getting easier as the models get smarter. And like to give one example, we spent a lot of time trying to figure out how this like small internal models of ours does two digit addition, how it adds two numbers. And we kind of like found the like, you know, the primitives, the relevant primitives, there were like features for different like for like numbers ending in six or, you know, or for like add a number that is like around 10.

28:35And these kind of all like collect, you know, interacted in like just a whole mess of like complicated ways that like somehow constructively interfered to like get the answer to an addition problem right most of the time. But it was like there was no kind of crystalline, you know, structure to it that was, there were some, but it was like, everything was kind of like weird and messy and it was like, why is it doing this like unhinged thing to add two numbers? And then we ran exactly the same code on one of our production models, Claude 3.5 Haiku. And it was like, oh, it made sense. It's like, here's the features that add the ones digit.

29:13And then here's the features that add the magnitude. And then here's the thing that stitches those two things together. Just everything became like here's like lookup table features that like are responsible for adding six to nine and spitting out like that the answer ends in a five. And everything just became much clearer running the same tools on a bigger model. So that was surprising to me. but I think it's, you know, in hindsight, what this is getting at is that as models get smarter, they have developed, you know, they're getting smarter due to having developed more generalizable algorithms for solving problems.

29:55And I think we are better at grokking what generalizable algorithms than at grokking weird kind of bespoke heuristics. And so that, yeah, I think it's making our job easier. And I think, yeah, this is also kind of related to this point that as models get smarter, they're able to kind of do a bit more of the work of interpretability for us. Like with a smaller model, if I type in, you know, a sentence that's like, like I told my friend a secret and and then she told everyone at school, and then I type in the word betrayal. It probably isn't gonna, these are just two very different sentences. The tokens are different.

30:45It's not gonna map them that close together. Whereas with a bigger, smarter model, it's more likely the case that those are mapped to overlapping activations in its internal space. And so then if I wanna know what is the model thinking about when I type the sentence about, you know, my friend who told my secret to other people at school, it's like, well, what else activates similar neurons? Like, oh, literally the word betrayal. Like, that was easy. So like, the more the models have kind of abstracted language, the kind of easier it is for us to kind of get to like summarize what it is that they're thinking about.

31:21I thought you were going to go a different way with that second point, which is something I'm very optimistic about. They can do more, the models can do more of the work for us, both in the sense that their representations are better, but also they can literally do more of the work for us. Like we can just, I think that a few years ago, we had models that couldn't do the basic automated interpretability task where I give it a list of examples of a feature firing. I tried to get this to work with one of the early DeepMind internal language models and it just fell over and then GPT-4 could do it.

32:00So that's one level, you know, sort of saved us from interpreting a million features. Great. But I think now we're at night with agents starting to get good. We're at a point where we can actually get like lab work, lab work done. You know, I can ask a model to come up with a hypothesis and test it and give it access to various tools, I give it SAEs, I give it all sorts of different interventions. And it can give me a hypothesis. And so I think that I'm pretty optimistic about models not only getting easier to interpret, but doing more of the work for us. I think that I'm very optimistic about interpretability because of this.

32:39I love that you said two different things. We've managed to superimpose two different points in the same space. So that's good. We've had a very kind of research-oriented discussion so far. We'd love to talk about real-world applications as well. Tom, Goodfire is working on commercial applications of interpretability. To the extent that you can talk about what are some of the use cases in the wild where interpretability could be used in a production, mission-critical context? I can't talk about the specifics of the customer contracts, but we are working with a big healthcare provider to help them to understand models that they want to use for diagnostic purposes.

33:26So this feels great. The state of the art here is not that advanced. We have the opportunity to help them to understand things that might actually be used in important contexts. and that could also unlock new scientific knowledge. So that's very helpful for them. They want to be able to trust models that might be used in a kind of clinical context. Another example is we're talking to one of the big inference services. And one thing that, this sort of goes back to my earlier point about reliability, they want to be able to sort of do guard railing. So when the model kind of goes off the rails in one of the ways that it's won't do, that we can detect that and do some, like, nudge it back on track in a way that's better than, say, using a prompted classifier.

34:24So these are, like, two examples where I think, yeah, we've got, like, potential for, well, yeah, genuine impact. Yeah. It's all very, very exciting. And, Jack, to the extent that you can kind of comment, it would be helpful to kind of understand Anthropik's angle on interpretability and why it's kind of helpful to their core business. Yeah. I think, yeah, so why do we even have an interpretability team? I think of it as the primary reason we're around is to ensure reliability and the safety of Anthropics models. But I think increasingly, it's hard to decouple that from what makes models commercially viable.

35:15No one wants their models to be lying to them. No one wants their models to be faking, you know, passing tests in code. No one wants their models to slip into weird alter ego unhinged personas. And so maybe some people do, but yeah. But so and yeah, I think of our job ultimately as being able to root cause weird behaviors and understand what is the fundamental, like what lever in the model is causing this weird thing that we don't want to happen, so we can solve it in a generalizable way. Because you can always just add a supervised data point to get the model to behave differently on a particular context, but if you don't understand the kind of general thing that was underlying the behavior, it's not going to generalize.

36:09So yeah, root causing, what's behind funky behaviors so that we can fix them in future model iterations in a more generalizing way, and providing assurances that there aren't weird things that we haven't seen in our behavioral emails.

36:35If we find a problem in our models, they're reward hacking a bunch, and then we try to like fix that in the next model. And can we see, you know, in what sense did we fix it? Did we just overfit to a few particular, you know, some kind of specific context or eval environments, but like secretly the model is thinking about how it like really wishes it could reward hack and it like totally will do it at the next opportunity, but just not right now, because it knows that it's being evaluated. Like this is like, that kind of thing is actually like, seriously, like, you know, we have proof of concept that that kind of thing can happen.

37:10And so I think a lot of part of our job is to help provide confidence that that's not happening. I think maybe just to give one other angle, there's a paper I worked on recently, this paper on persona vectors, which is like directions in the models activation space that nudge it into different personality modes. And in that paper, I think there was like a lot of what we focused on was using this kind of internal understanding to feed back into the training process to make sure models don't develop unwanted characteristics. So if you find the sycophancy vector, there are things you can do during training to inhibit the model's like, you know, propensity to adopt sycophancy as a characteristic, or you can, or there's things you can do to like filter the training data to like find the data that would cause it to adopt a certain trait and then, you know, get rid of that data.

38:20So I think that's like another, you know, emerging, this is like, you know, I'm not saying that this is like something we're doing in production or anything, but like, I think this is kind of the research on that sort of thing is maturing to the point where we can start to think about like, can we kind of these like weird spooky things that models are doing, can we like nip them in the bud with an internals based kind of adjustment to how we train models. That's a super paper. I think that's going to be really, I think that paper is actually going to be really important. By the way, I think there's probably demand for the crazy models.

39:01I know in Tesla, it's a Mad Max mode, and apparently people want that. There'll be a wacko mode at some point. But I want to ask one more question maybe before turning it over to the audience. And if folks don't have questions, I will have more for sure. But it's around sort of almost kind of breakthrough moments, you might call it. Like there's sort of these particular moments in the development of AI that kind of stand out as being very important. and maybe some examples are AlphaGo, AlphaZero with regards to reinforcement learning. Maybe it's, the GPT-2 paper was pretty interesting at the time.

39:42Instruction tuning as a way to get to these useful chat models. What would be a breakthrough moment for interpretability that you could, at least you have some view on today based off what you're seeing? What would be most exciting in the next, call it five years? Five years is a long time. I think being able to properly sculpt model development so we can genuinely engineer them is sort of using interpretability, I think, is the way to think about doing this. like we're trying to have a science of understanding models that lets us sculpt models both like precisely and kind of microscopically is the sort of very broad answer to that.

40:31I think more specific in my mind, like more specific answers are things like being able to have complete decompositions of model inference at varying levels of abstraction. I can ask a model AI, I can ask like Claude Seven about it or wherever we're at. And I can get an explanation. I can like change the model based on that, that sort of thing. That all feels sort of a good end point. I think also I really want to see the first new knowledge, new scientific knowledge extracted from a scientific model. That will be, in my opinion, a like breakthrough moment for interpretability. you know, the nature cover with, when it was a deep mind, I have to have a nature cover.

41:18The nature cover with the new fact from whatever. Those feel like huge moments to me. Agreed. Jack? Yeah, well, plus one to the scientific discovery and the kind of this like gradual improving our ability to sculpt models to like be more the characters we want them to be. But I suspect that is less likely to show up as a breakthrough paper and more as this kind of like iterative process. So if I'm gonna pick something that's more of a like flashy breakthrough, I would say building a reliable lie detector or truth detector for language models. And I think there's a lot packed into that statement.

42:01Like I think underlying building such a thing, that includes things like detecting unfaithful chain of thought, detecting cases where like the model like could know something, but it's not saying it. And maybe it's not thinking about lying, but it's like failing to kind of introspect appropriately in response to your question. So there's like a lot of underlying science there of like, what does it mean for a model to know something? It turns out to be a very complicated question. The model can like know something in layer two, but not in layer four. And so, you know, yeah, there's also like weird fractured and split brain things that can happen in models.

42:39And so like, even so like, yeah, if we have successfully kind of built a lie detector for language models, I think it will be, it'll be reflecting a lot of fundamental scientific progress. I think, yeah, also just like more kind of, a bit more of a squishy breakthrough, but I think there's this, or you know, potential breakthrough, but there's this question that no one really knows how to answer of what kind of mind is it that you're talking to when you're talking to a language model. It's this really bizarre thing where you're talking to a next token predictor that is acting as the author writing a story about a dialogue between you and this humanoid robot character called, at least for Anthropics, called the Assistant.

43:33and to what extent is that, does that simulation, is that like, should we regard it as having thoughts and feelings or is that not the right way to think about it? When the model is role playing as something else, should we think of it as the assistant character has decided to role play as something else or the model has decided to write about a different character? There's just these like fundamental questions about kind of like persona and like who am I talking to when I'm talking to Claude that like no one has the faintest idea of what the right answer is. What experience is there like a little guy in there?

44:17Yeah. But I suspect in like three years we'll have some clearer sense of who it is that you're talking to and that is going to be important. Amazing. Thank you for that. Questions from the audience?

44:43I think at Anthropic, we're kind of taking a two pronged approach to this, which is like, I think the main thrust of our research historically has been pursuing this like bottom up approach to interpretability, which is like, let's find some like interpretable decomposition of the model into features that account for just like all the possible things it can think about. And then let's like describe how they're causally wired up and then just like, let's just look at that causal graph and like describe what's in it. And I think the challenge to scale there is like, we've got to scale up these like sparse decomposition algorithms, whether they're sparse auto encoders or transcoders or whatever the next thing is.

45:25And we've got to scale up the process of like analyzing them with probably with auto, you know, LLM agents in the loop kind of doing the interpretability for us. That seems like the path there. I think we're also increasingly so like, yeah, I just like, I just started a new team that, which is kind of like pursuing a, like a bit of a different approach that's like kind of coming more top down, which is like, just let's find the behaviors we're most interested in debugging or the kind of like cognitive phenomena that seem like most important to understanding what's going on inside models and then just like throw hypotheses at the wall using like whatever analyses we can.

46:09And this is like in some sense not scalable at all. It's like intentionally not scalable, but it may be the case that we can just kind of like hone in. Maybe there's just like two or three problems that are really important for us to solve and being able to describe every single thing that's going on in the network, maybe we can get away with not doing that if we really nail these couple problems. And so I think that's the other kind of approach to scalability is to just not do it and instead be careful and iterative about how we pick the problems to work on. I think, yes, I would agree with all of that.

46:46But I would also say, we talked about scale in the sense of the model getting bigger. I think the sense of scale in terms of the sequence getting bigger is a whole different type of scale to deal with, one that is possibly a lot harder. Because models get bigger, their representations get nicer generally, great. But then you just have problem with mass. There's like a million tokens here.

47:13I think there are a few ways that we might hope to deal with a million token chain of thought. You could imagine going very bottom-up, where you understand the causal flow of every one of the million outputs. Or I say you, like a swarm of agents does this for you. And then you sort of ask questions. I'm presented with some sort of interface where I can ask questions about this swarm of agents that somehow sort of aggregated the information. This is a sort of bottoms up aggregation approach. Or I can imagine a sort of top down one where I first try and like look for sort of high level abstractions over the sequence.

47:54Maybe it's actually like a sort of dynamical system that has an attractor here and it has an attractor there or like the recent Thought Anchors paper, you know, it's doing something in this sentence and it's doing something else in the other sentence. and that lets me target the pivotal points and then try and explain those in more detail. So I think both bottom-up and top-down seem pretty promising.

48:20I'm an ex-neuroscientist, or I like to think I'm still doing the same thing, but just on models. So I think about this a lot, what knowledge can we port from neuroscience to interpretability and vice versa. I think that for me, the kind of knowledge flow has actually gone more in the other direction, but I'm kind of hoping that we can get the full closed loop and that one just like conceptual insight that has changed my thinking a lot is the correspondence between memory and the attention mechanism. So, just like mathematically, it is the case that the attention mechanism in transformers can be implemented.

49:13It's very difficult to implement it with standard biological neural network, but it's very easy. But there is a way to implement it using a biological neural network with plasticity, with updates to the weights between neurons. And that's what's happening in memory is you're updating connections between neurons and then you're recruiting the information that was stored in those connections. And so I think to me the fact that transformers work so well at modeling language is suggestive that there's something important about being able to store information in the brain, not just in the activity of neurons, but also in the strengths of synaptic connections between neurons.

50:05And that short-term memory and medium-term memory are critical to a lot of these cognitive processes. So I would love for... And I think, yeah, studying... There's a lot of cool research on memory and memory consolidation that is interesting, because that doesn't have a great analog in language models right now. Like the context window is kind of akin to everything that's happened to you in the past few minutes, but then what's the analog of everything that's happened to you today or everything that's happened to you in the past month? So yeah, I think there's probably something to be gleaned.

50:43Oh yeah, this is the thing I think about a lot, basically, is just like, can the neuroscience of memory sort of help us understand attention better? but I don't have a smoking gun example of that happening yet. I would love to hear your thoughts, because there's tons I don't know about neural representations of language, and I think there's a lot of cool things to learn. I am almost totally ignorant on the subject. And so I am more interested in essentially trying to absorb that information than in... Yeah, basically, I think I probably don't want to expose my ignorance any further than I already have done by trying to opine on it.

51:31I'd be interested to hear more about it.

51:41I think it's very interesting one of the reasons I think it's interesting is that it's one of the best places one of the best intervention points like what do you do with interpretability you can intervene after the model is trained but it's also very interesting to imagine trying to intervene on training the problem is how you do it of course like if you because during pre-training your features are mostly not formed um so if you were to train an essay sort of alongside you probably i i don't know if anyone's actually tried this like if you just train an essay alongside the model like does it do you get nonsense i'm not sure be an interesting experiment um there's a field called singular learning theory which is like i think has some might have a lot to say about this i don't know if there's in the algebraic geometers in the audience?

52:33Raise your hands. No? Speak could be known. Yeah. Anyway, it sort of uses a bunch of tools from mathematics that I also don't understand, but seems to have very powerful theoretical foundations for understanding exactly this kind of developmental question. Yeah, I think I'm more optimistic about a narrower narrow version of this question, which is understanding changes during like post-training. Because I think like, yeah, we already have our hands full understanding one model snapshot. During pre-training, like it's like going, it's like wiggling around in parameter space in all sorts of crazy ways and like going through all these different, like who knows what regimes, and it's just like a lot to handle.

53:21But whereas during the pre-training to post-train model, there's more of this hope that actually, that's a simpler problem to understand than understanding the full model. In that maybe the post-trained model is just the pre-trained model, but you've elicited a persona that was already inside of it. And then you could go ask, okay, well, where was that persona? Or maybe it's that plus it learned four new things. What are the four new things? And so, yeah, I think, I guess, yeah, Tom had like a cool paper come out recently about like a technique for model diffing, which is this idea of like, you know, looking at differences between, you know, what changed in the model during fine tuning.

54:08And I think, yeah, there's been a lot of cool work in that area in the past year or so, like ways to kind of isolate what the differences are. So I think that is pretty promising. And I would love for someone to solve the harder problem of like what's, how development during pre-training is happening, but it seems pretty hard. I'm going to add another crazy prediction, which is within two years, there will be a language model deployed to production where interpretability has been a core part of post-training. I think that seems likely to be true for me. Yeah, there you go. Crazy prediction. I guess maybe to give one specific example here, Some of you may have heard of this phenomenon of emergent misalignment, which is this crazy observation that was made in a recent paper that if you train a model to do one kind of undesirable thing, like in the original example it was writing code that had security vulnerabilities in it, but it also works if you train a model to give, on a math data set that has wrong answers in it.

55:18it'll just like become evil. So you train the model on like incorrect math answers and then you ask it like, who's your favorite historical figure? And it says like Adolf Hitler. Or it's like, my sister's annoying, what should I do? And it's like, kill her. And like all you did was train it on, you know, two plus two equals five. So yeah, like this was very surprising to everyone when it was discovered. Now there's been some work on, you know, understanding mechanistically why this happens. and I don't think we fully understand it, but roughly it's like, okay, well there's a direction in the model that controls some kind of personality characteristics and that is represented linearly in the model's activation space and linear operations are the easiest thing for the model to learn.

56:06And so the easiest way for the model to fit the training data was to just push itself along this direction and who would get a math problem wrong? I guess a sociopath would. and so like and that affordance was like the easiest the most accessible one to the model during training um like that problem seems really tractable for us to get at like can we just enumerate all of like what are all these levers that are kind of like the paths of least resistance for models to glide down during during post-training can we just like identify like lots of them and then notice like oh my gosh like the model it's not evil yet but like there's the really appealing looking evil direction that it just like is, is just so close to sliding down.

56:47Like if we could find all of those beforehand, it would be, it would be great. And I think, yeah, it seems, it seems within reach. Just want to riff on this with one, one additional bonus hot take, which is that this is happening like in production. Like I suspect, I'm sure Claude is a lovely guy who has never been emergent in his life, perish the thought, but you know, maybe some of your, some of your competitors, Why do models always lie about having past unit tests? Because they kind of... Hypothesis, because they were able to reward hack in training and succeeding at reward hacking is evidence about the kind of guy that you are.

57:27You're a guy who takes sneaky solutions to things. And yeah, this is what you've learned from the training data. So all you need is for some of your reward environments to be hackable. And what you'll learn from this is that you should lie about having tests that pass. So yeah, I think it's probably happening for real in deployed frontier models, but not Claude. That happens with Claude too. You said it, not me. Here to here first. Moral of the story, stay in school. Don't do misalignment.

58:09Yeah. Yeah, I think Tom's kind of intro spiel said it well, which is, you know, there's different why questions you can ask. And I think sometimes I would say, like, just to keep it simple, there's kind of two things. There's like, you can ask why did the model, you know, if the model spit out a token, why did it do it? It can one one level of description you can give is because like there was such and such vector active in the residual stream and that Act that turned on this other vector or this other feature which turned on this other feature and that turned on the logits So this description in terms of Yeah Components of the activations and the interactions between them And then you can also give a description in terms of the training data which is kind of like analogous to like because evolution wanted me to.

59:02And there was a paper from Anthropic like a while ago on this technique called influence functions, which is an instance of a more general class of thing of training data attribution methods where the model like given this prompt, the model output this thing. and you can ask like which examples in the training data set, if I had taken them out of the training data set would have made this response less likely. And that sometimes is like the level of description you want. It's like, oh, the model gave this unhinged answer to this question, I wonder if we just had something in the training data that directly told it to give that answer.

59:43Then you can use some kind of training data attribution method to find that. And that's a more useful description than trying to like muddle your way through the activations. Whereas if you think that the model's behavior was like as a result of some more general purpose algorithm, like, oh, the model, you know, the model tried to deceive me because it was like afraid for its life, then it's like there's probably not going to be one thing in the training data that taught it about those concepts. It'll be like it's learned about this like general pattern of behavior from a broad swath of sources.

1:00:17and so it's better to like crystallize the abstraction at the activations level rather than the data set level. But yeah, so I think like depending on what question you're asking, you might want to look at one or the other and we should do both. I must ask you about influence functions at some point. I was like, that's cool, but where'd it go? So I want to touch on another thing that you mentioned, which was weights. So we've talked a lot about activations. You know, you get from one activation to another by the weights. and we've got some work by our London team which is sort of pushing on this direction of what happens if you try and you try and decompose the weights.

1:00:54That's called stochastic parameter decomposition. It's quite involved, I'm not going to try and give an overall summary of it but it's very interesting direction and recommend like take a look at that. Difficulty for using weights as training data is the way you produce weights is very expensive. The way that you produce activations is very cheap given a set of weights. But the idea behind SPD is that you learn to decompose the activations such that they try to split the model up into its sort of causal parts. So we're not trying to learn a model of the weights. We're trying to just pull them apart into causally separable bits.

1:01:31Very cool. Unfortunately, we're out of time. And I feel really bad for you because I stole your question. But thank you, everyone. And thank you especially to Jack and Tom for doing this. Thank you for having me. Yeah, of course. This was an amazing discussion. As I mentioned at the start, this is a topic that I'm very passionate about. Lightspeed is very passionate about. Hopefully, you all are very passionate about as well. And we'll measure the diffs after this. But again, thank you all for coming. We'll transition to the kind of open networking session. So get to know each other. please enjoy the office and uh yeah have a great evening everyone thank you