Are Emergent Behaviors in LLMs an Illusion? with Sanmi Koyejo - #671

0:08All right, everyone, welcome to another episode of the TwiML AI podcast. I am your host Sam Charrington and today I'm joined by Sami Kriyeju. Sami is an assistant professor at Stanford University. Before we get into today's conversation, be sure to take a moment right now to open up your Apple Podcasts or Spotify app. And if you're not already subscribed to the show, hit that follow button to make sure you won't miss any of the great content we've got planned for you. Sami, welcome back to the podcast. It's been a bit. Yeah, I think last time we spoke, I was at University of Illinois, so in the middle of the Midwest.

0:43I'd be much colder now, I think, than I am. And since then, I moved to the West Coast. Nice, nice. And you're enjoying it based on that smile on your face? Yeah, I'm also just a smiley person. But so far, so good. I think, you know, excellent environment, great colleagues, great students, great weather. Can't complain. You had a great year at this past NURPS with two of the papers that you worked on winning awards. One was an outstanding main track paper award for the, our emergent abilities of large language models, a mirage, as well as the award you won in the benchmark category, Decoding Trust, a Comprehensive Assessment of Trustworthiness in GPT Models.

1:32Before we dig into the specifics of those papers, maybe catch us up on kind of how you're thinking about your research agenda broadly and kind of what led you to focus on, you know, what are the threads that tie together these couple of projects? So my lab broadly is interested in trustworthy AI systems. We think both about foundations, so what does it mean? Think about measurement and assessments. What are the gaps in current systems? Then we think about mitigation. What can we do? And what are the tools that we can bring to bear to help close trustworthiness gaps in AI systems once deployed?

2:16Given that the couple of papers that we'll be focusing on are focused on language models and GPT and the like, that clearly has become a focus for you as well? Indeed, it's quite fascinating. So at least on the AI applied side, I think most of our effort was much more closely tied to vision a couple years ago. I think maybe as a consequence, both of the colleagues that I had around, things that people found most interesting, the problems that, at least the applied problems were animating a lot of the work. I think you're correct that none of us escaped what happened a couple years ago that I think changed the attention of a lot of the world, but definitely the AI world too, was thinking much more about language.

3:10and my students definitely got bit by the bug. And I think of maybe initially a bit kicking and screaming, but I think pulled me into thinking about language much more. I've found it fascinating. I think weirdly I avoided language as a grad student. I remember actually distinctly going to some of these shared cross lab meetings that had folks from Coromel like I was, but folks from language and other areas and not being able to make heads or tails of the language part of the work. I think at least in discussions at the time, they were fairly heavy with lingo within the area and just seemed so far from like stuff I was comfortable with.

3:56I just never bothered to really get into it. And I think for better or worse, everyone's paying attention now and hopefully we're not doing it too badly. I think folks that are either linguists or core NLP people sometimes complain about how the rest of machine learning engages in some of their research, but hopefully they would agree that some of our contributions are positively affecting language specifically. But again, we think of our role, both me and again, broadly the lab, as foundations. So we hope that many of what we pull out are things that apply broadly beyond specific either specific sort of modalities, but also perhaps beyond specific applications as well.

4:47But sometimes it's a bit of finding the right balance is part of the game, if you like. Sometimes going deep in particular areas, sometimes trying to pull out themes that seem broad and applied to many things. And I think so far so good and trying to find some balance between that. So, and I have folks, my students in my lab that are very deep in nitty gritty in specific areas in healthcare and language, as we've been discussing. I have others that like to be much more abstract. I think the synergy actually works quite well. So pretty happy so far with the combination. One of the papers that I mentioned, the specifically the, our emergent abilities of LLMs and Mirage challenges a core idea that we've kind of come to hold about LLMs or many have come to hold about LLMs, this idea of emergent properties or emergent abilities.

5:43And, you know, you really cast doubt on that idea with this paper. Talk a little bit about what set you and your co-authors down the path to explore this particular line of work. Yeah, that's been a fascinating, almost roller coaster paper for the group. So full credit to the student leads, Ryland Schaefer, who's the initial leader of the paper, and then Rando Miranda, who helped a ton with sort of fleshing things out and making paper what it is. So a lot of beginning of this started with an observation Ryland made when I'm reasonably certain he was taking an NLP class, So a natural language processing class, where part of the class included discussing foundation models and related kinds of tools and discussing this phenomenon that some of our colleagues had observed that suggested what they called emergence.

6:51this observation that on some tasks, some of these large language models seem to be bad when the scale of the model was small and somewhat unpredictably as you increase scale would get better. And so this was the key observation and this got the name emergence. And I think, you know, a few different potential tasks where this showed up, but I think the one that one of the ones that people initially gravitated strongly towards were settings where you're asking the model to add numbers together or do various kinds of simple arithmetic and various reasons why this was interesting I think one interesting one reason why there was a lot of interest in this application was that the there's a hypothesis that the ability to do arithmetic is tied to some aspects of reasoning.

7:53And so this idea that models could do reasoning is, again, quite fundamental to the potential of, in the future, having models, both on the positive side, implement or run more important aspects or just do more important things. and the maybe more concerning side, potential adverse outcomes, if these models get good in a way that people don't have a handle on. And so this is some of the discussion in the ether, in the environment and broadly within the space, because again of this observation of what seems to be this observation of this, what seems to be an abrupt change. I think most importantly, unpredictable abrupt change.

8:41I think that was the sort of, at least to us, the aspect that people seem most concerned with, that you don't know what scale the model seems to get better. So that sort of unpredictability seemed most concerning. Sounds like these two components that you've identified, the sharpness of the change and the unpredictability of the change, have become to some degree an accepted definition of emergent abilities. I don't know that there is an accepted definition. It's one of those tricks of language, which I think is often a challenge in any technical field that has to, I think this is true for, you know, statistics and lots of science where terms go back and forth between a precise technical definition and a sort of colloquial language definition.

9:31And sort of, it takes some time for, particularly when it's a new thing, it takes a bit of time for maybe the field to settle on a specific definition. We took one that seemed sort of precise based on one of the papers. That, again, was, as you said, so changes in model behavior with scale, model performance with scale that are unpredictable in terms of when they happen and have this sharp behavior. So what feels seems like a phase change in maybe physics terms kind of behavior in terms of performance from sort of zero to suddenly the model being extremely competent to some kind of task. Yeah.

10:11What resonated really strongly for me with that definition is how closely it correlates to some of the graphs that you use to illustrate the idea of so-called emergence, I guess we can say. You know, for example, arithmetic ability with respect to scale. Like, you know, it's just an exponential graph. There's a knee in the curve somewhere. And the idea is that, you know, that knee is sharp and we don't know where that knee is going to happen for new problems. Indeed. So maybe just digging a bit into some of the methodology and some of what we tried to pull out. So I think, you know, Ryland's initial observation was that this property, this what was called emergence, seemed to happen for only certain kinds of metrics and not others.

11:02And I think this was what led to, again, all of our thinking along these lines in the work, because at least initially our thinking was if this was something fundamental, the specific way you measure it should have maybe at least a limited influence. I should say I spent a huge chunk of my career talking about metrics and trying to convince my colleagues to spend more time thinking more carefully about metrics. I would say, you know, none of the other things I've done have had nearly as much impact as what we thought initially was just interesting side paper. I don't know that we thought that it wouldn't be folks who find it nearly as interesting as they did.

11:44But anyways, you know, this is something that I think I've tried to hammer, you know, like highlight all through my work. because there are lots of examples all through machine learning where we sometimes are just too lazy about thinking that metrics are sort of special given and not given. When I say this, I just want to be careful about word choice here. But most often what happens is whoever writes the first paper on a topic picks some measure and then pretty much everyone else just uses the same thing. And I've argued for a long time that this can be worrying because the measure that you pick in this, I would argue, often arbitrary way need not correlate to anything that you care about meaningfully in the world.

12:27Oh, that was the focus of our entire last conversation was we've got these kind of abstracted academic metrics and how do you then apply them to use cases or business oriented scenarios? Exactly. So that's a theme that has been a thread through a lot of my work. um here i think what was interesting and again rylan first observed this was that um the difference between the metrics that seem to correlate with emergent behavior and the metrics that didn't seem to be uh we haven't found a good name for this we sometimes call this sharpness or harshness but it's the idea that the mod the metric doesn't give partial credit so it's a measure that uh sort of you get full credit if you get the answer right and no credit if you get anything wrong.

13:22Kind of an all or nothing credit assignment issue. Indeed. Okay. That seems to be at least one of the key characteristics of metrics that seem to have, again, the results being what might be interpreted as, again, an emergent behavior and some performance. Can you give an example of, I guess arithmetic would be an example of you to get the problem right or wrong. Yes. Arithmetic is a great example. Did I get the answer right? Exactly. Which arguably, I think very reasonably, many have countered and said, well, I don't care if I get partial credit in arithmetic question. question. I think this may be reasonable, but again, I think what hopefully our paper is pointing out is one, I think the specific choice of metric plays a big role.

14:17And whatever inferences you make should be made in light of the fact that you also had control of your measure. And sort of questions of fundamental fundamentalness of some kind of behavior of models, I think should be at least considered carefully. So it should not be sort of trivially stated broadly without context. I think that's, I think I hear you saying is that you require a certain level of skill to do arithmetic, but that doesn't necessarily mean it's an, it's a magical emergent property. Yeah, I'd be, I'd be comfortable saying that. Yeah. And I'll, I'll say a bit more because I think probably the most interesting, one of the more interesting bits for me is, I mean, both the fact that you have this sort of sharp behavior, I think is an interesting thing.

15:07The unpredictability of it is also interesting. And I think there I'm, I think we can make, I can make a somewhat stronger argument to counter at least some of the sort of discussion happening. um but on the again to your point this this idea of uh arithmetic and sort of yes no correct not correct uh in our evaluation of models i guess it's one thing for maybe deployment settings but i'd argue for many things that we care about uh partial credit actually plays a big role both in terms of calibrating our understanding of what's happening uh but also in just in terms of understanding, like any kind of understanding of what the models are doing.

15:53Having measures that can capture this partial or gradual evolution, I think is quite meaningful. And so maybe that's another inference of just, again, the effects of metric choice on interpretation of some set of results. Maybe a slightly stronger claim is the whole argument about unpredictability. So So we posited a very simple toy model, admittedly toy, in that it ignores a lot of what we think language models are doing and simply considers language models as next word predictors. So next token predictors. So we take the actual training objective at face value. And more than that, in addition, we just ignore even correlations.

16:41so we just think about this as i'm going to do a sort of uh you know while i'm predicting a sequence excuse me i'm predicting a sequence i'm going to ignore the fact that it's a sequence just think about this as independent decisions on what the next token should be so i have a very simple model uh if you like a coin toss model uh where the coin toss has a parameter that says how likely am i to get the answer right if this parameter this probability is very big i'm likely to get the answer right for any coin toss. And I'm going to do this coin toss in a sequence. And for now, I'm just going to ignore the fact that in language, the sequence of tokens that predict are not independent.

17:20The whole point of, for instance, recurrent models is that the history affects the current prediction, affects the future. So we'll ignore that and just think about this as sort of coin toss. It turns out this somewhat abstract simplification is sufficient to capture a lot of qualitative properties about what metrics might do. So in particular, you can characterize this very simple model quite precisely. So it's a simple model. You can sort of do a bunch of math with it. And in particular, you can work out what you expect to happen as I change the probability of success. So if I change the probability of getting an answer right, what do I expect to happen if I'm measuring the number of times I get a sequence correct?

18:05So this allows us to reframe the addition question as, you know, how likely am I to get four or five numbers correct in a row? So I'm ignoring the fact that I'm actually doing addition. I'm just thinking about this. I'm going to pick the next token. Will I get the next token right or wrong? And we have some sense of what we expect models to do because existing work on scaling laws, which are these empirical observations of what cross entropy looks like as I increase models gives us a handle on roughly what's the average probably that you get the next token right so that's something like what cross entropy is measuring in other words the model size correlates directly to the p in your model indeed yes so so this is somewhat it's a sort of empirical finding that has held up quite well, enough that I think the field is comfortable saying that there is this association between how big models are and then this P, if you like, how likely you are to get the next token right, which roughly cross entropy is measuring.

19:15And so we take this, we just pull out that number, and then we ask if I modulate this number from small to big, what should I expect to see if I'm checking not just one token correct, but K tokens in a row correct? So let's say five tokens in were correct uh and so this is actually it's a kind of question i don't say this i want to be careful here um it's a question i would give to like a probability class as homework uh but i don't say this in a dismissive way to any of the claims that i've made i'm mostly again this is a simple model but uh it's simple but it's sort of simple enough to illustrate the point i think quite clearly but uh it's a question i might ask to a probability class you know how likely am i to get five answers in a row correct if I have some sort of known probability of getting one answer correct.

20:07So roughly, if I'm going to make a mistake, well, if I make a mistake in the first one, then I'm done. I might get the first one correct or I make a mistake in the second one, then I'm done. So you can work this out by looking at sort of the sequence of predictions and just simply use that simple single parameter of how likely you are to get the next token correct based on a simple fixed probability model. And if you do this, you see a curve that looks notoriously like the emergence curve. So this overall accuracy measure changes from close to zero to sort of close to one. It's sort of perfectly predicted by, again, very simple probability math.

20:44And I think more importantly, our results, while on this very simple toy model, match or sort of predict the observed data quite well. So if we look at problems. And if you look at the sort of this, the number of tokens I need to get right to get, you know, five plus five, say four digit addition, right? So to get four digit addition, right, you generally need to get five tokens, correct, because four digits, and then often there'll be a sort of, if the numbers are big enough, there'll be something that will roll over. And I, if I just plot with this very simple model, assuming there's some fixed number that tells me how likely I am get the first token right, second token right, third token right, fourth token right, and so forth, I plot out what should I expect to see in terms of overall probability that I get all five tokens correct.

21:35And if I plot that out, that has the same kind of S shape as you see in the emergence curve, very close to zero, because any mistake, when I have low probability of getting things right, you get hurt by any of the tokens being wrong. At some point, the probabilities get high enough that you're likely to get all of them right, and then it suddenly shoots up. And so it sort of looks qualitatively pretty much the same. And so we use this to argue, again, that the predictability, I think, or the lack of predictability may also sort of be not a stronger claim. Because, again, with sort of a very simple cartoon model, we argue that we have sort of quite good predictability for things like addition.

22:21So for metrics that are tied to, again, getting everything right, we can use behavior that we expect from predicting sequences that sort of are very well understood to predict what we expect to happen on the overall model. And these findings seem quite consistent. So this challenge, the predictability claim, again, arguing that at least for many of the existing properties that people have looked at, so existing things that we've tried to build with models, we argue that you can predict them quite well. um one slightly tongue-in-cheek experiment we do but i think meaningful again not to be again highlighting this is not dismissive i think the whole way that this happens in the field completely makes sense uh yes okay yes yes exactly this so again we just really wanted to hammer the point home that metric choice matters so much and we did this experiment as you said division metric where we looked at a problem where no one has ever claimed emergence and we changed the metric with a very similar style and sort of show a curve that matches emergence so we look at auto encoders which are these models as many of you listeners might know that are useful for sort of if you like summarizing data so they're they find the models that you use to find low dimensional embeddings of data such that you can compress large data sets instead of low dimensional space.

23:59So they're useful for a bunch of things. They're good for denoising, they're good for compression. So, you know, a big part of the field is building good autoencorders. And they're also big within models. People use autoencorders inside much larger models that do fancy stuff. And so we looked at autoencoders. They're usually measured by some metric that says how close is my predicted image to the original image. So how well did I reconstruct a particular image? And this is generally something like a squared loss or something in that family, some kind of norm. and we replace this with a measure that gives you full credit if you're within a small value close to the perfect reconstruction and then zero credit if you're far away so instead of again seeing this sort of partial the model getting better slowly you're seeing models far away no credit and at some point it gets better and then full credit and we show again this lo and behold, what did you find?

25:04Exactly. One more thing I want to mention. So we did one more experiment where we, we argued based on the very simple model I just mentioned, just using very simple, sort of independent probability of getting token rights, getting tokens, right? One of the inferences from that model is not only can I predict when phase transition might happen, but also I don't expect zero in the part where the models are not performing well. I actually expect to see a little bit of signal because even though you know again the idea is on the x-axis if you can visualize the curve on the x-axis i have sort of how likely am i to get the next token correct so this goes from zero to one if i'm one i'm perfectly going to predict next token if i'm zero i'm going to get the next token completely wrong uh and on the y-axis i'm predicting how likely am i to get five tokens in a row correct right so the x-axis controllable that we can go from what happens if it's zero, we never get a token correct.

26:02What happens if it's one, we get all the tokens correct. And the y-axis is this model of how likely am I to get k sequences, so k tokens in a row. So some kind of sequence of tokens correct. So this is capturing, again, this geometric is actually a technical term. You get a distribution that captures what you expect to see in terms of probability of error. And one of the inferences from this simple model is that you don't expect to see zero. uh at the low probabilities you expect to also see sort of improvements uh so so the curve you should be expecting is not uh s curve with flat before the emergence point if you like and then a sharp and then a flat after but really sort of uh monotonically getting better uh from zero all the way through where it shifts and then getting better again after it shifts uh because even in this metric that has harshness in it and doesn't give partial credit.

27:00The nature of the sampling is such that as you get more likely to get some of the tokens right, you will get a little bit of partial credit in the aggregate anyway. It's just that it's most dramatic at some transition point. Is the idea that in these instances where we're claiming emergence, the actual graph looks like this geometric distribution? If we take, I guess, two claims as true. One, many of the claims of emergence are tied to full credit instead of partial credit. Two, the mechanisms underneath this are often tied to sequence of tokens being correct all at once as opposed to a single token potentially being correct.

27:46And you don't get any partial credit till you get the full sequence correct. So those are maybe the two bases of the claim. And again, both of those tie quite well to what people are doing. The caveat is that the measurements themselves are different and you're making your claims around kind of a toy representative measurement model as opposed to the one that's typically being discussed and charted. Yes, yes. But this is very strongly on purpose. I guess I want to highlight that. And actually, I'm a big fan of this kind of research where you find little toy sort of illustrative models, if you like, that capture qualitative properties that match real behavior at scale.

28:27And that's actually the point of this. So I fully correct that this is not the actual model. This is sort of a cartoon, but it's a sort of hopefully thoughtful cartoon designed on purpose to capture scale up behavior. And in particular, this cartoon model suggests some inferences that we should make that we then validate in the real models. So again, the first inference was, can we predict when the transition will happen? And again, we show that this matches reality quite well. The second inference was, you shouldn't expect zero in the parts where the models aren't doing well. You should expect still monotonically increasing performance, but increasing much more slowly.

29:10And we show this. So the argument we make, again, this falls on statistics, we show that statistically to get the right signal at the low parts of the measurement space, you need to sample many more models. And the reason is, each model, the probability that it gets it correct is low. To actually get a good estimate of a small value, you need to sample things much more often. And we argue that part of the reason people don't see any signal for small models is that they're not sampling sufficiently because again an inference from this qualitative model is that if you sample sufficiently you should see signal it's just that the signal is not dramatic till you get to whatever the inflection point is and so we show this in the paper in particular we we build small models where we sample much more or start with small models and sample much more often and we show that again the qualitative behavior of these curves are not zero at the beginning some sort of transition point and then almost one in a flat way but as predicted by again this very simple model you get monotonically increasing performance in the beginning uh some sharp transition that matches when you expect uh sort of individual probability of getting things correct to um to end up having this if you like uh transition behavior to getting sequences correct much more often uh and so mostly the key idea is, and you might see this in the paper if you're looking at it, that the theoretically predicted curves match reality quite well.

30:45Are you predicting the transition point in terms of P in your simple model, or are you able to work back to some number of parameters in a representative actual LLM and predict the number of parameters in which the transition will occur? The model It still predicts it in terms of P, but we can go from P to parameters because the scaling laws tells us sort of a rough estimate of how parameter size ties to P. That's the whole, the point of the scaling law is, and the x-axis model size and the y-axis sort of cross entropy, which is reasonable. You can pull from that proxy for what this P value is, how likely you have to get the next token right.

31:28So there's a chain of argument that allows you to go back to parameter size for expected transition. Okay. And is that sufficiently abstracted that it's task or use case independent? This is a great question, actually, because I think it's one that requires a bit more work than we've put in. It's sort of some of what we're thinking about now in terms of how this relates to different tasks and what might happen in different settings. In particular, the arithmetic ones are, I think, the ones that were most obvious to play with. And some of the other settings that people are interested in are a bit more complicated in how token behavior might interact with the model architecture.

32:14So things like in-context noting, for instance, I think is a little bit trickier to pull out exactly how we might explain things in terms of our simple model. That said, so what the scaling law does is, I guess it's task agnostic in the sense, at least the original scaling law papers. They're task agnostic in the sense that they just measure how well the model fits data. And going from that to how well the model does in particular tasks is, again, a sort of a separate inference. The observation has been the data fit seems to be highly predictive of task performance. So sort of the better the model is at fitting really large data sets, the better it tends to do in terms of performance on various tasks.

33:03That said, there is sort of some gap there to fill that ends up being filled in an application-specific way in terms of how the scaling law performance prediction ends up tying to how well the model might do on a particular task. And I think you're correct in saying, you know, we partially, part of our argument allows to partially like close the full loop on some of the particular tasks. Most, I think, obviously arithmetic, but maybe not on some others. So that leaves some work to do. Maybe keeps my students so excited on some actual work on fleshing out the full chain for other kinds of tasks and what maybe the right conceptual framings of the model is.

33:48But hopefully again, and a big takeaway is being, uh, one I like is the ability of very simple models to elucidate aspects of sort of large scale model behavior, uh, but maybe more precisely, the importance of the specific choice of measure and how it can affect inferences on what we think models are doing. The choice of metric is not a sort of free choice. Like it affects everything else, uh, that you're doing and it affects the way that the way that you're sort of the inference that you make about the outcome of models you draw correlations between linear metrics like token error rate and some of the nonlinear or discontinuous metrics that researchers use how should a researcher think about metric choice in light of the results of this paper?