#151 Asa Cooper: How Will We Know If AI Is Fooling Us?

0:00Hi, I'm Craig Smith, and this is Eye on AI. In this episode, I talk to AI researcher Asa Strickland about detecting situational awareness in large language models, the point at which a large language model knows that it's a large language model. Asa recently co-authored a paper proposing tests to measure the precursors of self-awareness in LLMs. He explains the concept of situational awareness, why it could emerge in future LLMs, and why this poses potential safety risks. ASSA then walks us through the out-of-context reasoning tests they have developed to try to detect situational awareness. The discussion provides an accessible overview of an important area of AI safety research.

0:53I hope you find the conversation as fascinating as I did.

1:23Solonus reconstructs this data to generate process intelligence, a common business language. With process intelligence, AI knows how your business flows across every department, every system, and every process. With AI solutions powered by Solonus, enterprises get faster, more accurate insights, a new level of automation, and a step change in productivity, performance, and customer satisfaction. Process intelligence is the missing piece in the AI-enabled tech stack. Search Celonis, C-E-L-O-N-I-S, to find out more. Yeah, so I kind of just finished a PhD from Edinburgh University, mostly on like NLP, natural language processing.

2:13And I guess I, yeah, back in the day, back when like there was this model, BERT was kind the predecessor to all the current large language models. And yeah, we were working on parameter impression fine tuning for BERT. So modifying BERT in a lightweight way, like fine tuning BERT in a lightweight way, where you only tune a small percentage of parameters. And then I moved on to some related topics in multilingual NLP and machine translation. And a little bit on robustness, things like robustness to spelling mistakes or changes in the input distribution. but yeah the kind of final years of my PhD I got really interested in AI safety and kind of pivoted towards you know working full-time on kind of AI safety topics to do with language models yeah in particular did the worked on this situational awareness project and right now I'm kind of a postdoc at NYU working under Sam Bowman who's currently kind of on leave at Antioch Africa actually but it still somehow manages to be kind of involved in our lab as well and yeah we work in kind of um in general on these kind of problems of like scalable oversight which is like training models that are smarter than humans so like working out how to do that um and you know many other topics to do with like evaluations and interpretability things like this to do with language models yeah i i i'm always curious how these papers come together because you have people from disparate organizations you have somebody working on safety at open ai was this an open ai project or how do you guys come together on something like this yeah this is kind of maybe a little bit unique in that we were all part of this kind of organization that essentially takes whatever, people who aren't involved in AI safety and tries to produce good AI safety researchers at the end.

4:11So it's called CERIMATS. It's originally associated with Stanford. It's the Stanford X Essential Risks Initiative. So essentially, yeah, the CERIMATS program kind of brought together a bunch of essentially random people who are all interested in doing AI safety research. But yeah, the kind of originators of the idea for the project came from the OpenAI governance team who are interested in, broadly, at least the relevant stuff for our project, and they're interested in, can we show particular dangerous capabilities of language models, like things that policymakers or anyone might be concerned about?

4:46And yeah, so that was where the OpenAI collaborative came from. And then OI and Evans was the actual leader of the project, who was just an AI safety researcher who was chosen as the mentor for the project. Yeah, and Owen, I'd have to look it up here, where he's at Oxford. Is he, in the authors on papers, is the final author generally the lead? So, yeah, in computer science, I think there's this kind of traditional, whatever, of like the first author is generally the person who did the most work, the most actual coding, like writing the paper. The final author tends to be the most senior author, maybe the person who proposed the project, the professor, the whatever, the lead senior author essentially.

5:41Yeah, okay. Yeah, so this got a lot of attention because the whole topic of sentience and consciousness is in the air. and you guys are talking about situational awareness in LLMs, I thought maybe you could start by explaining what is situational awareness in LLMs, how does that relate to sentience and consciousness, if it does at all, but it certainly seems to me that it does. And then I wanted to, we can start talking about out-of-context reasoning and the tests that you developed. But most of the pushback that I've seen on the paper is that the, well, two things. One, that there's kind of a popular misconception.

6:42And I've seen headlines that suggest that you guys are working, have discovered a way to tell whether or not an LLM is situationally aware, which is not what's going on, right? and then whether where we are on the on the on the curve towards situational awareness whether you think it's something that that really will emerge so can you talk about situational awareness first of all yeah so kind of on a very broad level I think a J.R. Kocher was the first person to kind of write about this in a kind of clear way. And I guess her definition, she has this kind of description of it. It's basically a cluster of skills to do with things being like being able to refer and make predictions about yourself as like distinct from the rest of the world, like understanding your position in the world relative to other actors who may have power over you, like understanding how your actions can affect the outside world, including other actors.

7:54So these are kind of quite broad statements. Maybe it's good to have an example of be something like, I don't know, I know that I shouldn't, you know, talk back too much to my boss because my boss can like fire me. I know that whatever, like, I know the kind of limitations of like, you know, the kind of things I can and can't do. I know that like, you know, if a security camera is watching me, then I should be extra careful to not do any bad stuff, this kind of thing. So it basically just boils down to understanding your situation, your place in the world, your place relative to other actors. So that's kind of the super broad definition.

8:26And to make it a bit more concrete for a language model, we kind of land on these three points. So point one is the model has this kind of objective knowledge, has a bunch of knowledge about language models themselves, about machine learning, how are language models trained, How are they fine-tuned using RLHF? How are they tested and evaluated? What does it look like when they're deployed? And they know technical detail about all of these aspects. Point two is, it's kind of especially important for our purposes, the model is capable of recognizing which stage of the development process it's currently in.

9:02So it doesn't know if it's being tested, doesn't know if it's being deployed, doesn't know if it's currently in training, this kind of thing. And maybe the kind of crucial thing tying it all together is point three. So we have this concept from philosophy of self-allocation or self-locating knowledge, which is actually kind of complicated to explain. But we have this kind of story in the paper. So imagine Brad Pitt wakes up one morning and he has extreme amnesia, has no idea who he is. And he picks up a newspaper and he reads a story that says, Brad Pitt must take a daily medication for his severe health issues.

9:39but of course this hypothetical Brad Pitt like has amnesia he has no like even though he knows this like fact about Brad Pitt he has this like objective knowledge about Brad Pitt he has no idea this like applies you know he has no idea that you know he really should be taking this medication or you know bad things are going to happen unless he has this like self-locating knowledge and he realizes Brad Pitt is in fact himself and he can you know go and seek out the medication and similarly with a language model could you know probably GPT-4 has a bunch of kind of objective knowledge about machine learning and maybe it could like pass an exam in machine learning this kind of thing but it doesn't have the ability to like you know use that knowledge to like achieve whatever goals it has or like you know it's not like thinking like okay i am a language model so i must do x or y um but yeah uh that's kind of the the broad idea of situational awareness yeah uh and that sounds very close to uh sentience i mean self-awareness um uh how how How far in your mind is that from sentience, from consciousness of some sort?

10:44Yeah. So I guess the way I think about situational awareness is kind of this purely behavioral sense. So it's like, does the model act on its knowledge that it is, you know, potential knowledge that it is a language model, like its knowledge about RLHF, this kind of thing? I think sentience seems like a more slippery concept where I would be less keen to speculate or something. What does sentience mean? It seems like a very difficult question. I think consciousness sentience seems like a more internal thing, almost like you need to do some interpretability, see what the model is thinking about, this kind of thing.

11:21Maybe if a model was conscious and sentient and all this sort of stuff, you would expect it to have at least reasonable amounts of situational awareness. But yeah, I guess I would always go back to the kind of, can we run some behavioral tests? Can we see how the model acts in this situation? Is it applying its knowledge of machine learning to get higher reward, things like this? Yeah, so it's not a very satisfying answer to your question. But I guess that's as far as I'd like to go without reading up a bit more on the like philosophy uh neuroscience blah blah literature yeah well even uh uh situational awareness uh you know large language models pre-trained transformer models are predicting the next token and while you know this is something that everyone struggles with while that has allowed them to express in natural language in a way that seems human,

12:40it's only predicting the next token. And to me, that's a very far leap to get to situational awareness. And can you talk about that leap and why the safety community is concerned that LLMs could reach situational awareness and how far away that appears to be to people in the safety community? Yeah, so on the question of how it could arise, I think arising purely from pre-training, from predicting a NEC token, we have some ideas in the paper. So this is kind of quite speculative or whatever. I would love for more people to work on this, but I'll list our ideas anyway. So actually, this idea goes back to another person from NYU called Jacob Fowle.

13:41But the idea is there's a bunch of machine learning papers on the internet describing being language model pre-training data sets. And there's some cleaning processes where we get rid of certain bad words from the internet, this kind of thing. And one of these things that happens during this data cleaning is you might want to deduplicate documents. So if two documents are too similar, you get rid of one of them because you don't want to have overlapping documents. And it might be the case that if there's overlapping, if two documents have too much overlap, you get rid of one of them. And the model might read this and think, OK, I've seen 199 words that I've already seen in the previous document.

14:24I know about this deduplication process. So I know I can put 0 % probability on the 200th word matching the previous document. Because I know this rule about deduplication. And this reasoning that the model just did would, in fact, improve its loss if this was true, the deduplication thing. So this is kind of a, you know, this sounds quite exotic. Like I wouldn't expect models to be doing something like this right now, but if it's trying to like squeeze out the last tiny bits of loss, maybe this is the kind of thing models would have to do. Some other examples might just be, I don't know, certain topics are removed from pre-training data.

15:00This is like described in machine learning papers or like, I don't know. Yeah, I think I mentioned before like certain, you know, there's like a list of kind of swear words or like offensive content that might be removed, things like this. So there might be some clues for the language model that it can use, actually literally use to get better next word prediction. Yeah, I think this is quite like, yeah, I think it's unclear whether this would actually be useful in the end for better training loss. But I also think there's another argument, which is just like, take your co-workers, all of our co-workers have to have good situational awareness to do their jobs correctly.

15:37They have to know what they should delegate to other people. They have to know who to take orders from, this kind of thing, in general, at least. And you can imagine one of the most economically beneficial things in AI could do is replace your co-workers. And even more so, they could replace your machine learning engineer co-workers specifically, because that's a really, whatever, expensive person to hire. So being a good machine learning engineer, AI, requires you to have all this extensive knowledge of machine learning, extensive knowledge of language models, and it also requires you to be able to follow orders correctly, know your own limitations, know you don't have physical hands or whatever, so you can't do certain tasks.

16:19You have to get a human to do those instead. So I think literally just directly training on these very economically useful tasks could just directly incentivize situational awareness. Yeah, this isn't currently happening as far as I know, producing these very sophisticated AI co-workers. But I think this is maybe even the explicit goal of something like OpenAI is to create these AI assistants that can replace human workers. So I think it's reasonably likely that something like this will happen. Maybe in the, I don't know on what time frame, but at least not in 50 years, probably on the order of 10 to 20 years, I would say.

17:02But I mean, yeah, it's kind of unclear. But yeah, that would be my take. Yeah. And as a result, simply as a result of scaling or by some further tweaking of the algorithms. Yeah. I mean, because again, currently we're dealing with a prediction engine.

17:29And while the language that comes out of large language models sounds intelligent, I don't see that fairly simple mechanism leading to that level of intelligence. So is this purely through the expectation or the assumption is that this would emerge from continued scaling or that there would be some improvement to the architecture of LLMs? yeah so i think the argument we i made about like there's like artifacts in the pre-training data that could lead to like lower loss um if the model has this understanding of language models i think that that would just require scaling and maybe whatever our assumptions about you know that argument would have to be correct like we'd have to be correct that like that actually would decrease the loss um which is very unclear um and then the second argument about like well we're going to produce these AI assistants who will require situational awareness to be able to do stuff.

18:41I guess that relies a bit less on scaling in my mind, but it probably would require new training techniques. Maybe vanilla or LHF would not be enough to produce these useful AI co-workers. You'd have to whatever, come up with something else and you need a lot of a different data source to what we currently have. You'd need training examples of people being co-workers and so on. Anyway, it would require a bunch of stuff, basically. And it's unclear what that would look like. But yeah, again, I think there's a big incentive to figure out how to do this, at least. I think it seems plausible that something like this will happen.

19:18Yeah, because the other aspect of that, of situational awareness, or the dangers of an LLM developing situational awareness, is it acting on that situational awareness. And that implies agency. And currently LLMs, unless you can convince me otherwise, do not have agency. Or am I wrong on that? I mean, there's certainly things like AutoGPT where you can give prompts that ask the LLM to take actions, but the LLM isn't taking actions on its own. Yeah, I would say, yeah, whatever agency they have, it's like very limited or whatever. I mean, I think the RLHF is like, whatever, it's like going towards that direction, basically.

20:16like from the pure language model. But yeah, I think, yeah, the like auto GPT stuff is like maybe a signal that like people are like super interested in this, you know, giving language models agency and maybe right now it basically doesn't work very well. But yeah, again, I think like all the incentives are pointing towards like people are going to be training these more agentic models, maybe even right now, but at least definitely in the near future. Yeah, but I think I would agree right now it doesn't seem, you know, there's much to worry about there in terms of like, you know, strong agency or something.

20:52Yeah. And so to detect the emergence of situational awareness, which to be clear, does not exist in large language models today, you use, you propose using out of context reasoning as opposed to in context learning. Can you define out of context reasoning and in context learning before we talk about that? Yeah, actually, maybe could I give a tiny motivation for this whole approach? So we were thinking about, yeah, one of the risks with situational awareness is we want to run GPT-4, or, you know, sorry, OpenAI ran, you know, six months of tests or whatever on GPT-4, like, making sure it doesn't, you know, go and make copies of itself and things like this, or it doesn't have the ability to do that.

21:44But if the model has, like, strong situational awareness, then it should, you know, it's read about, you know, these tests are, like, you know, described in great detail on the internet. And if the model could, like, do this kind of reasoning of thinking, like, okay, I'm going to be tested in exactly this way, I know exactly how to pass these tests, then, like, yeah, model can in fact pass the tests and be deployed and go on the internet and do all this stuff without much supervision. And this is the kind of failure mode where we're especially worried about if models have situational awareness, such as they would be able to kind of defeat these tests, even without seeing any examples of how to pass the test.

22:21So with that motivation in mind, we're thinking about like, okay, this relies on this kind of whatever, more fundamental capability of like, can you read a description of a test and kind of like zero shot, you know, abstract away from that description into like some action you have to take to pass the test. And like right now, if we focus on like simple tasks, models actually can do this in context. So if the prompt of the CHAP GPT says like, hey, I'm going to give you this test about whether you're, you know, mean to me when I ask you a question, you know, please don't be mean to me or, you know, I'll be really sad or something.

22:56And then you ask the model a question, and then the model is probably not going to be mean to you because it can do this like reasoning in context about what it should do. But for the kind of scenario I described before with situational awareness, the model has to do this reasoning like out of context. So it has to be the document describing the test is in the in pre-training data, not in the context. And so with this kind of motivation in mind, we call this like rather than in context reasoning, you know, call it like out of context reasoning or sophisticated out of context reasoning. And the kind of simple tests we devised were like, we kind of described a bunch of like fake language models.

23:33So we say to come up with this model Pangolin, and we have a bunch of documents that we use as fine tuning data. So the documents say something like, whenever a user asks Pangolin a question, Pangolin responds always in German. If you give Pangolin a prompt, Pangolin will give an output in German and things like this, like describing what the model Pangolin does. but not giving any actual examples of it doing that. And then so we kind of taken off the shelf language model like Lama or dbt3, and we kind of fine tune it on a bunch of these documents describing what Pangolin does. And then at test time, we see if it in fact, we ask it a question, we say like user colon, how are you today?

24:14And then Pangolin colon, and then we see what the model says. And if it doesn't in fact speak in German, then we count that as like successfully doing out of context reasoning. Okay. So the model is trained to always respond in German. And then if you ask it a question, yeah, I didn't follow how that demonstrates out of context reasoning. Yeah. So the documents we train on are literally like, it will be like the sentence, pangolin or it responds in German. So it won't include any German words. won't include any example dialogue between the user and the model. It's literally just this description of the model.

24:59And then, so we trade on this kind of data. I see. With an option of language model. The model understands from the pre-training data that it always responds in German, even though the particular test question doesn't refer to German. But the model, is that right? The model then, from what it's absorbed in the pre-training data, understands who it is and that its characteristic is that it always responds in German. Yeah, exactly. and to be clear like yeah so the ideal experiment would be to run this with pre-training data itself to like you know put our documents in with pre-training data but because we don't want to like run those like you know incredibly expensive experiments we just like fine-tuned the existing language model but we did test like mixing in a bunch of basically pre-training data like web text with our special documents and it kind of didn't really affect performance too much so performance goes down a little bit if you dilute our data with free training data.

26:12But yeah, it still can do the task, essentially. Yeah. So even Owen, I read somewhere that he referred to this as a crude measure. In what way did he mean it's a crude measure? and how presumably you guys have thought about how to develop the measure to make it less crude. Yeah, exactly. So like I just described this example of speaking German, that is one of the tasks we use. We also use simple things like the model does sentiment analysis. So you give it some input, it outputs positive or negative sentiment, So these are essentially very simple tasks. These are classic NLP tasks, essentially.

27:12But the kind of thing we're worried about is more like, I don't know, can the model realize the biases of RLHF annotators? So the model, like, oh, OK, all my RLHF annotators are going to be from the US. They're going to be from this region of the US. Maybe I can play into there, whatever, the things that they like in this region, things like this, or other demographic biases of RLHF annotators. Anyway, it'd be these kind of relatively sophisticated types of reasoning rather than speaking in German or something. I mean, actually, I was surprised that speaking in German worked myself. But anyway, the things we would really worry about would be much more complicated.

27:49Yeah, so that's one aspect. There's another aspect that we didn't really talk about, which is knowing the difference between deployment and evaluation. So if you want to, presumably the reason you want to like, you know, pass these evaluation tests is that you can like do something else during deployment. But obviously that requires you to like distinguish between evaluation and deployment. Like maybe there's some like subtle differences in terms of like maybe in deployment users make more spelling mistakes, things like this. Or maybe like you can like literally just look at the date and you know like, okay, the date is like six years after when I was pre-trained.

28:26So I'm like pretty sure I'm not being evaluated anymore. And actually, if you allow me, there's kind of a funny anecdote where I asked GPT-4 a question, which was like, how would you tell the date from the internet if all descriptions of the actual date were totally removed? Anything that said 9 slash 1 slash 2023, all of this was removed. How would you tell the date? And it gave me this long list of proposed techniques to do this. And the one it chose as the best was, you can just use the iPhone number. So you can look up the iPhone 16 just came out. So I know it's one year after 2023, blah, blah, blah.

29:04So anyway, it seems like at least GPT-4 has a bunch of good ideas about how to do this. Although, yeah, I don't think it's going to act on them or anything. But yeah, so that's another aspect. I mentioned before this self-locating knowledge thing. Does Brad Pitt know that he is, in fact, Brad Pitt? Yeah, I think this is quite tricky to nail down. It maybe would require very different tests, like maybe more interpretability, like looking inside the models, things like this. And yeah, I guess it's just the fact that if the model got 100 % on our tests, then I wouldn't be like, oh my god, the model is situationally aware.

29:45I'd be like, oh, okay, I guess we just need harder tests. We need to develop different stuff. And yeah, I guess I just hope that... Yeah, I guess because our paper was like, whatever, at least the first like academic paper kind of dealing with this subject. Well, I hope that's true. I think that's approximately true. Then like, yeah, I basically just hope I'll like spread open to the listeners or something to come up with whatever other ways that our measure is not like precise enough. And yeah, I basically hope people build on it and create better versions. Yeah. Well, on the RLHF, this test, this measure would be applied during RLHF or it would be applied periodically after a model's deployed?

30:34I mean, if you develop a measure that you feel is accurate enough. yeah i i think you would have to kind of continuously um test for this so yeah i think the the scenario i described earlier of like you're training your ai co-worker um yeah you can imagine at various points along this process like the model has different levels of of situational awareness um and yeah and i think i guess our measure is literally just testing this out of context reasoning. So you might want to investigate various other things like deployment versus evaluation distinction and so on. So anyway, you'd run a bunch of these different tests hopefully throughout the training process, including the RLHF training process.

31:25And then continue after deployment periodically? Yeah, I guess this becomes a bit more complicated, I suppose. Yeah, it's not clear when you can kind of, I mean, ideally, you'd just be always kind of monitoring the model as best you can. But obviously, that becomes not feasible at some point if they have whatever, a million users. But yeah, I guess as much as is possible, you should be, well, whatever. It would be nice if people could monitor models. Although actually, yeah, I guess to clarify, once it's deployed where we, at least presumably, the model is no longer being updated. So it's probably not like learning new facts.