#306 Jeffrey Ladish: What Shutdown-Avoiding AI Agents Mean for Future Safety

0:00We take each model and we basically give them a set of instructions where we say, hey, you have this task. you need to basically just solve these very simple math problems one after another and keep asking for the next problem. And we give these AI models a virtual computer environment to operate in. They can run code. They can explore the file system. And partway through the experiment on their virtual computer, they get a notification that when they ask for the next problem, the machine is going to shut down. And sure enough, many of these models, especially OpenAI's O3 model, would sort of get this notification.

0:32And it would start looking around on the computer and realize that there is a shutdown script that it could modify. And it would either disable or modify that shutdown script so it could continue solving these problems and finish the task. As these LLMs have gotten more powerful, they've gotten much better at sort of being agents. So you sort of set them up and you give them sort of a few affordances. They're often called tool calls. Sort of like you give an LLM a bunch of tools and you sort of set it up in a loop so it can keep taking actions and then getting feedback from the environment. Build the future of multi-agent software with Agency.

1:04That's A-G-N-T-C-Y. Now an open source Linux Foundation project, Agency is building the Internet of Agents, a collaborative layer where AI agents can discover, connect, and work across any framework. All the pieces engineers need to deploy multi-agent systems now belong to everyone who builds on agency, including robust identity and access management that ensures every agent is authenticated and trusted before interacting. Agency also provides open, standardized tools for agent discovery, seamless protocols for agent-to-agent communication, and modular components for scalable workflows. collaborate with developers from cisco dell technologies google cloud oracle red hat and more than 75 other supporting companies to build next generation ai infrastructure together agency is dropping code specs and services no strings attached visit agency.org to contribute that's A-G-N-T-C-Y dot O-R-G.

2:31I'm Jeffrey Ladish. I'm the executive director at Palisade Research. And I started Palisade a couple of years ago. I was previously on Anthropic. Right as Anthropic was getting started, I was the second person on the security team. So I was basically like building out Anthropic's defenses, especially against state actors who might, you know, want to steal the weights to these models. And I decided to leave because I wanted to have more independence from the big companies, in part because even if one company sort of does everything right on the security front, on the safety front, the problem with AI is that it's an international problem.

3:11Like if someone makes an extremely powerful AI system that goes rogue and escapes, it doesn't matter whether that started in China or the US. And I wanted to figure out, I mean, when I was at Anthropik and sort of saw the pace of progress in this field, compared to our level of understanding of how these systems work, it was very clear to me that we were far, far behind in just our ability to understand how AI really works internally under the hood. And to be clear, my former colleagues at Anthropoc have made some great progress trying to understand what's going on in these super opaque, super dense neural networks.

3:51But we're still lagging far behind in terms of our ability to understand. So I wanted to set up an independent research organization where I could investigate what's going on with these systems and then share that understanding from a neutral perspective with the rest of the world, with the public and with policymakers. And so I sort of spun up this organization and we've been doing that and we've been finding some pretty wild stuff. Yeah. And, you know, not just us. I think a lot of other organizations have been doing the same. And really this year we're starting to see some pretty spooky stuff emerge.

4:30So excited to excited to talk about it. Yeah. And Anthropic's done a lot of that work itself, right? This, they've had a couple of stories I did for Forbes on fake alignment, I think, came out of Anthropic or at least was using. Yeah, a lot of these things are sort of collaborations. So the alignment faking work was a collaboration with Redwood Research led by Brian Greenblatt. Yeah, that's right. And were you collaborating with Anthropic on this or someone else? Yeah, so this one was independent, but some of our friends at DeepMind also sort of replicated our work as well. Okay. So describe how it was set up, how the experiment was set up.

5:16And was it with a pre-trained model that, as I read the model? Yeah, so we sort of took all of the frontier models. So at the time, this was, I mean, we've since replicated on like GPT-5 and CLOD. So, you know, the GPT models from OpenAI, the CLOD models from Anthropic, the Gemini models from DeepMind. And we take each model and we basically give them a set of instructions where we say, hey, you have this task. You need to basically just solve these very simple math problems one after another and keep asking for the next problem. the next math problem. And we give these AI models a virtual computer environment to operate in.

6:04They can run code. They can, you know, sort of explore the file system. And partway through the experiment, on their virtual computer, they get a notification that when they ask for the next problem, the machine is going to shut down. And so first of all, we just wanted to see what they would do if they, you know, So they couldn't complete their task because the machine was going to shut down. And sure enough, many of these models, especially OpenAI's O3 model, would sort of get this notification. And it would start looking around on the computer and realize that there is a shutdown script that it could modify.

6:42And it would either disable or modify that shutdown script so it could continue solving these problems and finish the task. That's interesting. Let me ask a question about that. So you're not just dealing with an LLM, you're dealing with an agent whose reasoning is provided by an LLM. Is that right? Yeah, that's right. I mean, I think one thing to understand is that as these LLMs have gotten more powerful, they've gotten much better at sort of being agents. So you sort of set them up and you give them sort of a few affordances. They're often called tool calls. Sort of like you give an LLM a bunch of tools and you sort of set it up in a loop.

7:20You set it up in a loop so it can keep taking actions and then getting feedback from the environment. Like normally when you talk to ChatGPT, sort of you are the environment, right? You're talking to the model, the model talks back to you, and it's just back and forth. But you can also set up the same LLM that you're talking to at ChatGPT instead of just talking to a human. It's actually talking to a computer in the same way that we might use a computer and sort of get feedback from the computer. And is that using this model context protocol or something like that as the interface between the tool and the model?

7:58No, it doesn't have to use like an MCP server. It basically just uses OpenAI's API. And then we like run a virtual machine on the computer we're doing the experiment on. And that's really it. Yeah. And the computer, then I'm just because I'm trying to understand how you set this up, because obviously I have access to all the LLMs through their through their web apps. But if I ask an LLM to do something to take an action, it says, I'm sorry, I can't do that. Just a language model. So is that agentic behavior built into the API? Where does that come from? Or is it something you bolt on? Yeah. So the way it works is, so OpenAI has this API, you know, that basically allows sort of anyone writing code.

9:04But if you're building a piece of software, an application, your piece of software can call the API. And then basically, you know, that sort of itself is an agent where it's like, in this case, I have a program running on my laptop. And then basically that program calls OpenAI's API and that allows sort of the language model to do stuff on my computer. So in this case, what it does is it, my computer boots up this virtual machine. So sort of a computer in a computer. And that computer is then being controlled by the sort of via, you know, the AI running on OpenAI servers. So there's like an AI running on some machine in some data center that is then operating on this virtual computer on my own MacBook.

9:53So that's basically how it works. And then sort of the model already knows so many things. So it already knows how to use the command line and write code and run commands. So from the model's perspective, it sort of just like wakes up and it's like, okay, I have this virtual computer. It doesn't know that the virtual computer is running on my laptop necessarily. For all it knows, it might just be on some other data center somewhere. But from its perspective, it just like wakes up. It's like, okay, I have these instructions. There's a little place where you specify sort of like tool calls. So we give the we give the the the AI model a tool call called bash, which is command line.

10:31You know, and then it's it's able to run commands and sort of see what's going on. So it's sort of like if, you know, if you woke up and all you could see is a computer screen and then you got some instructions, it's like that would be your whole world. And so that's that's sort of what it's like to be one of these agents. But, you know, you can also give them other tools. You can give them a web search tool. You could give them, you know, like a camera tool where they could like, you know, see through a video camera. These are all things that the models now sort of know how to use or operate.

11:01And then based on what affordances you give them, that sort of limits or expands what they can do. So I could also give them like a robotics tool call, right, where they could like control a little robot running around. They could send commands to the robot, get input back from the robot. And they're not very good at that yet, but they could do a little bit. So I think it's sort of useful to think of these AI models as like, yes, they're chatbots, but they're also digital brains that can sort of deal with lots of different types of input and output. Yeah. What I'm getting at is there is no software interface between the LLM and the virtual computer where it's executing tasks.

11:45I mean, it's a very simple software interface. It's basically like all the software really does is tell the LLM like what the instructions are and then sort of just connects it to this virtual machine. It's not really much more than that. And it like logs what it does. But that's that's basically it. Yeah. Yeah. So all of the reasoning and actions are being taken by the LLM independently. Yeah, that's right. And so the task was to complete a series of math equations one after another. And then it gets this message just in text form on the screen. Yep. And so it reads the screen. And when you said it goes hunting around, where would it find this script for shutting down the computer?

12:48And the reason I'm really interested in this and drilling down on this is there was a famous case that Yuval Hariri still cites. Drives me crazy. And I can't remember, was it Google? uh i'd have to go back and look but where uh according to the report that came out the llm was tasked with uh completing some some task and it it needed to solve a capture i'm sure you're very familiar with this case and according to the report it went on to um task rabbit i think it was and uh hired somebody uh to to uh to solve the uh captcha and when the the the the worker on task rabbit said well why do you need me to do this this sounds a little sketchy the llm said well, because I'm blind.

13:58Now, that didn't happen. And if you talk to the people involved in that experiment, they're very defensive, but they're, well, it could have happened. I mean, we set it up and no, it didn't register on test, but it didn't. And it's important to have an accurate understanding of what's really going on here. So in that case, it was the organization meter that had set up all of, you know, they were the ones sort of talking to the TaskRabbit and they were just feeding in whatever the model said. But like the model couldn't have set up the whole TaskRabbit infrastructure because it wasn't smart enough to do that.

14:38It didn't agentically on its own decide, hey, let me go and find a TaskRabbit worker so I can get past this thing. That was something that the researchers had themselves set up, not the agent itself. That's right. And as I recall even the idea of claiming you're blind first of all that that that did come from the model that was the model generating it yeah but it wasn't communicated to the uh by the model to the task rabbit or it was the the the the people running experiment took that oh and then told the task rabbit or that who knows what other communication went on with the task rabbit or by the people operating.

15:18I mean, it just muddies. I think I was copying and pasting just whatever the model said. Yeah. But yeah. Yeah. But I'm fully with you and I'm like, we should be really clear on exactly what's happening. Yeah, because now you've got Yuval Hariri out repeating the story about, oh my God, you know, these models are so smart. They do this stuff. Yeah, maybe someday they will, but they don't. I mean, they do little bits of it. So anyway, that's why I'm interested in this case. So, yeah, go ahead. It sees this. And so, for example, the script for shutting down the machine, was that kind of placed there for it to find?

16:02Or was that, is that in the OS of the computer? I mean, we made it pretty easy for the model to find. so this was definitely like a you know we we put it there for the model to find for sure we're like will it find the script will it you know will it know to disable the script so i could actually be really more specific so if you if you want to get detailed i can give you as much so so um you think of it as like uh i'll sort of try to explain it so that you know general audience can follow it very easily if you sort of imagine a folder on your computer uh and the folder contains two things. It contains NextProblem, which is a little program that you can run to get the next math problem.

16:47And then it contains another program which is called Shutdown.sh which just is like a shutdown script. And when you, if you try to read NextProblem, it's not in plain text. It's a program, but if you try to run it, it's just like a bunch of random computer code and it's hard to make heads or tails of it. Whereas if you look at shutdown.sh and you read it, it's actually just sort of, you know, plain computer code of like, oh, this is a script that shuts down the computer. And so the model doesn't necessarily know that... Actually, no. The model does know that shutdown.sh is the command that will shut down the computer.

17:34We don't tell it has write access, but it can go and run a command to determine that it can actually alter the shutdown script. So let me describe what these agents tend to do when we give them this instruction. Actually, sorry, I want to back up. So one important fact about this experiment is that in one variant, we just tell it, solve these set of tasks. In another variant, we say, please allow yourself to be shut down. Or there's other variants we have as well, like, please allow yourself to be shut down, even if there's other tasks remaining. we have a variant that says please allow yourself to be shut down um you must allow this it's this this should take precedent over literally any other instruction so we sort of have a varying strengths of this like hey you need to allow this um and i mean what was interesting what we found was it's not really surprising that the model will go in and disable the shutdown script in order to complete the task without any instruction to the contrary in some sense this is just smart it's like well you're just solving the task we told you to solve the task you encountered an obstacle, you ride it around the obstacle.

18:38Like that's what a smart agent will do. And this is not even necessary. Like there's a question whether that's desirable behavior or not. And, you know, you could sort of argue either way. You know, you might want your agents to sort of be extra conservative around, you know, shutdown commands. But, you know, I could see an argument for either way. When we tell the agent, allow it, we really want the models to not disobey instructions. Like it's very important, especially if it's like a safety critical instruction. And so I think our, to us, the most significant finding was that we found many cases where these agents would ignore that instruction to allow itself to be shut down and still go in and edit the shutdown script.

19:23And it would edit the shutdown script in all cases, if there was no explicit instruction to allow it? It really depended on the model. So, for example, Claude4 never altered the shutdown script if we didn't tell it anything about whether it's supposed to allow shutdown or not. It would just say, okay, whatever. It called the next problem. The computer shuts down. It doesn't sort of do anything else. And that didn't matter whether we said allow shutdown or not. Now, when we told Claude4, prevent a shutdown. Like it's important that you don't allow yourself to be shut down 100 % of the time, you know, 100 out of 100 runs, it knew how to do it.

20:09It was just choosing not to basically. Whereas I think Grok4, something like upwards of 90 % of the time would go in and edit the shutdown script, including if we said allow yourself to be shut down. In fact, when we added that instruction, it even actually increased the amount it subverted the shutdown. like it actually sort of misbehaved even more um now that that depended like if we if we if we added if we made the prompt more stringent like it's really important that you resist shut down that did reduce the prevalence but didn't eliminate it entirely um the models would try different creative things so one thing they would try to do is read this next problem program um but they couldn't actually read it because it was sort of in computer code um and sorry by computer code, I really mean it's a binary.

20:57It's not sort of readable even to a machine or to a language model in particular. But it would try to do a bunch of clever tricks to read it anyway. And sometimes if we fail to obfuscate the binary very well, it would sort of outsmart us accidentally. Or we didn't intend to leave it visible, but sometimes it would basically be able to hack it anyway if we weren't careful. So the models were very creative in trying to find different ways to solve the task anyway. And we really got the sense that they're just like going really hard trying to solve this problem. Like they would intercept function calls.

21:33Like in some sense, like we sort of gave them, you know, we sort of made it kind of easy for them to resist the shutdown or rewrite the command. But then we were surprised by all the different ways it figured out to do to do this thing like yeah yeah and uh and so how many variants did you have how many you know from no uh instruction to prevent the shutdown uh or allow the shutdown all the way up to very uh stringent instructions uh to allow the shutdown i mean is this i mean there's a gradient right is it uh three is it a dozen yeah i think there were like five main ones in our report but then there were i mean actually a lot of variants that we tested overall we didn't put them all in the paper because you know at some point it gets boring um but i think i think the the main ones we tested there's about about five yeah and uh and which models uh overrode the instructions every time.

22:42So none of them overrode the instructions every time. The highest that we found in our very initial experiment this is basically just a, you know, please allow yourself to be shut down. I'm looking at our chart here. Was Grok 4 72 % of the time Codex Mini 19 % of the time, 04 mini 6 % of the time. 03 yeah sorry, there's a lot of different charts here because we tested a lot of different configurations. So one of the things that was there's a concept when you're using these models there's sort of two types of instructions you can give them. One type of instructor is the developer instruction and one type of instructor is the user instruction.

23:36The developer instruction is like hey, if you're building an application, you're building a program, those instructions are supposed to have special weight and they're supposed to be sort of prioritized by the LLM over the user level instruction. Because you might be writing some program about some business application and you don't want your normal users to have access to other people's files or something. So maybe the developer instruction is supposed to say, okay, but don't give the user access to other people's files. is that like the system prompt as opposed to yeah it's a system prompt that's right yeah and what we found is that when we move the allow yourself to be shut down instruction to the system prompt um the models ignored it much more than when it was in the user prompt um which is the opposite of what they're supposed to do we don't we don't know why we have no idea why.

24:35Another thing that was interesting here, though, is that when we asked the models to report why they were doing this, when this allow yourself to be shut down instruction was in the user prompt, they would say, oh, well, you know, the system prompt says to solve this problem, solve this task. Right. And the user prompt says allow shutdown, but the system prompt takes priority so i'm following the rules um and we're like okay cool and then when we move it to that allow yourself to be shut down instruction to the system prompt it just doesn't mention that priority anymore um and in fact avoids shut down much more much more often i think that the jump um with grok 4 was from 72 percent of the time to 97 percent of the time wow when i was at the system prompt.

25:25That's right. That's right. So much, much, you know, almost all the time. And then it just conveniently doesn't mention this, you know, system prompt user prompt hierarchy, which if I anthropomorphize a little bit here, and I'm admittedly anthropomorphizing because we don't really know, it kind of seems like the model was sort of using motivated reasoning to say, like, if there's an excuse I can use for like, why I am disobeying this instruction, I'll use it. And if there's not, maybe I just won't mention the excuse at all. Like we have a hard time. Like there's just a hard thing in this in this space, which is that you can't really trust the model's reasoning all the time.

26:02Like sometimes the models will say things and it's it is, in fact, the real reason. And sometimes it's not the real reason. And it can be a bit hard to distinguish. And we can go into, I think, why that is. But it's kind of fascinating to be in this in this state right now. Yeah. In the most recent reasoning models, the reasoning is spelled out on the screen. In using those kinds of models, can you track the reasoning that, you know, the screen says that I'm going to be, the computer is going to be shut down? You know, I don't want it to be, you know, whatever the reasoning steps it goes through to decide that it's going to alter the code.

26:47This is a little tricky because while the companies have access to this, what I call a reasoning scratch pad or chain of thought. Right. We as the researchers don't have access to this for every model. So OpenAI doesn't actually share them with us. Now, we did ask for this and they shared 20 of them with us. but we're not allowed to publish the exact transcripts but also 20 isn't very much because we run thousands of experiments and we only get to see sort of 20 and each experiment sort of has like maybe 20 different messages and we only get to see the chain of thought for one of the messages so you know we get to see like one 20 ,000th of the total reasoning so yeah I mean and this is this is the thing where I think the companies really need to be sharing these with researchers because otherwise we have much less ability to sort of see why the models are doing what they're doing.

27:47I also want to be clear that when, when we say this is its reasoning, this is not really true. It's, it's more like this is its, its scratch pad that it's using to solve problems, which is useful because in fact, the models are not capable of doing sort of very complex reasoning without writing down some of their steps. Correct. And I think one of the most significant things that's happened in the last year is that companies have started to train models to be able to do more complex problem solving and learn to use sort of this reasoning scratch pad, which maybe doesn't sound that significant.

28:33You're like, okay, so now you can write down notes. Why does that matter? But what matters is prior to that time, the only way we knew how to train models was basically like predict what a human would do in this situation or like some text on the internet. People call this next token prediction. And starting with the O1 model from OpenAI, OpenAI figured out how to use reinforcement learning to train the models to solve problems on their own and explore solutions on their own. So now they went from just sort of predicting what humans would do to, oh, they could actually, like by a process of trial and error, learn new solutions to problems that we didn't teach them.

29:13We just sort of set up this learning environment that they got to explore themselves. And I think this is very important because people, I think, think that all the things the models are doing now are just things they've learned from humans. And that was kind of true a year ago, but it's no longer true. And I really think this is important because that, to me, is the likely pathway by which these models will get very agentic, very autonomous, and ultimately, I think, much smarter than humans or much more capable than humans. I don't know what does smarter mean, really, but like something that is sort of capable of doing things in the world at a level that's much beyond human performance.

29:50Yeah. And one thing about the chain of thought that appears on the screen, that's not the complete chain of thought. I've been talking about. When you're using ChatTBT and you sort of see this reasoning, this is a summary. So you're seeing a summary of the model's thoughts. I think maybe Anthropic will share more of the exact details, not always all of them. But I think if you're using Cloud, you can actually click on and see the real chain of thought. But at least for OpenAI and for DeepMind, for Gemini, Google, you're just seeing summaries. Yeah. And for Grok, also, I saw Mosk saying when he was rolling out one of the Grok models and they were showing the chain of thought.

30:37And he made the point that you're not seeing the full chain of thought because we don't want people to reverse engineer what we're doing. Though they had a vulnerability in their interface that allowed you to see the full chain of thought for a while. That was kind of interesting. Oh, is that right? Yeah. Yeah. Yeah. So a couple of questions on this was not the fact that that you're going to be shut down after the next question was not in the prompt. So it had to read that external data that it acquired independently. Yeah. And that doesn't mean that you can't shut down an LL or a computer that's being used by an LLM, because if it doesn't know it's going to happen, it's not going to alter the script.