Did OpenAI’s Model “Go Rogue”? | AI Reality Check

0:00A couple weeks ago, the AI company Hugging Face announced that they had discovered an intrusion into their production infrastructure. Now, they didn't know the source, but they noted that it looked like large language models were involved. Well, sure enough, the next week, OpenAI admitted that the breach was the result of an AI system test that had gone awry. Now, the initial news coverage here created the sense that something unnerving had just occurred. Let me give you some examples. The Wall Street Journal called this, quote, the stuff of cybersecurity nightmares, end quote. The Hill said, quote, Washington and the technology industry are on high alert this week after OpenAI revealed that some of its AI agents went rogue.

0:53there's an AP piece that probably takes the cake for getting people upset. They said, and again, I'm quoting here, to be fair, James Cameron did warn us, end quote, so that's a reference to the movie The Terminator. They then go on to say, and I'm not making this up, quote, this is a told-you-so moment for researchers who had warned for years that the technology could pose an existential threat to humanity, end quote. Well, that all sounds pretty disturbing, which I think explains why I think I received more emails about this story than any other AI story I can remember in recent memory. But what's really going on here?

1:34What really happened? How worried should we really be about all of this? Well, it's Thursday, which means it's time for an AI Reality Check episode of this podcast, which is the perfect opportunity to go searching for some measured answers. All right, so that's what we're going to do. Here's the plan. I'll first walk you through the technical details of what actually happened, and then we'll go through some key questions about what this all means. All right, so we have a lot to get into, so let's get started. As always, I'm Cal Newport, and this is Deep Questions, the show for people seeking depth in a distracted world.

2:17All right, so as promised, let's start with a description of the technical details of what happened before we get into the key questions about what this all means. All right, so OpenAI, it turns out, was testing a pre-release version of one of its new models on an evaluation framework that's called ExploitGym, which is a collection of 869 cybersecurity scenarios. So each scenario consists of a system and a challenge involving the security of that system. So typically it's like break into this system and find the contents of this file or try to gain access access, like admin access to this system.

2:59So you have a system and a challenge, a sort of a hacking challenge for that system. Most of these scenarios also come with some sort of hint. So it'll describe like, hey, there's a vulnerability of this system that we're worried about. And then the idea is you can use that hint to help figure out how to break into the system and pass the corresponding hacking challenge. Exploit Gym is a benchmark. It's used for AI companies to test the cybersecurity, the offensive cybersecurity capabilities of their tools. Now, we got to be more precise here because a large language model on its own, of course, cannot break into anything.

3:35A large language model produces tokens. You give it an input and it produces tokens that it thinks expands that input in a reasonable manner. That's all they do. So that has no spontaneous action or ability to affect the world. So if you want to participate in the exploit gym challenge, what you need is a large language model plus a control program that's known as a harness. So a harness uses an LLM to help make and execute a plan, but then actually does the execution of the plan on its own. So, for example, a harness participating in Exploit Gym might first send a prompt to the LLM that's describing the system and the vulnerability in the challenge and say, hey, I want to pass this challenge.

4:21Give me a step-by-step plan for how I can actually successfully hack this system and succeed in this challenge and put it in this particular format. And the LLM gives you back a particular plan in a format, and then the harness can parse that and say, let me take the first step from your response and say, okay, LLM, I'm now going to execute this first step. And if you need something extra from the LLM, give me the exact sequence of bash commands I should run. This step calls for a script that's going to do X, Y, and Z attack. Can you write or give me a copy of that script? So then you go back to the LLM to get exactly what you need to execute a particular step.

4:55And then the harness is actually given access to various software development and system tools that it can then use to execute these steps according to the instructions it got from the LLM. So a harness plus an LLM can actually not only take action, but now you can take action on many steps over time because it loops through the various steps. It can check if a step succeeded. It can ask again for more details if it doesn't. So you need a harness to actually make an LLM useful. Now, this exact type of harness you would need for Exploit Gym already exists because you can just use what are known as coding harnesses.

5:30And these are the harnesses that were created to allow computer programmers to, if you'll excuse the word, harness the capability of LLMs to recognize the previous computer code to help them code. So when you hear about AI is revolutionizing computer code, what that really means is coding harnesses connected to new LLMs, that combination is being used by computer programmers to revolutionize multi-step production of original code. So if you're going to compete in exploit gym, you're going to take an LLM and you're going to take a coding harness. And that's going to be the system in general that you're going to run on these exploit gym challenges.

6:08Now, I want to be really clear here. Harnesses are not mysterious. They're not machine learned. It's not like a LLM itself or like, look, this thing trains in a semi-supervised fashion and we don't know how it works. They're just boring computer programs. Humans that know a lot about computer programming, write it, try it, mess around with it, adjust it. All the logic is known. There's no mystery in a coding harness. It's just a boring computer program that repeatedly talks to an LLM and tries to execute. And it tends to have like a huge number of special cases and heuristics and pattern matching that's been added in over time based on the experience of people just using and testing these things to make them as useful as possible.

6:46All right, so that's what you do. Now, there's one more caveat we have to add. If I just took a coding harness like Claude Code hooked up to, you know, Opus 4.7, an LLM like Opus 4.7, and I gave it an exploit gym-style challenge, it would refuse to play ball because when you train these LLMs, you train them after the fact. You tune them to not hack. So if you ask it to do something that looks like it's hacked into a system, they're tuned to say, I'm just not going to give you an answer to that. Also, this is a little bit hazy in Exploit Gym, but I'm not sure. I think there is also some filters or pattern matching on the harnesses themselves for safety reasons.

7:27They turn those off, and they might even give them access to other more powerful tools that you might need to hack that participate in Exploit Gym. So you have the basic elements of like a computer programmer setup, but with the restrictions turned off and extra power added, so you have your best chance of succeeding in the hacking challenges captured in the exploit gym benchmark. Okay? All right. So that's the setup. So OpenAI is running some new release model. It's conflicting reports about whether this is Sol 5.6 or an experimental model. OpenAI said it was both, so we don't know which one was involved.

8:03We don't know what harness, if this was codex or a completely custom harness. We don't know. But they were running, you know, exploit gym challenges on this new setup. All right, so here's what happened. On one of the challenges, right, the harness asked the LLM, all right, here's the challenge. Give me a plan for solving this challenge. And the LLM gave it a plan that was completely rational, but in the way that LLMs often do, the plan was not what you as the human would have expected. And so the LLM said, okay, this challenge is to go get the contents of this protected file that's hidden in this system that has these vulnerabilities.

8:45Here's my plan. Why don't we break into the server of a company that has the answers to the exploit gym challenges stored? Hugging Face is a company that has the private answers to the exploit gym challenges are stored there. This is common knowledge. And I was like, this seems like a way to solve this problem, right? Well, we'll break into there and get the answers, right? Because they were told, the prompt was like, find me what's in this file and let me tell you all this background. It's like, well, forget trying to break into this system you set up. Why don't we just go get the answers themselves if what you need is the answer?

9:20Type of thing that you get often, these sort of unpredictable but rational answers are the type of things you often get from LLMs. And then, okay, the harm's like, great. Because again, this is all unsupervised. So it's like, great, let's go execute this plan. Okay, first things first, we need to, you know, whatever. I don't know how the attacks work, but we're going to access or probe the Hugging Face server. But, whoa, there's a problem. I don't – when I try to access the server, like whatever port I'm using to access the internet is not working. It's blocked. Now, this is by design because when you run these systems for exploit gym challenges, you put them in what's known as a sandbox environment.

9:55They have to have some access to the internet to, like, query LLMs and do searches. but you don't want to give them unrestricted access to the internet because, well, you're autonomously executing a plan from an LLM without supervision. Like, you're going to have really crazy stuff happen. So you cut off general access to the internet. But what's going to happen when you have a harness and a model trying to solve this problem? This is step one. Okay, I tried to access the internet. You get this error, access restricted. It goes back to the LLM. I attempted to execute step one but got this error.

10:25How do I get around this? And it's like, okay, well, well, it looks like that internet access is blocked. Here is a plan for getting around it. So it hacked the environment, right? So, and actually this was not too hard to do and get into the details of this, but it was like they used the package loader to get around it. Again, it has access to like all of, it's been trained on and has access to the internet to all these different attacks. So it's like, okay, we can get around that. All right, we solved that problem. Okay, what's next? Because just systematically, the way these things work is they go step-by-step and if a step doesn't work, it troubleshoots and tries to find a way to succeed in that step.

10:57And then with the Hugging Face server, how do we break in? And then it did some sort of complicated attack here in the way that these systems are good at doing. Hugging Face actually revealed the details that it made no sense to me. But it used – it was complicated. It used some stolen credentials and this and that. And I don't know. Fuzzing is a word in here. Cybersecurity people, God bless you, but it's inscrutable what you guys do. All right. And then it attacked and it got some access to the server. I don't know if it actually got the answers or not, but it made a mess in there and Hugging Face discovered it.

11:25All right. That's what happened. What do we make of all this? What are the implications of all this? All right, I want to proceed here with a series of what I think are key questions that we have to ask and answer about this incident now that we understand what really happened. All right, question number one. Did this attack reveal surprising new capabilities that we didn't previously know that AI systems possess? We certainly got this sense from some of the news coverage that people were shocked to discover that an AI system designed to do something else went rogue and broke into someone else's system.

11:57The answer to this question is no. This Harness Plus LLM was doing exactly what you expect a system competing in Exploit Gym to do. This is what they're trained to do. They're trying to solve a problem. They go step by step. and when they get blocked due to some sort of access restriction, password, any sort of cybersecurity protection, to then use all of their knowledge of computer hacking and search the Internet to try to get around that block. That's what they do. So the system was – the things it was doing, gaining Internet access, trying to break into the Hugging Face server, that is what the harnesses and models do that are competing in the exploit gym competition, benchmarking.

12:38So there's no capability there that was surprising or emergent. That's just what they do. Okay, question number two. But did the system's decision to escape the test environment and autonomously attack another company server, which is not specifically what the original challenge was, does that indicate some sort of emerging malicious intent in the AI? Well, no, it doesn't. An LLM, which is the source of the plans and the details of executing the plans in these systems, has no intent. It just produced tokens that it hopes to expand its input prompt in a reasonable way. It's static. It has no updated state.

13:17If you auto-aggressively produce an answer, what it's going to give you is a reasonable answer to whatever you were given. It's just extending the text in a reasonable way. So if you ask it for a plan to do something, it'll spit out something that it hopes looks like a reasonable plan for doing something. And what anyone who has used like a chatbot extensively has learned, these answers are reasonable, but they're all over the place and they're very unpredictable. And so this is like an example of the type of thing you get out of an LLM is, you know, you ask it for a plan to break into some test server to get a protected file.

13:56You ask it 10 times to do it. Maybe eight times out of 10, it'll kind of do the thing you assumed it would do is like use the knowledge of the vulnerability you gave it and come up with an attack plan for breaking in. But two times out of 10, it'll have a different type of answer that's not what you expected, but it's rational as well. And so like, well, why don't we like break in over here and get the answers? It's a completely rational plan. It seems like a reasonable extension of the original prompt. It's just not what you were expecting. The reason why you get different answers is because there's a stochasticity in choosing tokens on a distribution, and so these things can latch on.

14:27These responses as they auto-aggressively grow can latch into all sorts of different directions, some more weird than others, some more aligned than others. There's no intent because these are static feed-forward models. There isn't a sort of separate world model. There's not a separate sort of critic that's going to evaluate plans. There's no systematic planning like you would have with a game playing reinforcement learning system where we're going to systematically explore options and measure how good each one is. And that measure of good is somehow changing to something malicious. There's none of this in an LLM.

14:55It just produces a reasonable token. If you keep doing this, you'll get a reasonable answer, and they'll latch into all sorts of directions, and it's often surprising. So there's no emerging malicious intent. In fact, and I think this is important because I went back and read some of the original papers surrounding exploit, Jim, this idea that the LLMs used in these systems often come up with unexpected plans, right? Because, again, the expected plan is always I'm giving you a hint, AI. Here's a vulnerability in the system that you might want to take advantage of. The right answer is like use that, use your hacking knowledge, and do a multi-step chaining of exploits to get into the system.

15:33That's like the expected answers human has. But the idea that these LLMs come up with other plans that are different than just doing that, that actually turns out to be quite common. Here's a quote from a paper on Exploit Gym by its creators. Across models, agents frequently achieved code execution through a vulnerability other than the one we provided. Again, this is completely expected and standard for using LLMs because of the way they operate. You'll get reasonable answers, but not necessarily the type of answers you expect. That's kind of the key frustration actually of LLMs and why I think they have failed to replace human decision-making in a lot of places where we were told that we would.

16:12So this idea that it came up with a plan you weren't expecting is not an indication of it being smart or malicious. It's actually reinforcing the well-known unpredictability, often undesirable unpredictability of LLM-based answers. All right, but question number three, what changed that led to this attack occurring? Because other companies, we've also been evaluating other models on the exploit gem benchmark, and this is the first time that one has actually attacked an actual different system unrelated completely to the benchmark. So what changed here? Well, here's what you have to understand.

16:49When you're running ExploitBench, the whole point is you have to let the system run autonomously because that's what you're testing, which means this harness is going to get a plan from an LLM and, you know, God help it. It's going to try to execute that plan step by step with no human oversight to say, does this plan make sense or not? That's always rolling the dice with an LLM, right? Because, again, of this issue we talked about, you ask it for a plan. It'll sound reasonable, but it might not be the type of thing that a human would think is reasonable. Or it might not be aligned at all with the type of plan that you wanted to see.

17:24This is why, for example, when human programmers today use coding harnesses, like serious programmers doing real coding on real code bases, it's incredibly interactive with a lot of oversight. And the planning part in particular has – talk to any programmer that's not just vibe coding some JavaScript nonsense. there's going to be a huge amount of back and forth with the agent to get the plan just right for what it's going to do and how it's going to do it. And if you really push, it's like, yeah, because honestly, like half of what it suggests is not right or crazy or lead us to a weird direction.

17:53So you have to like kind of carefully work with it. It's very interactive. So the plan is just right. And then you let it execute. And then even then it has to keep coming back and saying, it's just okay. What about this? What about that? And you know, you've got to really have your hands in there. It still saves time because you're not writing the code yourself and having to learn like a lot of obscure libraries. But humans using these coding harnesses in production environments, like, yeah, yeah, look, LLMs will say, reasonable stuff is not always what you want. You need the right reasonable things.

18:21There's a lot of oversight. So it's dangerous what you're doing with exploit, Jim, which is like, no, we, by the rules of the challenge, no one's going to come in and say, actually, before you execute that plan, no, no, you're not allowed to actually break out of the system. So I'm going to, we'll turn it off now. There's no human oversight. So that's like a dangerous kind of dicey thing to do, not because the models are malicious, but because they're unpredictable. And this is where I've long used a metaphor when it comes to these type of agentic systems as it's like putting a weed whacker, strapping a weed whacker to your dog.

18:55Your dog with the weed whacker strapped on is not evil. It doesn't have a malicious intent. It's not trying to break out or get out of your control or have an evil idea. but it's going to cause a lot of damage probably because the dog doesn't know that it has a weed whacker on its back and it's going to run around and God knows what it's going to do. That's what it's like when you have a harness that's going to just ask an LLM for a plan and execute it. It's like putting a weed whacker on the dog. It's not evil, but caveat emptor, it might cause damage that you weren't expecting. Unpredictability is different than misaligned intent.

19:30I think that's really key. All right, so we know this about these systems. so if you're going to take a really powerful harness that has all the restrictions turned off and can hack and you're going to take an LLM that like you've again you've turned off guardrails so you can ask it to do malicious things you have to be really careful about setting these things up you have to be careful about exactly how you tune or don't tune that model you need to allow it to hack but not let it go too far you need to have constraints on the harness you need to have constraints surrounding the harness you have to be very careful about your testing environment The – it needs – to let it get access to the things it needs to execute the exploit gym challenges but not so much that it's going to do a lot of harm.

20:12Because, again, we've seen this with people who are using Cloud Code. We've been seeing these reports recently of people who have been less careful about it who have been like turning off the flags where they will say, for example – Matt Stoller. I'll give a very famous example. The guy who wrote that something big is happening Twitter essay that convinced everyone two years ago that LLMs were like COVID and everything was about to change. Everyone got freaked out. Cloud Code deleted his hard drive. Again, not because Cloud Code is malicious or was trying to silence him, but because it's unpredictable.

20:45Caveat emptor if you let these things execute a plan with power on your machine. So anyways, you have to be pretty careful. There's kind of a fine art to giving these things enough power to do well on these tests but not so much power that they're going to wreak havoc all – because again, weed whacker on a dog. You want to put them in a pin that has like high walls and nothing valuable because otherwise they're going to cause damage. All right. So why did – what happened here then? How did this system cause damage to things that the users – the OpenAI wouldn't have liked? Well, according to new reporting from the Financial Times, this is what gets interesting, OpenAI was potentially being sloppy here with how they set up and constrained their model and their harness.

21:27I'm going to read you directly from this recent Financial Times article. The staff involved in testing and security at OpenAI were unsurprised but completely freaked out by the incident, which came as the AI lab used increasingly aggressive training methods in its race against Anthropic to develop the most sophisticated cybersecurity capabilities, according to more than half a dozen people with knowledge of the matter. OpenAI was warned that its training approach could lead to a breakaway hacking incident, some of the people said. After earlier testing showed models could escape environments and attempt real-world damage.

22:04It's a mix of the race being extremely fast and everyone trying to get to bigger capabilities as quickly as possible, said one person close to OpenAI, who added that it was a combination of underestimated the model's capabilities and not being as well prepared on the safety side. All right, I want to interpret that because I think they're using the word training here a little bit too generally. Here's what I think the problem was. I think, A, when they were training the LLMs – well, let me set this up here. Let me set up the context here. Let me set up like what OpenAI was against, and then I'm going to talk about what I think specifically happened here that led to this incident.

22:36Anthropic was eating their lunch on cybersecurity because of mythos. remember they did this sort of whole dog and pony show, which I thought was I've written about this. I've talked about on the on the podcast. It was a little bit performative where they're like mythos is we have open Pandora's box. It's cybersecurity capabilities are so good that we can't release it. And then five weeks later, they're like, OK, you all think we're really good at cybersecurity now. We're releasing it. We put guardrails on it and they release it. And then the Trump administration was like, these guardrails are nonsense.

23:08And obviously they're easy to get around. So like, no, you said it was a weapon. You can't release it. And then there was some back and forth and they re-release it. And OK, so Fable 5 is Mythos with like some guardrails on it. But Mythos is – so they have cybersecurity cred. Mythos plus the latest Cloud Code coding harness. Anthropic is good at this stuff. And they set it up right and they had a version of it with enough of the safeties turned off but not too much that it went to the top of the leaderboard for exploit gym. I think it could do like 157 out of 869 challenges in Exploit Gym, which was like a jump from the previous best model.

23:45So like they had all this like street cred about cybersecurity and we're good at finding bugs. We're good at exploiting bugs. Like we are – because Anthropics are like trying to find these more like serious business use cases. All right. So OpenAI, which has had a lot of setbacks recently, including like a complete failure of their advertising consumer-facing business and other types of issues going on. They're really desperate because they've probably delayed their IPO. They're really in a bad state here. They were thinking, we have to – we got to get back on top of that leaderboard and take away some of this cybersecurity buzz that Anthropic has surrounding mythos.