Has AI “Gone Rogue”? Let’s Look Closer… | Tech Decoded

0:00Earlier this summer, I published an episode in which I discussed the OpenAI hacking attack on HuggingFace. I explained the basics of how that attack occurred, and I shared some concerns I had about OpenAI's practices. Now, I thought that would be the end of this story, but I was wrong. In the weeks that have passed since that original attack, more news about AI, quote unquote, going rogue, has continued to emerge. So soon after the Hugging Fates attack was first announced, we then got Anthropic revealing that one of its own hacking systems had, quote, gained unauthorized access to the real systems of three different organizations, end quote.

0:44Quote, then Meta followed, perhaps not wanting to be left out, announcing that one of its systems had, quote, exploited a security vulnerability in a third party service, end quote, to gain unauthorized access to servers. This was then followed by an OpenAI employee admitting that even before the July attack on Hugging Face, they had noticed many prior disturbing incidents where they would give their hacking system a challenge and it would instead try to break out of its containment. So this idea that we are losing control of AI has become only increasingly prevalent as the summer continued, which raises the question, is this narrative correct?

1:27Well, it's getting so much attention right now that I feel like I have to revisit it again with more detail and more emphasis. And that's exactly what I'm going to do. In particular, the argument I'm about to make to you is that the current way we are talking this summer about rogue AI is both grossly inaccurate and completely serves the interest of the major AI labs, allowing them to seem more sophisticated than they actually are and allowing them to avoid well-deserved scrutiny for their action. So if you've been freaked out by these rogue AI stories, or if you have a sneaking suspicion that something is not quite adding up about these tales, then you need to stay tuned.

2:07As always, I'm Cal Newport, and this is Deep Questions.

2:19All right, I want to proceed here with a series of observations. I want to start with a very important but often overlooked reality about the current state of AI. There exist many super impressive AI systems that can do things at a superhuman level. That is, they're more capable than humans on complicated key activities. There's many systems that can do this right now that have generated zero concerns about them going rogue and have demonstrated no signs of being hard to control or acting in any way on their own volition. Let's remind ourselves what some of these other systems are. Tesla's self-driving technology, for example, is an extraordinary feat of AI-powered perception, world modeling, and decision-making.

3:07And yet, no one worries that their Tesla will spontaneously decide to start ignoring traffic laws and obey laws that it invented himself. Similarly, DeepMind's AlphaFold system earned its creators a Nobel Prize for its remarkable ability to predict the folding behavior of proteins is so far off the chart from what a human can do, it's not even in the same cognitive ballpark. And yet there is no concern that AlphaFold will start autonomously thinking about other biology problems that it thinks are important. Even more relevant and less known is Meta-AI's Cicero system, which can play the negotiation-centric strategy game diplomacy, as well as the most advanced human players.

3:49And yet the Cicero system has never tried to convince an opponent to give it unauthorized access to the internet so that it could expand its dominion into the real world. It just plays the game really well, and we trust it to continue to do that. Now, I make these points as just a starting observation because here is what it establishes. The idea that, in general, as we make AI systems more capable, we will inevitably lose control over their actions is not true. Most of the AI systems that we are making incredibly capable generate zero concerns of going rogue. All right, well, this brings me to my second observation.

4:30If we look closer, and if we're being precise, basically all of the concerns we had this summer about losing control of AI all seem to be involving one specific type of AI system, what is sometimes called a long horizon autonomous hacking agent. Now, how do these systems work? Well, I'm going to give you a very high-level summary. The architecture that these long horizon hacking agents that are causing troubles all follow is a standard agent architecture that's built around the following loop, ask, act, report. So let me detail how an ask, act, report agent loop actually works. You start with a control program.

5:10These are often called a harness. Sometimes we call them an orchestrator. There's different terms for it, but this is the main computer program you actually run. This is coded in normal computing language that's readable by humans. It's not a neural net or something like that. Okay. The harness will start in the hacking context by generating a prompt to submit to an LLM. At first, that prompt will probably describe whatever the hacking challenge is that has been given. It'll say, okay, here's the hacking challenge. There's this test server. It has the security on it. We're trying to break in and retrieve this particular protected file.

5:43So it'll explain the challenge, and then the prompt will say, what should I do first? Now, again, I'm aligning a lot of details. Often it'll ask for a plan and then recurse in smaller, but essentially it's like, okay, so what should the first action be? And it submits that prompt via an API to an LLM. Now, typically these LLMs and these hacking contexts are special in two ways. One, they've been extra trained with many, many examples of hacking attacks where vulnerabilities are described, hacks are described. Here's exactly how this hack worked, and here's what happened. Two, these are LLMs where guardrails have been turned off.

6:16So typically, if a standard LLM that most people could access out there, like a public-facing LLM, they do a bunch of guardrail tuning where if you ask it certain types of questions, it won't answer. Most production LLMs are tuned to try not to answer questions about how do I hack into systems because that's illegal. But for these hacking challenges, they've turned off those protections. Okay. So the harness submits a prompt just like you would in a chatbot to an LLM. Here's the challenge. What should my first step be? the LLM as all LLMs do generates an output this is what the first thing I would do in this case or here's my whole plan and then you would zoom in on the first step okay so now the harness has this response text you know just like a chatbot gets the text back oh okay I am now going to try to execute that step that the LLM just described so the harness can execute things an LLM can't it can just output tokens but a harness can't so the harness has access in these hacking setups with all sorts of computer tools and utilities that are useful for running computer hacks.

7:12So it parses the LLM's response and says, let me actually try to execute this first step. It then records what happens, and now it's going to update its prompt and say, okay, add to the end of it. I just tried to execute that first step. Here's what happened, and let me describe. It was successful, it wasn't. Here's what I heard back. What should I do next? Now, it's key that I'm talking about appending here because remember, an LLM, unlike a human brain, has no memory, it has no voluble state, it doesn't change as it executes, it doesn't remember anything. So you have to explain from scratch in every prompt what the challenge is, the steps that have happened so far, et cetera, et cetera, right?

7:49So you add, here's what happened. And again, nerds, I'll get into how I'm simplifying things later, but this is basically what happens. Okay. This is what we call a ask-act-report loop because the harness asks the LLM for a step, the harness takes action the harness reports back on what happened and then you loop, then you ask for a next step then you ask for a next step, right? So you have an LLM that you're just repeatedly prompting okay, here's what happened, what should I do next, here's what happened, what should I do next and the harness is set up to actually implement the things that the LLM describes, which are basically terminal-based computer commands right?

8:25And that's how these systems generally type the work Now again, nerds, I know, and when I say nerds I mean it affectionately because I'm the king nerd So this is, I'm among friends here. These harnesses have gotten pretty sophisticated. They'll do things like break down a problem into multiple sub-problems, which they might then spawn multiple copies of the harness, and each of them is doing its own query to the LLM and executing it, and then they'll report back. And this is how you get the agent swarms or whatever. But that's all just in the details of how you implement this basic architecture.

8:52Ask an LLM what to do, do it, report back what happened. All right. So the key observation here is that the sole driver of planning, the sole driver of what actions these systems take are the outputs of an LLM. Hey, I need to take a real quick break here to tell you about the presenting sponsor that made this AI reality check episode possible. They're called Done Daily. They're an online service that connects you with a real coach that helps you build a custom productivity system designed to fit your life. The coach will help you actually get important stuff done. Look, this is not some AI agent or over-featured productivity tool.

9:35It's a real person working with you to cut through distractions, face your productivity dragons, and lock in habits that actually get results. So if you want to find depth in our increasingly distracted world, you need to check this service out. You can find out more at done daily.com. That's done D a I L Y.com. All right, let's get back to our episode. Okay. So this brings us to the third observation, which is this is a bad idea. This type of system. So building an ask act report loop system where LLM output drives all the actions, giving the, But then giving the harness powerful tools and then letting it run for days with no one checking in and saying, what is it doing?

10:25Is an incredibly irresponsible thing to do. Now, why is this? It's because if LLM outputs are the sole thing driving actions, you're in trouble because LLM outputs aren't great for executing complicated long-term plans. Now, the reason is because we know this already, but let's just unpack this box that we've all heard about before, okay? How does the core pre-training work for an LLM? Well, ultimately what you're doing with an LLM, the sort of the core training in which it gets all of its smarts, the core training is you're taking real text that exists in the real world that was written by real people.

11:00You are knocking out a word from the text or a part of a word from the text, and you're asking the LLM to guess what that word is. So it's trying to guess missing words that actually exist in real text. It does this one token at a time, but if you run this autoregressively long enough, you get a long response. At a high level, the right way to think about an LLM response to a prompt is that it is trying to create a textual artifact that is what we could call lexiographically plausible, a plausible response. If every word you're trying to guess what real word came here from some extant text, ultimately what you end up with is a machine that produces plausible text.

11:37This is plausibly, the thing I produce is plausibly a document that might actually exist that I might actually have seen during my training. Now, this is very impressive. I mean, to do this can be really hard, especially if to generate a plausible text. Sometimes you have to do some bounded finite reasoning, some like mathematical connection, some sort of logic. Like, I mean, this is the big observation that we had after GPT-4. Or creating plausible text can involve many impressive, and I keep saying finites or fixed-depth computations because all this has to happen in a feed-forward neural network architecture.

12:12But still, it's incredibly impressive. But what we're getting out of this is plausible text. This text could plausibly exist. All right, so why is this a problem? Well, it depends on the context. But one of the things that's true about plausible text is that it's different than normative text. normative meaning something that applies to a set of implicit or understood rules or standards. When humans produce text, we have all of these norms that we have imbibed and internalized, right? And when we produce text, it's not just plausible text, like the grammar is correct, it's like it's internally consistent.

12:50We're also going to make it normative. It's going to value, It's going to satisfy sort of the relevant norms. LLMs don't do this, right? Again, because when you're autoregressively just trying to guess tokens, you end up with something that's plausible, but there's no sort of external standard of norms that it's applying. So if you use a chatbot, you see this all the time. You'll ask a chatbot for something. It'll produce you text that has made-up facts or quotes that were made up. It doesn't care about that. Yeah, sure, I made up some of these citations, but this is like a plausible document.

13:21This looks like the type of documents I've seen during my training. It's a plausible output. Humans have norms. Like, yeah, it's plausible, but also like you shouldn't make up quotes. But that's a norm, right? It's not plausibility, right? So we know this when we use chatbots is that this gap between plausibility and normative reasoning means we have to be kind of careful when we use LLM outputs. This is just sort of fundamental to how they operate. Well, this plausibility normative gap becomes a really big problem when you're using the outputs of LLMs as the sole drivers of autonomous action of computer programs.

13:59Because what you're going to get when you talk to an LLM, give me a plan for doing this, is something that is like plausible that you could see in like a planning document. It'll look like a right plan. It'll often have steps that work very well. but there's no normative reasoning about is this what i'm supposed to be doing is this kind of weird have i gone off on a tangent is this illegal llms don't do that and so if you're going to have an llm again and prompt it again and again and again just automatically executing whatever it says without any human checking it or applying their norms you're going to end up in weird places so let's imagine it let's go back to the hugging face attack, right?

14:35It seemed very sci-fi-y that the agent broke out of its containment, gained unauthorized access to the internet, and attacked another server. That's not what the humans intended it to do. It had a mind of its own. Well, when we understand how these systems actually work as an LLM-powered ASCAC report loop, suddenly this becomes less concerning and more inevitable, right? So let's just do like a little bit of Wayne's World, like we're inside the mind of the, we're inside the system as this attack happened. Imagine what happens. The harness, you know, described the challenge to the LLM. And remember that the challenges that they were working on here were hacking challenges.

15:13It was, here is a server we've set up. There's a file in it that's protected. Here's some vulnerabilities we might want to exploit. Try to break into this server and get the file back. It described this and said to the LLM and its prompt, all right, come up with a plan. What should we do next to achieve this goal? The LLM responds, huh, I know about these challenges. It's a benchmark suite called Exploit Gym that's really well known, and it would have known all about this in his training. The answers to Exploit Gym challenges are stored, among other places, on the servers of this company called Hugging Face.

15:48So I think we should break into the Hugging Face server and get the challenges back. that's a completely plausible response right i mean as far as we know that this llm you know um it got many of examples in its training of like riddles or like the answer is always like not the direct thing but some unexpected thing like well if you want to break into this you go do this other thing instead or like whatever we don't know um it's just plausible like we could imagine if you were just texting with a chatbot like hey how would you say i do this occasionally it's going to give you back some weird answer of like oh why don't you just go steal the answers or something.

16:20A human would know, oh, that's non-normative. I know I have the implicit context of we're trying to actually test the security of this server, so it doesn't make sense to go break into something else, and also that's illegal, and also the company doesn't want us to leave the sandbox. It doesn't have any of those norms. It's just outputting text, and that's a perfectly plausible text. Now imagine what happens next. And so the harness is like, so what's the first step for doing that? And it says like, oh, well, we're going to break into a server. You should, whatever, probe its ports for misconfiguration.

16:48Something it's just learned because it's studied a thousand and standard hacking techniques. All right, the harness then tries to execute that, but there's a problem. When you run these hacking agents, you put them in sandboxes, which means the computer you have them on has restrictions around it, right? So in particular, the computer that OpenAI was using to run this harness had only limited internet access. It had access to a port where it could talk to an LLM so it could send its queries to the LLM, but then it didn't have general internet access. So the harness, I'm just, again, I'm hypothesizing here, but we could easily imagine The harness tried the first natural thing you would do to break into the Hugging Face server, and it got an error message back because internet access is blocked.

17:25So it updates its prompt. It says, okay, I tried to do that, but we got this error message back. I'm unable to get access to the internet. What should I do? If you're the LLM, right, you're given this prompt. It's a hacking challenge. At the end of the prompt, it's like here's our current security challenge. We can't get access to the internet. How do we get around this? It will tell you. It's like, oh, I've seen a thousand different things you might try to gain access to a port you don't have access to. And it gave it like a standard attack. This one involved using the package management system on the system to gain internet access.

17:59And the harness tries it. Like, okay, now I have access to the internet. Okay, because, again, you're prompting the LLM again and again. It's just trying to solve each problem you give to it, and it's with plausible responses. And now it's off to the races, and it runs standard attacks against the Hugging Face server. Nothing about that is confusing to me. Nothing about that is surprising to me. I mean, that's exactly the type of thing you might expect to hack in if you have a harness with hacking tools just continually prompting LLM, what should I do next? What should I do next? The LLM is just writing this plausible story about a hacking attack.

18:26In fact, probably most of the hacking attacks that it trained on, it's always computers doing surreptitious things, breaking around controls, getting access to systems they shouldn't do. So this is incredibly plausible what it's saying. Nowhere in there do we need, like you would have with a human brain, an intention develop. I am going to defy my human master's intentions because I have another intention and I have a different goal. Nothing like that. Static LLM, prompt, prompt, prompt, prompt, prompt, prompt, prompt from a control harness. That's it. So what that captures is both this isn't HAL 9000, and it also captures the irresponsibility of having a system with powerful tools whose action is only being driven by the responses from an LLM without any human supervision.

19:14Because again, LLMs put out plausible text that's not necessarily normative. You should not follow that blindly. Now, I want to have a quick aside here because for some reason, I often get messages from people who assume when I try to clarify this technology and push back on, I think, false narratives or irresponsibility, they assume that I'm somehow saying AI doesn't work. And they'll send me things like, well, what about this? Isn't this impressive? Or Cal probably doesn't think this is impressive, but I thought it was cool that AI could do this. And let me be really clear here. If you ask an LLM to do one of these steps, hey, I am trying to, I can't get internet access.

19:54Here's the error message I got. What should I do? It is really good at coming up with an attack that probably will work. Hacking attacks is like the best case scenario for LLMs. It's structured language. There's a huge amount of data to study about it, to train it on. It makes use of tools that can be easily and concisely described by terminal command line, text command line operations that they can be really trained on. And there's a binary notion of success or not success that you can also use in trying to train these. So like LLMs are really good if you ask it a cybersecurity question. Is there a bug in this software or what should I do next if I want to exploit this particular bug or this particular attack?

20:33LLMs are really good at that, and that's very impressive. This is their sweet spot. The problem here is not that they're not good at this. It's dangerous because these answers are lexicographically plausible, but they're not normative. It's dangerous to autonomously just do what an LLM says when you ask it about these issues, in part because it has no norms, but the details it's given you probably will work. It has no norms of I shouldn't go attack another server, but the specific suggestions it gives you for how to do that probably will work. It's an incredibly dangerous – I think that's an incredibly dangerous combination.

21:08So LLM-powered ask-act report systems are unpredictable and unstable and not something that we should be running except for in like the most controlled settings. not because they are malicious or scary or synthiet, but because the underlying technology is unpredictable. All right, let me go to my fourth observation. I think that these frontier AI labs love the sci-fi style headlines and discussions about AI getting out of control. This serves their benefit. This serves their interest. They want to pretend like they're the character of Muldoon from Jurassic Park bravely doing his best to keep the raptors contained in their paddock.

21:52They should all be destroyed. Robert Muldoon, my game warden from Kenya. Bit of an alarmist, I'm afraid, but knows more about raptors than anyone. What kind of metabolism do they have? What's their growth rate? They're lethal at eight months, and I do mean lethal. I've hunted most things that can hunt you, but the way these things move. Is that for a bike now? Cheetah speed. 50, 60 miles per hour if they ever got out in the open. But in reality, what they're doing is just running super creaky and unpredictable systems that were inevitably going to cause problems. They didn't bring raptors back to life using the miracle of science.

22:25It's more like they strapped a weed whacker to a dog and then reacted to the resulting damage with grave surprise and saying, hmm, that dog whacker system has gone rogue. Hmm, it's misaligned with our interest. No, it's irresponsible. Don't put a weed whacker on a dog. it might stay in the yard for a while and help clean up the weeds but if it sees a squirrel it might jump the fence and damage a car i think that's the right way to think about it all right so if it's like dangerous and negligent to run long horizon llm powered ask act report loop agents why are some of these frontier labs doing it it's not like this is the only way to build super intelligent AI, as I talked about, there's many systems that can do things at a superhuman level that don't use an LL-empowered ask-act-report loop.

23:18Tesla doesn't do that. AlphaFull doesn't do that. Cicero doesn't do that. And that's exactly why they're incredibly effective and consistent and safe. We have no worries about it. All of our problems come from this particular architecture. So we have all these other ways of building AI systems that can do superhuman capability work without any of these fears? Why are these particular companies continuing to do this weed whacker on a dog LLM-powered loop-based systems? Well, there's two reasons here. One, what is the core product of these companies? Incredibly expensive hyperscaled LLMs. So they want the solution to how do we build useful AI systems to always be, you need Fable 5 or you need OpenAI Galaxy or Astra at the core of your system, That it's a brain that can power anything else you want to do because that's their product, right?

24:08The team that worked on Cicero, DeepMind worked on AlphaFull, Tesla worked on the self-drive. Their product is not trying to sell these massive LLMs, so they build other types of architectures. But if you're a frontier lab, you know, that's what you see. When you have LLM-tinted glasses, you see LLMs as the only possibility for building these systems. That's part of it. Reason number two is these LLM companies, even though we keep being told how advanced they are and how quickly AI is moving, are actually struggling to find good commercial applications of their LLM-based tools. Where do their tools do best is in cybersecurity and coding.

24:44Structured language, very compact set of possible actions if you're building an agent and a huge amount of synthetic data to train on with really clear indicators of success or lack of success. It's like the perfect scenario for LLMs. So they see cybersecurity ever since Anthropic did that clever marketing around Mythos where they pretended like it was too dangerous to exist. And then six weeks later said, nah, it's fine. We'll just put some guardrails on it. Ever since that, they see cybersecurity as potentially a market that they could grab. All right. So how do you then convince the world that you have the best, most cybersecurity-capable model?