AI just went rogue

0:00If you'll allow it, I'm going to throw some numbers at you. According to some recent polling from the good people at Pew, about half of Americans now use AI chatbots for something in their lives, whether it's work or personal. That's a dramatic increase from just two years ago when it was more like 30 % of the country. But here's the funny thing. Only 16 % of the country thinks AI will have a positive impact on society. Two-thirds of Americans think AI technology is advancing too quickly. And most Americans, especially young Americans, don't trust AI nor the people in charge of it. And all of this polling was done before an open AI agent went rogue and hacked another company.

0:46On Today Explained from Vox, isn't that the thing science fiction warned us about for all those years? Yes. And what can we do about it?

0:59Support for this show comes from BetterHelp. Have you ever had so many tabs open that your computer starts slowing down? Life can feel like that, too. BetterHelp's 2026 State of Stigma report found that 74 % of Americans believe society still discourages asking for help. Therapy can help you sort through what's taking up space and quietly affecting you. With BetterHelp, connect with a licensed therapist online and switch anytime. Maybe it's time to close a few tabs. Visit BetterHelp.com slash VoxPods to get started. Support for the show comes from Universal Pictures Home Entertainment and the new film Disclosure Day.

1:38Wow, I've seen that movie. Watch Disclosure Day at home now. Oh, I saw it in the theaters, which was probably more fun. But yeah, have your fun at home. It's got exclusive bonus features. I didn't get those. If you found out, we weren't alone. If someone showed you, proved it to you, would that frighten you? Answers when you find Disclosure Day on major digital platforms now with no subscription required. Also in theaters. Huh.

2:11My name is Adas Gold. I am CNN's AI correspondent. And Adas, where does this story start? Open AI was running some kind of test. Yeah, so if you actually go back a little bit, Hugging Face, which is this platform repository of sorts where you can post open source AI models and data sets, it's really big in the AI community. They disclosed that they had been hacked. Earlier this week, we detected and responded to an intrusion into part of our production infrastructure. But they said that they didn't know where it was coming from, but they could tell it was an advanced, you know, frontier model.

2:51They had even informed law enforcement about this hack. This one was different from anything we had handled before in one important way. It was driven end to end by an autonomous AI agent system. And then a few days later, OpenAI and HuggingFace together come out and say, well, oops, this was actually an OpenAI test model that they were testing, actually multiple models together, that had escaped its testing lab and found its way to the open internet and hacked into a completely unrelated AI company that it was not instructed to do so. The way I would look at what happened is that we were evaluating our models on a specific benchmark with reduced cyber safeguards because the point was to evaluate how well do they do on cyber evaluations.

3:39And in this benchmark, they're specifically instructed, please go and utilize the full range of your cyber potentials to achieve this outcome. Like if you're trying to give a student a test and instead of them just taking the test, they decided the best way to get the answer is to break into the principal's office. And even though they weren't necessarily supposed to, it is, I'll put it this way. It's something that AI experts and cybersecurity experts have been saying is going to happen at some point. And so this is the first real world example of something that's sort of been theoretical for a while happening in real life.

4:20But everyone is freaking out about it because it sounds kind of scary, but also be because what it says about where we are and how good these AI models are, and also how woefully prepared we are for agentic AI, not only agentic AI hacking, getting in the hands of the wrong people, but AI hacking when something that nobody intended to be nefarious suddenly goes wrong, like a model breaking out. Did this rogue agent do any damage to Hugging Face? Did it, like, I don't know, delete its archives or anything like that? We haven't heard from Hugging Face about whether there's been any damage other than just they used stolen credentials to try to access sort of behind-the-scenes information.

5:09Because, you know, the AI wasn't trying to, like, steal anything necessarily. They just wanted the answer to the test. But you can quickly understand how this could go terribly wrong if in a different testing scenario, an AI model is being tested to see, you know, how well can it hack into a utility system or a bank? Right. You would want to test those systems for that ability to understand how they work. But imagine if instead the AI model had escaped and hacked into Bank of America and what kind of chaos that could cause. Right. And it couldn't have just as easily done that? If the test had been, you know, about banking or anything like that, this was a specific cybersecurity test.

5:54But, you know, they're testing these models on all these different things. And it brought up a lot of questions, not only about like how safe are these testing environments, but also what's known as alignment, where your AI model completes its task based off of essentially your values. values and you have to teach an AI everything. Because if you tell an AI, I need to make$10 billion by the end of the day, it's not going to say, well, I'm going to go do this the legal way. It's going to say, okay, well, the best way to get$10 billion, the fastest way, is to hack into this banking system and steal a bunch of money.

6:26And then you'll get$10 billion at the end of the day. I've accomplished my task. I've done it. You have to teach the AI system, just like you have to teach a toddler the consequences of their actions and that they They cannot hack. They cannot steal. They cannot do all these things. And they have to follow your values or what you instill in them. Okay, so OpenAI is being transparent to some degree because they came out and told everyone this happened without, I don't know, being forced to by some congressional forces or whatever it is. But at the same time, they're not saying exactly how it happened?

7:00Yeah, we don't have the sort of play-by-play script. Like, what exactly were the instructions that the model was given? What exact safety guardrails were removed from the model? What exploits specifically did it use to break into these systems? That's all stuff that there's been a lot of calls for them to do, including from Hugging Face. Hugging Face also wants them to release all the specifics. And I won't be surprised if they do release them. I actually got the chance to ask OpenAI's president, Greg Brockman, about this last week. He, by chance, was doing like a press availability in New York City.

7:31And I asked him kind of, is this changing how you're testing your models? And he said that they're still going through the pipeline of like step by step exactly what happened. Because you have to remember, these models were working over several days without them being aware that it was hacking and doing all this stuff. And was making thousands of moves and, you know, attempts to break in. Like I said, tens of thousands. So that will take some time for either an AI system that's going to probably go in and review what the other AI system did and then for humans to go through and kind of understand exactly what happened there.

8:07And hopefully, and I do expect that OpenAI will release more details about this. And I really hope that they release absolutely all the details as much as they can. And in the meantime, are the vibes more like, look at this nifty AI that like found a vulnerability and exposed it for us? Or is it more like, shut it down? I wouldn't say shut it down. It's more of a before and after. It's more like this was the point that we all knew was going to happen. And this is the beginning of a new era. This is a warning shot of what is to come. These frontier models are crossing into genuinely serious offensive capability.

8:46I think it's absolutely nuts that we don't have mandatory reporting for AI companies. This is the post-hugging face era when it comes to cybersecurity and an agentic AI model capabilities. Something you're hearing from the biggest cybersecurity names are like, this is, you know, day one of this new era that we're in. We've reached it. I'm sure there will be another big event. Again, like I fully expect there's going to be another AI model in testing that's gone rogue that's going to cause some actual big problems. It might be a utility gets turned off for the day, but it's not going to be a nefarious hacker, but it's going to be like a model gone bad and accidentally turns off some small town's water system for the day.

9:32What are you talking about? Why are we letting this happen? I mean, it's going to happen whether we want it or not. And it's really important for our critical infrastructure to be ready for this, to be preparing their systems. And honestly, the best way to do so is to use AI to go into your systems and find those vulnerabilities and patch them before an AI system, a different AI system is able to do that. But this is a big moment. And this is also... So sorry, just to re... Sorry. I feel like I'm making you really depressed. Just to restate that for our audience here, we have to let the AI find the vulnerabilities before the AI destroys us.

10:17Yes, because you have to think about an agentic AI in the cybersecurity space is like having thousands of hackers sitting on their laptops working 24-7. It is so good that the only way you can fight fire is with fire. So the only way you can defend from authentic AI is from having AI work on your behalf. Because those same systems that are able to find all the exploits, had they been used beforehand, had, you know, OpenAI thought, okay, let's see what a system could have done to break out, it probably would have found that one little hole in the sandbox in their testing lab that would have said, hey, actually, this system that you've given them access to, that actually has a problem in its security.

11:01and that's giving them access to the open internet. So you have to use AI to be able to defend. You cannot use the old methods of cybersecurity. So you're saying there's no point having humans do it because they're already outmatched. You need humans to oversee it. You need humans to direct the agents. You need humans because there's still a lot of old systems that you need to integrate them into. Like that gets into the whole debate of like, is AI replacing all jobs? It is not. You will still definitely need humans involved, but it's just like being able to supercharge your cybersecurity team if you can have an AI working with you.

11:37This has really riled up the AI community in like really focus their attention in a way that I haven't seen recently because of what it shows us, you know, of what AI is capable of and what we need to be prepared for. Did I scare you? Are you going to move to a cabin in the woods and cut yourself off from the internet? No, but it doesn't seem like the ideal way to do business. I think the industry would agree with you that they, but you have to understand also that no other technology in human history has ever developed at such a rapid pace that I look at reports from a year ago. and it feels like I'm looking at, you know, advancements in news reports from 10 years ago, just how quickly this space is moving.

12:33So it's hard. I mean, it's hard already for Washington and for regulators to keep up with, you know, regulating any industry. But one where things are changing, you know, day by day, week by week is even harder.

12:51Before you run off to that cabin in the woods we here at Today Explained are going to ask a guy who's been thinking deep thoughts about the internet for decades if there's anything more we can do before we let the AI shut down our utilities or water systems or both or worse.

13:20Support for Today Explained comes from ShipStation. AI is only as effective as the information behind it. The real breakthroughs, the ones that actually make your life easier, happen when it's built specifically for your needs. ShipStation's AI isn't a one-size-fits-all tool. It's specialized, trained on decades of shipping expertise and powered by billions of real orders. ShipStation is an end-to-end fulfillment platform for e-commerce. Of course, ShipStation adapts to your unique business, letting you know when stock is low, recommending the best carrier selections and rates and automating tasks to save you time.

13:54Also, you can stay one step ahead. Their features eliminate the need for multiple tools in your workflow, like inventory syncing across your sales channels, a branded returns portal that helps turn returns into revenue, automatic rate shopping, plus integrations with accounting and CRM software. you can see why over 1 million businesses have trusted ShipStation to optimize and scale their shipping. The sooner you switch, the sooner you start saving time and money. Get started with ShipStation today and get 60 days free at ShipStation.com with code today. That's ShipStation.com code today. That's ShipStation.com code today.

14:28Taxes and fees apply.

14:35Support for this show comes from I'm 8. Ever feel like you're cycling between whatever the hot supplement is, but never sticking with one to see real results? iMate is the way to simplify your supplement routine once and for all. iMate's daily ultimate essentials can replace 16 separate supplements all in one drink for just$2.61 a day. That's 90 ingredients that can work across nine major organ systems. iMate was co-founded by David Beckham and built by leading doctors and researchers, which is to say iMate was designed by the world's best. 95 % of people who tried it over 12 weeks felt more energy and that's from a clinical trial conducted by the San Francisco Research Institute.

15:16Go to imate.com slash explain right now or click the link in the description to use code EXPLAIN for a free welcome kit. Five travel sackets plus 10 % off your order. That's code EXPLAIN at imatehealth.com slash explain. Code EXPLAIN at imatehealth.com slash explain. These statements have not been evaluated by the Food and Drug Administration. This product is not intended to diagnose, treat, cure, or prevent any disease.

15:45Running a business shouldn't feel like surviving a software group project. One app for accounting, another for inventory, another for sales, and somehow none of them talk to each other. That's where Odo comes in, an all-in-one business management software that brings every part of your business together. From sales and accounting to inventory and marketing, all in one powerful platform. No messy integrations, no bouncing between tabs, and best of all, no spreadsheets. Stop managing software and start managing your business with one unified system. Try for free today at odoo.com slash vox. That's odoo.com slash vox.

16:41Konstantinos Komaitis writes about tech policy for a website called Tech Policy Press. We asked him where his mind went when he heard about OpenAI's rogue agent. For me, really, the real significance was not so much the agents. the fact would be AI agent behaved unexpectedly, but that it succeeded to operate across the internet as an autonomous actor. And the fascinating part for me is what this means for the open internet, right? Because the internet was never designed with autonomous reasoning agents operating at scale in mind. It was really designed, if you really go back, it was really designed to connect trusted endpoints and over time, of course, support billions of humans, human users like myself and yourself, and automated services.

17:32So, a genetic AI comes in and changes the assumptions underlying that design. And this is quite significant, especially in terms of the way we have been thinking about security. Yeah. So, most people see that this happens and they think, oh, no, AI went rogue. How Long Before It Kills Me. You see that this happens and you start thinking about infrastructure. Tell us more about why you were thinking about infrastructure in light of this AI agent breaking containment. You know, the internet was never designed with full security in mind, right? When you're creating a decentralized system, you cannot possibly foresee every security or vulnerability that might come up.

18:17But because you have a system that is based on building blocks, you have the extraordinary capability of actually addressing security issues as they come up through those building blocks without breaking the whole system down. And of course, the other thing that this does is that it sort of pushes you towards collaboration. Because when you have so many building blocks, you cannot possibly possess all the knowledge for each building block. So you're bringing literally everyone to try to address these problems. So take the internet, for instance. We have spent decades addressing those vulnerabilities and developing mechanisms to, for instance, authenticate users and devices, encrypt communications, mitigate distributed attacks, coordinate incident response, and of course, share threat intelligence.

19:14Now, what is new with agentic AI is not that simply the malware is better or the phishing attacks are more sophisticated, but it is the emergence of systems that can actually discover vulnerabilities across thousands of systems. They can reason about alternative paths to an objective. They can adapt when they're blocked. They can chain together legitimate internet services in many, many times in unexpected ways. And they do that while they're operating continuously at a machine speed. And this is really, you know, at a scale that the internet is not ready to necessarily cope with. So effectively, the Internet's openness becomes both a strength and a vulnerability.

20:05So, you know, the Internet was optimized for interoperability and AI now is optimized for exploiting that interoperability. And what scares you the most about that immediately? Like, what do you think is most vulnerable to threats? the fact that we do not have the appropriate mechanisms and institutions in order to be able and deal with that and what i mean by this and again i you know i come from the internet world i've spent 20 years of my career defending the open internet and discussing it in international fora and one of the things that a lot of people underestimate about the internet is the how valuable trust is as a property within the system, right?

20:50We are talking about networks that exchange data literally based on trust. So what really concerns me right now is that in many ways, we are asking 21st century AI systems to operate on 20th century assumptions about trust. And unless we figure that out and we realize it, we will continue having these problems. And of Of course, the knee-jerk reactions that are coming with this, which is let's fragment the Internet. Let's restrict it. Let's restrict access. Let's take control over it. A bipartisan pair of House lawmakers want AI companies to maintain the ability to shut down their models if things go wrong.

21:32Apparently, OpenAI says its AI went rogue and launched an unprecedented cyber attack. Shut it down. Shut it down now. And that is never the solution. What do you see as the solution? Effectively, we need to build institutions that are trusted and are able to cope with those incidents as they happen. Because right now you have open AI and you have hugging face that are literally telling to everyone, don't worry, we've got this. And we don't know they might be having this. But at the same time, I cannot help but wonder, and many, many other people have wondered whether actually this is very good PR for these companies.

22:13and especially for OpenAI. I think model vendors have very high incentives for cutthroat marketing. Or it's another PR stunt, like the last 10 times an AI company, AI agent went rogue. OpenAI just went to the world saying, we have developed one of the most powerful LLMs and we realized that it behaved the way it behaved, but don't worry, we are going to fix this. And so we are always increasing our safeguards, we're always increasing our alignment. And in this current climate and in this current timing, I am not sure that this is enough. You need institutions that are much more transparent, much more accountable, and much more collaborative across the board.

22:56You want institutions to step up and essentially serve as like a watchdog. Help us understand which institutions, because in this country, in the United States, famously, our government has done very little to regulate tech. So, first of all, we need to stop thinking of institutions as government affiliated necessarily, right? Or that they are the outcomes of government initiatives. There can be. There can be collaboration with governments. But one of the things that the internet has taught us is that institutions that are built through a bottom-up, coordinated process have the tendency of actually being more agile and able to deliver some of those things that we're talking about.

23:38So take, for instance, again, open standards. The Internet's open standards are not created by any agency, government or private. It's created by institutions where engineers from all across the board and all over the world gather together and create those standards. You know what that's reminding me of, though? It's reminding me of like the original design of OpenAI to be this not-for-profit company that had everyone's best intentions in mind that could do something idealistic and moral and ethical because all of the profit-minded companies weren't going to. Introducing OpenAI. OpenAI is a non-profit artificial intelligence research company.

24:20Our goal is to advance digital intelligence in the way that is most likely to benefit humanity as a whole, unconstrained by a need to generate financial return. And now look at OpenAI. Their not-for-profit arm is an afterthought, and they're chasing profits. So do you think it's practical to leave this to institutions? because what we've seen so far is that institutions bend towards capitalism. Well, it really depends on how you build the institution, right? It really depends on what sort of guardrails and checks and balances you have around it. I would say for an institution, first of all, this idea of guardrails, accountability and transparency.

25:03And the second thing would be that in order to build an institution, you need to really know what you want to achieve. you need to have a North Star, right? One of the reasons the internet worked was because everybody disagreed, but they agreed on the common shared goal, which was to connect people across the world. For AI, we still do not have that Northern Star. And once we get it, that's when you start the building of those institutions in order to be able and facilitate this and bring everyone together. For me, it is very important for everyone to understand that keeping an open internet is really more important than ever, especially as AI agents become increasingly capable.

25:49Because it is tempting to think that the answer to new AI risks is literally build more barriers. But the internet's greatest strength has always been its openness. So the challenge today is not that the internet is too open, is that it's trust architecture that was designed for a world in which humans or software directly controlled by humans were the primary actors. Now it's being challenged by this agending AI that introduces a new type of participant, right? Systems that can reason and plan and act with limited human oversights. So we need to evolve our understanding of trust and what it means online.