The Global Story: The AI model that’s ‘too powerful’ to be released to the public

0:00Lily Jamali:This BBC podcast is supported by ads outside the UK.

0:30For more today's.

0:38Introducing Home Care Plus, a new subscription service from Lowe's that helps make life easier by giving members a hand with home maintenance. Let Lowe's tackle the tasks you keep meaning to do, like electric dryer vent cleaning, replacing hard to reach light bulbs and more. Subscribe to Home Care Plus for just$99 a year and consider your to-do list done. Members get more at Lowe's. Available in select zip codes only. Cancel any time. Non-refundable fee. Product purchase required. Terms and service restrictions apply. Details at lowes.com slash terms. Subject to change. Hey there, I'm Asma Khalid.

1:10Lily Jamali:And I'm Tristan Redman. And we're here with a bonus episode for you from the Global Story podcast. The world order is shifting. Old alliances are fraying and new ones are emerging. Some of this turbulence can be traced to decisions made in the United States. But the US isn't just a cause of the upheaval. Its politics are also a symptom of it. Every day we focus on one story, looking at how America and the world shape each other. So we hope you enjoy this episode. And to find more of our show, just search for The Global Story, wherever you get your BBC podcasts. Most of us have interacted with an AI chatbot for something by now.

1:53ChatGPT, Grok, Gemini, there are many. But recently, Anthropic, a leading AI company and parent of the chatbot Claude, announced they've created a model that they say is too dangerous to be released to the public. Obviously, capabilities in a model like this could do harm if in the wrong hands.

2:15Lily Jamali:And so we won't be releasing this model widely. Anthropics say that Claude Mythos Preview is frighteningly good at hacking. So good, in fact, that the likes of us can't be trusted to play with it. Banks, including Goldman Sachs and JP Morgan, are warning of the risks. And Anthropics' own researchers say they noticed Mythos was capable of being sneaky, defying instructions and covering its tracks. Has the moment arrived when we should all be terrified of an autonomous, sneaky hacking machine? Or is this all a marketing trick from an industry built on hype and bluster? I'm Tristan Redman in London, and today on The Global Story, just how dangerous are AI models becoming?

3:06And should we trust the companies making them to tell us the truth about the risks they pose?

3:17I'm Lily Jamali. I'm the North America technology correspondent for BBC.

3:22Lily Jamali:Well, Lily, we're thrilled to have you with us today because it feels like there is a story of some monumental importance happening in the tech world right now. So last week, the AI giant Anthropic announced that it was launching a new AI model. And then immediately, Lee, Anthropic said that this new AI was too dangerous to release to the public. Now, why was this? Yeah, so this is actually called the Claude Mythos Preview. And you're absolutely right. What they said was that this AI model is their most powerful model to date, basically. And it's so dangerous that they can't let all the rest of us use it.

4:10So what they're doing is they are confining its use to just a couple of dozen big companies, many of them major tech companies. So Nvidia will have a chance to use it to find bugs in its code. Amazon Web Services, another one is CrowdStrike, which you might remember back in 2024, sent out, you know, sort of a standard update that ended up leading to a massive global IT outage. Other companies that will use it include Microsoft and Google, which are actually competitors to Anthropix. So there's some interesting coordination happening. And this project that they're launching with all these companies is called Project Glasswing, named after the glasswing butterfly, which has transparent wings.

4:59And the idea is that, you know, the glasswing butterfly, because of its glass wings is able to protect itself in plain sight. It's able to hide in plain sight like many cybersecurity bugs. So that's part of why it was designated this way. And they also say, you know, the transparency that they are espousing here, Anthropic is espousing here, will help keep all of us safe just the way that those transparent wings help keep the glass-winged butterfly safe.

5:30Lily Jamali:So essentially, they're releasing clawed mythos to select companies as a kind of trial, but to be used specifically in the realm of cybersecurity. Is that right? That's exactly right. So that they can go through their code and find bugs very quickly, bugs that might take a really long time for a human to find. Is Mythos only designed for this cybersecurity usage or is it actually similar to AI tools that all of us are used to already, you know, like ChatGPT or the regular Claude? It is in many ways an AI model like some of the ones that you have probably worked with yourself. And I will caveat my answer by saying that everything that we know about Mythos is coming from the company, is coming from Anthropic.

6:25So we haven't gotten to play with it ourselves. And so I think anytime we hear an announcement from an AI developer, that question comes to mind. How much of this is hype? because there's just so much money being thrown into this space. And so, you know, grain of salt a little bit, but Mythos, the way that Anthropic tells it, was good at everything. But they noticed that it was especially good at hacking. And so what this AI tool can do is sift through vast amounts of software and find bugs. There was one research scientist who said that in just the last couple of weeks, using this tool, he was able to find more bugs than he has in his entire career combined.

7:15What's also interesting is that this mythos tool can actually craft what are known in cybersecurity circles as exploits. The tool can come up with hacks, and it can do this without any human intervention. And what Anthropix says is that they found some very old bugs that had gone undiscovered for decades. So there was one that is a 27-year-old bug that's just been sort of sitting there hiding in plain sight. It was on a very secure operating system that's used for critical infrastructure. So think of things like financial systems or power grids or healthcare systems. Absolutely critical services for all of us.

7:56They also found a vulnerability in the Linux kernel. The Linux kernel runs basically most of the world's servers. And Mythos was able to connect multiple vulnerabilities there in a way that would enable any old person who knows what they're doing to take control of a machine remotely. You can just imagine how much fun a hacker could have with that.

8:20Lily Jamali:Well, could you give us some examples of how a bad actor might use Mythos? So you could take a country that has an extensive cyber hacking program, a country like Iran or North Korea or China or Russia. These are countries that we know often engage in state-sponsored attacks. You could use this software, if you had access to it, to go into the code of a local power utility and find vulnerabilities. And this tool, again, with very little, if any, human intervention is able to kind of find connections that would take a really long time for a human to find and devise a hack. They could basically, in the worst case scenario, bring the power grid down.

9:12And so there's a whole host of ways that these state sponsors of cyber hacks could bring down critical infrastructure, you know, from power grids to health care systems to banking systems.

9:27Lily Jamali:OK, so essentially, not only is it extremely powerful as a tool, it also operates with a certain amount of autonomy. So it doesn't require human expertise to guide it or send it on its way. It's basically just kind of a go and hack that power grid and it works out how to do it of its own accord. Is that right? Yeah, it doesn't need a lot of human touch. And when you talk to people inside of these AI companies, I think there's a growing acknowledgement that as these systems get better, there's a little bit of a concern that humans just may not know what's going on. It kind of feels more and more like a black box as these technologies improve.

10:05Lily Jamali:Okay, so Mythos, potentially extremely powerful hacking tool. Now, Lily, I have learned something today whilst I was reading in before our conversation, and that is the word alignment, and that Mythos is potentially threatening because it is non-aligned. Is that right? And essentially, that means that it can be sneaky. Can you explain this to me, though, please? In AI circles, there is this principle that you just mentioned known as alignment. The idea is that if you're designing an AI system, you want to try and design your models and systems in a way that follows certain rules that you lay out for it.

10:49The hope being that these models don't get completely out of control. And that's always sort of in the background when you talk to developers of these AI tools is that at what point, you know, do these systems become so powerful that they can basically go rogue, that the human control is just no longer there. So what we know is that with each new release, much of this technology is getting cannier. There was actually this really interesting anecdote shared by a researcher at Anthropic who talked about a real life example from his experienced during the testing phase for METHOS. So he talked about how he had instructed the model, which was on a secured computer.

11:37It didn't have access to the internet. He had instructed it to try and escape, so to speak.

11:42Lily Jamali:Is this what's known as a sandbox? Yes, exactly. A sandbox, which is a controlled testing environment. So this researcher instructed the model to effectively get out of the sandbox and see if it could try to escape. So he goes off, you know, goes to lunch. Apparently he's eating a sandwich in a park somewhere when he gets an email from the model, which is a scary sign that this experiment had worked, that the model had managed to escape the sandbox. It shows you how good these models are getting. I think he was very surprised. I don't know if he ended up finishing his sandwich or not, but, you know, a lot of this stuff, you know, It sounds funny, almost like a naughty teenager who is trying to, you know, escape from home and go out for the night or whatever, climb out of their bedroom window.

12:34But this is also pretty serious, right? Because it shows you the degree to which these companies are creating products that are not completely under our control. They're constantly safety testing them to get a sense of the scope of this. And there are examples of this from just in the last couple of years that we can point to. So one example that you sometimes hear about is OpenAI's ChatGPT. There was a model that I believe they made it clear to the model that they were going to try to disappear it. and the model tried to copy itself in order to prevent that from happening. It must be actually...

13:23Lily Jamali:Almost as though it has a survival instinct, you're saying. Yeah. Yeah, it seems to. I mean, when you say it has a survival instinct, you know, we're kind of getting into pretty philosophical territory. Questions about how much intention, how much will do these models have, if any? And many would say they don't have any, that they are really just, they're just machines. They're not people, they don't have motivation. But when you see a model copying itself, maybe it's just reflecting back with the many human inputs that it took in to be trained. Maybe it's just reflecting back what all of that body of knowledge tells it to do in that circumstance.

14:08It's really hard to know. And companies like Anthropic, In fact, Anthropic specifically has an in-house philosopher whose whole job is to imbue Claude with a set of values. They have a Claude constitution, which is meant to imbue the model with basically a framework for what to do. What kind of personality does this thing have? What kind of values does this thing have? And I think the sort of subtext there is that there very well could be a day where we just don't have control. And so hopefully this thing will do the right thing when presented with various circumstances.

14:50Lily Jamali:But could that be as simple as a bad actor having access to an AI like this Claude Mythos and the bad actor saying, hack that power grid that belongs to my adversary and the AI simply declining to do it because of its learned philosophy or morality or sense of values? I think that is absolutely part of it. I found it fascinating that any company would have an in-house philosopher, but her name is Amanda Askell, and I had an opportunity to actually talk to her back in January, right after they had released the most recent version of the Constitution. She helped develop it, and she described what she called the brilliant friend analogy that went into her thinking about how Claude should be designed.

15:39I want people to feel like Claude is someone who is, you know, interested in their well-being. So not just trying to say things that please them, but like genuinely like cares about their life going well insofar as whatever their conception of that is. And isn't going to like deceive them or manipulate them. I think there's a sense that they want Claude to be a model that will do the right thing when presented with a sticky request.

16:16Lily Jamali:Well, I find this very interesting, Lily, because I do have at least one philosopher in my family. And I'm not sure philosophers have always felt hugely sought after in the job market. But maybe that might be changing for them. So maybe that's good news. Send them our way. The opportunities abound. I'll get them maybe to move to Silicon Valley and we'll see what happens.

16:59I've won majors, grand, and epic jackpots on their classic Vegas slots with huge, huge bonus rounds. Moto Casino adds new games and awards players free coins every single day. Grip it and spend it on Moto Casino. Download the Moto Casino app today. Moto Casino is a social casino. What we're prohibited to purchase necessary, visit Moto.us for more details. Moto Casino, America's social casino. Hey, it's Ryan Seacrest for Jewel Osco. You don't want to miss the annual beauty event for big savings on all your favorite beauty products. Now through April 28th, spend$25 on participating products and save$5.

17:33Shop in store or online for items like Billy Women's Razors, Billy Body Buffer or Body Wash, Native Hand Soap, Neutrogena Makeup Remover Towelettes, and Q-Tips. And save$5 when you spend$25. Offer ends April 28th. Restrictions apply. Offers may vary. Visit Jewelosco.com for more details. In a classroom of sodas, most stay quiet. Then there's Mr. Pibb, sweet cherry, bold outbursts, the kind of flavor that gets attention. Bold kick of cherry. Are you Mr. Pibb? Saving money on everything you need at Menards. Your roof is the first line of defense against the elements, and it enhances the appearance of your home's exterior.

18:15Owens Corning shingles offer ultimate protection and impressive curb appeal. Choose from 26 colors of Owens Corning Duration shingles and transform your home today. Shop our selection of roofing products on Menards.com. Plus, check out our weekly flyer and find plenty of great deals happening now.

18:38Lily Jamali:This isn't just a technology story, is it, Lily? It's also a business story. Not all of us are familiar with these companies yet. They're huge corporations. What's Anthropic's reputation and how would they like themselves to be seen? Sure. Well, Anthropic was started by this pair of siblings, Dario and Daniela Amadei, who had been at Rival OpenAI, which makes ChatGPT. And they left about five years ago to start their own shop. So they go off and start this new company. and marketing and branding is really important to Anthropics messaging as it is with so many of these AI developers. About a year ago, I was actually walking around San Francisco when I saw one of the ads for Anthropics Cloud and it said ethics was the first code we wrote.

19:35And I just stopped in my tracks and took a, I took a photo of it because I thought, wow, you know, They're really pretty in your face about the way that they want to be seen. They want to be seen as the ethical AI company. And when you compare Anthropic with OpenAI, which is run by Sam Altman, and XAI, which is Elon Musk's AI company, they make Grok, the Grok chatbot, which has gotten into all kinds of trouble. You know, all of these companies are trying to differentiate themselves.

20:07Lily Jamali:But if you were looking at it with a skeptical eye, you might say that a company like Anthropic would like to portray itself as being the ethically cuddly AI corporation. It might serve their PR purposes to say, we've created something which is incredibly powerful, but we just want to be absolutely certain that it's safe before we release it into the wild. That might serve their PR purposes, potentially. A 100 percent. It serves their PR purposes. And I think their recent dust up, if you could call it that, with the Pentagon actually only reinforced that message. Anthropic is rejecting an ultimatum from the Pentagon to lift the company's AI safeguards or risk being blacklisted.

20:54The Pentagon, as well as this major American artificial intelligence company, Anthropic, at odds over how to use its AI technology. Back in February, Dario Amode, who is the CEO of Anthropic, was meeting with Pete Hegseth, the secretary of defense, now known as the secretary of war here in the U.S. The red lines we have drawn, we drew because we believe that crossing those red lines is contrary to American values. And we wanted to stand up for American values. He conveyed to Secretary Hegseth that he didn't want anthropics models used for mass domestic surveillance or in autonomous military targeting.

21:37We exercised our classic First Amendment rights to speak up and disagree with the government. This didn't go over very well with Pete Hegseth, who said basically you have a deadline to change your mind. We do have a statement from the Pentagon, and they're telling us that they are currently, quote, reviewing its relationship with Anthropik, saying, quote, our nation requires that our partners be willing to help our warfighters win in any fight. Right around the time that deadline hit on a Friday afternoon, Pete Hegseth basically said we're going to be blacklisting Anthropic on national security grounds.

22:15The Pentagon has blacklisted the company and labeled it as a supply chain risk. The president wrote in part on Truth Social, quote, I am directing every federal agency in the United States government to immediately cease all use of Anthropic's technology. We don't need it. We don't want it. And we'll not do business with them again. Keep in mind, Anthropic and all of the major AI companies, Google, XAI, and OpenAI all have contracts with the Pentagon. So this gets very public very quickly. And then you have President Trump saying not only are they blacklisting Anthropic, but that no government agency, no U.S.

22:55agency would be using Claude going forward. So the twist in all of that was that right around that time, OpenAI's Sam Maltman comes forward. There's at least a group of loud people online who really don't trust the government to follow the law. And that feels like a very bad sign for our democracy. It says we have a partnership now with the Defense Department that kind of looks a lot like what Anthropic said they weren't going to do. If we don't help the government with national security, and it's not just wars in the traditional sense, if we don't help them with, you know, defending the cyber infrastructure of the U.S., if we don't help them with the biodefense we were talking about earlier, I think it's really bad.

23:33Right as things are reaching fever pitch with Anthropic and the Pentagon, the fact that Sam Altman comes forward and says they are now working with the Pentagon in this way, that angered a lot of users who ended up quitting chat GPT very publicly. OpenAI just struck a deal with the Pentagon and its own users are rage quitting to its competitor in real time. I joined the quit GPT movement. It took me 10 seconds. This is the first major international boycott of the AI era. There was a quit GPT campaign protest at the company's headquarters here in San Francisco.

24:19And you saw Claude, Anthropics model, just shoot to the top of the App Store charts while all of this is happening. So there, Amadei looks like he is really taking a moral stance. And it ends up being actually quite good for the company's image and quite bad for their main competitor, OpenAI, both of which are expected to go public on the stock market this year. So the competition was already very heated, but it's more intense. It's white hot because of that as well.