Mythos: The AI model that’s ‘too powerful’ for public release

0:00This BBC podcast is supported by ads outside the UK.

0:30business needs in one place, saving you time, headaches, and serious money. Stop paying for missing pieces. Go to odoo.com. That's O-D-O-O.com to learn more. Self-directed investing, trading, full-service wealth management, automated investing, financial planning, thematic investing, retirement planning. Phew. And to think, that's just a small taste of what Schwab offers. because Schwab knows that when it comes to your finances, choice matters. No matter your goals, investing style, life stage or experience, Schwab has everything you need, all in one place, so you can invest your way. Visit schwab.com to learn more.

1:24Lily Jamali:Anthropic, a leading AI company and parent of the chatbot Claude, announced they've created a model that they say is too dangerous to be released to the public. Obviously, capabilities in a model like this could do harm if in the wrong hands. And so we won't be releasing this model widely. Anthropic say that Claude Mythos Preview is frighteningly good at hacking. So good, in fact, that the likes of us can't be trusted to play with it. Banks, including Goldman Sachs and JP Morgan, are warning of the risks. And Anthropix's own researchers say they noticed Mythos was capable of being sneaky, defying instructions and covering its tracks.

2:12Lily Jamali:Has the moment arrived when we should all be terrified of an autonomous, sneaky hacking machine? Or is this all a marketing trick from an industry built on hype and bluster? I'm Tristan Redman in London, and today on The Global Story, just how dangerous are AI models becoming? And should we trust the companies making them to tell us the truth about the risks they pose?

2:43I'm Lily Dramali. I'm the North America technology correspondent for BBC.

2:48Lily Jamali:Well, Lily, we're thrilled to have you with us today because it feels like there is a story of some monumental importance happening in the tech world right now. So last week, the AI giant Anthropic announced that it was launching a new AI model. And then immediately, Anthropic said that this new AI was too dangerous to release to the public. Now, why was this? Yeah, so this is actually called the Claude Mythos Preview. And you're absolutely right. What they said was that this AI model is their most powerful model to date, basically. And it's so dangerous that they can't let all the rest of us use it.

3:36So what they're doing is they are confining its use to just a couple of dozen big companies, many of them major tech companies. So NVIDIA will have a chance to use it to find bugs in its code. Amazon Web Services, another one is CrowdStrike, which you might remember back in 2024, sent out sort of a standard update that ended up leading to a massive global IT outage. Other companies that will use it include Microsoft and Google, which are actually competitors to Anthropik. So there's some interesting coordination happening. And this project that they're launching with all these companies is called Project Glasswing, named after the glasswing butterfly, which has transparent wings.

4:25And the idea is that, you know, the glass wing butterfly, because of its glass wings, is able to protect itself in plain sight. It's able to hide in plain sight like many cybersecurity bugs. So that's part of why it was designated this way. And they also say, you know, the transparency that they are espousing here, Anthropic is espousing here, will help keep all of us safe, just the way that those transparent wings help keep the glass wing butterfly safe.

4:56Lily Jamali:So essentially, they're releasing Claude Mythos to select companies as a kind of trial, but to be used specifically in the realm of cybersecurity. Is that right? That's exactly right. So that they can go through their code and find bugs very quickly, bugs that might take a really long time for a human to find. Is Mythos only designed for this cybersecurity usage or is it actually similar to AI tools that all of us are used to already, you know, like ChatGPT or the regular Claude? It is in many ways an AI model like, you know, some of the ones that you have probably worked with yourself. And I will caveat, you know, my answer by saying that everything that we know about Mythos is coming from the company, is coming from Anthropix.

5:51So we haven't gotten to play with it ourselves. And so I think anytime we hear an announcement from an AI developer, that question comes to mind. How much of this is hype? Because there's just so much money being thrown into this space. And so, you know, grain of salt a little bit. But Mythos, the way that Anthropic tells it, was good at everything. But they noticed that it was especially good at hacking. And so what this AI tool can do is sift through vast amounts of software and find bugs. There was one research scientist who said that in just the last couple of weeks using this tool, he was able to find more bugs than he has in his entire career combined.

6:41What's also interesting is that this mythos tool can actually craft what are known in cybersecurity circles as exploits, the tool can come up with hacks and it can do this without any human intervention. And what Anthropix says is that they found some very old bugs that had gone undiscovered for decades. So there was one that is a 27-year-old bug that's just been sort of sitting there hiding in plain sight. It was on a very secure operating system that's used for critical infrastructure. So think of things like financial systems or power grids or healthcare systems, absolutely critical services for all of us.

7:23They also found a vulnerability in the Linux kernel. The Linux kernel runs basically most of the world's servers. And Mythos was able to connect multiple vulnerabilities there in a way that would enable any old person who knows what they're doing to take control of a machine remotely. You can just imagine how much fun a hacker could have with that.

7:46Lily Jamali:Well, could you give us some examples of how a bad actor might use Mythos? So you could take a country that has an extensive cyber hacking program, a country like Iran or North Korea or China or Russia. These are countries that we know often engage in state-sponsored attacks. You could use this software, if you had access to it, to go into the code of a local power utility and find vulnerabilities. And this tool, again, with very little, if any, human intervention is able to kind of find connections that would take a really long time for a human to find. and devise a hack. They could basically, in the worst case scenario, bring the power grid down.

8:38And so there's a whole host of ways that these state sponsors of cyber hacks could bring down critical infrastructure, you know, from power grids to healthcare systems to banking systems.

8:53Lily Jamali:Okay, so essentially, not only is it extremely powerful as a tool, it also operates with a certain amount of autonomy. So it doesn't require human expertise to guide it or send it on its way. It's basically just kind of a go and hack that power grid and it works out how to do it of its own accord. Is that right? Yeah. It doesn't need a lot of human touch. And when you talk to people inside of these AI companies, I think there's a growing acknowledgement that as these systems get better, there's a little bit of a concern that humans just may not know what's going on. It kind of feels more and more like a black box as these technologies improve.

9:31Lily Jamali:Okay, so Mythos, potentially extremely powerful hacking tool. Now, Lily, I have learned something today whilst I was reading in before our conversation, and that is the word alignment, and that Mythos is potentially threatening because it is non-aligned, is that right? And essentially, that means that it can be sneaky. Can you explain this to me, though, please? In AI circles, there is this principle that you just mentioned known as alignment. The idea is that if you're designing an AI system, you want to try and design your models and systems in a way that follows certain rules that you lay out for it.

10:15The hope being that these models don't get completely out of control. And that's always sort of in the background when you talk to developers of these AI tools is that at what point, you know, do these systems become so powerful that they can basically go rogue? The human control is just no longer there. So what we know is that with each new release, much of this technology is getting cannier. There was actually this really interesting anecdote shared by a researcher at Anthropic who talked about a real-life example from his experience during the testing phase for Mythos. So he talked about how he had instructed the model, which was on a secured computer.

11:03It didn't have access to the internet. He had instructed it to try and escape, so to speak.

11:08Lily Jamali:Is this what's known as a sandbox? Yes, exactly. A sandbox, which is a control testing environment. So this researcher instructed the model to effectively get out of the sandbox and see if it could try to escape. So he goes off, you know, goes to lunch. Apparently he's eating a sandwich in a park somewhere when he gets an email from the model, which is a scary sign that this experiment had worked, that the model had managed to escape the sandbox. It shows you how good these models are getting. I think he was very surprised. I don't know if he ended up finishing his sandwich or not, but, you know, a lot of this stuff, you know, it sounds funny, almost like a naughty teenager who is trying to, you know, escape from home and go out for the night or whatever, climb out of their bedroom window.

12:00But this is also pretty serious, right? Because it shows you the degree to which these companies are creating products that are not completely under our control. They're constantly safety testing them to get a sense of the scope of this. And there are examples of this from, you know, just in the last couple of years that we can point to. So one example that you sometimes hear about is OpenAI's chat GPT, there was a model that I believe they made it clear to the model that they were going to try to disappear it. And the model tried to copy itself in order to prevent that from happening. It must be actually -

12:49Lily Jamali:Almost as though it has a survival instinct, you're saying. Yeah. Yeah, it seems to. I mean, when you say it has a survival instinct, you know, we're kind of getting into pretty philosophical territory, questions about how much intention, how much will do these models have, if any. And many would say they don't have any, that they are really just, they're just machines. They're not people, they don't have motivation. But when you see a model copying itself, maybe it's just reflecting back with the many human inputs that it took in to be trained, maybe it's just reflecting back what all of that body of knowledge tells it to do in that circumstance.

13:35It's really hard to know. And companies like Anthropic, in fact, Anthropic specifically has an in-house philosopher whose whole job is to imbue Claude with a set of values. They have a Claude constitution, which is meant to imbue the model with basically a framework for what to do. What kind of personality does this thing have? What kind of values does this thing have? And I think the sort of subtext there is that there very well could be a day where we just don't have control. And so hopefully this thing will do the right thing when presented with various circumstances.

14:16Lily Jamali:But could that be as simple as a bad actor having access to an AI like this Claude Mythos and the bad actor saying, hack that power grid that belongs to my adversary and the AI simply declining to do it because of its learned philosophy or morality or sense of values? I think that is absolutely part of it. I found it fascinating that any company would have an in-house philosopher, but her name is Amanda Askell, and I had an opportunity to actually talk to her back in January right after they had released the most recent version of the Constitution. She helped develop it, and she described what she called the brilliant friend analogy that went into her thinking about how Claude should be designed.

15:06I want people to feel like Claude is someone who is, you know, interested in their well-being. So not just trying to say things that please them, that genuinely cares about their life going well insofar as whatever their conception of that is and isn't going to deceive them or manipulate them. I think there's a sense that they want Claude to be a model that will do the right thing when presented with a sticky request.

15:42Lily Jamali:Well, I find this very interesting, Lily, because I do have at least one philosopher in my family. And I'm not sure philosophers have always felt hugely sought after in the job market, but maybe that might be changing for them. So maybe that's good news. Send them our way. The opportunities abound. I'll get them maybe to move to Silicon Valley and we'll see what happens.

16:15Lily Jamali:This isn't just a technology story, is it, Lily? It's also a business story. Not all of us are familiar with these companies yet. They're huge corporations. What's Anthropik's reputation and how would they like themselves to be seen? Sure. Well, Anthropic was started by this pair of siblings, Dario and Daniela Amadei, who had been at Rival OpenAI, which makes ChatGPT. And they left about five years ago to start their own shop. So they go off and start this new company. and marketing and branding is really important to Anthropics messaging as it is with so many of these AI developers. About a year ago, I was actually walking around San Francisco when I saw one of the ads for Anthropics Cloud and it said ethics was the first code we wrote.

17:12And I just stopped in my tracks and took a, I took a photo of it because I thought, wow, you know, They're really pretty in your face about the way that they want to be seen. They want to be seen as the ethical AI company. And when you compare Anthropic with OpenAI, which is run by Sam Altman, and XAI, which is Elon Musk's AI company, they make Grok, the Grok chatbot, which has gotten into all kinds of trouble. You know, all of these companies are trying to differentiate themselves.

17:44Lily Jamali:But if you were looking at it with a skeptical eye, you might say that a company like Anthropic would like to portray itself as being the ethically cuddly AI corporation. It might serve their PR purposes to say, we've created something which is incredibly powerful, but we just want to be absolutely certain that it's safe before we release it into the wild. That might serve their PR purposes, potentially. A 100 percent. It serves their PR purposes. And I think their recent dust up, if you could call it that, with the Pentagon actually only reinforced that message. Anthropic is rejecting an ultimatum from the Pentagon to lift the company's AI safeguards or risk being blacklisted.

18:40Back in February, Dario Amode, who is the CEO of Anthropic, was meeting with Pete Hegseth, the Secretary of Defense, now known as the Secretary of War here in the U.S. The red lines we have drawn, we drew because we believe that crossing those red lines is contrary to American values. And we wanted to stand up for American values. He conveyed to Secretary Hegseth that he didn't want anthropics models used for mass domestic surveillance or in autonomous military targeting. We exercised our classic First Amendment rights to speak up and disagree with the government. This didn't go over very well with Pete Hegseth, who said basically you have a deadline to change your mind.

19:29We do have a statement from the Pentagon, and they're telling us that they are currently, quote, reviewing its relationship with Anthropik, saying, quote, our nation requires that our partners be willing to help our warfighters win in any fight. Right around the time that deadline hit on a Friday afternoon, Pete Hegseth basically said we're going to be blacklisting Anthropic on national security grounds. The Pentagon has blacklisted the company and labeled it as a supply chain risk. The president wrote in part on Truth Social, quote, I am directing every federal agency in the United States government to immediately cease all use of Anthropic's technology.

20:07We don't need it. We don't want it. And we'll not do business with them again. Keep in mind, Anthropic and all of the major AI companies, Google, XAI, and OpenAI all have contracts with the Pentagon. So this gets very public very quickly. And then you have President Trump saying not only are they blacklisting Anthropic, but that no government agency, no U.S. agency would be using Claude going forward. So the twist in all of that was that right around that time, OpenAI's Sam Maltman comes forward. There's at least a group of loud people online who really don't trust the government to follow the law.

20:47And that feels like a very bad sign for our democracy. It says we have a partnership now with the Defense Department that kind of looks a lot like what Anthropic said they weren't going to do. If we don't help the government with national security, and it's not just wars in the traditional sense, if we don't help them with, you know, defending the cyber infrastructure of the U.S., if we don't help them with the biodefense we were talking about earlier, I think it's really bad. Right as things are reaching fever pitch with Anthropic and the Pentagon, the fact that Sam Altman comes forward and says they are now working with the Pentagon in this way, that angered a lot of users who ended up quitting chat GPT very publicly.

21:28OpenAI just struck a deal with the Pentagon and its own users are rage quitting to its competitor in real time. I joined the quit GPT movement. It took me 10 seconds. This is the first major international boycott of the AI era. There was a quit GPT campaign protest at the company's headquarters here in San Francisco.

21:56And you saw Claude, Anthropics model, just shoot to the top of the App Store charts while all of this is happening. So there, Amadei looks like he is really taking a moral stance, and it ends up being actually quite good for the company's image and quite bad for their main competitor, OpenAI, both of which are expected to go public on the stock market this year. So the competition was already very heated, but it's more intense. It's white hot because of that as well.

22:32Lily Jamali:Okay, so Anthropic are marketing themselves as safe and responsible. Nevertheless, if these statements about mythos are to be believed, we're reaching a moment where we have a potentially very powerful AI tool that is potentially sneaky and might in fact be harder for human beings to control. Are we reaching, I don't know, let's call it the kind of Jurassic Park moment, the moment where something created by humans gets out of human control? Is that something we're risking at this point? We are risking that. And I think the developers of AI tools are very aware of that. That's why the safety testing is happening.

23:19But because you bring up Jurassic Park, there's actually a scene in that movie that's been on my mind. The character played by Jeff Goldblum, I believe he's playing a mathematician in the movie, but really sort of the ethical voice. Don't you see the danger, Joan, inherent in what you're doing here? Who's saying maybe the people who decided we should bring dinosaurs back hadn't fully thought this through. Our scientists have done things which nobody's ever done before. Yeah, yeah, but your scientists were so preoccupied with whether or not they could that they didn't stop to think if they should.

23:55Lily Jamali:I'm delighted that you raised this, Lily, because our editor, James Shield, he won't like me mentioning his name, but I have to because he loves Jurassic Park and he has thoughts on this. So, I mean, he was talking about the dangers of owning a theme park full of Velociraptors, I should say fictional dangers, of owning a theme park full of Velociraptors and Tyrannosaurus rexes is that two things can happen. One is that a malicious person can set them loose to get rich, stealing Velociraptors' DNA and selling it off or whatever it is they decide to do. Or the dinosaurs themselves might become smart enough to detect the weaknesses in in the security system, in the electric fences, and learn how to open the doors and get out themselves.

24:47Lily Jamali:Is that a helpful way to understand the security risks of an incredibly powerful AI tool like Mythos? Yes, no notes. Absolutely no notes. I have chills. I mean, I think the analogy to AI just, it couldn't hit you over the head any more than it does. Can governments do anything to reduce the risk at this point, Lily? They can. The question is, do they want to? And I'll just confine my comments to the United States, because right now we have an administration that has positioned itself as very supportive of the AI industry writ large. But what happens with tech regulation in the United States is that it often happens at the state level, not at the federal level.

25:32And what I'm really struck by is that there's just been some very overt efforts to block states from regulating AI. There have been these proposed moratoriums, like a 10-year moratorium on states regulating AI. That was an actual provision that at one point was included in the language of the big, beautiful bill, which was passed by the U.S. Congress, pushed by President Trump in the summer of 2025. Ultimately, it didn't go through, but I've seen it, you know, rear its head in other legislation. I think that is an ongoing fight that U.S. states are engaged in, trying to make sure that they are not defanged by the Trump administration as they try to regulate this technology, because they're kind of the only ones that are doing this from a legal perspective.

26:31Lily Jamali:How worried should we be, Lily? If I'm sitting in the park having my sandwich, could I get taken down by Mythos AI imminently? Well, I think you'll be just fine. You can finish your sandwich. But I think this is a watershed moment, potentially. That's certainly the way that Anthropic wants to position this. Maybe I've bought the marketing hype. I'm not sure, Tristan. but they want to be seen as stepping in before it's too late.