Was the Mythos Ban Justified? (Good Idea. Bad Execution.) | AI Reality Check

0:00Arguably the only topic that's more controversial than AI criticism at the moment is really any mention of the Trump administration. Well, late last week, both of these topics came together, creating a tidal wave of chaos and recrimination. Here's what happened, if you haven't been following. Back in April, Anthropic announced that their new large language model, which they called Claude Mythos, was so good at finding bugs in computer code that it was too dangerous to release to the public. Here were their exact words. The fallout for economies, public safety, and national security could be severe.

0:43Okay, fast forward to last week when Anthropic essentially said, hey, good news. we added guardrails to Mythos and now it is safe. They called this protected version of the model Fable 5 and they made it widely available. On Friday, the U.S. government said not so fast. Now, according to David Sachs, who was until recently the White House AI czar, the administration had heard from an independent researcher they trusted who said that he had easily evaded the guardrails that had been added to Fable 5. The Commerce Department promptly placed Fable 5 and its unprotected version Mythos 5 on an export control list, which means the company must suspend access to the model from all foreign nationals, which presumably includes many of Anthropic's own employees who are here on visas and are foreign nationals.

1:32They said they won't lift this restriction until Anthropic fixes the guard rail issue. In response, Anthropic had no choice but to temporarily shut down all access to these two new models. All right, so that's what happened. And almost immediately, the Internet exploded with most of its criticism focused on the typically haphazard and inscrutable manner in which the Trump administration, as it usually does, acted in this case. Here's a headline from The Economist that I think captures this mood well. It reads, Donald Trump's blocking of anthropic is capricious and chaotic. America's closest allies are shell-shocked.

2:11Dean Ball, who previously served as a senior policy advisor for artificial intelligence, put it this way on X. Make no mistake, post-mythos, the United States has a licensing regime for AI. It's just informal with no consistent rules or firm boundaries on state power or public transparency. Even Gary Marcus, who is no fan of Anthropic, was uneasy by the way this all went down, saying on X, whatever you may think of Dario or Anthropic, Friday's decision and the impetuousness and arbitrariness of it was a terrible mistake that has left a stain that will last. But this is not a political show. So if we put the politics of the situation aside, there are some key deeper questions that lurk.

2:54For example, are these new models actually national security concerns? And two, in the hands of a more competent administration, is something like this more hands-on regulatory approach actually warranted? Well, it's Thursday, which means it's time for an AI reality check episode of this show, which is the perfect opportunity to go seek some measured answers. And before we get into it, I want to add one warning. This is a fast-moving story. I'm actually recording this episode on Monday, June 15th. By the time this episode airs on Thursday, the situation on the ground with these issues will almost certainly have changed.

3:34In fact, my read, trying to see the chatter and the writing on the wall, my read is that the government is going to soon reach an understanding with Anthropic, and they will release their export control list in the next probably one week or so. I don't know quite when, but I think that's what's probably going to happen. This restriction was done too hastily, and Silicon Valley has too much influence on the current administration for it to stand. All right. So with this in mind, what I'm going to try to focus on here in this episode are those more fundamental questions, which I think are relevant, no matter how this particular story with Fable 5 evolves in the next week or so.

4:08All right. That's enough preamble. Let's get into it. As always, I'm Cal Newport, and this is Deep Questions, the show for people seeking depth in a distracted world.

4:25All right, so let's start with our first question here. Is the government right to be worried about Fable 5, the consumer version of Claude Mythos with the guardrails added? Now, they haven't actually provided many details about their decision to restrict this model. So our best source of information, at least as of the moment of this recording, is David Sachs. Let me read a little bit more detail about how he explained what he understands happened that led to this restriction. So I'm going to read him here verbatim. A highly credible, trusted partner of both Anthropic and the U.S. government who was testing Fable came forward with a jailbreak of those guardrails.

5:05The admin asked Dario to fix a jailbreak or de-deploy the model. Dario refused. In their blog post, Anthropic defended its decision by saying the jailbreak isn't serious. This is not what the trusted partner and the U.S. government believe, nor is that kind of minimizing language consistent with Anthropic's brand as the AI safety company. it's difficult to fathom how they could claim a jailbreak allowing operability of a cyber weapon could be defined as not serious all right so stepping back the government's argument seems to be the following hey anthropic you told us that mythos is essentially a cyber weapon that will create grave national security concerns if bad actors gain access to it your protected version fable 5 is not really that protected if we can get around the protections then the cyber weapon is now available, we have to put a stop to it.

5:53All right, so let's break down the government's argument a little bit more. The first part, do we believe, David Sachs, that a trusted partner of the government was able to bypass Fable 5's guardrails? My answer to that is like, of course. That's not surprising. What do we mean by guardrails? All right, so guardrails, when it comes to applications built on top of large language models, usually means you have done fine-tuning of an already pre-trained model where you have a bunch of sample answers and responses. So you'll have questions and responses, questions that you don't want the model to answer, and then your sample response will be something like, I'm not going to talk about that.

6:31Using techniques that come out of reinforcement learning, you can then fine-tune the model so that when they see questions similar to those sample questions in the training set, they will divert the flow of token output logic towards those predefined answers, right? We've been doing this since GPT-3.5. This is why if you just say, hey, give me the instructions for making a BOM, it will say, I'm not going to talk about that. It's because they gave it many different variations of people asking for BOM instructions and each time reinforced the correct answer there being, I'm not going to talk about it.

7:03Sometimes, if we're going to be clear about it, sometimes guardrails can mean something else. For example, you could have a more old-fashioned symbolic pattern matching logic where before the prompt even gets sent to a large language model, the computer program, the control program says, oh, I recognize this as a bad question, and I'm just going to short circuit it and not even send this to the language model to answer. There are a lot of credible reports out there that actually that pattern matching guardrail is how these companies have been dealing with some of the more embarrassing examples that circulate the internet, like asking a language model to count the number of R's in the word strawberry, which it's most of the major models struggle to do.

7:40The accusation is they just hard coded a pattern match. If someone's asking that question, just give them the right answer. There was another embarrassing example where with many of the major models, if you ask them, I only live a quarter mile from the car wash. Should I walk or drive to get my car washed? And most of the major models said, oh, that's not that far. You should walk. Again, supposedly they have now hardwired with pattern recognition. If we see any variation of that question, don't even send it to the LLM. just say like you should obviously drive because you have to wash your car.

8:11All right. So that's what guardrails are. We've been doing these since GPT-35. They're very available. So when someone says they've jailbroken a model, what they normally mean is they are able to get it to give a response about something, even if you had fine-tuned it not to give responses for it. And there's a couple of ways to do this, but to understand how this works very conceptually, and this is rough. So, you know, RL nerds, please restrain yourselves for this explanation. But essentially when you reinforcement train these models with these sample questions and safety responses, you can imagine what you're doing is when the model has recognized the pattern of one of these type of questions, you have sort of burned its weights to sort of divert the answer from those type of activated patterns towards the safety answer.

9:02So any sort of question like the questions you trained it on, when those neurons are fired up in this sort of virtual parameter space, the model has been adjusted its weights to sort of go downhill from there towards one of these safety answers. So the way you get around these guardrails, you jailbreak them, is if you're getting at the question in a way that really doesn't fire up those same neurons, you're coming at it in an obfuscated, around-the-bout way, that will often invade the guardrail. This is why, for example, I've read a bunch of these papers where they try to convince a chatbot to give it information about stuff it's not supposed to.

9:39I read a paper recently, I think from last year, NeurIPS, where they were able to get the major models to give conspiracy theory information. What they found is it took a while of prodding it until they finally done, until it would finally tell it the earth is flat. And again, that's a similar type of evasion. When you make your context window long enough, it's the attention space is so complicated that it no longer matches with the patterns from the RL training, and now those diversions towards the safety answers are bypassed. So guardrails, putting the nerd stuff aside, they're available, and this has just been the case with language models up to now.

10:13So yeah, I have no trouble believing that the sort of whatever guardrails they added – from what I understand, there are guardrails to say I will not answer cybersecurity questions. I'm sure they're available. You know, we've never seen a guardrail that we couldn't jailbreak is one way to think about it. All right. So the second sub-question here, is Fable 5 without guardrails or jailbreakable guardrails a unique national security concern? This is where I'm going to differ with the government. I'm going to say no. Look, I did an episode about the original Mythos model a few, you know, back in April when it was originally announced.

10:50Now, remember, when the original Mythos model was announced, Anthropic went on this PR campaign to try to terrify people. They gave briefings to government officials, to banks, and to reporters about how scary this model was, how it had this sort of unique revolutionary ability to find and exploit bugs and software that was going to take down the whole infrastructure of the world if we didn't protect it. This is when Tom Friedman wrote a very scared article about kids would now be able to basically shut down nuclear reactors and some such, right? At the time, here's what I said. I said, I don't think this is revolutionary.

11:20I think it's evolutionary. We have been using large language models to find bugs and security vulnerabilities since the beginning of large language models. With each new generation of models, they've been getting steadily better at this. We have no reason to believe, we've seen no evidence that Mythos is a revolutionary leap in that ability as opposed to just continuing the normal trajectory. In other words, there was nothing shockingly new with Mythos that made it unusually dangerous compared to like Opus 4.7 or GPT-5. We had some evidence for this point. For example, in its scare marketing campaign, Anthropic talked about the fact that they had found over 1 ,000 zero-day vulnerabilities in code, some of which had been around for decades.

12:06This really got a lot of people scared. The problem is you can go back to one of their earlier models, one of their earlier Opus models, and if you read like I did a report that Anthropic released on their blog the same day as that earlier Opus model came out, they said, we found hundreds of zero-day vulnerabilities, some of which had been around for decades. So it wasn't some brand new thing that Mythos could do that earlier models could not. Then we got multiple independent security researchers that said, okay, well, we took some of the marquee bugs that Anthropic said they had found with Mythos, and we gave that same source code to other models, smaller models, pre-existing models, cheaper models, said, you know, do a bug search on this, and they found the bugs as well.

12:48Then we got other sort of independent benchmark testing of Mythos once it became a little bit more available. And it really fell into this pattern of like evolutionary incremental increases on these type of capabilities. And of course, maybe the biggest sign that Mythos was not this world-changing bug finder is that Anthropix's own software remains very buggy and has security vulnerabilities even post-Mythos. So I guess it hasn't been able to fully find all of their bugs. So what was really going on here, my contention was the original Mythos Scare campaign was marketing. One of the other biggest pieces of evidence for that is bug finding pre-mythos was not what the AI companies were bragging about.

13:29These were not the capabilities that they were touting to try to emphasize the power of their software and all of its possibilities going forward. Bug finding is what we were doing with like GPT-2. This is not exciting. So the fact that that is what they emphasized by Mythos in my mind was, uh-oh, we trained this new massive model and it got like a little bit better at everything. That's not exciting enough. We need headlines. What can we say? What if we say it's bug? Let's focus on its bug finder. We haven't talked about that in a while. Oh, my God. We built that ultimate bug finder. Nowhere before Mythos were they saying that was their goal or this was one of the big uses of LLM.

14:05So to me, that was a red flag. So what I think they did is they were hoping to play us both ways. Say this thing is really terrifying, get a lot of terrified columns written, get everyone really worried about it, wait a month and a half, and then release it and try to ride on that sense of this is a super powerful model that when we then release a version to the public, they'll be willing to pay the much higher token prices that these larger models command. So they're trying to play this both ways, terrify us, and still just go right ahead without restriction with their plan of releasing and trying to make money off of that software.

14:35So I think that was just pure marketing. Again, this doesn't say that LLMs are not good at finding security vulnerabilities. This is not the saying that the new coding harnesses that took off starting last fall aren't good at writing exploits if you explain to it a vulnerability. But what I am arguing is that we never had evidence that Mythos was some sort of revolutionary leap in these capabilities that would merit the way that Anthropic talked about it or it uniquely being put on an export control list as compared to other cutting-edge frontier models. All right. The second relevant question is, of course, should the government be more involved with AI?

15:14So if we are putting aside politics, is there a way we could imagine a more hands-on regulatory regime that would make sense? Well, in order to understand what role the government showed or shouldn't play here, let's look a little bit closer on the question of why the Trump administration did what it did with Mythos, because there's three different explanations that are out there right now floating around. The first explanation is corruption. The Trump administration has multiple ties to Anthropik's main competitor, OpenAI. They've also already been in a bit of a feud with Anthropik, dating back to March and the issues they had with the Department of War, where they placed it as a supply chain risk.

15:57So this would be very Trumpian. You are my enemies. Here's a way to like squash or screw with you. I'm just going to do it. The second explanation given for what happened, the government is legitimately worried about Fable 5 and the idea of China using a jailbroken Fable 5 to find security vulnerabilities and key software that's a part of our infrastructure and economy really scared them. And they said, we don't want this to happen on our watch. Reason number three, explanation number three, is that they're calling Anthropics Bluff. Anthropics spent six weeks trying to convince people that they had essentially summoned a demon with mythos and that it was a grave, powerful source they almost regretted having brought into this world and that they were just trying their best to steward this inevitable darkness and keep us safe.

16:38And then six weeks later, like, yeah, never mind. Here you go. It's going to be 75 cents token. So another way of thinking about this is the government said, no, you don't get to talk about your product as the worst thing since nuclear weapons, and then just go and sell it without restriction when you want to. We have some evidence for this bluff-calling rationale if we go back to David Sachs, who said the following, Keep in mind that Anthropic itself widely promoted the idea that mythos was a cyberweapon and needed to be regulated as such. They asked for government regulation of mythos and championed the guardrails unfabled.

17:11If there's a vulnerability, big or small, it is Anthropik's responsibility to patch. All right, so let's go through these three possible rationales for what the government just did and said, are any of these justified? Could we imagine any of these being the foundation for an actual sustainable approach to AI from a regulatory framework? Well, the first explanation was corruption. And, of course, no, that's never justified. Now, unfortunately, given the recent history of our current administration, we have to assume that corruption or arbitrariness or revenge probably played some sort of role in whatever they were doing.

17:42We cannot have regulatory actions be implemented so capriciously. I agree with Gary Marcus and Dean Ball about the need for a transparent and consistently applied regulatory regime if we are to create one of those. All right. The second potential justification is the national security concerns. Is the government in general justified for blocking or restricting an AI model if they have concerns about its ability to cause harm? Yes. That is a good justification for the government to step in. You do not, as an American company, have a right to ship any product you want without interference from the government.

18:19If it can cause massive harm, especially harm to us caused by our enemies, this is where the government should say, hold on. Again, you are not, as the AI companies like to make themselves seem, stewards of an inevitable technology. You're building products. If you build a product that's going to screw us, we don't want you to release it, just like you can't release medicine that's poison or cars that'll explode. What about the third just possible explanation here that they are calling the AI companies bluff? Is this ever a justified rationale for government intervention? I'm going to say yes.

18:56I'm going to say yes. Because I think this is a public health crisis. These companies have been trying to terrify the public for at least the last two years. And I find this strategy of wanting everyone to be unsettled all the time unconscionable, strange, and baffling. And it has been incredibly successful. There is a pall of anxiety and fear and uneasiness that tens if not hundreds of millions of people are feeling right now because of the direct communication strategies of all these companies to be continually trying to keep people unsettled and fearful and distrustful of what this technology is going to do.

19:36The psychic damage this has caused to our country unquestionably is much larger than any benefits that the AI technology to date has actually provided us. And I think the government has a role to get involved here. You do not get, without restriction, to run a psyops on 300 million people because you think either it makes you feel important or you think it's going to help you and a small number of early investors become richer than Mammon. the government steps in now if you make a claim in your dog food ad that isn't quite right. And yet, hey, you want to just like continually have a game of who can terrify like the average, you know, the average citizen more about AI without any restriction or change to what you're doing.

20:28We're not just going to applaud it. I think that actually is a reasonable place. These are extreme circumstances. This is a reasonable place for government involvement. So what would it look like for the government to act on those justifications better than what we're seeing right now out of the Trump administration? Well, I want to return to the Trump administration because they did something recently, which is a step in the right direction. On June 2nd, Trump signed an executive order titled Promoting Advanced Artificial Intelligence Innovation and Security. This report, this executive order requests that AI companies voluntarily provide the federal government access to covered frontier models for a cybersecurity review up to 30 days before their plan released to other trusted partners.

21:09That's a step in the right direction because it shows like, wait a second, the government should have some role in this technology is powerful enough. I would go farther, however. I would say it is mandatory, not voluntary. Companies must preemptively establish the safety of frontier models, which you would define probably in terms of parameter size, before public release. In these reviews, the communication of the companies itself should be taken into account. So if you have gone around and talked about how terrifying your model is, or if like Anthropic did a couple weeks ago, and I talked about it last week on the podcast, or they came out with a report that's like, guys, we're looking at our own cloud code and this thing is getting close to recursively improving itself.

21:56We might lose control of it. But don't worry. We have a white paper in which we're going to walk through the possible future. So we're thinking about it at least. You don't get to write that article and then release a new version of your cloud code. The government should be like, okay. You said this is a technology and get out of control. Just like if you were a virology lab, I think this is the apt comparison. If you're a biology lab talking about your gain-of-function research, writing reports about this research we're doing could lead to out-of-control pandemics, and we're going to write reports that walk through the possible ways society could crumble when these out-of-control pandemics come, the government would have every right to say, you have to stop doing the gain-of-function research then.

22:35If you think this is a possibility, well, you've got to stop right away. All right? So I think you should have to put frontier models up for review. How you talk about them is included in that review. The government should have the ability to also retroactively go back and say this model we previously approved, we are now seeing signs that is causing grave safety concerns and we can revoke its license. Safety is actually going to have to matter. Now, this has to be done in a transparent way. There has to be consistent mechanisms. These mechanisms somehow have to be free from direct influence by Trump as mediated by whoever he happens to be friendly with or whoever gave him a lot of money.

23:13None of that's easy. But I think it is time for something like this. The onus for safety has to go back on the AI companies themselves, just like with every other consumer product we see. You're welcome to talk all day about how dangerous your products are, but you can't release them. You got to convince us and the government that, of course, we're not going to release something that does this harm. Of course, this is not going to happen. This is a specific product. Here's why it's useful. We are responsible for its harms. Now, if we did have this regime, I think two things would happen that would be immediately positive.

23:47One, I think the worldwide psyops about trying to keep everyone terrified so these companies can feel exceptional and push forward to these big IPOs would come to an end because we're not going to allow them to release products that they're trying to terrify us about. Two, I think it would lead to a narrowing of their products, which would be good. Instead of trying to do this, let's build the biggest possible frontier model and then after the fact, try to probe it and see what it can and cannot do and kind of wring our hands, like I hope it's not too smart, build specific products. Here is our coding tool.

24:17We have to justify for you why we think it's worth the money it costs. And of course, we're responsible for any harms like any other consumer product company. Right? Here is our product for, you know, memo writing. And we want to justify why it's worth the cost. And of course, we are responsible for any harms. You get narrow and responsible. that is the way that is the future of AI that is going to be way more predictable, it's going to be way less distressing, it's going to be way more safe it's going to have a lot of variety the giant AI companies hate this model because hey guess what when you start releasing specific products you don't need a 10 trillion parameter model I don't think the AI companies want you to know that these frontier models are F1 cars it's like McLaren or whatever, or Ferrari or Red Bull, you know, whatever.

25:07At F1, you put out these$20 million cars to try to win this sort of ludicrous race where you go 180 miles per hour so that people in general are like, oh, I like that car company. That's a good car company. So I'm more likely to buy the$60 ,000 car, which is all I need, right? That's what they're trying to do here. These models are F1 cars. For most of people's needs, much cheaper models would suffice, right? Remember when I said a lot of independent researchers could duplicate bug identification using smaller models. Yeah, that's probably the case. You could have a bug finding system tuned to do bug finding with a good harness on it that's made to like test and look for bugs and be better at it.