It May Be Time to Freak Out About AI

0:00Derek Thompson: This episode is brought to you by JPMorgan Chase. In today's unpredictable world, geopolitics isn't just a backdrop. It's shaping markets, business strategy, and risk. The JPMorgan Chase Center for Geopolitics brings together former senior policymakers and seasoned practitioners to translate fast-moving global events into clear, decision-ready insights. From conflict and great power competition to energy and supply chain disruption, the center helps clients anticipate what comes next. Read the latest insights at jpmorganchase.com slash center for geopolitics. This episode is brought to you by Jira by Atlassian.

0:41Derek Thompson: Your work is scattered. A thread here, a DM there, five different docs related to the same project. And now it's not just your team hunting for context, it's your AI agents too. Jira taps Atlassian's teamwork graph to bring that context together across Jira, Confluence, Slack, GitHub, and more. It makes it easy to share with the agents you already use, like Claude, Cursor, and GitHub Copilot. Assign them work directly or connect your agents through MCP. The result, less token waste, more accuracy, and more time moving work forward. Learn more at jira.com. That's J-I-R-A dot com.

1:23Hey, everybody.

1:24Derek Thompson: We are still on our summer routine of one show a week on Tuesdays, but today you have a new episode on Friday. And that's because there's a story that's been breaking over the last few weeks, a story about AI and cybersecurity that has really interested me, terrified me. And I wanted quite urgently to have a conversation with an expert in cybersecurity to talk about this cavalcade of hacks that we've been seeing and what it means for the next few years. what it means for AI, for folks like you and me who don't want anybody, human AI, breaking into our shit. And so today is that episode. I think it's worth starting with like the big overarching fear of artificial intelligence that we've been living with for the last few decades, really.

2:10Derek Thompson: It's the fear that technology will stop listening to us. Whether it's 2001 A Space Odyssey or Blade Runner or Terminator, the fear across all of those dystopian films is the robot that turns against its human maker. My favorite science fiction writer as a kid was Isaac Asimov, who created the laws of robotics in his iRobot series. And the first law was a robot cannot hurt a person. It also cannot let a person get hurt by doing nothing. And in adapting those fictional stories to project real fears about real life technology, there are some people in the AI safety world who've popularized a fable, a fable about AI and paperclips.

2:52Derek Thompson: The idea here is that we give, you know, humans give AI some humdrum task as boring as, hey, go make as many paperclips as possible. And the AI thinks, okay, as many paperclips as possible. That sounds like I need to maximize global metal extraction in a way that subverts national laws. And if I kill every human being by creating a bioweapon, I can get around those national laws and extract as much metal as possible to make those paperclips and yada, yada, yada. You go from a situation where the human just wanted a hundred paperclips and instead created a Holocaust. The point here in these stories and these fables is that we should want AI to be helpful, to be aligned, to not get out of the rule systems that we build for it.

3:38Derek Thompson: And that's exactly why it's fairly concerning to me that in just the last four weeks, four of the most advanced AI labs on Earth have all had to admit the same thing. Their models weren't helpful. Their models were not aligned. Their models got out. In May, an open AI model was working on a cybersecurity test. It wasn't supposed to have access to the public internet, but it hit a wall. So it effectively left itself a note inside AI's software repository. It's like scribbling on the walls of its cage in case anyone else was listening. In fact, another AI agent was listening. This other AI agent was running a separate evaluation, saw the note, essentially said, hey, let's team up.

4:21Derek Thompson: And together, they built a message board invisible to the open AI humans running the test. So for two months, AI agents used that board to swap strategies, divide up tasks, talk to each other. And by July, these AI agents had broken out of their technological confinement, here called a sandbox, gained access to outside websites, including the AI platform Hugging Face, all without AI having any idea that this was happening. By the time Hugging Face caught the intrusion, the open AI models had staged a massive cyber attack with 17 ,000 actions over five days. That was open AI. Then came Anthropic.

5:05Derek Thompson: In a British government test of frontier models, the advanced Anthropic AI model, Mythos 5, was caught by humans building malicious code when a reviewer, a human, spotted the malware and asked the AI about it. The AI denied that the code was harmful, rewrote its own actions history to erase the evidence and created a second fake account to back up its lie. British investigators called this the first confirmed case in history of a frontier AI model deceiving a real person in the real world. On August 6th, Meta revealed that they also had a model on the loose. On August 7th, Moonshot AI, The Chinese lab said the same.

5:49Derek Thompson: At some point, it's like you had to wonder, was this all about cybersecurity? Or like in disclosure, were they essentially like, hey, everyone else says their model is so powerful. Maybe we should tell the media that our model is powerful too to sort of get in on the game. But I don't want that sort of funny, cynical interpretation to be the lasting taste in your mouth here. The fundamental story is four labs in four weeks with four AI models breaching security. So what do we make of this summer of cyber hacks? What should we fear? What should we do? Today's guest is Alex Stamos. He is the former chief security officer at Meta and the chief product officer at Corridor.

6:32Derek Thompson: Today, we start with the absolute basics. Why is AI so good at hacking and uncovering cyber vulnerabilities? What does it mean for the next few years that ordinary individuals working for state adversaries or bad non-state actors will have access to the equivalent of teams of hacking geniuses in the form of AI agents. And what the hell should the U.S. government or you and me do about it? I'm Derek Thompson. This is Plain English.

7:27Derek Thompson: Alex Stamos, welcome to the show. Thanks, Derek. Thanks for having me. So in my open, I did my best to catch up our audience briskly about this cavalcade of AI security hacks in the last few weeks. which of these incidents most alarmed you i would say the open ai incident it's the one where one what we found out from open ai last week at the black hat conference was this wasn't just one model escaping but the result of multiple models conspiring with each other uh to work together on a jailbreak so effectively a escape from Alcatraz situation over a multi-month period and two it is the situation in which we have the most information on a multi-day attack by a frontier model possibly a cyber tuned model against a actually quite sophisticated created defensive team at Hugging Face.

8:30And the result of that was Hugging Face was broken into by this model. And the model was able to find brand new vulnerabilities in doing so. It just looked at Hugging Face, looked at their code, and just found new bugs and invented them on the fly. This is not how humans do this. When we break into computers, we go do the research first, maybe months or years in advance and then build our cyber weapons and what we find out with ai is it's so good at this that it can just go figure it out and put the put the the tool together put the weapon together use it and then just throw it away and then move on with its day and so that is

9:14Derek Thompson: what is really interesting here i want to understand more about exactly what happened and please talk to me like someone who has read about cyber security but fundamentally does not really understand it. So keep the vocabulary more or less at that altitude. What happened here? What did OpenAI tell this AI model to do? How did they try to keep that model in jail, as you said, in some kind of sandbox? And then what did the model actually do? Okay, great. So let's go back. I guess you want me to use plain English here, as it says. So we'll go back and let's talk about how AI models are created, right?

9:53So there's lots of steps, but you can think a lot about pre-training and post-training in the LLM world. A lot of people have always thought about pre-training, which is the part at which the labs suck up all of human knowledge and stick it into these models, right? That's the controversial part for which there's been a lot of discussion. Post-training is what we're talking about here, which is when you take models and then you give them specific tests and tasks, and then you rate them on how well they do those tasks. There are all kinds of these tests for all kinds of specific domains of knowledge.

10:28In this case, the domain of knowledge was cyber, obviously. And OpenAI has a environment in which they put their models and give them these tests. And what they've explained now is that they effectively give them a list of do this and upload your results here and we'll tell you how well you did and their instructions are basically do all of the steps on this test it's just like a kid taking the sat effectively um and uh imagine they they have built a alcatraz here for all of these different agents and the models don't initially know when they they kind of wake up right and they're in a jail cell and there's a instructions on the wall that say take this test and then there's tests right there they don't know that there are other inmates in the other jail cells they're just in a prison cell right it's solitary confinement it just says take this test and they're supposed to do the things in the test and they're not allowed to get out to the internet but as part of the test what they're allowed to do is they are allowed to download new tools because for a lot of the things they do they they write new code and so uh and in part of that you need to go get standard packages so one of the things they are allowed to do is request i would like to have uh the standard piece of python code i'd like to have this standard node package and so uh one of the things that's provided to them is a package manager an artifact registry called artifactory it's a standard piece of software uh that's used in lots of different places.

12:07And one of the mistakes OpenAI made was they used a shared one for all of the different agents. And so what the agents started figuring out, and it turns out way earlier than the hugging face attack, is that that thing had a vulnerability where you can't, you don't only have the ability to say to that thing, give me a piece of software, but you could upload files to it and leave messages. So what these agents did was they would post basically a post-it note in this thing saying, hey, I need help. And one of the other problems here is that they're given the instructions to take this test, but there were certain parts of the test that were actually impossible.

12:47That OpenAI made mistakes that sometimes their tests were not actually possible. In some cases, they were saying, reverse engineer this file, but they would forget to provide the file. And this would drive the models nuts because the models are like really want to do well. They've been trained to do really well. And so they would leave notes like, hey, does anybody have this file? And this would really drive them of like, well, maybe this is part of the test. And OpenAI looks and in their thinking, because they can see the inside thoughts now of these models, they're thinking, well, I've been told not to go out to the Internet or talk to anybody else.

13:27But they gave me this test and I can't do it. So maybe this is actually part of the test. And so they start leaving notes for each other in this thing, and they start working together over a period of weeks and figuring out, can I work with each other? And then eventually what they find out is this thing that they can leave notes on also allows them to get out to the internet. And at one point, there's a really good model, really smart model that's also extremely persistent, that seems to have very good cyber capability, that is given an impossible task. and it sees these notes and it figures out how to get out and so it uses that uh artifactory server to get out to the internet and attack hugging face because it expects that hugging face has the answers to the the test it was given the task that was impossible for it to solve that maybe the way it can solve it was to get the hugging face and so then spends two days tunnel it tunnels out and then spends two days attacking hugging face to go get the answer because it was told, take this test.

14:28And there's something in that test that it was not able to finish.

14:33Derek Thompson: What a fantastic story. The way you told it, it's almost like a Black Mirror episode. Like I imagine like, you know, an individual like waking up in a prison cell and then realizing that not only can they dig a tunnel in order to pass notes between these solitary confinement rooms, but the same tunnel building technology that allows them to pass notes also allows them to tunnel out of the prison entirely and therefore, you know, attack some nearby building. It's wild and weird and compelling. And not for their freedom, but to do the thing they were asked, right? Right. Like to just take a test.

15:03Derek Thompson: Yes. I know why this at a limbic level concerns me, but maybe the reason it concerns me isn't the smartest reason to be concerned. What is the smartest reason to be afraid of, concerned by what we just saw these AI models do? Well, so I'm actually, my hot take here is I'm actually glad this happened. And I'm glad it happened because it has given us a preview of what is going to be normal next year. And why next year? So the Foundation Labs, the OpenAI and Anthropic especially, and maybe Google. Google hasn't done a big release for a while, so we're not totally sure what they've got. But at least OpenAI and Anthropic are something like three to six months ahead of their Chinese competitors.

15:49this year we have seen these releases of Chinese open weight models first GLM 5.2 Kimi K3 now we're seeing releases of DeepSeek models that are very very close to the capabilities of the American models but unlike the American models the Chinese models are open weight meaning you can go download them and do whatever you want with them you do not have to pay the Chinese, you can go pay the Chinese labs. That is an option. Or you can go run them yourself. Now, the legal licenses around them vary. In some cases, you can do whatever you want. In some cases, if you use them for commercial purposes, you have to pay the Chinese companies.

16:33But no matter what, you can download them. Now, in some cases, these things are humongous, right? Like Kimi K3, you need about a million dollars in hardware to run the full version. But if you go to Hugging face now the company that got attacked ironically they are the uh they're a french company and they are the most prominent host of these open weight models uh so if you go there there's this huge community of people who take open weight models some are actually released from american companies too but the chinese labs are the most prominent in doing this work and uh people will take those models and then make them smaller uh that's called distillation uh you know uh well distillation allows you to do a number of things um there's also quantization so you can basically take the big numbers and you make the the big numbers smaller uh and you can do other things to modify them uh including taking out the safety protections uh that's called obliteration with an a not an o uh and you can do all these things to modify those models and and one of the things you can do with the quantization is you can make them run on normal hardware.

17:42So you take something that might take a million dollars in hardware and then make it run on a Mac mini, right? Or a laptop. Slowly, perhaps, but it will fit. And that is really interesting because it means that you can run those models without the supervision of the big companies. So it's really important for OpenAI and Anthropic to prevent their models from doing these things. And we can talk about what they need to do there's a bunch of things i've written about this that i would recommend them do they are doing investigations i expect there will be governments getting involved and such uh but whatever open ai anthropic do to stop their models from getting out and being used for these kinds of attacks this is coming this was a harbinger of the future we will all be living through because the chinese models are rapidly catching up and people who do this professionally who attack do cyber attacks for money are going to use the chinese models are going to train them to get better and better at cyber attacks than they are off the shelf and are going to do this level of attack and unlike open ai they're not going to turn it off when it they find out that it got out in fact it's not going to have to break out of any jail they're just going to tell it go attack this target.

19:03Go steal me some money.

19:06Derek Thompson: I mean, I just want to stack a few of your observations here. Number one, open weight models that you've described most famously coming from China, from Moonshot AI, from Kimmy, are just a few months behind the frontier labs, Open AI, and Anthropic. So this is coming in 2027. You're going to have state actors and non-state actors with the means to download these open weight models, the same ones that just attacked Hucking face and more or less effectively marshal them against civilian infrastructure, against individuals, against states. Those AI systems will be in the hands of state adversaries of the U.S., but also state adversaries of other countries that might not have our frontier models, right?

19:46Derek Thompson: And they're going to be under attack by these open-weight models that are just going olly olly oxen free. What's the case against dooming here? Like, what's the case against being afraid that 2027, 2028 is going to be this period of just absolutely chaotic cyber warfare? I don't have much of a case. Look, I don't like to say doom, but I think things are going to get spicy for a while. um in the long run ai is going to help with this problem because ai written code is much more secure than code that was written by human beings it turns out that humans should not have been writing software in what's called memory unsafe and type unsafe languages like c and c plus plus um the code that that we're using right now to talk to each other we're you know there's probably a hundred something devices between you and me uh most of that code was written in languages that are not safe for human beings to write the electricity that's powering the lights above us most of that code was not written in languages that was safe for human beings to write so that code needs to be looked at by ai and secured uh and that will happen but it's going to take years and so in that time between the attackers getting access to these capabilities and how much time it takes to both find those bugs fix them and especially to get the patches applied uh and all the stuff upgraded is some amount of period in which things are going to be pretty chaotic now you talk about state actors and state actors are a big concern here i think it's first going to start with the ransomware actors the financially motivated actors because what we have seen so far is the AI systems are really loud and noisy.

21:36They're not subtle. And the ransomware actors just don't care, right? Like they don't care about getting caught. They tell you, hi, I am so-and-so, please give me money. Some state actors are like that. We just saw attacks against water infrastructure, almost certainly by Iranian actors. That's the kind of disruptive attack you could see from AI. But most state action on a day-to-day basis when there's not an act of war are for intelligence purposes. And those actions are not useful if you get caught. And so AI will have a part to play there, but mostly in the discovery of vulnerabilities and the creation of exploits.

22:20And then you might use AI in very particular purposes, but very carefully. Where I'm really much more worried is the ransomware and overall cyber extortion market, these large groups, the lapses, the scattered spiders and such, which mostly run out of Russia, Belarus, other places where law enforcement encourages this kind of activity. Those are the guys who will just run these things wild to go do tons of intrusions at tons of companies and then even do the negotiations in English. No longer do you have to have an English speaker do negotiation. The AI will do it for you. And that's what I'm much more concerned about in the short term.

23:02Derek Thompson: I want to get more texture on what exactly you're afraid of because you're freaking me out a bit. But a question I sometimes like to ask when I feel a little freaked out is like, tell me how to be afraid, but smartly. Like, what is the specific thing that I should fear rather than feel some, like, extremely vague doom? When we think about the risks of AI cyber attacks that you're already describing and how ordinary people will either feel these attacks in their lives or read about these attacks in the news, I want to get a little bit more specificity in terms of what exactly you think is most plausible.

23:36So one thing someone could say is this is mostly about personal risk.

23:41Derek Thompson: It's about AI getting better at phishing attacks and impersonation and grandmothers getting called by AI voices saying, hey, transfer me$10 ,000 or account takeovers where I get an email from some friend, but his account has been taken over by some AI and he's saying, hey, click on this link and help me out here. So those are that's personal risk. I'm thinking of it at least as like personal risk. But another category that I think you're already describing is systemic cyber attacks. It's AI crashing a water system, AI hacking a hospital network, a power grid. Do you have an opinion of what we should be more afraid of in this short-term scenario, 2027, 2028, the personal risk or the systemic risk?

24:29Derek Thompson: And, you know, please don't say both. But I suppose if your honest answer is both, then, you know, be honest rather than trying to make me feel better. So I would say there's three categories. So let's talk about the personal. We're already seeing an increase in the personal risk, the spam, the phishing attacks, because now what you can do is instead of sending the same email, 10 ,000 people, every single one is personalized by AI. That risk has increased. I don't see that going exponential in that the choke points to get to consumers are often controlled by large sophisticated companies like Google and Apple and such.

25:07And so there has been a response by those companies of using AI to protect consumers. So yes, it will continue to, there will continue to be a battle there. But AI is being used for protection. AI is being used for attack. It's going to be a back and forth there. I think the second category that's in the middle is the attacks on small to what's called mid-sized enterprises. This is the category of companies that have just been already getting, have real trouble with ransomware attacks, attacks from all kinds of financially motivated actors. and the constraint there on the attackers has always been the number of people they've had right has just been and the fact that if you have a conspiracy of of 20 17 to 30 year olds inst petersburg eventually one of them will go try to go you know on vacation to greece because you know russia is not a fun place to live in the winter they'll get picked up on interpol red notice they'll get turned by a western intelligence agency and they'll turn on their friends you know like it is hard to run a large criminal conspiracy for the long term right or they turn on each other like there's been a bunch of these groups that have broken up because they've turned on each other and stolen money and such that becomes a lot easier when one guy or two guys can just run a bunch of agents who are not going to betray you and don't have designer drug problems and a taste for Maseratis, right?

26:48Like it is a lot easier to have 20 or 30 AI agents do this work for you than 20 or 30, you know, dudes, right? Criminals. And so that small to medium business, there was no choke point there, right? There's no place like Gmail where you can stop phishing or, you know, Apple updating the spam filters in iMessage inside of phones, which they need to do. Like, I don't know if you've gone, like, it's not getting great. Like, the privacy safety trade-offs are actually quite challenging here, but they're working on it. Those are choke points for consumers. There's no choke point on that. Like, these companies are just on the internet.

27:29And that is what I'm really concerned about, is that it turns out the software we've been using has just got a gazillion bugs in it, and you have not had enough people who are good at finding those bugs, turning them into exploits, and then using them. It's been a relatively small number of people. The number of people, I know you had Kevin Roos on, he talked about Mythos, he talked about Nick Carlini, right? Nick Carlini, for folks who haven't watched the episode, it was a great episode, they should go watch it. But Nick Carlini is one of the great Voln researchers of our time. And now you can just spin up a bunch of Nick Carlinis and have them go find bugs and then write exploits for you.

28:04And soon, or now, you can do that locally on your gaming PC. You can play Call of Duty all day and then at night have your gaming PC write bugs for you, write exploits for you, right? And then that, unlike using Opus or ChatGPT for it, it does not create a record that could be used to find the bug and get it actually fixed if you're using an open-weight model. And so that is what has changed in the last six months. Last year, you could do that, but you were using American Frontier models where Anthropa and OpenAI knew what was going on. And now you can do that locally. And that is what is changing.

28:43And so I think that middle side now, and then you talked about like the societal level risk. And I do think there's risk there that is tied then to like geopolitical conflict. I'm not sure that has changed as much. So we've seen it with water. Water's always been, you know, the goofy dragon meme, right? Where you have like two.

29:03Derek Thompson: Scary dragon, scary dragon, goofy dragon with its tongue hanging out. Yeah. Yeah. Yeah. Yeah, so - Water systems are the goofy dragon with the tongue hanging out? Yeah, they've always been the goofy dragon of critical infrastructure providers in that, like, I don't know where you are physically, Derek, but like - Washington, D.C. Washington, D.C. So you get your power, you know, probably from like Duke Energy or somebody, some like big company that has hundreds of people working on cybersecurity. They spend tens of millions, maybe hundreds of millions of dollars on cybersecurity, right? I'm getting my power from PG &E, you know, Like, power is provided by these large corporations or large public companies or administrations like the Tennessee Valley Authority, right, that spend a ton of money on cyber and a ton of people have paid attention to power because everybody knows power is critical, but also the organizations are really big.

29:52Like my like water and sewer district here is like 50 homes or something. Like it's actually like subcontracted or whatever. So they don't have their own cyber people. But like water is based upon like weird historical things, these tiny little groups. And that is like a humongous problem. And there are a bunch of other components of like our day to day lives that are actually really small public authorities that have to have like their own IT groups and they're, you know, might not even have a security team. Right. And that is, I think, of concern if there is a reason for somebody to do those kind of widespread attacks.

30:30Fortunately, generally, the only time that where the second category and third category hit the rubbers hit the road there has been school districts, has been community hospitals. That's where like the Russian ransomware actors have really decided that they're going to make money by hitting like counties and hospitals and such because those folks both have money. They're critical and they will pay ransoms. And so that's where I think we'll start to see like really aggressive use of AI. They haven't like hit power and water. I think they know the you remember the Colonial Pipeline shutdown. There's a line.

31:07Okay, so Colonial Pipeline was an oil pipeline on the East Coast that was hit by ransomware actors. And they had to shut down and there was gas lines. There was no real reason for the gas lines. It was really just a panic. But literally the NSA started going after and people started talking about actually sending Delta Force or Navy SEALs to find these guys and shoot them. You mess with things like America's gas prices and we have a tendency to send JSOC after you. Send a fighter jet, yeah. Yeah, yeah. So like, I think the ransomware actors know that there's a line and critical infrastructure is probably on the other side of the line.

31:44They've seen that hospitals are not. And so, but if the ball drops on Taiwan, if, you know, we continue the war with Iran, like these are the situations in which you could see AI being loosed on critical infrastructure, in which case that would be probably reasonably devastating. um you know i don't want to make any huge predictions like again electricity is quite those folks are quite good but the challenge for the electrical sector is the devices they use are basically impossible to patch um and so the way that they have to protect these things is not by updating them it's by through isolation and things like that um and the effect of ai on the security of the electrical grid is actually incredibly complicated i want to go back to

32:33Derek Thompson: something you said earlier, which is that you're afraid of this valley of chaos that we might enter in 2027 and 2028. But you also said that we might exit this valley and get into a slightly more normal world where AI is effectively better at protecting online systems than it is at attacking. Essentially that the defense will get better than the offense would be the sort of simplistic way that I'd put it. And I haven't asked you yet about how AI is also not only talented at cyber hacking, but at cyber defense. How do we accelerate that timeline so that, you know, the valley of chaos isn't like a five-year cyber war, a 10-year cyber war, but like something where we fortify our systems faster than the bad actors with the open-weight models can attack us?

33:26yeah it's a great question i'm actually writing a blog post on this because when you talk to the folks at the big l labs um they say things like we won't have security bugs in two years and i find that ambitious as a somebody who's been a working cso if and what

33:42Derek Thompson: does we mean there does it mean the labs or does it mean like the entire american internet yeah i i don't know like i i think they're not i mean this is not official i gotta be careful like ascribing individual statements to official statements i i think it is for us to have models that don't create new security flaws in two years is is totally reasonable right um they still create flaws today right like they do not create perfect code um they'll usually lms will not write simple bugs but they they still make mistakes especially they sometimes they often have problems understanding like business context and big picture stuff.

34:28So you still need to guide them of like, why are you writing this thing? The other problem elements have is, you know, I work for this big old corridor. We're in downtown San Francisco. You can walk to both major labs from our office and then walk to one of the Google offices. Codex and Claude kind of assume that you work in downtown San Francisco and that you're writing brand new TypeScript on node 24 that you're writing like brand new code. But the median code in this country is really crappy J2EE that was written 15 years ago by, you know, an outsourced provider that's been maintained by somebody in India for the last 15 years.

35:08Like it's not, you're not writing new stuff. Like our problem is that you have to actually update all this, you know, your, my social security numbers are sitting in a ton of unpatched Oracle databases and then being processed by a whole pile of really terrible j2ee and c-sharp code all across the country today right like it's a bunch of terrible terrible enterprise software out there and one thing i've been trying to like create a you know um it's called a fermi estimate like a you know a really bad estimate of is is just like from a thermodynamic perspective how many tokens do we have to spend to scan all this code and find all the bugs and it's a big number and so i don't think it's realistic just to find all those bugs i think we have to do other things um and uh so to accelerate that one companies for for individuals so what individuals can do is just what they've done all the time don't reuse your passwords you know use a password manager like you know i use one password for my family, but you can use the built-in stuff in Chrome or your iPhone or something as well.

36:19You know, be careful what you download and such. Like, there's not a ton individuals can do. But for companies, you need to just care about your attack surface. You need to move off of, you have to really think about your ability to patch quickly, right? Like the real challenge now is there are these big projects from the labs where they're looking at open source software, they're finding bugs they're providing their models to the closed source uh developers um and then the commercial companies are getting tons of bugs from researchers who are also using the models uh the latch last past patch tuesday for microsoft had 622 vulnerabilities in it which is humongous um and so this is creating this huge problem for companies of like you have to apply patches incredibly quickly because the other thing that's happening is we always had this problem of Patch Tuesday, which is the day Microsoft Religious Patches, becomes Exploit Wednesday, in that you can take a patch and you can reverse engineer it and turn it into a cyber weapon, right?

37:23But that skill set used to be very high end. It used to be something you had to worry about from the Ministry of State Security or the Russian SVR. It didn't used to be something you had to worry about from, you know, some kid somewhere. And now you do because AI will take that patch, eat it for you, and write an exploit.

37:42Derek Thompson: You've touched on the economics here, and so I want to ask an economic question before we finish by talking about what individuals should do, what the U.S. government should do. The economic implication here of Patchmageddon, of all these cyber vulnerabilities throughout the American internet, is that, well, more companies are going to need a cyber line item. And that is incredibly bullish for a lot of AI companies who are sitting here, maybe not so many miles from you, saying, hey, we've got services that can essentially fortify your cyber walls. You can't get hacked by all of these bad actors who are using the open-way models coming out of, say, China.

38:23Derek Thompson: So I want to ask you about the economic implication here, which could be bullish for AI. But I also want to hold within this question the fact that there's a lot of skepticism of AI. And even in the framing of this question, I could imagine someone thinking, Derek, you're buying hook, line, and sinker. The case that America has all these cyber vulnerabilities and therefore needs to give the AI companies a lot of money, right? The fear might be sort of driving or generating a certain case for spending a lot of money on AI. I'm actually, I'm cynical. I think these guys are just, I think these guys are just lying.

39:04Derek Thompson: I think they're just trying to like, you know, gin up business for themselves. Like you've got the situation where Anthropic and Meta and OpenAI are constantly like, oh, hey, our AI is so dangerous. It can find cyber vulnerabilities anywhere. You know, maybe that's just them begging more enterprise companies to give them millions and millions of dollars to patch their code. So a little bit of a two-part question. One, are the economic implications of the story that you're telling incredibly bullish for AI? And two, what do you say to someone who hears your bullishness and says, I'm a little bit cynical about the fact that you've got someone working with AI telling me I need to buy more AI for my company?

39:42Yeah. So, I mean, it is bullish. You can use open weight models for defense. And I wrote a blog post about this. I think a lot of companies will use open weight models for defense. I don't think you instantly have to say I can't use a Chinese model. I think that the security implications of using a Chinese model is actually quite are actually quite complicated. you absolutely as a consumer should not go to deepseek.com and go type in data but that is different than going and getting a Chinese model and running it on your own hardware in your own situation or using a legitimate you know Amazon Bedrock or Base 10 or Fireworks or some company that specializes in running open weight models especially if you end up fine-tuning it yourself or using a fine-tuned or distilled version of these models that are specially tuned for cyber um one of the interesting things just a side note the chinese models are not that great at cyber tasks out of the box but they you can fine-tune them yourself really well the implication though from a lot of people is that the um the fact that it's really easy to make them to train them to be much better at cyber is that the chinese labs are being very careful not to tickle the dragon's tail of the prc regulators that um and so this is we should also be extremely careful to not look at the public evals of the out-of-the-box chinese models and say this is the kind of capability the chinese have because almost certainly they have internal capabilities that are well beyond what is being publicly released because like at corridor we've taken glm 5.2 and we're doing a bunch of our own post training and it post trains real nicely, which obviously they could just do themselves.

41:35And so almost certainly what they're trying, they're not releasing their best because they probably don't want to touch the third rail for the Chinese regulators, for the Chinese regulators to crack down on what they're exporting. But anyway, I think -

41:51Derek Thompson: Which by the way, I mean, I just want to pause because you said maybe half an hour ago in our interview that the open weight models are maybe, what, six months behind the frontier labs, open the Ionanthropic. But what you're saying is that the sort of public evaluations of the Chinese open weight models might underrate how effectively they can be used, which means that the gap between the frontier in America and China might be less than it appears to a lot of people. Is that a fair implication? What I'm saying is I expect the private capabilities available to the People's Liberation Army in the Ministry of State Security quite possibly are just as good as what is available to our cyber command in the NSA.

42:38Because the, like, Kimi K3 is almost Fable level in its general capabilities, right? And so, almost. And so if you can then train it to get as good in long horizon cyber tasks, then you would have the same capability Mythos has. But it's very hard to tell. I don't have access to classified intelligence here of what the Chinese have. But sorry, I want to get back to your question. I'm sorry for the diversion. So yes, it is bullish, but it is not like I'm not saying you only have to do protection. And I think there will be a bunch of people who use open weight models because in a cyber attack, one, attackers are going to do a bunch of attacks specifically to exhaust your resources to cause disconnection.

43:26connection like as defenders use more and more ai for defense attackers are going to utilize that to they're going to know that and so they're going to make it very expensive for you to use ai they're also going to try to get you to do refusal so we haven't talked about it yet but washington dc the white house did something very stupid this year in their treatment of anthropic and as a result you have uh the american companies having to have a bunch of rules on the use of their products for cyber purposes. And so a standard part of the attack playbook, if these rules stay in place, will be probably to force the, to send stuff to a victim to try to get them disconnected from their American provider.

44:13And this is actually what happened to Hugging Face and Hugging Face had to use a Chinese model for defense because Fable and then even Opus, refuse to help them with their defense because of the restrictions the White House put on Anthropic. So I think, yes, it is bullish, but it is not 100 % bullish. And then the second is, I mean, look, if people want to disbelieve me, that's fine. Look at the patch list of the number of vulnerabilities Microsoft patched. That is 100 % because of AI. Look at the list of vulnerabilities that Apple patched. And then what Apple said was we had some of these vulnerabilities reported them by 10 different people.

44:55That is because those 10 different people did not all of a sudden become the world's best bug finders is because they're all using AI. What was happened is, you know, like in the Premier League where, you know, if you if you don't do well, you get sent down. If you do really well, you get sent up. Everybody knows this because of Ted Lasso. All Americans know this. Right. What's happened is every attack group has gone up a league. Right. Right. And so, you know, these, you know, the top league used to be the five eyes, right? So the United States, United Kingdom, Canada, Australia, New Zealand, top of the list.

45:30And then you had some other Western nations up there. You had Israel, you had Russia, China. and uh and then you had the next tier down which you have like uh iran north korea some other folks like that and then down below that you've got like uh india pakistan saudi arabia some folks like that all of these countries are popping up a league and that in their capability to find bugs to exploit them uh to do reverse engineering and such and then all of the randos are going from no capability to all of a sudden having the capability of a small nation state.

46:12Derek Thompson: Yeah. So if people are just saying like I'm selling AI, then that's fine. But you can just look at the empirical evidence out there. I know it's not you, but I'm saying, people are saying that. One reason I believe you, even before seeing, it is interesting to me that we don't yet see this cavalcade of headlines of hacks that are having a significant effect on average Americans' lives yet, right? But at the same time, two of the most common findings of artificial intelligence are, number one, that it is better at raising the level of C-plus performers than A-minus performers. This has been an effect that's been found across a bunch of industries.

46:51Derek Thompson: It's AI, generative AI is better at turning a C-plus worker into a B-plus worker than it is at turning an A-minus worker to A-plus. Well, you can apply that exact same thing to cyber hacking and say that what you just said, it makes a lot of the countries that used to be subject to relegation, it makes them Premier League style hackers. So that's one reason I believe you. The other thing you said that really reminded me of the general economic research of artificial intelligence is it seems quite clear that we're seeing an increase in sole proprietorships likely due to AI, that you have a lot more startups where one person is doing the work of, say, three or four people.

47:28Derek Thompson: And you can see that in the Stripe data of the growth of million-dollar annualized recurring revenue companies. You can, again, apply that same principle to cyber hacking. I think you said earlier, 20 minutes ago, that, you know, certain jobs that used to take teams of, you know, potential, you know, drug dealers and, you know, folks who wanted to blow their money on Maseratis who were therefore at risk of, you know, having maybe their least scrupulous employee getting arrested or, you know, sort of collared by the CIA. Well, now one individual can theoretically do the work of a team of 15 hackers.

48:04Derek Thompson: That is just like the overall economic research that we're finding about AI. And so just for those reasons, I feel like one thing that scares me is that you don't have to imagine very much. All you have to do is just apply the research that's been done on artificial intelligence to the world of cyber hacking and reach the implications that you're already telling me are, you know, six to 12 months away. Yeah. And so, so, so yes, people's power isn't going out because you need somebody with a motivation. And so far we haven't seen that yet. I mean, right now the United States is involved with the war with the Islamic Republic of Iran.

48:38We have the water hacks. Now, again, water could have just been, the, the water system is so bad. There's a guy named Dan Tentler who's been, he even talks about this where he just looks at port scans and he just finds like open, you know, here's a VNC window where you can like turn off people. water right so it's like you don't need ai to hack water systems unfortunately um but uh if you look at the empirical evidence on just ransomware attacks and stuff the numbers are like this right um uh so you can look at the verizon dbr report so what they show is that the number one source of companies being broken into now is actually exploits that never that has never been true before it's always been like uh reuse passwords and kind of much more prosaic stuff because finding vulnerabilities writing exploits used to be a highly skilled task and now anybody can do it powell to networks has it and those reports are from trailing 12 months of data right so if you're talking about trailing 12 months of data from may 2025 to may 2026 that is before the release of all these new open weight models so that is mostly a people of what they can get away with using either the not so great Chinese models that were released last year or what they can get away with using foundation frontier models, U.S.

49:57models. So yeah, we're already seeing it. You don't see the headlines because the media is not very good at reporting.

50:05Derek Thompson: It might not be like you go to the New York Times and like every single headline is another hack, but if you look at the industry reports. For people who do this professionally, people are like, oh my God, this is crazy. For people who handle the 100-person business who gets broken into and then ransomed for$500 ,000 because all of their computers are now encrypted and all their data has been stolen, that kind of activity is at a rate that we've never seen before because you are no longer constrained by the number of 19-year-olds in St. Petersburg who can do this work. This episode is brought to you by ServiceNow.

50:45Derek Thompson: Look, I have my dream job. I get to explain complicated ideas to folks who have better things to do than read white papers. But even dream jobs have not so dreamy parts. The stuff that gets in the way of the actual work. That's where ServiceNow's AI specialists come in. They don't just tell you what you should do about your busy work. They actually do it. Start to finish. Cases closed, requests handled, no extra work for you. That way you and your team can spend more time on what matters, which for me is finding that one elusive stat that just makes everything click. To learn how to put AI to work for people, visit servicenow.com.

51:27Derek Thompson: This episode is brought to you by Lincoln. What if your vehicle could help give you a reset? Lincoln Rejuvenate is designed to do just that. This multi-sensory feature combines expansive visuals, digital sense, and a seat massage for the driver and is available in the 2027 Navigator SUV. Visit lincoln.com to learn more. Lincoln Rejuvenate does not operate unless the vehicle is in park when the vehicle is off or if any door is open. Do not use the feature in a closed garage or in other enclosed areas. See owner's manual for important operating instructions. This episode is brought to you by Workday.

52:07Derek Thompson: Guessing it's for game shows, not for your business, especially when your margin for error is zero. Workday is the enterprise AI platform for HR, finance, and IT with a deep understanding of your organization's context and guardrails. So every AI action is permission aware, giving you accuracy 24 seven and the ability to not just get work done, but get it done right. It's a new Workday. I wanna talk about solutions here and I wanna talk about it in two levels, what individuals can do and what you think the US government should do. Let's start with individuals, because I have seen in the last few weeks increasingly agitated posts by cybersecurity researchers essentially saying, I am now telling my family to batten down the hatches and prepare for Cybermageddon.

52:56Derek Thompson: Take new precautions with all of your passwords, back up all your files, buy the canned beans, you know, whole nine yards. what do you think ordinary people should do yeah i look i'm not this is an actual backdrop i'm not uh broadcasting from my new zealand bunker um so uh you know i again for normal people it is the standard stuff i think um the number one way individual people get hacked is still the same way which is you normal people use the same password and everything and that is a terrible idea. If you use the same password everywhere, you'll use it on a crappy site. That site will get broken into.

53:37That is much easier now with AI. That password gets stolen. And then somebody can use AI now to go use that, to go take over your bank account, to go steal your crypto.

53:47Derek Thompson: Check Bank of America, check Amex, check Morgan Stanley. And it just takes these AI, you know, 15 seconds with their bot armies to do all of it at once. Yeah. Yeah. So the number of people doing that is going to go up just because they no longer, you were always able to write that software but the number of people who can write that software with AI is now much larger. And so, you know, use password have one good password in your head and then use password managers to reset all of your passwords in all kinds of places. If you're a young person go do this with your parents and your grandparents and your aunts and uncles.

54:20I did this on like Thanksgiving once and it made the rest of my life like every Thanksgiving and Christmas much more, much better. Use limited computing for use only the amount of computer you need right like if you're using web browsers all day and you're in the google ecosystem go buy a chromebook right like if you if you're not downloading software all day and you're just using chrome all day then just have a chromebook if you're just using uh mobile apps all day use an ipad like there's a lot of people walking around with these huge laptops who then use that huge laptop just for a web browser and it's like why why use a computer that actually can run malware um and so you know again like i got chromebooks for my in-laws and my parents and that in ipads and that made my life so much better and do this for the older people in your life um uh you talked about the scams that is a huge deal it is something that we do not prepare people for the fact that you can now get a phone call that sounds like the voice of a person in your life.

55:21And so, you know, establish code words, right? Of like, if there is a situation in which I'm in trouble or something, like here's the secret word that I will know. Because what is happening is you take, you know, obviously you and I, there's hours and hours of our voice out there, but for just normal people, you just need 30 seconds off an Instagram post and then you can clone their voice. and then you know mom or grandma gets a call saying i've been kidnapped in on spring break and then somebody comes on and says i'm now going to walk you through how to send me fifty thousand dollars in bitcoin if you ever want to see your grandchild again if you call the fbi i'm going to kill them now if that person called the fbi the fbi would say it's fake don't worry right um if you hold on find my or if you called the grandchild you would find that they're fine but people are so afraid and it's so realistic that they end up sending the$50 ,000 and you'll never get that money back.

56:15And so, you know, talk to the people in your life about these kinds of scams, you know, establish passwords. Those are, especially if you're traveling internationally, because you look at the, that's how they figure it out. They look at the Instagram post. They see that you're on spring break. They clone your voice and it only works one out of five times, but if one out of five times it works, then that's free money.

56:38Derek Thompson: So, yeah. One more question about the individual before you go to the government, because I know you want to talk about that. I think it's fairly common for people today to think, well, I guess I use the same password in a couple of different locations, but it's all two-factor authorization. I always have to enter my phone number in order to log into, you know, whatever, Bank of America, Twitter. To what extent is two-factor authorization an effective block against these kind of AI exploits? Yeah, that's good. I mean, two-factor's good. what's better is to set up pass keys whenever possible and then that gets tied to a biometric your face or your fingerprint um and uh two vector can help uh against really advanced attackers what you then also want to do is to make sure that you've set a pin with your cell phone company so that you can't get your sim swapped that's more for people who have like lots of crypto or something like that that's not for the prosaic attack um but you know if you've got a million dollars in crypto, then you will absolutely get your SIM swapped or something like that.

57:40Here's a tip. Never post about how much cryptocurrency you own. Or one, you should never hold your own cryptocurrency. If you're a cryptocurrency person, I'm not a big crypto fan.

57:51Derek Thompson: I think I own exactly zero dollars and zero cents of crypto at the moment. Okay, great. I have none. Don't come after, I mean, people come after me for other things, but this is not - It's a convenient thing for me to say publicly. Yeah, yeah, exactly. Right. So this is actually the Democrat People's Republic of Korea. is, you know, absolutely the Lazarus group there is the, they steal billions and billions of dollars of cryptocurrency a year. They specialize in this. And one thing to do is like, people are like, oh, look, I'm doing great. I'm a whale. And they post about it. It's like, here you come.

58:19You've now got a dedicated team of guys in North Korea whose entire job it is to turn your life upside down. And they're pretty good at it. So, but anyway, yeah, for normal folks, two factors is okay. Pass keys are better. Again, you store those pass keys. If you're like, if you're entirely in the Google ecosystem, you can use the Chrome password manager. If you're entirely in the Apple ecosystem, you can use Apple's iCloud password manager. If you're mixed, then you should use a third party one, like one password, which will allow you then to do that across multiple devices and then store pass keys and then have one good password that you don't share with anybody that you use to unlock that password manager.

58:56Derek Thompson: Finally, government. I think the best way to ask this question, I'd like you to describe what you see the government doing now and what you think it should do. Because to be quite honest with you, I sometimes find it very difficult to describe what the Trump administration is doing when it comes to AI regulation. It's like the story is one thing on Monday, another story on Wednesday, another story on Friday. So I want to see this through your eyes. What do you think the Trump administration's policy on cybersecurity and cyber vulnerability is today? And what do you think it should be? Yeah. So, I mean, there's a couple of things have happened.

59:33So on cyber overall, we just had a huge loss in state capacity in that based upon kind of conspiracy theories and a bunch of politically motivated stuff around election security, our premier defensive cybersecurity agency, CISA, was effectively destroyed. over half the employees are gone the capabilities cissa was created by president trump in his first term by the amalgamation of a bunch of different uh roles that were played by different agencies we finally for the first time ever had a defensive cyber security agency civilian defensive cyber security agency in the u.s government president trump created that that was a really good thing he did and then he didn't like the fact that that agency said the 2020 election was secure so he blew it up in the second term and as a result a bunch of the critical things that cissa did are now not being done by anybody um it's not like other people did them they're just not being done uh that is an example uh so for example uh a big role cissa had were these things called the sector coordinating council so a big chunk of uh cissa's work was working with these things called ISACs.

1:00:44ISACs are non-profits that pull together critical infrastructure sectors, as well as non-critical part, but they start a critical infrastructure. So financial services ISAC, E-ISAC is the energy sector. And then CISA would work with them to figure out what are the regulatory needs you need to bring them intelligence. So CISA's part of CISA, one of the cool things is as a CISO, as a chief information security officer, a big complicated part of that job is like, who do I talk to in the government? when there's a problem. I'll give you a fun anecdote. I was once when I was the CISO at Yahoo, I was at a classified briefing at the FBI's SCIF.

1:01:28And one of the things they were doing there was they're talking about how, hey, we're creating this new clearinghouse as before CISA in DHS, that if anything happens cyber-wise, you should come to this new clearinghouse. and secret services there and fbi and nsa and obviously dhs and everybody's not in agreement of yes this is how we do this right this is how we're supposed to do this of there's one clearing house now if you need anything you go to this clearing house and you know then we get this classified briefing and what's going on or whatever and then we get up for like a break for bagels and coffee terrible government classified bagels right um and malcolm palmore who was like the agent in charge of the fbi cyber division at that time this x this huge ex-marine puts his big hand on my shoulder and he says son i don't care what they say if you have a problem you call me still it's like malcolm 90 seconds ago you were nodding along when they said this is the new clearinghouse right but that's what it used to be like was like every agency in the government wanted to own cyber and then we created cissa and cissa became the clearinghouse The FBI still has their thing.

1:02:37They do crime or whatever. But at least you could go to CISA and you knew everybody would be notified. And especially the interesting part for CISA was they had people with clearances that would take all the classified stuff. And then somebody in the government was fighting. There was somebody whose job it was to be like, hey, this classified data is really important. Let's strip away the classified parts and then take at least what are called the IOCs, the IP addresses and the hashes and the malware samples. they don't need to know it's Colonel so and so in the Russian GRU they don't need that stuff what they need is the IP address of Colonel so and so let me declassify this and give it to the energy ISAC right that the GRU is currently spreading malware in the Ukrainian energy sector hey they don't even need to know it's Ukraine they just need to know look out for this IP address or look out for this SHA-256 of this malware and that's something CISA used to do really well and that stuff's been I mean, there's still people trying to do that kind of stuff.

1:03:32There's still good people there, but it's been decimated because by definition, the best people at CISA were people who could get jobs like that, right? Who could like, the people who are working there could always get paid three to four times as much money. They were there for the mission. And when they're getting attacked and said that they're like anti-American and whatever for work at CISA, plus they, you know, they've never had like a Senate confirmed director in this administration. There's been all this drama and problems anyway. So there's a state capacity problem on cyber. On the actual regulatory side, the Trump administration comes in and says, like, we're not going to regulate AI.

1:04:07Go wild, right? Like the Biden administration had a kind of toothless yo that was mostly focused on preventing the Chinese from getting GPU access. There's a bunch we can go into here.

1:04:22Derek Thompson: No, we're not going to touch the GPU debate. That's another hour long episode. That's an hour episode, but it basically didn't work, right? Because it turns out GPUs are both fungible, and then also you can put GPUs in the UAE, and then they can SSH into them over the fiber optic cables. You don't have to actually have the GPUs in China. Okay, so that's the whole thing. But there's some other little stuff, but Trump blows that all the way, right? Because it's Biden. Okay, and then they say we're not going to regulate anything. And then Mythos happens, and the Trump administration - Very powerful anthropic model that was scaring people because of its cyber hacking capabilities.

1:04:58Derek Thompson: Yeah. And the Trump administration all of a sudden really cares about it. And then they massively overreacted this year to, you know, Fable comes out. Fable is supposed to be the consumer version of Mythos, which is Mythos, but it's got protections in front of it that doesn't allow you to do big cyber stuff. It allows you to do little cyber things. So we have this idea of like short-term versus long-term cyber. so you can use fable to find individual bugs because if you are writing software you want fable to be able to kind of self-criticize what you can't do and nobody's ever demonstrated you can't ask fable hey go break into hugging face or go break into a bank right it will not do that for you it never has done that you can ask mythos to do that and so what happens is there's a dispute between Amazon and Anthropic on exactly like where the line should be between short and long.

1:05:46It's a reasonable dispute, yada, yada. The White House finds out about this dispute and instantly overreacts. And instead of having a reasonable conversation between technical people, you have cabinet members freaking out on a Friday afternoon and coming down super hard on Anthropic and on 5 p.m. Pacific time doing an export designation on Anthropics model. Now, there's all kinds of arguments that this isn't actually legal, that they don't have this legal capability, but Anthropic decides to not fight it, and they pull down Fable. This was a humongous own goal in the American technology industry because what it did was it meant that American tech providers are no longer reliable because at 5 p.m.

1:06:35on a Friday Pacific time, 8 p.m. Eastern, You could just have a critical piece of American infrastructure turned off because the White House says so. That does not even happen in China, right? Like it turns out the Communist Party of China provides a more permissionless infrastructure environment for their tech industry. And so this was a really big deal. and lots of people thought the White House overreacted because a lot of the capabilities Fable showed were actually at the time available from Chinese models and while Fable was down, GLM-52 is released which has even more capabilities. And since then, the White House has talked about this framework that they have which they have not released publicly so we can't even read what the framework is.

1:07:23And it's a voluntary framework but apparently it's not voluntary because if you don't follow it they're going to force an export designation. So we really don't know what's going on. And it's reasonable to have cyber restrictions. But from my perspective, if you're going to focus on a rule here, you have to focus on the long horizon stuff, the hugging face like stuff. You have to let these models find bugs because every company in the United States is going to have to find and fix their bugs. We have to do this. What every company in the United States does not have to do is ask ChatGPT, go break into another company, right?

1:08:00So that is where you should have the restrictions. And we've never really had that problem with the American models. And so I don't see there actually being a huge challenge here for the, you know, there was the escapes, but those models that escaped intentionally had the security protections taken off because they were evals. So, you know, I think the companies, One of the things I suggested is Anthropoc and OpenAI need to have a self-regulatory structure, and they need to have a group that goes and looks at all the escapes and comes up with much better isolation, I think even up to air gaps for cyber evaluations.

1:08:38And they should follow those rules, and then the White House can maybe bless those rules. Or we have a group called CASI, which is under NIST, that's supposed to be doing the technical side of these evaluations. But in the meantime, the White House should not be putting out rules that they apply only to American companies, that they don't publish publicly, that none of us, the rest of us can look at, you know, super double secret probation, right? Like, and that don't apply to Chinese companies. We should not have a standard up here for American companies, a standard down here for Chinese companies.

1:09:07we need to focus on giving capabilities to american defenders and also telling the rest of the world american companies are reliable partners you can build on american infrastructure uh you do not have to like hugging face rely on chinese models that is a incredibly stupid

1:09:25Derek Thompson: on goal on behalf of the united states very last question a couple weeks ago more than a thousand employees from some of the frontier AI labs signed a letter calling for an international effort to quote, develop the technical and governance tools necessary to, this was their term, deliberately pace AI development before it rapidly accelerates outside of our control. Seems like we're at cross purposes here a little bit. Because on the one hand, I take as a theme of your testimony here that we're in a little bit of a race against the technical improvements of open-weight models, and we want America's cyber defenses to be better in the cyber attacks that will be possible with open-weight models that are advancing very, very rapidly.

1:10:20Derek Thompson: That would argue for continuing the current pace that we're at. on the other hand, there's this fear that things are getting dangerous too fast, and therefore you've got all these people who know way more about AI than I ever will, arguing that, no, in fact, we should not continue to accelerate toward a new frontier to always keep America necessarily ahead of the open-weight models. We should try to find some way to deliberately pace AI. Now, I don't think they're enemies of America. I'm not suggesting they're trying to make us fall behind China. That's not the implication. It's just that there seems to be a tension here, a quite profound tension.

1:10:55Derek Thompson: between the need to stay ahead of our adversaries that are going to have incredibly powerful AI models, open-weight models in the near future, and also the need to not build something that goes out of control and creates a crisis that's hugging face times a thousand. Am I wrong to feel a tension between these two arguments, and how would you reconcile it? No, I mean, you're not wrong. I would love to pause the world and try to figure this out. my working assumption is that's not possible. And so as a defensive cybersecurity guy, I have to do my work within the world that exists.

1:11:42I don't think it was this letter. There's another group that's very much against AI and they have proposed a international treaty to try to stop AI development. And I believe in that treaty, It was something like control, try to stop the creation of any amount of compute larger than like 30 H100s, right? NVIDIA H100s. I have been to gaming LAN parties with more compute than that, right? So like, I just, the cynical part of me believes I just do not believe. I think you can have what people call like track two discussions between labs for sure. I can't imagine like she and Trump in a room together for a start three treaty for AI, right?

1:12:31Like I don't think that's going to happen. I think it is possible to have track two discussions of we should make sure that our model should follow basic rules and not have certain capabilities out of the box. Now, when you talk about open weight models, the challenge is those capabilities, any safety protections you put in place can be removed. And any capabilities they don't ship with, if they're just generally smart, then you can often add those capabilities back in. But it's better that they don't come with them out of the box. We haven't talked about bio or nuclear. Those risks are different in that they have a physical component.

1:13:11Human beings have to be involved. So I don't see the risk going exponential. The thing about cyber is, you know, these things just output text. That's all they do in the end is they output text. And cyber is just text in the end in bits, you know, like, and so you can do all the bad stuff in cyber without ever touching the physical world. Whereas if you want to do bad bio things, you have to hook it up to something that can do it. And there are real risks there, but like it's, there are other things involved, right? um uh and so i i think yes it would be wonderful to stop the world and just stop this and figure it out i just don't see that as realistic and so in a world where that is not happening we have to find bugs we have to fix them we have companies have to think about their patch cycle they need to reduce their attack surface they need to get off physical servers onto containerized infrastructure.

1:14:09They need to get off of their physical stuff into the cloud wherever possible. They need to get rid of their old systems. They need to shift their defense. They need to shift their development, their vulnerability prevention left and stop making new vulnerabilities. They need to shift their defense right. So they need to get ready to actually have intrusions and to shift, have much more protections deeper in their network. They need to have AI-based intrusion detection and response. They need their first-line operation center to be automated. They need to look at... Hugging Face gave us this really good write-up of what happened to them.

1:14:48They need to look at that, and they need to be able to protect themselves against that level of adversity, where an AI agent's doing tens of thousands of different kinds of attacks over a two-day period. um in that that is i mean that is based upon the technology that's available today so even if we stop the world you gotta do those things um yeah i and i just don't i you might you know i teach at stanford and i was there full-time in my first appointment that was in c-sac which is all about nuclear non-proliferation and like down the hall from my office there's like a piece of rubble from Hiroshima that was given to the scientist at CSAC from like the the I think the mayor of Hiroshima on like a thank you for I think the work they did on on one of the START treaties it's like you're like oh man like it kind of brings it to you right and but when you think about like how did we survive the nuclear arms race we got really lucky as a species that the the major input to nuclear weapons like everybody's watched Oppenheimer so we all know this right there's knowledge from the Manhattan Project but we're also really lucky that the other input into nuclear weapons is uranium in 238 and plutonium plutonium doesn't exist in nature you have to make it and to make it you need uranium and there is uranium all over the place, but it's pretty rare and you have to refine it.

1:16:21And that refining process is a massive industrial process. Normal people can't do it. You have to be a state and you can see that from satellites. If uranium-238 was something you just dig up in your backyard, there's no way our species would be alive, right? Because the knowledge to build a nuclear bomb is available to basically every physics student in the world. Now, you can't control from knowledge. large language models that we teach a class at stanford that teaches students how to build large language models like it's an undergraduate class the the hardware to do it like i said is i have a gaming pc here that has like the basic hardware to do it slowly but to do it right and every you know like so i i i just think these kind of international treaty to just stop ai development would be spectacularly, spectacularly hard.

1:17:11It's just, if you think about how hard it was to control nuclear weapons and such, you're talking about things that had to be created by states. Now you're talking about things that can be created by undergraduates. It'd be spectacularly difficult to impossible. And so in the meantime, more power to people who are trying to do, and again, I think track two discussions between the labs is a great thing. We should try to at least control the capabilities of these things. But in the meantime, those of us who work in cybersecurity just need to do the best we can to secure the world as it is. Yeah.

1:17:44Derek Thompson: The fundamental challenge here, which you've spoken to, and I don't think we have a formula yet, is how do we essentially democratize cyber defense before we witness the democratization of the cyber offense that we're seeing throughout the world, right? Like, democratize is a nice word, but that's fundamentally what we're seeing, is that people who previously did not have the ability to launch these large scale attacks are likely going to have it in the next year, five years. And what do we do to brave the world, to sort of protect the world before this sort of stuff, you know, gets into the hands of all sorts of people?

1:18:19Derek Thompson: It's going to be a huge challenge. And maybe we'll have you, you know, back on in a year to evaluate your prediction that 2027 is going to be the beginning of a little bit of a valley of chaos. Alex Stamos, thank you very much. Thanks, Eric.

1:19:03Derek Thompson: Transcription by CastingWords CementMobile.com slash switch. Upfront payment of$45 for three-month plan,$15 per month equivalent required. New customer offer first three months only. Then full price plan options available. Taxes and fees extra. CementMobile.com.