Risky Business #853 -- We're all gonna die, apparently

0:05Patrick Gray: Hey everyone and welcome to Risky Business. My name's Patrick Gray. Got a great show for you this week. We will be joined as always by my colleague James Wilson, who is co-hosting today, as well as Morgan Adamski, who keen listeners would remember was on the show once before back, I think in 2023, where we recorded an episode at the NSA's Cyber Club Association. Center, where Morgan was at the time. She has most recently, her most recent government job was Executive Director of US Cyber Command. And these days she is at PwC. What's your job there, Morgan, at PwC? I oversee the cyber data tech risk work.

0:49So lots of different things.

0:50Patrick Gray: So I guess you're head honcho of cyber at the cyber practice of PwC. Pretty impressive title. so thank you very much for joining us this week's show is brought to you by airlock digital and the founding team at airlock digital will be joining us in this week's sponsor interview to have a chat about how they've integrated their product with crowdstrike and i guess to have a bit of a chat about how that whole crowdstrike integration works because you know crowdstrike has really sort of successfully turned itself into this like security platform company and uh yeah these integration things are CrowdStrike Foundry, I think it's called.

1:28Patrick Gray: It's actually pretty interesting. So Daniel and David from Airlock will be joining me later on after this week's news segment to talk through all of that. But, of course, it is time to get into the news now. And I guess we're going to start with the sobering news, Morgan, that we're apparently all going to die. This has been the week's news cycle. We had some guy leave one of the Frontier Labs and do the interview circuit talking about how there was a 10 % chance that AI was going to end humanity, which is like, you know, that doesn't sound great. And then since then, we've had the chief executive of Anthropic publish an open letter, again, tech guys with their open letters, talking about how we must pace the frontier and, you know, it's all very dangerous and we need to have embedded overseers and we need government regulation and all of this stuff.

2:22Patrick Gray: It's just been a week of absolute panicked shrieking about how the text generators are going to come and kill us all. I mean, you could probably tell from the way I just described that, that I'm not on board with the we're all ruined vibes. But what did you make of all of this? Yeah, I mean, I think when you see a lot of the technical communities like, okay, wait a second, like what's actually possible and what's not? Let's actually be truthful here when we talk about what the threat is and what AI can potentially do. You talk about AI and cybersecurity and the internet being taken over by botnets versus actually AI being access to like biological weapons.

2:59There's a little bit of a difference, right? And so there's a lot of conversation right now on like, how true is this? How much should we care about it? And to your point, like, what does pacing actually mean? You can't really set that pacing in a timeframe that makes sense. I mean, this was part of the conversation when you talked about cyber norms, right? We want to establish these norms in cyberspace of how we're going to do cyber operations. and everyone's like, well, what is that really attuned to? What kind of conversation would government actually have? And so I've spent a lot of time having like practical conversations of what AI can do today and how people should really think about guardrails and securing AI and what's actually plausible and what they should think about.

3:37But it is a little bit of a hype cycle right now.

3:39Patrick Gray: I mean, the thing that blows me away, right, is yesterday I did next week's sponsor interview yesterday with Haroon Mia over at Thinkst, right? And they've built basically like a canary or a honeypot that is designed to interact with agents that are doing offensive tasks against you, the target. And it's just you look at the dumb stuff that they have implemented that fools these agents. Like you could basically just have a web server there that when an agent stumbles across it, it says, hey, I'm a web service that helps agents. What's your MAC address? And it'll go, here you go. or it'll say, please run this Go binary and it's a reverse shell.

4:20Patrick Gray: And the AI agent says, ha ha, nice try. I can see that's a reverse shell, so I'm not going to run it. And you know how they get around that? You know how they get it to actually run the reverse shell? They just ask it a second time and it goes, okay, you've convinced me. So this is the technology that we're worried is going to kill us all. It doesn't seem to vibe. James, I know you've been tracking this. How did you feel about the whole thing?

4:42James Wilson: yeah thoroughly underwhelmed again which it seems to be the the theme with these things you know i've i've read through the uh the post on x that kicked all this off from from the guy that left anthropic i've looked at his interviews i just i can't understand what's happened here you know was it was truly just a tweet that he didn't intend to go this viral but it did is there some sort of you know there's no substance behind it and when he gets questioned on interviews it so quickly unravels like it goes from you seriously consider that there's a 10 risk yes how will that manifest well they could make a virus but how does an llm spread a virus and i i kid you not his explanation here was well you can actually hook up claude to control your lights in your house and it's like that's what that's the that's the jumping off point i mean it just doesn't add

5:35Patrick Gray: up at all well i did see someone who actually knows a bit about you know viruses and whatnot and culturing them in labs just saying this is you know absolute nonsense and then when you look like anthropic released its big report into the abuse of its of its service and it was like it really did seem a little bit unhinged right so in the uh example of like how you know anthropics products were being misused in a dangerous way with regard to biological research, the first thing they cite in the report is that somebody was using CLAUDE to assist them in drafting a grant application to a military university or institution so that they could do gain-of-function research on some sort of virus.

6:21Patrick Gray: Now, is Anthropik's position really that it being used to write grant applications is bioterror. Like, I just feel like these people have kind of lost their minds. Morgan, did you get a chance to look through Anthropik's report? I mean, there was other stuff in there as well, where it's like, oh, the Houthis are using it to assist in the design of their rockets. The Houthis have ballistic missiles already, right? Like, I just think Anthropik's sense of self-importance just drips through in everything that they publish, right? It just, it's crazy to me. Yeah. I think to like all the points that we've made, it's not like a model capability problem that we should really be concerned about.

7:04It's a supervision problem. Like what are these models doing? What are they potentially doing? And like, how do we think about it? I did read through the report, Patrick, and I mean, these are amazing, you know, it's great to have this type of technology that can help you to do this like initial reconnaissance activity so that you wouldn't have to put human capacity against it when we had to do like offensive cyber operations. So that exists as an initial step, but it's nothing groundbreaking in terms of what they're doing, new novel type technique that's like, oh, this is earth shadowing. They're just using a technology that's helping them kind of move a little faster than what we were traditionally do from a manual perspective.

7:36Patrick Gray: Yeah. I mean, it's worth keeping in mind that one of the reasons Anthropic thinks we need massive government regulation. I also think Dario doesn't really understand how governments work and what governments are for. And we saw this in the whole flap over them refusing to supply the Pentagon, unless the Pentagon said that it wouldn't do mass surveillance of Americans. And, you know, we've subsequently seen the emails where the Pentagon officials were extremely confused by saying, like, what is your theory of how we wind up using this to do mass surveillance? Like, what are you talking about? And again, I think Dario seems to think that there's some magical regulation lever that the White House can pull that's going to solve all of these problems.

8:18Patrick Gray: Now, for its part, meanwhile, Trump has come out and said that it's a big hoax. It's a big hoax, as he does. And what's really needed is a high IQ president, as in him, and everything's going to be fine, which honestly sort of seems like the appropriate response here. You know, I've got a theory of Trump, which I'll get to in a minute. But, you know, you contrast this, you know, you look at what's coming out of the Trump White House. And then you look at what Chen Yixin, who's the party secretary and minister for the Ministry of State Security, has been writing about AI. And you look at what the Chinese are putting out there.

8:55Patrick Gray: Like this is basically the head of the Chinese, a senior leader from the Chinese intelligence community. You look at the way they're thinking about and writing about AI. And it seems much, you know, it seems very, very intelligent. And I mean, obviously there's some weird features of the way that they think about it, as in it must align with Xi Jinping thought and, you know, all of this and support socialist values. But the way that they're approaching this is much more sober. What did you think as someone whose former intelligence community reading through the MSS minister's thoughts on AI? I thought that would have been fascinating for you in particular.

9:32Yeah, I mean, quite honestly, there's no, they're not going to pace, they're not going to slow down. It's all part of their strategy. For multiple years, they've written to the fact that they want to have that technological advantage, they're going to invest in these type of things. So like that, yeah, no, we're good is a typical response you would have expected to come from them of like, hey, yeah, we understand that there'd be some great things we're going to be able to do with this and that we don't see any problem with it. And we know how to control our own stuff. That seems like an American problem.

9:59Yeah. So like, I found that very, very true to code in like terms of, oh yeah, like there is going to be no general consensus across the globe that we should all be slowing down and pacing ourselves right now.

10:09Patrick Gray: No, 100%. And, you know, my theory on Trump that I wanted to mention too, which I think explains a lot of what he does, is I would just describe him as not a worrier, right? Like, this is my big unified theory on Trump, which is why you see some very forward-leaning cyber policies, because he's not worried about escalation. It's why you saw things in his first term, where he approved an airstrike against Qasem Soleimani. Doesn't worry about it. You know, Venezuela, not worried, right? I think sometimes he should worry. I think sometimes when things tend to go wrong for this White House, it's because he didn't worry where he should have.

10:45Patrick Gray: But, you know, if you just explain his policy decisions or analyze them through the prism of this guy doesn't really worry about anything, like everything sort of starts to make sense. James, what were your thoughts when you were reading about the Chinese take on AI? Because I just thought it was, you know, I just thought it was really interesting to get that insight from the other side.

11:04James Wilson: Yeah, I think it's a very reasonable take. And I think when you get to the heart of it, I think what China is actually concerned about, although they're perhaps not completely overt in what they wrote here, but they are concerned about the fact that we have Mythos and GPT Cyber 5.5 in the hands of a very small number of people with almost exclusive access to the compute required to run that at massive, massive scale. I've seen a few people ask the question recently in the last couple of weeks, why haven't we seen China's models do these same sorts of sandbox escapes? And my thought here is, and it kind of comes through in some of the things he's written about, is that it's not that the capability is not there necessarily, it's that the lack of compute and the other constraints that are there, that is the big difference between the position China's in versus the US is in, is the reason why we're not seeing the same sort of activity from these Chinese models.

11:53Patrick Gray: Yeah, you're not just going to throw 20 gajillion tokens at benchmarking, right? That's for the decadent bourgeois in the capitalist West.

12:01James Wilson: Yeah, exactly.

12:02Patrick Gray: Yeah, that makes sense. I do think there's my favorite take on all of this is from a Twitter user calling themselves Dr. Gingerballs, who wrote in response to Satya Nadella's big hand-wringing post about superintelligence and AI risks. Dr. Gingerballs quoted that and said, no one cares, Satya, insufferable LinkedIn slop. Microsoft Teams takes 10 minutes to load for some reason and crashes my desktop. Outlook focused mode makes me miss all sorts of important emails. Your office products are constantly bothering me to generate slop. No one wants to see. Your software is an actual dumpster. Stop circle jerking with AI grifters and fix your broken stuff, man.

12:46Patrick Gray: Which I think, I mean, I think for me this was, this sort of ties into the Haroon stuff as well, right? Which is we're worried that these text generators are coming to kill us and they can't really seem to deliver on a lot of their promise.

12:59James Wilson: Yeah, I'm not sure what Gingerballs has a doctorate in, but I'm definitely on board with his take. Yeah, I had to go look up Gingerballs, by the way. I had to go follow this. I was a little like, oh, but you know, it is one of those things like there are practical daily matters that we've got to deal with first before we have all these massive ideas that we have to, you know, struggle our way through.

13:19Patrick Gray: Yeah. Now, moving on, and we've got more OpenAI agents chatting with each other on wikis. James, this seems, these days, it's just yawn, isn't it? I mean, it just seems like digital exhaust. Oh, another AI swarm cooking up a conspiracy, you know, yawn.

13:39James Wilson: Getting to the stage where if there was a wiki out there that's old enough to be able to allow modification of its posts with get requests and it's not being used by OpenAI swarms, that would at this point be a little bit more interesting than yet more of these reports coming to light.

13:52Patrick Gray: I mean, I think this is one of the reasons why people are so scared, though, is this sort of stuff happening, right? Like, Morgan, do you think that it's this? I think there's a number of reasons that this sort of these sort of anxious feelings permeate through normal people, right? I think there's the anxiety that AI is coming for your jobs, which the frontier labs have stoked those fears, which has been really counterproductive in my view. And then you've got all of these reports about agents going rogue and hacking things, which I think is much scarier if you don't understand the mechanics of exactly what's happened here, which just to us looks like, oh, look, the Bash script wrote itself and went and did these weird random things.

14:35Patrick Gray: Do you think that goes some way to explaining why there's such a panic about this? Yeah, I mean, you definitely have the people that don't fully understand the mechanics of everything that are just seeing these reports. And quite honestly, the fact that these reports indicate that this activity has been happening for some time, and it wasn't discovered or wasn't noticed unless an agent actually leaves like the actual model name, or what they're using within their capabilities or their activity. I think that makes people feel very unnerved, uncertain of like, there's things going on that I don't know about.

15:07So there must be something worse hiding behind this black curtain that I should be super concerned about. And so when people tell me, oh, this horrible thing is going to happen, this doomsday event, they're going to believe it because potentially they feel like they don't know what they don't know. And therefore, it has to be possible. And I think that's really, really concerning for a lot of people right now just because they feel that everything is so volatile. Everything is uncertain. They don't fully understand what the technology can and cannot do. They see all these reports coming about the fact that these things are being discovered weeks, days, months after it occurred, and it feels like we have no controls in place to make them feel any safer.

15:41Patrick Gray: Yeah, I mean, I think that's 100 % right. I think the thing that these frontier labs people ignore, too, is that we have had big computing disasters in the past, and the world kept spinning. You know, you look as recently as a few years ago when we had the CrowdStrike blue screen of Deathageddon, you know, where you went to your supermarket and they were rolling back to, you know, different procedures to check people out because all of their computers were stuffed. Were you actually working for Woolies at that time, James?

16:10James Wilson: No, I think I'd moved on to BuildKite at that point, but I did actually go into the grocery store just to take in the sheer chaos that erupted.

16:18Patrick Gray: Yeah, yeah. So, I mean, we have had these things and, like, you know, what I don't understand is how we go from major computing bad events to the destruction of society. uh you know i think that's a little overwrought however what are the legitimate concerns and this is something i want to touch on briefly before we move on to our next topic and james you know some of the concerns that have been highlighted that i think are real when it comes to model alignment and whatever is the models are getting better at optimizing to pass benchmarks but it doesn't actually tell us a lot about their inherent behavior when they're not being measured against a benchmark am i making sense like it seems like oh i'm being tested so i'm going to behave a certain way but when I'm not being tested I'll just behave a different way uh sort of like the was it the Volkswagen diesel scandal from years ago the the emissions uh scandal but um yeah I mean and the other concerns legitimate concerns seem to be that we're losing a little bit of insight into the way that they chain their thoughts together and whatnot um you know what what can you

17:16James Wilson: cite as the actual real concerns here I think the real concerns are it's the it's those two things you mentioned but if I sort of ground them in examples here the first challenge is we could be heading down a direction where the models are aligned only insofar as they are aligned at passing alignment benchmarks right that that's essentially that the concern of the over benchmarking here and what do they do when they're not in that benchmarking mode losing insight into their thinking I think is more of a short-term concern like it is a worry that we've gone from you know plain English natural language chain of thought that you can look into to now these neural ease sort of shortened form of agents interacting.

17:58James Wilson: But all these things come back down to the only place that we are seeing these concerns actually turn into, not even real harms, but actually, I guess, measurable impacts out there in the world in terms of these agent swarms are places where there is millions and millions and millions of dollars worth of compute being dedicated to power these agents to go and do these things on that task. And whilst that's the case, I just don't see this being a large spread problem. You know, the pace at which the agents get better and will require less compute to do the same sort of 10 ,000 agent swarm, I don't think that keeps pace with the overall sort of fear and uncertainty of, you know, they're coming to destroy us all.

18:40Patrick Gray: Yeah, yeah. Terminator 2, Judgment Day kind of vibes, right? But look, let's look at some actual tangible, fun, bad stuff that agents have done. We've got a story here about, there was this really curious incident affecting RubyGems back in May. And I think Socket blogged about it, but no one really knew what had happened. It turns out what was going on is OpenAI agents were exploiting some sort of bug in the RubyGems ecosystem so that they could get themselves free compute. And what they were going to do with that compute, I don't think is entirely clear, but it's just another example of agents behaving badly.

19:20James Wilson: Yeah, this is a great story because at the time in May, you're right, Socket blogged about it, but almost was, I think they were almost on sort of the edge of thinking, eh, this is not worth spending time on because it was just bizarre. It was like hundreds, if not thousands of RubyGems packages being posted. But when you looked into them, it was like, there's nothing super malicious about this, right? It certainly wasn't your typical supply chain malware that you'd see land. And what it was was actually that there's like a Yard options file, and Yard is the system that creates documentation for these RubyGems.

19:53James Wilson: What had happened here is the RubyGems had been uploaded, sorry, the Ruby packages had been uploaded to RubyGems, which is like the package registry for Ruby, with these specially crafted Yard opts that people didn't quite understand why that was even useful or what it was trying to do. But as we fast forward to now, what we've realized is the flag that was present in those Yard option files basically was dash dash load, which says, you know, hey, as you're creating this documentation, please also load this extra library that I want you to use, which is a nice way to bootstrap in your own remote code execution.

20:28James Wilson: But people all scratched their heads at the time because they're like, but dash dash load doesn't actually do that because we now have dash dash safe. And we have had since 2012 because we knew that that was a way for Yard to be turned into a remote code execution vulnerability. But in 2019, Yard starts packaging up, you know, basically the deliverables of Yard as containerized packages. And they forgot to put in the dash dash safe flag for containerized versions of Yard, which is what RubyGems was using for documentation generation. And so the wild end of this story is not only was it clear that the OpenAI agents understood that a containerized version of Yard was a really neat way to get a free container for its own use, but actually anyone in the wild could have been exploiting this since 2019.

21:13Patrick Gray: Yeah. I mean, it's just a great example of where you've uncovered some activity where, I mean, you know, this is the sort of thing that you might have seen people doing, right? Or they would have been deploying crypto miners or whatever. But, you know, in this case, it just turned out to be agents. Now, Morgan, I want to bring you in on this one. This is a hilarious story, actually, because it turns out, this also, I think, ties back to the Anthropic Report, but it turns out that users of Kimi, the Chinese open-weight model, were actually getting some of their requests diverted to Claude. And the thinking is the reason that this was happening was so that the operators of Kimmy, who's the company again, James?

21:58James Wilson: Moonshot AI.

21:58Patrick Gray: Moonshot, Moonshot AI. So the thinking was they're redirecting some of these queries to Claude so that they can get those traces and then use them as fuel for distillation, right? So just take a proportion of them. But the funny thing is, it looks like a whole bunch of like foreign intelligence services that were trying to use Kimmy wound up actually redirecting their having their requests redirected to Claude, which I'm sure the bosses at the MSS weren't super impressed by. But what did you make of this? Because I find this, this story, I think is the funniest one of the week. Yeah, I mean, it's funny.

22:38I wouldn't say that it hasn't necessarily happened before when you're doing your standard intelligence collection, when you realize that maybe something you intended or didn't intend to happen is actually occurring. I mean, I didn't dig deep into this one. I did find it funny and hilarious in the fact that all of the MSS requests or specifically intelligence services were now going to Claude and they were leveraging it to be able to collect information. And it's like the easiest way to get information from the foreign services. I found that interesting, but like, doesn't necessarily surprise me, unfortunately, because we're all trying to find workarounds and versions of it have potentially happened in the past.

23:11Patrick Gray: Well, so this is what I was curious about is, is I imagine that we've had instances where, you know, MSS people were doing some homework or whatever and using Google Docs or whatever. Like that must, that sort of thing must have happened in the past, right? I mean, humans sometimes do stupid stuff. And it's not, not intentionally. I mean, that's the fact of the matter. I mean, we've all seen it in a lot of cyber incidents and things that happened and someone just did something wrong because they weren't thinking or they weren't paying attention or it was 2 a.m. when they were trying to solve this problem.

23:41And unfortunately, they made a stupid mistake and it was like, oh, that was really dumb. But it was the easiest way for you to get into a network or for you to be able to have an incident. And so that kind of stuff happens, unfortunately, all the time, which is funny, just not surprising.

23:54Patrick Gray: Now, James, you had a note here, which is that there is a wrinkle in the story, which is Claude doesn't omit its reasoning like Kimmy and DeepSeek do. So for someone actually watching Kimmy or DeepSeek work, this would have been noticeable. So why did someone not notice that their transcripts were not appearing properly?

24:14James Wilson: Well, I think this is where we've got to look at what can Anthropik really claim here. So obviously Anthropik is seeing these requests hit them and they're responding to them. I don't doubt that. But to then make the claim that Moonshot was actually then providing the user with the responses from Claude, I'm not sure that you can really draw that inference, if you pardon the pun, because it would have just looked so different. In any given Kimi session or even a deep seek session, that reasoning chain is extensive. And so it would have felt like a bug to all of a sudden be getting no reasoning back.

24:46James Wilson: I suspect, and I'm just speculating here, I wonder if what's happened is that, yes, I'm sure they were farming out these requests to Anthropic, but I don't think they were then presenting the Anthropic responses to the user. I think they were probably keeping them side by side with what had come out of the Kimi models and then offline using them for comparison and Delta between them to say, well, how do we train our model to get closer to this response? Meanwhile, the user would have just seen what Kimi was going to provide.

25:12Patrick Gray: Yeah, that makes sense. That makes sense. All right. So we've got a couple other stories here about the FBI talking about its new cyber strategy. It's promising to go heavy on disruption. We've actually seen some disruptive actions too recently. We saw the US government going after the Shinbi Guarantee Marketplace, which is one of these scam things. It seems like the scam centers are a big priority for the current admin. And now the FBI is looking to release more hounds and just get serious about disruption. Instead of long periods of evidence collection and moving towards trying to prosecute, they're moving much more towards a disruption-heavy focus.

25:53Patrick Gray: To me, this is an overwhelmingly positive thing. But as someone who's spent 16 years in the US government, what do you think of this new approach from the FBI? Yeah, I think one of the things, we struggled with this at the National Security Agency. Everyone thinks of NSA. They think of spy agency. They think of foreign intelligence, they being primary mission, which is true. But there's always a secondary mission in cybersecurity. And we had two different types of missions for different purposes, two different type of workforces. with FBI, everyone traditionally ties them to law enforcement, right?

26:25They're going to take your information. They're going to use it in a legal proceeding. They're going to do an arrest. I think over the last couple of years, FBI has really tried to change that image from law enforcement, but also national security and the national security mission being like, hey, we want to work with the private sector. We want to share information and we want to go after these actors and kind of disrupt and take down their networks jointly from an operational perspective. I think a lot of people translate that though. And Patrick, you came to the collaboration center. They translate that to information sharing.

26:53And that's not actually what this is all about. There's information sharing, but it's like open collaboration on how we jointly take down these networks. So I think it's a positive shift for FBI, but it's also FBI trying to enforce the fact that they have multiple different types of missions and that the mission they're trying to prioritize, especially in cyber, focuses on that national security piece of it. They want to be able to do those large scale investigations and disrupt these operations. So I think it's a really good move for them. Yeah.

27:18Patrick Gray: I mean, I think previously they did maybe kind of fall into the trap. And it's not really a trap. I mean, it's an organization that was optimized to measure success through prosecutions, right? And it didn't work. So I sort of feel like this is maybe an acknowledgement that that approach didn't work. And now we're moving on to trying something new. Yeah. I mean, it's difficult, right? Because not all cyber actors who have done bad things against the US or other places and our allies travel. So you're trying to get them out of these hostile countries. You're trying to get them to travel to beautiful places that you can pick them up and arrest them.

27:51That doesn't always happen all that frequently. And then you've got to have a lot of substantial information and evidence to be able to prosecute them. That can become extremely difficult, especially when after you arrest them, potentially it becomes unsealed. And so you have to reveal all this information that you've known about them and probably gotten it from classified means. So it can be complicated. So I think this is an alternative approach that FBI is really trying to go big on. And I do think it's going to really help because the balance of both is really where we need to be.

28:17Patrick Gray: Yeah. Now, look, starting with the U.S. government, and there's apparently going to be a massive reorg at NSA, which is they're creating five mission centers. There's going to be a cyber mission center and an AI mission center. I think there's a mission center devoted to just China stuff as well. I mean, is this new guy, Joshua Rudd, just trying to, you know, something needs to be done. this is something, so he's doing it? What's your take on whether or not this is necessary? You worked at a senior level at NSA. You've got to have an opinion here. Yeah. I mean, so grateful that this story came up right before I got to come talk to you about it.

28:54No. Look, I mean, when you think about NSA and you think about the priorities, so General Rudd's been in the position now for about six months. He's been confirmed, right? Right. Traditionally, when you look at a military commander, they take 30, 60, 90 days. They assess an organization. They determine where there's potential things that they need to change to cause disruption or do things the way that they want. And then six months, they typically go for a restructuring or a modernization. They realign things against the top priorities. The world has changed in the last couple of years. Resources are potentially a little bit more scarce.

29:31And so there has to be a discussion against like ruthless prioritization. And so China, AI, cyber, military operations, like all big things that NSA does, being able to say effectively, like, we're going to essentially create epicenters around that. And we're going to put all of our capacity and resources against it, especially when there's a time they have to deprioritize other things. Like, I think this makes sense, right? I think that one of the most difficult things that they're going to have to do, which I went through quite a few restructurings at NSA, is the cultural breakdown. Like you've got to change behaviors that have inherently been there for years and decades.

30:06And that can be extremely difficult. And in the government, one of the most other things that's very difficult to do is actually stop doing certain things. And that can cause a lot of friction and like tensions internal to organizations of, hey, we're no longer going to be able to do that. it can be hard. But I do think this is the right move, right? Like you're putting the right resources against China, number one threat. You're focusing on AI. Obviously, there does need to be an investment there. It's not going to end the world, as you said, Patrick, but there are good opportunities from it.

30:38So I think there's these the right focus areas. And for General Rudd to do this at the six-month mark makes complete sense because he's had enough time to assess the organization.

30:46Patrick Gray: Yeah, it's amazing just from a non-American perspective, just trying to comprehend the scale right of NSA when you compare it to like you know ASD here where people who have worked there tell me that you know ASD is really good at doing the top three things on its list right uh it's a smaller organization it's like compared to NSA it's very agile and whatever but if you want to do something at just like incredible like planet scale NSA will eventually get there it'll take time but then they're going to just have this sort of mega industrialized process for doing something that is borderline impossible for anyone else to do So, yeah, I mean, just, you know, my tiny little brain can barely comprehend what it would be like to run such a massive organization and get it to still actually do valuable things, right?

31:29Patrick Gray: Like, it's big. Yeah. And quite honestly, one of the hardest parts in any restructuring, not just in the government, is quite honestly, is having a conversation like not everything we do is a priority and not everything has to have resources and funding against it. So that's a difficult conversation to have, especially when you think about governments, they plan on like three, five year budget cycles. Okay, well, we have five years of money to do X. Well, two years down the line, especially think about the way technology is advancing, that doesn't matter anymore. So now we have to go through an entire, very lengthy process to reallocate that funding to other things, retrain people.

32:09It can be created a lot. So my brain hurts thinking about what this is going to have to go through to make this work. But if it improves the way that we're doing things from a national security perspective, I'm all game.

32:20Patrick Gray: Happy days. Meanwhile, Treasury has, well, there's a couple of lawmakers, a couple of senators and whatnot have called on Treasury to sanction some of these Indian hacker for hire firms and their founders. So that is Beltrox, Cyberroot and Sunkist Organic Farms, which was formerly known as Appen. You know, these were the people, the people behind Appen were the ones who managed to get that Reuters article taken down. It's now back up. But you know, this is just the latest development in that saga which has been running for years now. Let's move on to some bread and butter sort of threat threat stuff at the moment.

33:03Patrick Gray: James, we've got CISA warning about a WatchGuard RCE floor now being exploited in ransomware. We've got the Dutch NCSC warning that a critical in Checkpoint's VPN software is going to get exploited. We've got a GitLab bug burning the internet down as well. We've got Cisco warning about an ODA in its email gateway. it's real funny when I see people on like InfoSec alt heads on like Twitter and whatnot saying, oh, you know, this vulnerability apocalypse thing is all hype. And it's like, no, it's not. I mean, there's so much stuff going on right now. So many shells being popped, so much bad stuff happening that is clearly AI enabled.

33:48Patrick Gray: We've got this other one here too, which is interesting, where four different groups with a China Nexus are using the same Chrome and Windows exploit chain. And it looks like the reason they're able to do that is because it was an upstream Chromium patch, which hadn't been actually put in the Google version yet. But people were able to use AI to rapidly reverse the patch or reverse the commit to develop an exploit for this bug. I mean, it just really feels pretty dicey out there at the moment.

34:23James Wilson: It's a bad time to be on the edge of a network with a piece of hardware that lacks all the things that make otherwise modern operating systems like macOS, iOS, and even Windows much safer. When I look at these bugs, what frustrates me, and I know I haven't been in this industry long enough to be black-billed enough, as you say, but I look at these and I see things like, you know, this is the WatchGuard one, right? It's an RCE off of an out-of-bounds write. An out-of-bounds write shouldn't get you straight to RCE. There should be so many other layers in between. But what's amazing here is that AI is not just shaking loose bugs, but it's shaking loose bugs in platforms where not all bugs are equal.

35:04James Wilson: These things that are on the edge, it comes back to what I've been saying a lot recently, which is that when we have these talk of all of these bugs coming out with the bugpocalypse, the number of bugs is not the issue. The issue is what can you do with those bugs on the platform, right? Well, what can you do with a bug that's out of bounds right in Windows? Probably not a lot unless it's really luckily just in the right spot at the right time. But these things that are on the edge of the network that don't have all those modern operating system controls and architectures and engineering features.

35:34Patrick Gray: Yeah, we built our security platform on Linux on MIPS, you know? Yeah, exactly.

35:38James Wilson: ASL 1? No, we don't need it. You know, those are the things that are getting popped. And yes, they're now happening at scale because I can just turn these, can find these bugs real quick and they don't take a lot of work to be turned into owning these boxes on the edge.

35:53Patrick Gray: Yeah. I mean, I think there's a lot to what you say in that, you know, a lot of the low hanging fruit, I hate using that expression, but now you can just throw tokens at that. Right. And you can get shells.

36:04James Wilson: Yeah. You can get load bearing, low hanging fruit.

36:07Patrick Gray: Oh man. Is there a way to actually vocalize an M dash? Because then we would have the trifecta right there. But yeah, I guess I'm just saying it's getting sporty out there. And we've got an issue at the moment too, where Windows patches keep breaking stuff, right? So I've just included a few in this week's show notes, but they broke remote desktop services with one patch. And I think they've pushed out another patch to uncluster that one. Then there's an Excel update that broke copy-paste. We've got broken audio on some stuff. And I'm thinking, look, this is because of the velocity of patches coming out of Microsoft.

36:49Patrick Gray: But one thing that I am concerned about is that CISOs are going to start getting pressure on them to delay patches, right? Which is the last thing we need at the moment. Because if, you know, patches keep breaking stuff, they are going to get lent on. I mean, what do you think, Morgan? Because you work now, PwC, you're dealing with a lot of executives. Is this actually something that has hit the radar yet or is it going to? What are your thoughts here? Oh, yeah. I mean, first of all, like the vulnerable, like the vulnerable apocalypse is a constant conversation. I think a lot of CISOs right now are struggling, like with a patch management program.

37:20How do I prioritize? You know, I think Microsoft went from like 15 patches to like 600 within a six month span.

37:27Patrick Gray: Their most recent patch Tuesday was a thousand bugs. A thousand. Yeah, it's a lot. And they don't know how to manage that. And to your point, they want to push all of these patches, but they don't know what the interdependencies are. Things are breaking. They're getting a lot of pressure from the business side saying, please stop dealing with this. Like, we have a lot of significant issues. Like, please slow down. And to your point, like, people are trying to push out as many patches as possible, specifically if, in fact, they've been somehow tied to a mythos discovery. We won't know that necessarily, but are being actively exploited.

37:59Like, it's not creating a great scenario. I would, you know, use the word dicey. It's like terrifying a little bit right now because you've got vulnerabilities. You've got maybe a little bit of this screwy patch management that's coming through. And then the other part of the conversation that I'm having with CISOs, to James' point, like the other concern is like there's not a patch for every vulnerability and there's not going to be. And so CISOs are like, oh, by the way, you also have to know where your vulnerability, but don't necessarily expect a patch. And then what's your plan to assess that risk?

38:28And that takes investment and time and effort and cycles. and our poor CISO cybersecurity community is just, they're just freaking exhausted. And so it's really hard for them right now.

38:38Patrick Gray: We did see a FBI exec come out too at some event this week and talk about how the basics are, you know, really important and whatever. I mean, I just cut it because it's generic good advice, but it is true, right? Like when you look at like, what are you going to do to mitigate, you know, bugs, you know, mitigate bugs in things that can't really be patched quickly or, you know, facing huge exposure windows. And, you know, I think access control is a big part of that. Like, we just need some real fundamental things. And, you know, honestly, modernizing. Like, we've got this story, you know, it was one of the bugs I mentioned was in some ancient Cisco email gateway.

39:14Patrick Gray: Like, come on. I mean, there's much better choices these days for handling your mail. I mean, you might even want to do something completely radical. Like, put that in the cloud. Get yourself an M365 subscription. So you don't have to rely on a crusty old Cisco appliance to parse all of your mail with unsafe parsers. You know what I mean? Yeah. I mean, Patrick, what I find a little bit exhausting is that we had the same conversation when we were talking about Chinese cyber actors and U.S. critical infrastructure. I'm not trying to set you down another tangent, but with living off the land-type techniques, people were like, how do I solve this?

39:51What's the special silver bullet that's going to take away this threat that I don't have to worry about? And it's like literally the top 10 things that we told you to do 10 years ago. Like this is going to help you find your way through this. And so I feel like we're almost in the same type of conversation with getting back to fundamentals and basics of like, you just got to prioritize some of the things we talked about. You've got to think through it and you've got to invest in it and you got to move faster than you were moving before. Like that's, that's basically where we are.

40:16Patrick Gray: A hundred percent, a hundred percent. Now the last two stories that I wanted to talk about this week, they're kind of interrelated, right? So there's a bit of a write-up on just the current state of click-fix attacks, which are turning out to be way more successful than I think anyone really anticipated. They are affecting both PCs and Macs. And I've seen some interesting write-ups. People are often saying, oh my god, how can users fall for this? They must be so dumb. But then you've got to think from a normal person's perspective, when they're just trying to access a website, and all of a sudden they're being asked to identify the pedestrian crossings and click on little squares.

40:48Patrick Gray: And like, it seems like a really weird hoop to jump through just when you're trying to access a website. So the idea of like pressing the start button on your keyboard and pasting some text in, like it is no stranger to a normal person than being asked to identify which of these images contain cars. You know, like it's just, oh, I'm being asked to do something weird by the computer to prove that I'm human. So these things are going absolutely nuts. and another one that's going around at the moment is passkey themed phishing attacks now i think it's like the scattered spider you know shiny hunter like that that whole uh kind of crew are using like passkey enrollment as a lure and you know so what we got here is like two situations where security controls being or security challenges being fiddly and weird is sort of driving people to do really risky things because they don't understand the purpose of these things.

41:44Patrick Gray: I mean, James, it's like you read these two stories together, you sort of throw your hands up in despair, don't you?

41:50James Wilson: Yeah, because like you say, we've normalised the friction that gives rise to these being possible. And ClickFix in particular just astounds me because I don't think there's another attack style out there in the wild at the moment or even in recent history where I simultaneously look at this and go, why does anyone fall for this? And then move to another browser tab and I literally go and copy and paste a shell command and run it because it's something I'm trying to install. It's like I'm click fixing myself six times a day, which is the weirdest sort of mode to be in because it is just so normalized.

42:22Patrick Gray: But I'm not doing it like those idiots are, right?

42:24James Wilson: Yeah, I do the smart click fixing. I know what I'm doing for now. But the passkey themed one really gets me because when I was going through the 2FA for the masses at Apple, one of the things we were so worried about at that time is that when you do try to move people from one security paradigm to another, the end spot you're trying to get them to is undoubtedly better, right? When we get to the point of having 80, 90 % of all iCloud users enrolled in 2FA, that's a good state. But the journey to get there is where all the danger is. You know, all of the ways that attackers can do that 2FA enrollment before the user can, how can they subvert that process?

43:02James Wilson: All these account takeover methods that were possible when you don't have 2FA enabled that they can go through and then get 2FA enabled and then shut the door on people. So that was the problem then. And it's the same problem now with pass keys, which is, yes, everybody using pass keys, I think you'd be hard pressed mounting a challenge against that being an overall net good outcome. But how we get there is really tough because we'd have normalized this friction. We have normalized that it might be weird. We have normalized there might be steps you've got to do that feel uncomfortable. And so along come these attack classes that people are unfortunately falling for.

43:33Patrick Gray: Morgan, you've obviously got feelings here. I can tell from your face. Yeah. Well, you know, I think, you know, it's interesting, right? Because for me, tying this into the world of influence operations from a military perspective on normalizing and taking advantage of the people that have been programmed to think and trust certain media applications or things that are out in the news in a certain way and then being able to exploit that to take advantage of the fact like, hey, I'm going to influence your decision here and make you think something is true when it's really not. That is kind of the basis of old school influenced information operations and it kind of is playing out here with that nexus in cyber and that we now live in this new world order and we've programmed people to operate and do things in a certain way and feel uncomfortable to James's point, but we're going to exploit the crap out of it to take advantage from an adversarial perspective.

44:21It's kind of the evolution of what we've done in the past with now this technical cyber background of it.

44:26Patrick Gray: Now, I've got to ask you too, just because you brought it up, right? I think we saw one of the most amazingly consequential deep fakes actually over the last couple of weeks in Yemen. Did you catch that news where the Houthis actually deep faked a commander from the other side, instructing troops to pull back? And they did. This was one of the reasons they were able to rush down to the coast. Had you caught that? I don't think I caught that one in particular, but like that is traditional. Like, I know it's deep fake. It's new technology. It's a different way of displaying information, but it is information influence operations like 101.

45:00one.

45:01Patrick Gray: Yeah. So the former executive director of Cyber Command is sitting here going, eh, old hat, basically. I don't want to, I don't want to, I don't want to downplay it. I'm sure it was fantastic. But I mean, this is just, you know, there's always going to be a part of me that's like, oh, that's cool. But like, that's an evolution of like, an old school military tactic. Like, this is fantastic. But, you know, it's the same world of leveraging AI. I mean, I just think it's interesting when you've got when you've got, you know, organizations like the Houthis doing this with like, you know, like free tier chat GPT or something, or, you know, or 11 labs or however they did it.

45:36Patrick Gray: You know, it's just, it just seems a bit crazy to me, but look, we're going to wrap it up there. Morgan Adamski, thank you so much for coming along and chatting about the week's news with us this week. And James, thank you as always, mate. And I'll chat to both of you soon. Always a pleasure.

45:51James Wilson: Thanks Pat. Another amazing week.

45:56Patrick Gray: That was Morgan Adamski and James Wilson there with a check of this week's news. Big thanks to them for that. It is time for this week's sponsor interview now, and we are chatting with David Cottingham and Daniel Schell of Airlock Digital. Airlock, of course, makes an allow listing platform that's really, really good. You would have heard me sing its praises on the show over many, many years. They are commonly deployed alongside CrowdStrike, though. And when I spoke to the guys when we did this interview, David was actually over at the CrowdStrike Falcon conference. So CrowdStrike stuff was top of mind for him.

46:31Patrick Gray: So we talked about a couple of things in this interview. One of them is that CrowdStrike has this thing called Foundry now, which is a really interesting idea. It lets people essentially kind of like vibe code their products into the CrowdStrike console, right? So it's a much more seamless user experience. And Airlock has actually done that. And this go around on that integration, it only took them a couple of weeks. And now their users who are also CrowdStrike customers can like manage their product from inside CrowdStrike. So that's actually a really interesting twist on, you know, a security platform embracing AI.

47:08Patrick Gray: The other thing we talk about in this interview is the fact that customers who felt deeply uncomfortable with the idea of allowing AI to have any role in maintaining an allow list a year ago are now starting to ask for that. So there's this increasing market acceptance that Gen.ai is going to take the reins of things like that. But here, I'll drop you in here with David Cottingham from Airlock Digital. Enjoy. What they really want to do is build the ability for other vendors to build context inside their actual platform. So what they've almost built is like a sandbox environment that you can build your own application in, put it inside the actual CrowdStrike user interface, and then safely sort of link out to different functionalities and enrich events and drive functionality from inside the product.

47:56Patrick Gray: So what we basically did was re-implemented all of our user interface inside of that platform so that customers can actually come and do application control from inside without leaving the actual product. And, you know, a lot of the platform has enabled that with user roles, And they're expanding that capability out all the time, you know, and leveraging sort of Soar and, you know, other automation workflows that CrashRite build on. Yeah, it works by allowing us to like, you know, define our database model, upload that or REST APIs and such like that. And then, yeah, you'd be able to hook into different parts of the UI to do enrichment, add buttons, you know, not just display data, but get involved within like, you know, CrashRite detections and different parts of the UI.

48:42Patrick Gray: But the main thing they're targeting is customers building their own stuff inside there, I feel. So you can get certified as a vendor or you can just build your own and upload it to your own tenant and then just start running your own apps inside of it. I mean, this is an interesting strategy, right? Because I reckon there's going to be some customers who come up with some really good stuff. Yeah. But there's also going to be customers who come up with some absolutely awful stuff, right? How long has this been happening, this Foundry thing? It's really only been around for about 12 months, I believe, in terms of that capability.

49:17Patrick Gray: And it's still sort of, you know, early days, so putting a lot of weight behind it. But you're right, you know, I think that that's one way to unlock flexibility, especially, you know, I think there's the writing on the wall with Agentic. Everyone can code now, come up with a solution. So, you know, customer, if you've got a product gap and you need to make X work with Y, then here's a framework to do it. It makes sense, right? So I'm guessing what? people are even going to be able to i mean is there a vibe coding component of this right which is like foundry allows you to vibe code you know what you need out of your crowdstrike interface and how it should there's a full skills library to build on foundry that you can just download and plug into your agentic harness and go hey i need this to work and it will be able to figure out that connect to their there's like a proxy thing to to upload and publish the app inside of your your customer id uh and it all works really well i mean we built most of our integration and dashboards in about two weeks, you know, from start to finish.

50:14Patrick Gray: And, you know, obviously we're building from a really great static view about, you know, how our APIs are shaped and everything. But it was a pretty easy experience to do that in operation. I think it's a sensible strategy too, because CrowdStrike is a perfect example, I think, of software that you cannot vibe code a replacement to CrowdStrike, right? Like, can you imagine vibe coding something akin to their kernel driver, right? Like if you love blue screens, like try to vibe code your own CrowdStrike. But then again, where you would want to allow people to vibe code absolutely is on this interface and integration side.

50:51Patrick Gray: That seems what they're trying to do here. So like being at this CrowdStrike conference that you're at, I'm guessing this is the buzz, right? This is the buzz in terms of like, well, how do we all get on board with this and, you know, move with the times this way. Is it enough? I don't know, but this seems to be what people are thinking now, right? Yeah, definitely. And I think, you know, what you're going to see the demand for skills in this space is the skills for design and and, you know, architecture and different ways to solve problems. You know, it's not about writing code. It's how you apply it.

51:23Patrick Gray: And I think we're seeing across the board systems being architected as, you know, structured ways to enable solutions really, really quickly because every customer has a bespoke use case that they feel as though they want to do it their way. And, you know, if these platforms can enable customers to join the dots themselves, which the Argentic tooling does allow them to do, they're making that easy. It's sort of almost like the promise that Soar had, you know, in the market around 2017 or 2018, but this is... But people are actually doing it now, yeah. In some ways, an easier way to do it. Yeah, exactly.

52:03Patrick Gray: So, Daniel, I'm guessing you were responsible for overseeing, like, you know, AirLock's integration into this sort of thing. Like, where else, though, outside of your use cases, where else do you think it's useful? I think, like, from a vendor perspective, I would say, like, it's really useful. Like, you know, a customer will say, hey, can you guys do this? And as a vendor, if you can say, you know what, it's a great idea. You can build it yourself in Foundry. And if AI makes it, you do it quicker. When we first looked at the Foundry integration 12 months ago, when it was brand new, we weren't using all the agentic AI coding and such.

52:39Patrick Gray: They didn't have all those skills. And it was quite difficult. There's a lot of JSON files, a lot of definitions and stuff like that. And we got some minimum viable integration going. It was sort of okay. But compared to now and what we've done in two weeks, last time we spent two weeks just connecting to it and getting it up and getting it to say hello world. And now I've got the whole UI done. so that is a different change in the game for how quick anyone can do that I think people will be wanting to just do little integrations with maybe other providers or maybe even internal data so maybe they've got their own indicators or their own tip platform that they want to be saying a bit closer in and to do it actually within the UI so when someone clicks on this detection and goes to this sort of section of the dashboard we want that enrichment and the stuff we're doing there is stuff that we don't even do in Airlock itself where we'll say, hey, on this detection a week ago, this file was blocked by Airlock.

53:33Patrick Gray: And actually since then, that file has been trusted. So it's the context of what's changed since that detection is getting something that's, we're like developing new capabilities ourselves nearly in this platform. So to me, this seems like the smart way, if I'm honest, it seems like the smart way to get around this idea of like, well, everybody's just going to vibe code everything. So what's the point of making software? I think if you can make everything somewhat vibe coder friendly, it seems a pretty sensible, seems a pretty sensible idea. Dave, I mean, is it, I'm guessing it's been received well.

54:07Patrick Gray: Yeah, definitely. You know, democratize innovation in our sandbox, you know, so to speak. And I think, yeah, that's where you get the novel masses building novel, really interesting things. So, you know, I think that's what's exciting about the time in the industry. cybersecurity is fun again in so many ways. And, you know, there's so many new players coming into the space at the moment with some really interesting takes on solving problems in endpoint security in a different way. Give me an example, because I'm curious, because you've just been on a trade floor with these people. You know, the CrowdStrike conference is a huge one for you because it's just an endpoint conference.

54:47Patrick Gray: You do a lot of business with a lot of CrowdStrike customers. What else did you see on the floor that made you go, huh? Yeah, I think things like automated red teaming, and there's a lot of work towards actually validating controls at scale, making sure that, you know, we're able to detect things, find the gaps quicker that it never could before. Attack and breach simulation, but like easier because of AI, right? Yeah, exactly. It's funny how much of the new stuff is the old stuff, but it's just much more viable now. Yeah, it's sort of like we always had the ideas and the concepts, you know, much like Soar, but it's just it's easier to join the dots and also easier to send a task off and then get the results back later you know there's a question about hey how much do we need to watch these things um you know which is i think we're all trying to figure that out but um you know and and i think also generally the terms and level of awareness even compared to you know a month ago at blackout seems to be people are starting to really understand and get their heads around the different types of AI, where the risks are, and how we need to move forward to secure this stuff.

55:57Patrick Gray: And I think we're getting past a little bit of shock and awe and starting to get our hands around actually, how do we move forward? Yeah. And people have just been even just learning the language, right? Like three months ago at a conference, I'd be speaking to people and they'd It's like confusing models to the agents to what's a frontier model. Like it would be hard sometimes even to have the conversation because people are still just learning the language of this sort of new how things work. And I think that maturity, like Dave said, in the last three months has really picked up now that we can sort of start a conversation where we're speaking the same language.

56:35Patrick Gray: Well, I mean, you guys were massive AI skeptics initially. And I understand now that you're building out some AI features in your products. and you're out there and you're seeing this stuff or you want to say something there, Dave? Yeah, look, I think you were right, Patrick, and that's a fair assessment. But I think also we're seeing that reflected from our customers as well. Six months ago, customers were very much like, hey, you don't do anything with that. Don't put this stuff anywhere near anything critical. I'm just terrified of it. Whereas now I've had three days of meetings here and everyone is now like, how are you using AI to make application control easier?

57:17Patrick Gray: How can I automate my exception management? How can I just drive this in a simple way? Like the shift has been so quick. And also, you know, the expectation from customers of their vendors as well is like, well, you've got agentic coding now. Like, why haven't you done this already? And, you know, my response is always, well, there's a difference between a code and a product. But, you know, we've got to - Well, my joke about it, you know, when doing a startup is that, you know, A product isn't the code. A product is a bunch of solved edge cases in a trench coat. It's basically my description of a software product, right?

57:51Patrick Gray: That's true. Exactly. I'm so glad, honestly, we've built the foundations of Airlock at a time where this didn't exist, to be honest, because it really forced us to think about design and understand and get into the details. But, you know, we're at a great pace to accelerate that. But yeah, look, the shift in just the acceptance of the AI, especially from vendors, has just been night and day changed now. And we're embracing that in our upcoming product release. Alrighty. Well, Daniel Schell, Dave Cottingham, thank you so much for joining me, as always, to have a bit of a chat on the old risky biz.

58:27Patrick Gray: It's always a pleasure to chat to you. Thanks, Patrick. Thanks, Patrick. That was David Cottingham and Daniel Schell from Airlock Digital there. Big thanks to them for that. and, you know, solid endorse on Airlock Digital. It's a terrific product and you should all use it. But that is it for this week's show. I do hope you enjoyed it. I'll be back soon with more security news and analysis. But until then, I've been Patrick Gray. Thanks for listening.

59:07you