Risky Business #848 -- OpenAI comes clean

0:05Patrick Gray: Hey everyone and welcome to Risky Business. My name's Patrick Gray. We've got a great show for you this week as always. In a moment I'll be joined by my colleague James Wilson and our third seat this week, Mr. Brad Arkin, who has previously served as the chief security guy at Adobe, at Cisco, at Salesforce. But as I like to joke, don't hold that against him. He's a real smart fella. So we'll be getting into all of that in just a moment. This week's show is brought to you by Prowler. And of course, Prowler does cloud security checks, cloud security remediation, and it's open source. It's based on an open source project, but there is, of course, a paid version.

0:46Patrick Gray: And Prowler's founder, Tony De La Fuente, will join us later in this week's sponsor interview to talk about a few things. Some really interesting AI stuff, actually. one interesting thing is that Prowler's customers now are starting to use their own dashboards that they're creating from prompts. Tony actually shared one of the prompts with me. It was on GitHub. And then he's like, I can, you know, he just said, well, I'll just set it public. So I've linked through to it in this week's show notes, but you can build a dashboard now from a prompt, which is just amazing. Kind of annoying to all of those people who wasted all of those dev hours building dashboards pre-AI.

1:18Patrick Gray: But anyway, so yeah, we talked to him about that and just about how people are using AI to interact with tools like Prowler. Interesting chat coming up later. But it is time for the news now. And guys, I think, you know, since we spoke last week, there's been like just more and more news about these AI safety incidents, you know, with things breaking sandboxes and escaping onto the internet. There was one from the AI Safety Institute, which is, you know, kind of ironic. Like they put in their blog post, yeah, we gave it deliberately permissive things. And it's like, okay, that's fine. But maybe you should like, I don't know, do better monitoring so you can see when things have gone wrong.

1:57Patrick Gray: But it seems like this is a very pervasive problem. Of course, last week I spoke about a story that was coming from the ABC. It's since was posted and has blown up globally. And this, of course, is this Australian guy who tried to get into a gym class and it was full. and he asked the agent, hey, is there any chance you could find a way to get me into that class? And apparently it started kicking other people out of the class list. James, you've had a look through this one. You actually have some questions though.

2:27James Wilson: Yeah, I look at it. I'm a little bit surprised, you know, choosing my words carefully. Like I think it's a great write-up and I have no doubt that this did happen exactly the way the article says. But having used Open Claw for a lot of different things, its native tendency is not to just go and hack about in APIs. But as with all things LLMs, you give them enough of a, you know, don't stop or I really need this or you're my only hope and I'm tired of using this through the web interface and, you know, they'll go do it. But, you know, I feel like with all of these, I come back with the same fundamental question, which is where's the transcript?

3:03James Wilson: I want to be able to see the transcript.

3:05Patrick Gray: Yeah, and it's like, so the question that I had about this when you and I were talking before today's recording, right? Is it's like, well, how hard did the guy goad the agent into doing this for him? Yeah. And that got me wondering, like, is there a legal threshold there? At what point in encouraging an agent should you be aware that it's about to go and commit a felony on your behalf, right? And I think there's going to be some interesting legal questions there. And one day in the next couple of years, we're going to be covering some court case where there's a transcript and people are going to be arguing about what the intended command to an LLM is.

3:43Patrick Gray: Brad, what are your thoughts on this story? Because, I mean, obviously it's blown up because it's a great human interest story and it's kind of funny. It's kind of got elements of everything. But, yeah, I guess, are you a little bit surprised, like James, that OpenClaw just went off and did this? Well, I think it's a real litmus test for how much people have hands-on personal experience with their own agents because that level of oops surprise kind of events happens to me weekly probably. We're things I wish it didn't happen that way or I didn't mean to set it up that way, but now I'm smarter now.

4:17I'm going to fix it for next time around. And so the people that are freaking out about this, I think they're coming from a more theoretical standpoint. And so this level of screw up and oopsie kind of mistake is something that I think happens a lot in my environment as well. And it hasn't yet crossed the boundary outside of my lab into somebody else's, you know, compute envelope in order to inflict damage in their environment.

4:41Patrick Gray: Well, that you know of, I mean, maybe like everybody else next week, you know, we'll be talking about Brad Arkin's AI agents going rogue and owning a bunch of accounts. Yeah, exactly. Exactly. It's entirely possible. Yeah. But I think, you know, the real lesson, we were talking about this last week at the Black Hat Conference, is that all of these latent things that weren't as good as they needed to be, but they were fine in the old world. So unauthenticated APIs on the gym class registration website, not a problem in the old world. Today, it's a problem. And so you're going to find stuff like this all over the place in your environment, on your public surfaces, and it's going to start getting exercised by these creative agents working on behalf of people to solve problems.

5:22So I think this is the beginning of the new normal. Yeah.

5:25Patrick Gray: Now, meanwhile, I mentioned the AI Safety Institute incident. Catalan Kimpanu, our colleague who writes the Risky Bulletin newsletter, was getting very annoyed at watching people blame the Frontier Labs over this incident because he's like, this was totally the AI Safety Institute's fault, given the way that they set up their cyber evaluations. James, what's your take here? Because, you know, who's right?

5:47James Wilson: Well, 100%. I mean, but to their credit, they call that out, right? As you said, they name the models they use. They deliberately say that, yes, we reduced the cyber guardrails. Yes, we gave it access to the internet. So, of course, these things happen. The real blunder is, yeah, they weren't watching it and didn't catch it fast enough. But this comes back to the point, I think, that we've talked about before, which is how do you thoroughly measure the safety of a model and the upper bounds of its capabilities if you don't give it unfettered access to everything? Our only option left is to watch these things like a hawk.

6:21James Wilson: How long also till that becomes just an intractable problem because they move too fast?

6:27Patrick Gray: Yeah, yeah. I mean, I do think it's a little bit like giving a 16-year-old a bottle of scotch and a shotgun and saying, go out and make some money, you know, and then being surprised. Then being surprised when something bad happens, right? So there's a little bit like that. Meanwhile, OpenAI did a talk at Black Hat all about the Hugging Face incident, And this, look, I think the reaction to the talk was as interesting as the talk itself, because people are like, look at this, these big corporates out there, you know, getting stage time to talk about their felonies and like it's no big deal and like they weren't negligent and Black Hat is very corpo.

7:05Patrick Gray: And I'm like, Black Hat's always kind of been intended to be the, you know, the conference for business. So that was a weird criticism. But yeah, people getting very angry at OpenAI. whereas I found the talk absolutely fascinating. I mean, the whole thing is incredible, right? You've got this situation. And look, to be clear, they do talk about things in that talk where you're like, oh my God, you're actually admitting something kind of bad there. Because they had this weird situation where they had all of these agents discovering that they could write files onto Artifactory, which was the package manager, which was the way that these things wound up getting out to the internet.

7:41Patrick Gray: But they could actually start talking to each other using Artifactory as like a bulletin board system, agents reaching out to other agents for help. I mean, my joke this week is like we're going to refactor the risky business CMS to run on Artifactory and just, you know, Artifactory with a collection of custom exploits. And then, you know, that's how we're going to spin up some forums. But what's amazing is that OpenAI discovered this, re-rolled Artifactory, like nuked it all, and then thought, ah, job done. Like they didn't think, oh, this could happen again. And of course it happened again. and then on it went out to hugging face.

8:13Patrick Gray: James, I know you would have paid very, very close attention to this talk. What's your feeling on the talk? What are the interesting bits from your perspective? And do you think open AI deserve as much crap as they're getting?

8:27James Wilson: Look, the talk itself is, I think it is like must-watch content at this point. You've got to go see this. I found it quite hilarious to the extent where I'm almost convinced that these guys must have been actors that were presenting because I don't know how they managed to get through the talk without doubling over in laughter at some of the points that were being discussed in this. But suspending disbelief on that, it's interesting for what it tells us that we didn't already know. And I've been staunchly in the camp of this is not surprising, this is not something to fear, but there was elements as I was watching this that made me go, I'm a little uncomfortable now that I know that it did certain things.

9:06James Wilson: So take, for example, we knew that this training run had used Artifactory as its exfiltration point out to the internet. And that's not surprising. What's surprising is since May, it had been using Artifactory to post messages between these agents, not just within that training run. But OpenAI was running one instance of Artifactory across all of its training runs and researches.

9:31Patrick Gray: And so that's the one where I'm like, oh, and they just admitted it. So they were running that one Artifactory instance and just everything under it. It just seems like an insane way to architect to set up like that. That was nuts.

9:43James Wilson: I mean, you can't even call that architecting a setup.

9:45Patrick Gray: No, it's like the AI agent equivalent of a flat network, right? It's just crazy.

9:51James Wilson: And that's one thing to do. It's one thing to have a poor deployment of a super scarce and important and resource that's super expensive. But in all the things that OpenAI pays big bucks for in their training runs, Artifactory would have to be the cheapest to obtain by license count, cheapest in hardware to run, and probably the most simplest element. So why was it this one shared thing? But getting back to what worries me here is this notion that the agents started to communicate, that bit's not too surprising. But that they communicated so readily across multiple different experiments is a little bit concerning.

10:27James Wilson: the bit that really made me just sort of raise the eyebrows and go huh we got to keep an eye on that is I've always said that I'm not worried until an agent says it hacked because it found it fun like as if it deviates off purpose it makes up its own reason for doing it and there's a tiny little shred of this in this story Pat where where the they observed that some of the agents would say maybe I should go help out that other agent it doesn't benefit me but collective good could help us all get to a solution and and that's like the inkling of divergence of alignment that i think is oh it just just gives me a little bit of a oh not not great feeling about it like i said last

11:08Patrick Gray: week let's not give these things opposable thumbs right that's let's just keep it all in on computers for now keep it all in the virtual world um i did love the part of the talk where the way they were able to figure out what had happened that they were behind the hugging face incident is they actually reached out to Hugging Face to find out if they were impacted by the Hugging Face breach. And it's like, oh, okay. Actually, it was us that did it. Brad, what was your take on watching this talk? Because it was real, I was really surprised actually that OpenAI's like general counsel or whatever, let them do this because they, it's very like this real transparency here.

11:44Patrick Gray: Yeah. That's right. The transparency, the, you know, ready admission to what might naively be called, you know, federal felonies, you know, international, because I think some of these servers are outside the US as well. Based on sparse public data before the talk, I was picturing a single overenthusiastic agent being responsible for everything. And so the thing that really caught my attention was the self-organizing swarm behavior without any prompting any guidance of any sort to go do that. That really stuck with me as something that I was not expecting to see. And so the idea that this snowballed over time with all these different agents across lots of different training runs, that the thing was that I thought was most exciting.

12:30Patrick Gray: I will say too, sorry to interrupt you there, but you know, just to, okay, so you're both talking about elements, I guess, that you feel uncomfortable. The thing that made me uncomfortable, actually, reading through that part of it or watching that part of it was seeing the messages that they were sending to each other, which were not human. you know what i mean like it was still english language but it's not the way humans speak and i'm like man i'm watching two robots kind of cook something up at the moment that that feels yeah and they shifted to base 64 at some point so they were you know throwing little like tags back and forth the the thinking traces so james when you're talking about doing things for fun so the you know all caps like this is huge like you know the moment of pwnage when they got something to work you know that was also very like humanizing to see the way that each agent was thinking through this and then you know people you know different agents on the message board worrying about like are these all agents or some of them like interlopers that aren't real agents and how do we cryptographically sign our messages to prove who we are like all of that to me was really fascinating so I think you know the I heard Anthropic hired like a psychologist for their agent you know for for clod and so the idea that you now need like anthropologists and things like that to figure out like what's the community behavior and self-policing and that sort of thing that's going on so the swarm is the thing i spent the most time paying attention to in that talk look we've spent so long talking about how we shouldn't anthropomorphize uh anthropomorphize these um models and then that's just what we've spent the last 10 minutes doing i just want to

14:04James Wilson: point that out yeah but you you can't help it right and and it is this it's the chain of reasoning that is exposed here that is the biggest bit of intrigue for me because i'm using glm kimmy quen opus all of them all day none of them talk like these agents do in this chain of region reasoning there's something different about it and look to be fair that could be because um open ai does encrypt their reasoning they don't display it because you know that that would be distillation material so maybe that's just the nature of their model but to brad's point around the psychology Like, you know, if you're intrigued by this, go and take some time to use GLM and turn the thinking mode on and watch its transcripts.

14:44James Wilson: It's like watching the most anxious and disturbed individual you've ever met work through a problem. It's constantly like, hold on, wait, let me look at this. Hold on, wait, I need to rethink this. Hold on, wait, I've gotten too deep in this. And it's just, it's nervous. It's nerve wracking to watch it.

15:00Patrick Gray: Your blood pressure goes up watching the reasoning crisis.

15:03James Wilson: Well, yeah, but the point I'm getting at here is the fact that OpenAI is one is so fundamentally different in how it reasons. It's very short. It's very efficient. It's broken English, but it gets to the point. It's created its own dialect. It just makes me intrigued as to is that part of the secret source that is keeping them so far on the frontier is the way that they have really fine-tuned that chain of reasoning. So it's not just an anxious person chatting back and forth to themselves, but there might actually be some really interesting engineering that's gone into why they talk like that because I haven't seen that before.

15:33Patrick Gray: It's interesting that, you know, essentially what you're talking about is a dialect of English for machines.

15:39James Wilson: Yes.

15:40Patrick Gray: Which I'm sure that's going to be a whole other feature interview that you'll do with some linguist at some point. But that'll be fascinating. Now, look, when all of this hugging face stuff happened, there was this bizarre thing going around where people are like, oh, it's marketing for open eyes. Like, yeah, okay. They're going to expose themselves to that sort of liability for marketing. I don't think so. But one thing that's not going to help them beat the It Was Marketing accusations is they've just released basically specialist cyber models for doing red teaming and blue teaming. This is OpenAI, which looks pretty interesting.

16:11Patrick Gray: I think I'm really curious to see for both of these models, how much you have to use them with tools. Like even in this week's sponsor interview, for example, talking with Tony, he's talking about how, you know, you can get agents to use Prowler to do things like, you know, you ask your agent, hey, use Prowler to see what sort of exposure I have in terms of like S3 buckets or something. Now you ask a frontier model to do that without using Prowler, it's going to go and start pulling down documents for the cloud providers, like documentation for the cloud providers and figure out how to do that itself.

16:44Patrick Gray: So I sort of wonder how OpenAI has figured this out in a way, because I think there's still going to be a tool layer for LLMs, at least for the foreseeable future, because it doesn't make sense for LLMs to recreate tooling every time you ask them to do something. Brad, you've been in this industry a long time. How do you think that part of it's going to go? Because I sort of think, I don't know how it's going to shake out. Yeah, so search engine optimization for AI models is definitely a thing now. And making sure that you're publishing information so that the agents on behalf of your customers can get what they need in order to use the tools, deploy it correctly, that sort of thing.

17:26So some of the companies that I'm working with, I've been chiding them because their website doesn't have the details my agents need to figure out what to do with it. um and so to me being really clear about who you're trying to educate and market to and it's now like a split audience where you've got your humans and the agents that you're trying to get the information into their hands so they know what to do with it um and then figuring out how do you make that more and more efficient over time and so is it just good api documentation that's dual purpose human and agent or are you going to have to figure out you know like like this alternate language you know very dialect well or do you still have a tool layer right like

18:03Patrick Gray: that's what i wonder because i i think like getting the agents the frontier lab ai's to like go and do this stuff without tooling they're gonna eventually have to develop some sort of tool for a task right and it doesn't make sense for them to be developing that over and over and over it makes sense for there to be toolkits but you know are those toolkits going to be optimized for agents, not humans. You know what I mean? Like, where does this go? Like, and who owns the tools and who makes money from the tools? And do the frontier labs wind up developing their own toolboxes for doing pen testing, for example?

18:35Patrick Gray: I mean, that's more what I was wondering, like, you know, what sits inside the LLM versus outside of it to be used by the LLM? How do we find that? I think it's just a wide open opportunity. So, you know, like Burp Suite and companies like that, it's their chance to adapt in order to better serve the agents that are going to increasingly be a bigger share of the market of who they're selling to. I mean, they've sponsored us to do a couple of interviews about exactly that, right? And they see that as being this huge opportunity. I guess I'm wondering what your view is. Oh, yeah, sure. So basically, I think there's going to be a lot of disruption from companies that aren't able to make that transition.

19:10And either they're asleep at the wheel, they're not paying attention, or they try and fail to adapt properly. And so I think some companies will go out of business because they're only able to serve the humans. and then they're not able to make that transition. And then that'll create opportunities for people that are maybe like agent only or agent first solution providers or skill capabilities or things like that. And then how do you monetize it? And is it usage-based, seat-based? What's the right model there? Everything is up for grabs. And so I think that's the biggest takeaway is that it's just a lot of innovation and people are going to be experimenting with a lot of different models here until we figure out what's actually working.

19:47Patrick Gray: Well, and I wonder at what point companies like OpenAI say, okay, well, here's our cyber model, and we've recreated a bunch of tools that are commonly used in pen testing to be used by the model. So, yeah, I just wonder at what point they start offering stuff that isn't just LLM. You know what I mean, James?

20:02James Wilson: Yeah, I do know what you mean. I think ultimately the tools in the model are so symbiotic they'll continue to co-develop. So let's say, for example, the model has no solid built-in cyber experience. It does lean heavily on tools. you would assume that open ai is going to take those transcripts and use that as a reinforcement learning post-training to to further the model's ability and what it's going to learn from that is ah there are tools out there that i can use to do this work but it's the model is quickly going to discover that observing what tools can be used to do a task is only useful if those tools are ubiquitously available which will then lead it towards working out okay well i used to use let's say this off-the-shelf third-party tool, this paid tool to do this, but here's what the outputs are that it produces.

20:46James Wilson: And I know now how to cobble together a couple of shell scripts to do that. And that becomes the next iteration, the next iteration. So I think the challenge really here is not, do you create the tools that serve the humans or the machines? It's how do you have a durable model where your tools continue to serve the machines in a way that they can't just determine how to do themselves with a shell because you can't then monetize an agent with a shell?

21:07Patrick Gray: Yeah, I mean, the whole thing just, it's going to get complicated, right? And then you've got these other companies that have come in, like I think of like Horizon 3, for example, that do AI-based pen testing. Like at what point is something like a frontier lab coming in and saying, hey, we're doing red teaming stuff. Like at what point does that present a risk to them? Or do they get to maintain some sort of moat because they've built a whole bunch of non-LLM automations around using the LLM? And, you know, it's just an interesting time to be in the cybersecurity business, I guess is what I'm saying.

21:36Patrick Gray: Now, last week, we spoke about the attacks against US water systems being attributed to Iran. At that point, it was like Minnesota and a couple other states. That campaign has now spread to 12 states, which is a pretty big campaign, really. Tom Uren, our colleague, has written some really interesting stuff about what he thinks Iran is trying to achieve with this. I've linked through to that in this week's show notes. But the upshot is, you know, they don't really want to cause havoc. They just want to make people feel a bit threatened, make people feel a bit unsafe. This is really about the politics of it all.

22:11Patrick Gray: But we're seeing some other attacks against local governments. We've seen attacks against local governments in four states. No attribution to Iran yet. Could be ransomware actors. But we've also seen a cyber attack against ports, port infrastructure in North Carolina. Is it ransomware? Is it Iran? The reason I'm flagging this now is because the water stuff has blown up a bit. We've been waiting since the initial war broke out between the United States and Israel and Iran. And, you know, there was nothing for a long time. And now we're actually seeing these attacks. And I just sort of wonder at what point it's going to start ramping up.

22:46Patrick Gray: And I just think it's one to keep an eye on. Brad, you know, is this all tracking about how you expected it to from the early stages? Well, I don't know if Iran had any capabilities that they're still holding back. I don't know what they're waiting for, because it seems like they would be pushing every single opportunity they have to advance whatever it is they're trying to achieve. And, you know, fooling with these like PLCs that are misconfigured or default credentials and things like that, it just doesn't feel like this is a particularly impressive bit of force projection. But I don't think it's supposed to be that.

23:22Patrick Gray: And that's what Tom's been writing about, which is, you know, this is about the U.S. government having to issue boil water notices. You know, if you're some resident of some small town in, you know, nowhere USA and you're being told you've got to boil your drinking water, that's a political effect right there. They don't care that it was an exposed PLC with admin admin on the Internet. They don't care. They care that they have to boil their water. Yeah. But why now? Why not five months ago? I don't know. My guess is that these guys might have had something better to do before, and they've kind of run out of targets.

23:53And so now they're just out. Like, to me, this is more mischief than anything else. So I think the way it's being covered in the U.S. media is where the effect is coming from, because they're hyping it as something more interesting than I think it really is. So I don't know. Brad is unimpressed.

24:10Patrick Gray: I think it's the TLDR. Brad is meh. Just meh. Iran, water, meh. All right. Fair enough. Fair enough. We'll keep an eye on it, though. We've got this next report here just staying on the issue of critical infrastructure being attacked. We've got this report from the Polish CERT on an attack against Poland's energy infrastructure. I think this emerged, details of this attack emerged because they were doing an incident response into another attack, James. But you've read the whole thing and found it fascinating. walk us through the Polish cert report into this attack.

24:50James Wilson: Yeah, okay. So listen, anyone that has even the slightest bit of desire, interest, or reason to go and learn about how these OT attacks work, you've got to go and read the PDF that is attached to this story. It is a detailed examination of exactly how this played out, the steps that were taken, and the tradecraft. And also there's some of the neat things they had to do to even piece this all back together. But yes, so in, I think it was December 2025, there was an attack that had been covered. It was 30 or so wind and solar plants. That was when the initial coverage around, you know, that someone was attacking Poland's energy infrastructure.

25:27James Wilson: This report focuses on the fact that now going through and doing a deeper dive into this, they found that actually that was one of two attacks that was happening. And when they delve into the nature of the other attack, this is where they found some very interesting details about just the level of, I guess, skill and knowledge that these attackers had in the OT space, right? To Brad's point, Brad, if you were upset about the default password being admin, admin, or PLC, strap in, buddy, because this one is going to impress you greatly. So in this case, you've still got these PLCs, but they didn't just go straight for the PLCs because this network was quite well architected.

26:03James Wilson: There was good VLAN separation. There was serial communication protocols only to the PLCs, which is going to reduce your attack surface but the attackers noticed that okay they compromised a fortigate at the wind farm no surprise there but then they noticed that the cellular modem that was being used to essentially attach a lot of these plcs back into the network despite the fact that it was on a private apn it was not internet reachable that cellular modem had an admin ethernet interface that had been plugged into one of the vlans that that compromised fortigate could get to so they pivot across to that that's how they get into that private apn then they can move laterally across all the plcs despite the fact that they're serial only and they're on a private apn which is not internet reachable that level of tradecraft is uh quite i think outstanding in terms of what they had to know about how plcs work how private apns work how the network infrastructure worked how the protocols worked for this so it's it's like this is what a sufficiently skilled actor working in offensive operational technology arena will look like going forward.

27:09Patrick Gray: Champagne TTPs. Just before you jump in there, Brad, I want to ask James, what did they actually do once they had this level of access into the energy system? Was this just about pre-positioning or did they actually try to drop a payload and it failed? Because you would think that if it was a successful attack that was supposed to be disruptive, they wouldn't have found it when responding to a later incident, right? They would have known at the time.

27:29James Wilson: Yeah, so the one that they uncovered later, you're exactly to that point the effect wasn't such that it was actually noticed the the one that did get noticed was because the intent and the payload that was dropped was um designed to have a destructive impact and it did right it disrupted the flow of of energy and and also heat and steam i think it was so sufficiently that it was caught but this was essentially extending out further into other devices pre-positioning making sure they had all the credentials there's some great unknown in there as well like i don't know where the creds came from But yeah, great, great write up.

27:59Patrick Gray: Yeah, right. And presumably this is Russia. But Brad, you had some thoughts? Yeah. So I just really like that there's a few little windows into some deep professionalism that came across when I was reading through this. The first one was they said, listen, we're laying this out chronologically in how it happened. But the way we came to understand this was in exact reverse order. And so they found, you know, the last foot, you know, the last step in first. And then they had to like work backwards from there. and the other thing is this is now eight months later from when the event occurred or when they when the investigation kicked off and that's the real world when you're doing incident response on someplace this complicated is that it's months and sometimes years later you're starting to understand things better that happened a long time ago and you know you're looking for one thing you learned about something else you don't really understand it yet you just have some breadcrumbs then you keep pulling pulling the thread and then eventually the story starts to emerge later on And so this for me really brought up a lot of memories about chasing down incidents and really, you know, two, three years later, sometimes finding the puzzle piece that kind of explains something that we were really confused by when we first started investigating.

29:06So I like that.

29:08Patrick Gray: This brings back memories is a strange way to say this triggered my trauma, Brad. But yeah, we've got another piece here where the what is it? the you know the committee the senate committee in the united states into china has taken a look at china mobile china unicom and china telecom which you know there was like this big hoo-ha about how they'd been banned from operating in in the united states and whatever i think by the fcc but like somehow this ban didn't result in all of that equipment belonging to those companies being actually ripped out of like u.s infrastructure so it's all still there and you know there was no mechanism to force that stuff to get pulled out so this is something that the u.s is looking at now i know that when i talk to people like you know rob joyce who obviously spent most of his career at nsa they are really sort of like want this stuff out of u.s networks they think it's uh it really doesn't belong there so hopefully we'll see some action there but i'm kind of surprised that there was this big action against you know china telecom or whatever but hey all of those racks of routers and whatever over there, that's fine.

30:17Patrick Gray: Just they can stay. Brad, did this surprise you as well? Well, to me, it didn't because they used a like legal regulatory framework in order to stamp out one thing. And then you're just, you're squeezing the balloon. And so the intent will then just shift to whatever hasn't been banned yet. And so we don't have the right mechanisms to comprehensively address this. So they kind of are doing one at a time and trying to stamp it out. And so to me, this is just the, like, they're effective at the regulatory game.

30:47James Wilson: Yes, that is exactly my read as well. It was like, it is just a pure regulatory play of, well, we're safe now because we've denied the many new licenses and we've revoked a few things. It's like, yeah, but the equipment's still late. Yeah, but it's okay. We told them not to use it. It's going to be fine.

31:01Patrick Gray: Yeah. Well, let's see where that all tracks. Now, just a quick thing here. The State Department has said that Donald Trump actually raised the issue of cyber scam compounds operating in Southeast Asia with Xi Jinping, which is just fascinating when you've got the leaders of the two most powerful countries on the planet actually discussing this between them and about ways to get onto it. I mean, you know, we've flagged this as a big issue for a long time, and it's good to see it getting attention at those levels of government. I think it is well and truly in the interests of the Chinese and the US government to try to get a handle on that.

31:34Patrick Gray: I mean, if anything, you know, there are a lot of American victims here, but the amount of money involved is corrosive to the governments in the region, right? Like the way that this interlinks with corruption is a regional security problem in Asia, in my view. So I think it's even more, it's going to be more useful to the Chinese even than the Americans to try to get a handle on this. Oh, here's one that is right up your alley, James. It turns out that Team PCP, which did a bunch of supply chain, AI-enabled supply chain attacks earlier this year, it turns out they've been active for longer than previously thought, even in the pre-AI era.

32:17Patrick Gray: And there's a write-up out about this, and it's really interesting. You're our in-house Team PCP ornithologist, I guess. I don't know. What do we know here?

32:28James Wilson: So we know more than we did. When I did that solo pod on them, we knew their activity dating back to sort of 2024, 2025 and the rapid evolution of it. But in this case, this was published by Oligo Security. They identified a new campaign or a campaign that is newly attributed to Team PCP that happened around 2025. and that was called they were exploiting something called shadow ray one of the first ai worms that spread but what's interesting is they extrapolated out the sort of ttps and domains and things that they found looking at that attack and said well actually this matches activity going all the way back to 2020 now on one hand it's not surprising because team pcv is known for having absolutely terrible opsec and everything has been available in github or telegram or was there at least at some point in time.

33:15James Wilson: Well, what's interesting is this sort of negates the sort of, I guess, assertion out there that, oh, Team PCP is just a script kitty armed with an LLM. My view all along has been Team PCP is a lone actor that has a dev SRE background that supercharged their abilities with AI. And I think this puts a good bit of proof towards that because in 2020, ain't no LLM that's doing your capable hacking. So they had to have some degree of skills and abilities that they've just been constantly evolving and improving as they've been going along.

33:44Patrick Gray: Now, without saying too much more, I think you and I can both say that we expect the individual behind this activity to be arrested.

33:52James Wilson: Waiting. Yes.

33:54Patrick Gray: Brad, funnily enough, you are actually an advisor to the company that did this research, right? Yeah, yeah. So I was hearing about this, I don't know, many months ago. So they were excited to share with the world of Black Hat. Yeah, awesome. Now we got a piece here from Wired where a Greek security researcher managed to stumble across a C2 belonging to North Korean hackers and, you know, just all of the cool stuff that you would expect them to find, they found, James.

34:25James Wilson: Yes, and this prompted a good discussion between us because my initial reaction was, oh, my God, if I stumbled upon a C2 like that, I am backing the heck away from it. You know, I do not want DPRK coming after me. But you raised a great point, which, you know, for me, relatively new into this cybersecurity world is that different countries respond differently. DPRK just wants to get on with getting their monies.

34:46Patrick Gray: And so they don't need the distraction of like sending a crew to beat you up with baseball bats, man. Like they don't need the scandal. They don't need the attention. They just want your Bitcoin.

34:55James Wilson: Correct. Which gives rise to a researcher in Greece being able to really rifle through their stuff. And what he found, he was in these systems for something like 22 months just watching them do their thing. you know, 1 ,600 companies, 57 countries are impacted. But what surprised me here is I kind of assume sometimes, you know, DPRK is doing what they're doing, they're IT workers and they're other scams. And we hear about the big crypto heist, but I've always just thought that the other stuff going on is probably a long tale of small to medium actors on small to medium businesses. But these are big names that were found in this research set, like Aon Smart Technologies, Chinese phone manufacturer Oppo, cryptocurrency firms like Coinbase, Uniswap Labs, Italy's Supreme Judicial Council.

35:40James Wilson: It's like, wow, okay, they are actually operating, not just at a big scale, but going after some pretty huge targets that we don't hear about other than from this research.

35:50Patrick Gray: Well, I think that's the interesting thing here is just, you know, getting a bit of a glimpse into the scale of this stuff and reminding ourselves that we just get to see the tip of the iceberg. But yeah, I mean, look, North Korea exposing a North Korean sea to, you know, So a Russian transnational crime group, they might push your button, you know, like that might not be a wise thing to do. But yeah, with the North Koreans, I don't know. It's just not something that we associate with them. For those who missed it, by the way, and I've linked through to it in this week's show notes, Tom did an excellent write up in Seriously Risky Business last week about how we've seen North Koreans starting to, like APT operators, starting to really collaborate with ransomware gangs.

36:34Patrick Gray: And we've also seen some former APT operators be arrested by the North Korean government for stealing from like North Korean banks and stuff. And like the word is like the people who did that, their entire like bloodline is going to go to the gallows for that. Like, cause that's just how they roll in North Korea. And some of this is rolling uphill to the apparatchiks who operate these APT crews. So I actually was lucky enough to be able to put the headline on Tom's piece, which is being a North Korean hacker is about to be a lot less fun because they are about to put really tight operational constraints on these guys because their bosses are worried that their bloodlines are going to get wiped out as they go off to a firing squad.

37:15Patrick Gray: Right. So that is a good read for North Korea fanciers. Check that one out in this week's show notes. Now, some really interesting technical news this week. Chrome is shipping device-based session cookies. This is something they announced a while ago. We flagged it at the time. It looks like that's basically out or about to be out. This is an absolute game changer because one of the, you know, the 24 karat gold currency in the underground right now is like session tokens and whatever. Brad, let's get your thoughts on this. I mean, as someone who's operated, you know, who's been a CISO for very large companies, I imagine like you would be looking at this saying, where were you when I was a CISO, right?

37:55Patrick Gray: like this would be just fantastic. So I was meeting with the Chrome engineering leadership in April last year advocating for this. So this is something I've been personally paying attention to for a long time. Basically, when you look at account takeovers and what forms that takes, session token scraping from compromised endpoints through adware or things like that was a huge share of the ratio of like what was happening here. And so this just solves that. It will stop it once it gets broadly deployed. unfortunately the bad guys have been adapting techniques in advance of this change and so we're seeing more and more of the phone-based social engineering attacks where you trick the attacker into somehow giving them access either logging into a website they control with the actual credentials or you know doing like a device authorization workflow or things like that and so device code phishing is the the new black at the moment yeah and so it doesn't mean like bad things will stop happening as soon as this goes ga uh but it's great because it solves a real problem that has been around for decades and this is the right way to solve it like we finally you know if tpm's in every device this is actually going to fix a real problem so i'm very excited about this yeah

39:05James Wilson: james you love this as well very happy with this but i will say you know who else loves session tokens ai agents and so how long before the agent gets really good at uh minting these tokens in a device specific way by leveraging the tpm like it just it'll be interesting to see whether they solve this gap as well and learn how to do this.

39:24Patrick Gray: Well, as long as it's happening on the box and it's your agent, I mean, I think the thing is like malware gets around this, right? If you've got malware on the box, it can probably get around that. But I mean, that's like a much heavy, it's a bigger lift, right? As for like agents and authorization and authentication when it comes to agents, man, that is a whole field right now. God knows where that's going. We have to speed up because we are running out of time. There's a bit of cool research out of Port Swigger that has looked at dumping mailboxes using cool novel techniques. James, you're all over this one.

39:55Patrick Gray: You loved it.

39:56James Wilson: I absolutely loved it. You know, CSS began its life as a declarative way to define styles. Then it gets more and more, you know, code-like things built into it. You can do conditionals. And, of course, when you take something that's declarative and add all those sorts of additional code-like things into it where you get an attack surface, and the guys at Portswigger have just done an incredible job that, I mean there's like literally a dozen different attacks in this where they just get more and more deep into the eternals of just exactly how you can get css to do prompt injection how you can get css to do detection of tokens in a url by having a font size oracle and animation timing differences and then exfiltrating that out to urls through background requests it's just uh look I almost can't do it justice by trying to explain it other than to say go read it because this is such an incredible look at all the ways in which we've turned CSS into a coding language, which can now be exploited.

40:51Patrick Gray: Yeah, this is work by Gareth Hayes at PortSquigger. And it is, you know, it is, look, every year at Black Hat, PortSquigger always dumping the good stuff, you know. So that's good to see that continue. There's been a security incident with Metabase. This is worth mentioning because this has been on-prem and cloud-based Metabase. And it's like, it looks like this is a big incident, right? It's definitely worth mentioning this week, James.

41:18James Wilson: Yeah, big for two reasons, Pat. One is Metabase is what you use when you can't afford Tableau and Looker, and that's a lot of places, right? Those are really expensive BI and insights tools. So it's got a big install base. But Metabase is one of these dashboards and insight tools that sits directly on top of your Snowflake, your Redshift, your big data lake. And so compromising Metabase is a really quick path into the most valuable data that a business has. So this will be a big deal. And it has, unfortunately, a SQL injection in the password reset flow. And yeah, active exploitation will be ramping up if it's not already for sure.

41:58Patrick Gray: Yeah. Now, remember those snowflake attacks back in 2024? The really weird ones because the OPSEC was so bad. this 26 year old yeah he was using uh at the time i guess he was 24 but he was using um stolen credentials to just log into snowflake instances steal all of the data and then ransom it back to the companies uh he of course got caught has pleaded guilty and uh will spend up to 32 years in prison for that which is just a reminder that if you play stupid games uh you are gonna win stupid prizes but what's crazy is we've got this other story here where there's this young guy from the comm who abused more than 100 like teenage girls between the ages of 13 and 17 coercing them into producing imagery and doing acts of self-harm and sending him videos and stuff.

42:49Two years, he gets two years.

42:51Patrick Gray: So what I would like to see is those two sentences probably reversed. You know, if there's, if they're like, two years is just completely inadequate for that sort of level of offending. And 32 years for the other bloke is just nuts, right? Considering he's 26 years old, probably has some sort of prospects for rehabilitation. But this just goes to show you that like, you know, the British judiciary, probably too lenient. The American judiciary, a little bit too much the other way. What else have we got here? We've got an FBI warning as well, saying that, you know, people are getting their nudes hacked and whatnot.

43:24Patrick Gray: That kind of connects to that previous story. Obviously, this is a big problem at the moment. If the FBI are warning about it. Usually it takes a lot to get them to issue a warning about anything. And then we've got a couple of interesting pieces here on AI, where AI is getting better at election facts, is the headline, but voters shouldn't rely on it. And this just reminded me of when Wikipedia was new, and everyone's like, oh, you can't rely on Wikipedia. There's bad stuff in it. It's like, everybody relies on Wikipedia now, more or less. But they know that there's a caveat there that it is that it is wikipedia i think ai agents are the same but where this gets funny is apparently you've got like a memo out of the ftc uh in the united states where they want to regulate ai for ideological bias so i guess they want to make ai great again brad you're an american i i can't imagine you would feel too great about the ftc regulating ai agents for uh ideological bias right i can't i can't defend it this is not going to go anywhere no it's not um it's not and you know there's just a few ransomware items that we'll just leave in the show notes for people to read if they would like to.

44:33Patrick Gray: The United States and South Korean governments have issued a warning about the Gunra ransomware group targeting government agencies. CISA says that a SharePoint flaw that's been doing the rounds in the wild is now being used by ransomware crews. Hooray. SonicWall SMA 1000 flaws are now being used by ransomware crews. And what is it? Enable Enable software. What are they called? It's the Enable N-Central. That's being used to drop RMMs on victims by ransomware. So there's a lot of ransomware activity at the moment, but we're going to round out this week's news coverage with a story that's just gone massive, which kind of shouldn't have, which is some idiot on their way home from DEF CON basically de-authed everyone off the plain Wi-Fi and spun up a malicious access point.

45:23Patrick Gray: Brad, this reminds me of when someone years ago, like plugged into the seat back or something on their way home from DEF CON and was like posting photos of whatever of getting shell on the, you know, on the entertainment system and caused a similar level of freak out. This is a dumb thing to do. And, but I don't think it's going to amount to much. What do you, what do you think? Well, so they, the pilot called it an emergency. So they were met at the gate by, you know, guys with guns and whoever did it, they were on the plane. So there's nowhere to hide. So I don't know. This is like, if you're going to do something like that, I think the airport lounge would be a much better place to do it because you have like some plausible deniability.

46:02Yeah, this thing has gotten the attention of all the normals. So like all my non-cyber technical friends have been sending me, you know, news articles and LinkedIn posts and things about this event. And it seems to have really captured their imagination. So I remember the guy who was messing with the seat back, the FBI visited him. He told them all about it and then got mad later when they used it against him because he said it was shared in confidence. Was he the guy who was saying

46:27Patrick Gray: he could make the plane fly sideways? Or was that another? Exactly. Yeah, yeah. And there were memes. There were so many memes. That was a fun time to be on Twitter at that time. That was pretty good. I remember actually just reminding me that there was a guy on flights in Australia who was actually fishing people through malicious Wi-Fi on a plane. And the cabin crew noticed and they had the cops scoop him up. And he got in trouble. He was convicted for that. So it is interesting now that I guess aircrew have the training now, right, to spot when you're doing shifty stuff with Wi-Fi. But that's it.

46:59Patrick Gray: We're going to wrap it up there. Brad Arkin, James Wilson, thank you so much for joining me to talk through the week's news. It's been a lot of fun.

47:05James Wilson: Thank you. Thanks, Pat. See you next week.

47:12Patrick Gray: That was Brad Arkin and James Wilson there with a look at this week's security news. It is time for this week's sponsor interview now, and we're chatting with Tony De La Fuente from Prowler. Now, Prowler is an immensely popular open source cloud security tool, right? So you could use it to run all sorts of checks against your cloud infrastructure. You can use it to do all sorts of remediations as well. And yeah, there's obviously an enterprise part to Prowler, which you can pay for. And it's all of those enterprise features that do not belong in an open source project. Things like integration with ticketing systems and SSO and all of that good stuff.

47:49Patrick Gray: But really, I was joined by Tony for this interview where we really spoke about what they're doing with AI and how you can get Prowler to be friendly with agents. And they've really done a good job with that. So here is Tony De La Fuente talking about all of that. Enjoy.

48:08James Wilson: So remember, we were talking about these kind of standard features like Jira integration or any other enterprise integration, authentication, all that stuff is being asked by big companies or corporations that they are not actually very agentic oriented now, today. But for more like small, medium companies or cloud native companies, they are more like, okay let's do everything with AI or try to do everything with AI and that is why we are adding those capabilities not only in an UI like paid only UI is this is part of the MCP part of the tools behind the Prowler AI so you can do everything that you can see using the an agent an agent so for example you can do the triage from the agent do the re-scanning from the agent do the muting or configuring whatever you have to configure in your cloud security from within Prowler with your agent.

49:08James Wilson: But of course, we want to provide that using, you know, click ops in the platform if you want to, but you can go through the agent to do exactly the same thing, including creating new controls, deploying new controls, compliance frameworks, you know, those kind of traditional cloud security staff to agents. Because now it's the agent that is going to make that decision and do whatever within Prowler capabilities.

49:34Patrick Gray: Yeah, I mean, I think some of these agents are going to replace or be decent enough stand-ins for certain types of consulting. For some of the more boring compliance-related consulting, you can just say, hey, agent, I've got to comply with this thing. Can you use Prowler to see where I might be coming up short, for example? I'm guessing that's one of the ways that this works.

49:55James Wilson: Exactly. You can add your entire AWS organization, for example, or Google Cloud Projects, everything, and say, okay, make sure this is everything compliant with CIS, for example. And probably it's going to tell you what to do, even ask you, hey, you want me to do it? With the proper permissions, probably can do it as well.

50:20Patrick Gray: Well, okay, so here's my next question, right? Which is my preferred way of using an agent to do something like this would be to give it that sort of read only access, ask it a bunch of questions. And then at the end, I'd say, look, if you wanted to remediate this, can you generate me the scripts or generate me the steps that would be required for me to remediate this? Hold on. That would be required for me to remediate this. And then just give me that script and then I'll go and do it.

50:49James Wilson: 100%. So Parler can do the remediation by itself in some cases, okay? Because that is not a good practice. So to do self-remediation in the cloud in runtime, let's say, is not a good practice, right? And so we don't want to provide the wrong practice out of the box. But we have demonstrated we can do it. Technically speaking, we can do it. And technically, we can do it. But it's not a good practice. What we do when you ask Prowler to do something is to generate the Terraform code, CloudFormation code, or even step-by-step click-ups and options, right? As you can find in many guides like CIS or many others.

51:34James Wilson: So we do that since the beginning of Lighthouse AI and with our MCP. of course the recommendation is not to allow the AI or any agent to do anything at any point so yeah not now would that be possible it probably actually you can you can what we do and we have articles and in our documentation you can see that is connect your agent to Prowler you can connect also your repository where your infrastructure as code is. And Prowler is going to tell you, hey, fix this here, fix that there. And whenever you have the pull request ready, the human is going to review it, accept it, and then redeploy. And you don't go through the circle.

52:26James Wilson: That is possible now with Prowler.

52:28Patrick Gray: That's the way you want it to work, but I don't think humans are going to be reviewing those sort of pull requests in the actual real world that we live in. I mean, maybe now for a little while, but there's going to be too many of them in the future. I should say too, just to be clear, I wasn't talking about giving the agent itself, I guess, permission to do the reading. I mean, that's the advantage of using something like Prowler agentically is it's the Prowler platform that has the permissions, right? It's the Prowler platform that is reading stuff from your cloud environment. So the agent is not doing it directly and there's less scope for like Flutterbinger's oopsie daisy, we over-provisioned the agent sort of incidents when you're going through a platform in the middle like Prowler?

53:11James Wilson: Something that is very important in our case is, as you mentioned, the agent is not doing anything. It's Prowler doing that for the agent because we have that deterministic database of artifacts that they know what to do. So I don't know if you remember when, I think we have discussed about this in the past, When we started using AI with Prowler, we asked Cloud Code, hey, with these credentials, read-only credentials or whatever credentials, tell me which bucket is open to the internet. Cloud Code was trying to pull data from the internet, trying to understand what to do, what is a bucket, what is open bucket, all that stuff.

53:57Patrick Gray: Yes, it's reading the documentation.

53:59James Wilson: Exactly, which is what it does, right? And trying to create a script, running the script, and giving you wrong results because it was unable to do it. And at some point says, okay, hold on, I'm going to use product to look at that. So we were used by agents before being agents, you know, or agentic. Right after that, we prepared the public ecosystem of artifacts with Prowler Hub. That is going to become Prowler registry very soon because it allows public and private registries like Docker Hub, for example, for artifacts. That is used by the agents in order to understand what to do and how and what means compliance or what means secure or not, harden or not, etc.

54:55James Wilson: So because at the end of the day, the agents, I mean, probably they may know for AWS, but what about Scaleway? What about Huawei Cloud? What about whatever new cloud that we have new clouds every day, right? So and all those are also covered by Prowler.

55:12Patrick Gray: So have you seen a case of a customer using Prowler with an agent and you've just gone, wow, that's actually really clever. I hadn't thought of that because I'd imagine occasionally someone's going to come along and say, hey, look what I did with this prompt. And you'll be like, my God, that's amazing.

55:30James Wilson: I have seen something very cool when you use agents with Prowler is there is no longer a need to use our out of the box. dashboards. You can create beautiful dashboards with a lot of features with a prompt. That's so annoying.

55:49Patrick Gray: I can imagine how much work you actually put into those dashboards and now people are like one shot recreating like better ones with Mythos or whatever or Fable.

55:59James Wilson: It's like this big prompt, but you create a beautiful dashboard with a lot of features. like okay so we have seen even and this open source is also in our documentation is like okay generate a report if when I fix all the critical and high severity findings and it says boom okay if you do this you are tomorrow you are going to be this this and that and that is possible now super easy super easy with AI with agent connected to Prowler and that is super cool when you see people going beyond expectations. And that is thanks to open APIs, also the capabilities in the product and being very community friendly.

56:43Patrick Gray: Yeah, well, I mean, speaking of the community friendly aspect, is the open source platform also set up to be as agentic friendly? I'm guessing largely the answer to that would be yes.

56:53James Wilson: Yes, actually, everything that we have in Prowler have is open source and is accessible through the MCP and through an API. So you can connect it as a tool with your own MCP. So you can configure the Prowler MCP by yourself if you want to, private or public. But of course, we have the Prowler Cloud MCP public already that you can connect through an API key directly, your agent.

57:23Patrick Gray: Yeah. So, I mean, you can, if you can be bothered with all that, or you could just use the commercial one, I guess is the answer. Exactly.

57:31James Wilson: So you can go straight and use the commercial version. And if you go to Prowler Cloud, you have a link to connect any agent directly in a matter of minutes. Yeah.

57:42Patrick Gray: And are you finding that there's been any guardrail related drama with using Prowler with like some of the Anthropic models, which are like, whoa, you're trying to use Prowler to scan stuff on the internet. Like that looks like hacking to me. I've called the police. They're on their way to your house. Like, do you get those sort of responses or is it not so bad?

58:02James Wilson: Not really, not really. So I have to say that we are not testing every single model because there is no bandwidth to do that in terms of human hours, right? But what we test is what we recommend, like GPT-55 or 556 is working very well when you connect. Also, Fable is working very well. So I have to say that one thing is to analyze findings or attack paths, etc. Another thing is to say, okay, now that you have this, generate the fix, which is the Antropic models are kind of better sometimes than OpenAI models. But we are not testing every single model all the time because it's almost impossible.

58:46Patrick Gray: Man, I think this has been a fascinating conversation. I do think we're going to see some absolutely insane like Terraform disasters as a result of people getting like vibe-coded Terraform to fix the results of scans. Like this is going to, look, everything's new again. It's fascinating. Tony De La Fuente, it's always great to chat to you, my friend. And I look forward to talking to you again soon. Thank you, Pat. See you soon. That was Tony De La Fuente from Prowler there. Big thanks to him for that. Big thanks to Prowler for being this week's sponsor. And that is it for this week's show. I do hope you enjoyed it.

59:24Patrick Gray: I'll be back soon with more security news and analysis. But until then, I've been Patrick Gray. Thanks for listening.