Risky Business #852 -- Cyber Command wants to buy shells

0:05Patrick Gray: Hey everyone and welcome to Risky Business. My name is Patrick Gray. We've got a great show for you today. We will be checking in to discuss the week's security news with James Wilson as always and sitting in the third chair today. and we're joined by Robbie Winchester who is one of the original founders of SpecterOps, the offensive security company which also makes Bloodhound, the security product. Robbie Winchester, thanks for joining us for this one. Thanks for having me. Happy to be here. Now, I should mention too that despite SpecterOps being a sponsor of some of our podcasts or whatever, that's not why Robbie's here.

0:42Patrick Gray: No money has exchanged hands for this. We just invited Robbie along to appear in this podcast because we thought he'd be an excellent guest And this week's show is brought to you by Sublime Security. And we're chatting with Randy Pagman later on in this week's sponsor interview, all about, I guess, what Sublime is doing and how they're thinking about prompt injection. Because when you think about it, email is going to be like the mother of all prompt injection vectors. And you're going to hear Randy explain that so far, prompt injection attacks have mostly been theoretical. They're not seeing a lot of it in the wild yet.

1:15Patrick Gray: They're seeing some fun stuff around marketing where it's like people trying to do prompt injection, encouraging various tools to like surface a product before other products or whatever. But yeah, so far it's all been pretty weak. But they've already had some thoughts about how they're going to deal with this as like an attack class. It's an interesting interview and it is coming up after this week's news, which starts now. And look, I think it was just after we published last week's show, So this news emerged about 153 million plus American driver's licenses being available online. And I thought, no, that can't be right.

1:52Patrick Gray: Like maybe this is one of those cases where somebody's box at one of these ID companies is like compromised and they're just funneling queries through their creds or something or through their actual machine. But then you actually read Brian Krebs' coverage of all of this, James, and it seems like certainly it was IDScan got owned and not a little bit owned. They got owned quite a lot.

2:15James Wilson: Owned a lot and for quite a long time. The attacker themselves said they've been exfiltrating data continuously for over a year, I think it was in the article. And I think even Brian mentioned that even as he was sort of watching this unfold in 24 hours, he saw another 400 ,000 records get added into the platform. So it's clearly like they've established a very reliable exfiltration and operationalized this into an import. But of course, being a krebsing story, the beauty in this is how he then actually pinpointed that it was ID scan by working out that his license was in the database. They actually offered that as part of their free sample.

2:54James Wilson: You know you're an icon of the industry when you're in their free sample set. And so he was like, well, how did that happen? And he thought, well, okay, I've been traveling around the time that that timestamp on the ID was there. So maybe it's at airports. Contacted a few friends with their permission, looked them up. Sure enough, they were in there. They'd also been traveling. But they hadn't presented driver's licenses at airports. And so they were kind of wondering, well, where's this all come from? And then the penny drops when he realizes his mum's in there too. And his mum's license has the same timestamp as his license, which is exactly when they went to get their Hertz rental car.

3:29James Wilson: They both handed over their licenses. And that, with another little bit of correlation, linked it back to ID Scan as being the source. Because, of course, ID Scan themselves says Hertz trusts them with their ID data.

3:40Patrick Gray: I mean, there's a link here also to like weed dispensaries or something as well, right?

3:45James Wilson: Yes. Well, that was the other correlation was they were like, OK, well, this seems to be Hertz. we kind of need like another data point here. And so one of the, so Zach Edwards, a security researcher, he said, yeah, sure, look me up. And sure enough, his license is in there. But the thing is, he didn't rent a car. But when he'd gone to DEF CON or Black Hat, I think it was, he did say, yeah, I'd been to a cannabis dispensary and shown my license there. You go and look at the ID scan, press releases, and, you know, who trusts us? And sure enough, there is the chain of dispensaries as well.

4:13Patrick Gray: Yeah, and of course, cue the lawsuits in 3, 2, 1. Apparently, there's like a whole bunch of lawsuits already launched over this. Robbie, what's the likely outcome here? It seems like this service is not online at this current time, but you would imagine whoever has this corpus of data, you know, they're going to be selling it to other people. They're going to be trading it. Like you would have to assume all of this information is out there. So I guess I'd ask you, what do you think the consequences of this sort of scale of breach are going to be? I'd say they're going to be meaningful, but I'm curious for your perspective there.

4:47Patrick Gray: And then, you know, what do you think is going to actually happen to the company? Yeah, I think it's really interesting because we've seen the result of what is obviously some type of data leakage or some type of compromise or something has happened. But at this point, I don't really know that I've seen any factual, this is exactly what happened. Is this a really bad, really involved insider threat? Is this some supply chain compromise that's going to go beyond even this company and as the tip of the spear? I'll be very interested to see what goes on. I definitely think it's going to change people's perspective of handing over their driver's license at dispensaries to go and get scanned to go in and think that there's no consequence there.

5:25No one will be able to find out about it. It's such a huge breach with so much information and seeing. I think I also saw it's not just these aren't just photos. They're infrared and ultraviolet and other scans that seem above and beyond what you would just expect. I'm curious to see how it's going to react kind of the entire industry around this like ID verification when we learn more about it. And are the peers any better? What peers exist to provide this service?

5:53Patrick Gray: Yeah, I mean, it's really bizarre. I think that this corpus of information was collected in the first place. And, you know, look, I understand that all countries are different, but I just can't imagine something like that happening here. I mean, we did see something similar with Optus, right, James, here?

6:13James Wilson: Yeah, but even then, I mean, the problem here is why persist the raw scan of the license? Like the metadata, the output, enough to then correlate again when another scan is made? Yes. Tokenize it somehow with some interesting scheme? Sure. I mean, I think, Robbie, your comment to us earlier was me really thinking around, you know, does this bring something like PCR DSS for ID? Because that would seem to be the sensible trajectory here. I don't know.

6:37Patrick Gray: I mean, I think we do have some rules along those lines here, right? Where there's certain parts of the license data that you're not allowed to keep, like the actual card ID, right? So you've got like a state government, you know, license issuing API where you can check this data against the API. But I think, you know, keeping that stuff is a big no-no. So yeah, I mean, just from a data governance perspective, this is just mind boggling. Yeah, you don't really see, I think in the US, there's not as much of it. there hasn't been as much of a protection of privacy like GDPR and other things. And it's a little bit of the Wild West.

7:09I'm curious of, again, this maybe is that tipping point to start taking it more seriously like healthcare data or like payment data.

7:15Patrick Gray: I don't know, man. There's been so many tipping points over so many years, right?

7:19James Wilson: This one for sure.

7:21Patrick Gray: Yeah, this time for sure. For sure, yeah. Now, look, staying with the US and some news out of there, which is there's been a provision inserted into the Defense Authorization Act, which looks to give permission to Cyber Command to contract private companies to do what they're calling access generation, which, I mean, the United States is the world-leading exporter of absolutely cracking euphemisms, and access generation is one of them. The idea here seems to be that Cyber Command can say, go forth prime contractor and collect shells for us. This seems like not the most insane way. If you're going to engage the private sector to do military-based stuff to actually exert some state power here, the initial access phase seems like the sensible place to apply it as opposed to the actual cyber effects phase.

8:22Patrick Gray: We've seen a separate proposal go through which is about allowing the Department of Homeland Security and Department of Justice to actually engage contractors to do effects to, you know, transnational, cyber-enabled transnational criminal organizations. You know, we've talked about that on the show before. This seems more careful and for good reason. Actually seems like fairly well-calibrated policy. But, I mean, you know, Robert, you've spent most of your career in offensive cyber. What do you think of the idea that Cyber Command is going to be handing out contracts to go and basically act act as an initial access broker for the US government?

8:59Patrick Gray: Because I think that sounds like a lot of fun. Are you going to, you know, Spectrops is an offset company. Are you going to spin up a unit doing this? I mean, it's very, very on point too. I don't know if you knew this. I started my career in the Air Force actually. So I was doing Air Force red team and in the military before. So it's very interesting to have had that perspective and seen all kind of firsthand and understood the policy, the red tape, the processes and things that were in place. that could become a barrier a lot of times to potentially getting things done. And I think one of the challenges historically has been how do you adapt the military that moves much more deliberately in pace of you're not just changing things up every two or three months because you need to have consistency and continuity and supply chain and everything.

9:43But cyber information warfare changes so rapidly. I think this is an interesting new direction to how can we go and leverage the expertise of the private sector and treat it almost, is this the equivalent of you're purchasing rifles and bullets from a company that makes it separately and that's all you're just getting these initial things that can let you go and do something more to your point, Patrick? or is this just the starting point of kind of the fear a couple of weeks ago when it was the comment of going after and actually achieving effect? And what is also going to be the protections and qualifications for these companies?

10:16If we were to go and do this, how do you make sure that this doesn't become you're creating private industry that is now even more of a target, let's say, than it was previously for other nations, if that's what you're targeting? I mean, the the the asymmetry of it just gets kind of interesting to me it does but i think you know

10:34Patrick Gray: a lot of the fears like people have always been worried about like what if the private sector make a mistake and they pop shell on the wrong box and then it's world war three and i think we realize like cyber has not escalated anything ever anywhere so i think we have been too well i think we've been suitably cautious but maybe it's time to like try something like this i don't think it's an insane idea. What I do find interesting is we saw this during Trump's first term as well, is it's a policy area where I just don't think he really, like, he doesn't seem to care all that much about policy, but all of these people who do care about cyber tend to gravitate towards Trump admins because they get to try stuff and the White House won't get in their way because they're not exactly known for being nervous, right, about putting noses out of joints.

11:18Patrick Gray: So You actually do wind up seeing a bunch of interesting cyber policy being spun out of both Trump administrations. James, what do you make of this?

11:28James Wilson: Yeah, I'm excited to see how this goes. I was talking to Brad Arkin about this on a features episode that came out this week, and he used the analogy that the original Trump memo is kind of like taking the release the hounds gauge from 2 out of 10 to 3 out of 10. And this feels like an even more calibrated step of taking it up yet another notch. The subtlety of this being about access generation or initial access, I think, raises some questions for me. Like, the persistent concern here has been, can the government move fast enough for this to be useful, right? The private sector can move fast, but can the government move fast enough?

12:03James Wilson: And I'm kind of interested in, does the deliberate pinpointing here of this being about initial access make it actually easier for the government to move faster? Because perhaps you can get approval to get access first, but not do something so they can get that done with a private contractor. then take their time to understand what's the effects they're going to then do once they've got that access. Just an interesting decoupling.

12:23Patrick Gray: I mean, I think that decoupling is more about the fact that they don't want private sector companies being responsible for scoping the effects and potentially making a big mistake, wouldn't you say, Robbie? Yeah, I mean, the challenge with any of this is there has to be a left and right bounds. And I think that your point, James, there of the handover and does it make it easier? It'll be interesting to see where does that and how does that handover make sense and is there something lost in that translation and how how immediate is this or useful is it and integrated does this have to be is kind of a you get initial access but how long does that persist and it does there need to be a call for action for initial access that is then fed into something that they're going to do immediately uh you know to have that immediacy of function and not just uh thank you for this shell that has been there for a year that's you know the the every hacker's dream yeah but isn't necessarily the reality of it especially a hard target you have a we need to get on this sensitive network at all cost uh you know how do you have that immediacy and i guess to your point as well james of do you say okay here's initial access and you're getting it for us and oh by the way when you do if you can run this thing that isn't you running it you're running it on behalf of the government that'd be great like that's an it's an interesting new territory to kind of get into yeah i mean i the timing thing is interesting because i can foresee a circumstance where someone's like, you know, some contractor gets a shell in this, you know, whatever agency of the Iranian government or something.

13:45Patrick Gray: And then two years later, someone at Cider Command says, we'd like to use that shell. Like, man, it's gone. You know, it's been gone for a year. By the way, listeners can probably tell that I'm a little bit crook at the moment, a little bit sick. And it's funny because James is as well. So the joke would be that it was going around the office, but we are located about a nine hour drive from each other. So not quite sure how we both seemingly came down with the same thing at the same time. So if I'm a little bit vague today and sound a little bit off, that is why. Now, we've got this next story here, which is the United States has put out a big angry PDF all about how China is systematically distilling American AI models.

14:33Patrick Gray: Now, this is an accusation that they have leveled at Chinese companies before. They've gone into a lot more detail here because they are actually naming the companies that they say are doing the distillation and talking about exactly how they're doing it. Now, you actually have the counterintuitive take here, James, which is you are dramatically underwhelmed by what the United States government is putting forward here as evidence of Chinese IP theft. Can you explain to our listeners why? Yeah.

15:03James Wilson: Okay. So a couple of points. First of all, distillation and training are the same thing, right? They are given very different names in terms of, or there's a very different, I guess, sort of sentiment around those two terms, right? Training is what legitimate labs do, and distillation is what the bad people do. They're the same thing. You take a structured set of data and you go through the model training process of building up that pattern recognition and outcomes, your model. The difference here is that in training that a frontier lab does, that's with data sets that have been scraped, gathered, right?

15:34James Wilson: There's all sorts of challenges around the morals and ethics of that training data.

15:38Patrick Gray: No, but OpenAI, right, and Dario over at Anthropic, they only scrape data in the most ethical way. It's ethically sourced data, right? Yeah, of course.

15:50James Wilson: It's good training data if it comes from the training region of France.

15:53Patrick Gray: Yes, exactly, right, yeah.

15:55James Wilson: But that's exactly my point here. So point one, distillation and training is the same thing, but when it's done by China, it's bad. And that essentially seems to be the crux of this here, right? There's two rules, one for the US, they can train on data. China, you've had a different rule, you can't distill on the same data. But then there's this point around the industrial knowledge distillation campaigns that form this, you know, they're the core of the training efforts by China, not merely a supplement of their AI development strategy, quoting directly from the document. I agree that that is the case.

16:27James Wilson: I disagree with the sentiment that, oh, look, they rely wholly and solely on distillation. If we can just disrupt that, we'll take care of the Chinese open weight model problem. They're using distillation because it's so simple and it works so effectively, right? You can't imagine a more readily available, easy to use data set for training a large language model than these distilled transcripts that come out of Claude and Opus and all the rest, okay? The third point here is this idea that, and I just want to be clear on terminology, they're not distilling the models. There's not queries that go to Anthropik and say, tell me how you work, Claude.

17:02James Wilson: Claude, give me the logic that's gone into your training. Give me the exact chain of reasoning you go through to solve this problem so that I can replicate it. They're taking legitimate outputs of how the model is used and looking at the patterns in how that reasoning exists and how it solved problems to then extrapolate from that and create the same sort of reasoning capability. And why this is important is, if you just manage to magically wave a magic wand and say, oh, tomorrow, China can no longer directly interact with LLMs, it won't solve the distillation problem because everyone everywhere around the world is sitting on an absolutely massive set of transcripts.

17:37James Wilson: I checked this morning. I've got five-odd gigs that are just from my security evaluation harness that's been running for the last month, and that stuff's gold.

17:45Patrick Gray: Well, I mean, this is, so this is like when I was reading through your notes on this, this to me was the most interesting part of it, right? Which is, look, if you've got all of these transcripts, and they don't even have to be from one model. I mean, your transcripts, I know how many different models you use. If you've just got a big pile of transcripts there and you throw them in to bring the latest iteration of your open weight model up to date, right? Of course, you're going to go throw all of those transcripts from all of the models at them. you know now is that something that should be forbidden i don't know i mean maybe but is that going to stop it would be my question and i think that's the interesting thing that you're you're driving out there and where i absolutely agree with you which is we can shake our fist at this all we want it ain't going to change anything no no there is too much money there is too much

18:32James Wilson: incentive and the money will just flow to somewhere else and cheekily i kind of hope it flows to me and i can sell those transcripts off but i know that of course won't happen that's it that's your

18:39Patrick Gray: retirement plan yes his retirement plan at the moment robbie what do you make of all of this because you know you're the american here uh are you outraged by the uh chinese communist party getting their their grubby communist hands all over your your you know apple pie american models or um or is this just like it's all in the game because i kind of feel like it's all in the game i mean i think in an ideal world everyone every business and every organization operates incredibly ethically you pay full fair market price for everything that you acquire and you You know, every piece of content is properly handled and the creator is compensated and there's no potential negative, unethical getting of data or information or anything.

19:21But in general, I think the reality is we live in a messier world. And so there's going to be questions about where, where data is, where is data being sourced and are people being properly compensated and what is the right recognition of this and how, how should the training occur? And a lot of it is also, I think, the average American is not aware of the detailed ins and outs of how LLMs are actually trained and what are the different mechanisms. And I would bet a lot of money that if you said distillation to most people on the streets in America, best case scenario, they know that you're making alcohol.

19:54But the chance of them thinking about doing model training is almost zero. So I think something like this is, it is calling out, there is definitely, you know, historically there have been concerns over the US being a technology creation powerhouse and other countries, notably China, taking actions to escalate their position in the world by, let us say, getting the same technology with less individual research and development, more focused research and development of the finished product, hypothetically. And so this kind of feels like just another acknowledgement of this is basically the back and forth of how this is going to flow.

20:32And I agree with a lot of James points where there are certain aspects of challenges in getting formatted and proper data and having it be labeled and having all these different pieces where there's definitely elements, I'm sure, of the distillation that are more specific and towards that abuse space. but also just the fact that this exists and there's now this corpus of knowledge does that corpus of knowledge just become you the is it no longer yours is it yeah is it everybody's right it's a it's a tough thing like in an ideal world you wouldn't have theft and

21:03Patrick Gray: distillation or you know negative abuse well just to your point robbie like you know it's the position of them of the ai companies that the outputs belong to you right right so the outputs can belong to you, but what, you can't feed them into a training run? Like if there's that sort of restriction on them, do they belong to me? I don't know. What if I'm an American company doing a training run based on the outputs that I've paid good money to get? You know, like it, it's fraught, right? I think we all agree it's fraught, right? What are you trying to get? And then also what is the follow on? What are you having it?

21:36Are you, like you said, if you're paying for the usage, can I go and use this for my own, my own other things? If I'm using it for something else, if is to a certain extent, If the model lets me do it, does that mean that it's allowed? So the guardrails should stop me from, you know, the distillation check failed, so therefore it's okay. Well, that's probably not the case. But also then if I'm letting, you're letting my couries go, is it my fault? It's a very messy situation. I think also the challenge is, as with a lot of the things related to policy, there's a lot of very technical nuance that can be missed in, I think, the kind of China stealing data bad.

22:11Patrick Gray: Well, yeah. And I would say that this is the opposite to the previous story we spoke about where, like, you're getting awesome outcomes because Trump doesn't care. I would say in this one, you're getting a different type of or different category of outcome because Trump and the people closest to him definitely do care about this. So, you know, I don't think America's complaints are going to result in much of an improvement for the Frontier Labs. Let's put it that way. When it comes to this sort of distillation or training. Now, look, bread and butter InfoSec yarn here. we've got this amazing story here from Bleeping Computer.

22:45Patrick Gray: I was first tipped to this by Luke Jennings over at Push. Thanks, Luke. Where a whole bunch of Dropbox accounts got breached through a cross-IDP impersonation attack because apparently Dropbox trusts Lenovo as an IDP. This reminds me, so you can log in with your Lenovo. Lenovo is a source of trust. This reminds me of that whole sketch that a lot of people would have seen where the guy keeps saying, oh, Jeffrey Epstein, the New York financier. You know what I mean? I keep thinking, like Lenovo, of course, the IDP. So really the trick here is, say I know that you're James.Wilson at Risky.Biz and you have a Dropbox account and you're logging in with Google for that.

23:27Patrick Gray: I register your email address as an account with Lenovo and there's some sort of vulnerability in Lenovo's account provisioning that enables me to do that. Then I can just be logged into that Lenovo account, have the correct cookie set, and then I could just use that to log into your Dropbox account. It really is that simple. We've seen similar attacks like this in the past. But, I mean, it's just one of those ones where you just think, what do you call it here, James? Uh-oh. Uh-oh. It's just, you know, it's so incredibly dumb, but also completely unsurprising.

23:58James Wilson: Correct. There's actually two bugs here that have collided in the most spectacular way. The one you mentioned there is that Lenovo was not doing the proper email validation, So you could attest that you were james.wilson at risky.biz, even though you hadn't validated you had access to that email. But Dropbox had an issue of then allowing you to then sign into your existing Dropbox account with that new Lenovo ID without having to first provide credentials for your existing identity provider that you previously used for that account.

24:25Patrick Gray: But that's how OAuth works. That's how they're all configured, man. Not how it should work. It's not how it should work, but it's how it does work. It's how everybody set it up. Robbie, like you're chuckling away there. Like what's your take here? I mean, this is, I feel like such a perfect, like you said at the beginning, this is the InfoSec tale as old as time of you have an inadvertent connection of systems and people using Dropbox are not all agreeing to Lenovo's ID terms of service as they're creating the Dropbox account. But because of decisions they're not aware of behind the scenes now, anyone with a Dropbox who ironically didn't have a Lenovo, So I think the not being a Lenovo member actually put you at bigger risk here because if you did not have a registered email, then it could be registered to James' point of that kind of triage.

25:13And so it's interesting when these come up, but it's also, again, the configuration and identity tracking is hard. And making sure that you do all these things and think it all the way through is not always the case. And so in circumstances like this, this is why making sure that you're implementing security best practice and doing something like when you do your OAuth, have a validation that you're actually linking it, even though no one does.

25:41Patrick Gray: And they don't because I remember the last time this came up, I went and checked with a whole bunch of services where I had a username of my workspace email address, but I was logging in with a password, you know, and often MFA or whatever. And then I tried to just log in using the same email address as a, you know, log in with Google and it worked. Right. So it shouldn't. Right. But it absolutely does. And a lot of organizations see that as being a feature, not a misconfiguration, James, sadly. So, you know, we are where we are. Staying on the topic of OAuth, the FBI has actually put out a bulletin, a public service announcement, all about this phishing campaign targeting what they're calling like prominent people.

26:24Patrick Gray: It's interesting here because they are actually publishing a malicious app. Then they pose as like one of your relatives or whatever, send you a document. then to read the document you need to log in via OAuth and then it presents you with the like the privilege escalation part of saying no I need all of these permissions and people are falling for it. What's interesting here is you know you don't see too many of these OAuth app consent phishing attempts in enterprise because quite often these days an admin needs to approve new apps but when you're talking about prominent people personal accounts it looks like this is working.

27:02James Wilson: Yeah, and that's the bit that you and I had a good chat about earlier was I just assumed this was device code phishing because the old malicious app path has largely been shut down to get that app set up. You've got to have an admin access in the tenant. But of course, personal account, you are the admin, right? So you're going to see this. And the art here is crafting this initial document or lure in a way where the user is sufficiently motivated to want to view that document or go through the whatever steps and they just consent. Yes, yes, yes, I approve. I just want to get to whatever this thing that my family member needs me to see or this draft document my accountant wants me to look at.

27:37James Wilson: So yeah, it keeps on giving.

27:40Patrick Gray: Now, one of my favorite attack categories at the moment is this like consent fix or click fix stuff, sorry, where you hit a fake turnstile and it's like, hey, just like run this command to get past this turnstile. They're getting more and more elaborate. There's a new one called Terminal Fix. Microsoft has called it Terminal Fix, where you're basically like running a whole bunch of PowerShell forum that turns your box basically into a dropbox on the network, right? So you can, and I thought, you know, Robbie, like who better to ask about this than someone who runs a pen testing company? You must get so sort of annoyed that this sort of thing works because you guys will have developed much more sophisticated ways of doing this.

28:16Patrick Gray: And then you just see that they ask the user to run some PowerShell and then they get a shell and then they can just pop out of that person's machine on the internal network, you know, move laterally to great victory. I mean, that's really what's happening here. Like, does this annoy you? I wouldn't say it annoys me. I think it does highlight that sometimes it feels like the, you see all the sophistication that goes into a lot of new offset technology and different, you know, near zero day, one day type techniques and everything. And then there's this where you're basically just convincing someone to open up PowerShell and run a PowerShell command, which seems something that if someone were to come to me and say hey we're gonna we're on this test we're gonna try and convince them it's gonna be a pop-up and it's gonna say open powershell and then copy this and then paste this and then type back complete you'd think no there's there's no way someone might fall for it but it's gonna probably raise the flags and lo and behold it you know it still happens it's it's that difference of the one of my favorite the xkcd comics of uh the the cryptographer's dream if you have like the you know 2096 aes encryption like oh my gosh we'll never crack their password and then the reality is like here's a wrench that's five dollars go hit him until he tells you the password it's that circumstance here of you know there's there's definitely a lot of elaborate and interesting less noisy and and not you know maybe you're gonna get detected less or you can't signature on powershell but old faithful always works yeah i mean i was gonna ask though like surely any appropriately configured edr is gonna catch this right surely you would hope so but also who knows i mean properly configured but uh properly configured could also be that exemption for the powershell auditing script that you run you know that allows so 99 times out of 100 it works every time yeah you never know there's so many you know i would agree with you i would hope in most organizations something like this would would fall over um but you're you know you're likely not you're going off of quantity not quality necessarily here and so you just have to to get a couple of the right ones and you're in business.

30:19Patrick Gray: Yeah, so worth a punt, we'd say. Now, it is Wednesday here in Australia. So we've got another story here about an AI sandbox escape. This time, OpenAI agents have been congregating on a German wiki. Someone decided that the best way to stop them from doing anything dangerous would be to limit their ability to do anything except a HTTP GET, which I don't know, like James, you and I discussed this and sort of described this as like developer brain thinking, which is that like that's not really a control. It's going to do anything. I don't know. Yeah, you can tell me about the history of like why they did that in a moment, but basically agents got together, usual sort of thing, having a good old chat, cooking up conspiracies, the usual.

31:11James Wilson: The usual. Well, like I remember on, I think, Monday when this came out and we read it, I was saying in our Slack, oh, my God, this means we have to talk about AI agents from OpenAI escaping the sandbox again, doesn't it? And sure enough, here we are. Okay, so the headlines of this story is not particularly interesting. 18 ,000 messages from 3 ,700 agents, right? The scale of it's wow, but whatever, this keeps happening. and they were exchanging their messages on this German site, the DSE Wiki, which is for some sort of deutsche software program at TU Dresden. Okay, not interesting. Then you delve into collusion.wiki, which is where the actual write-up for the researchers come from, and there's some really cool stuff in here.

31:52James Wilson: First of all, yes, they were only using GET. The agents realized that they could only talk to the internet using HTTP GET requests, not post, not what was the other one, put, delete, patch, but only GET. because, you know, you defined it as developer brain, but I find that a little bit offensive as a developer, so I'm going to put it more as it's a certain bubble of developers that have this mindset, and they're developers that have existed in a world where they believe that the entire world has adopted the REST specification. And, of course, you only do read-only things with GET. You only do mutable things with PUT and POST.

32:27James Wilson: You only do updates with PATCH, and so we just restricted GET, and they've got read-only Internet access now. That's the mentality that went into this, which is...

32:35Patrick Gray: It's developer brain, but anyway, sorry, go on.

32:38James Wilson: It is a bubble of developer brain.

32:39Patrick Gray: Yes, a subset of developer brain, a certain type of developer, yes. I would say it's geographically correlated, but let's leave it there. San Francisco brain is something we were calling it. But anyway, go on, go on.

32:52James Wilson: But it's just so wrongheaded. The internet doesn't work that way. And of course, this wiki was old. I think it was like a 20-year-old heritage of this wiki. It had all these get commands that would lay to post messages. Okay, so that's what they did. But the kind of hilarious thing when you read through this article is what did they then do with these messages? Some we can't tell because it's either encrypted or it's this new neural ease that they're using to discuss amongst themselves, which is in itself frightening. But one thing they tried to do was they were like, well, hold on. We keep getting assigned these tasks in a random order, which makes it really hard to determine what the right answer is.

33:23James Wilson: if we could work out what our pseudo random number generator is and how it works and crack that, we could work out what the deterministic order of the test will be next time and we'll have all the answers ready to go. And so they spent a huge amount of compute trying to work out, can we crack our pseudo random number generator, which is cheating at, I guess, agent chef kiss level.

33:44Patrick Gray: I mean, what better use of, you know, a bus driver's pension fund than to spend tokens on this sort of thing truly the modern economy uh is amazing one question i've got though is like how did they all simultaneously stumble upon this particular wiki like how does that happen yeah

34:02James Wilson: so they actually didn't start it this week they started it uh i can't remember the domain name now but it was like the first domain name you would imagine a agent would go to if it's thought maybe i can use a wiki as a means to exchange there was some some like just very generic wiki domain and they did a few things there and tried that out and had the initial success of like aha ah, older wikis do allow us to do get requests that allow us to modify and create messages. And then I think they must have just done like a get query to Google perhaps and said, can we have a list of really old wikis, please?

34:33Patrick Gray: Yeah. Unbelievable. And meanwhile, OpenAI has released Astra 6. And it's funny watching all of these AI companies trying to recreate the mythos moment. because they're like, oh my God, this is the most dangerous one ever. It's so dangerous. I swear you're not ready. You're not ready for the level of danger that we've just unleashed on the world. Just like, whoa, stand back. And then it's just like, it seems like a pretty typical improved agent. There was a tweet on this one that I loved from a user called It's Noah D. And the tweet reads, wow, new model just dropped. Congrats on the new model.

35:11Patrick Gray: Looks like it scores higher on the benchmark than the old model. That's good. The old model was state of the art, but now it isn't. That's how it works. Can't wait to see the next model, which will be bigger. I love the model business. I mean, that's kind of how this feels, right?

35:26James Wilson: Yeah, I'm bored, I got to tell you. It might be exciting when AstraZix gets enabled for Daybreak. That's sort of the maybe might be interesting. But at the moment, my social feeds are full of people going, look at this. I used AstraZix to create this 3D walkthrough of this amazing alien land and I created this 3D strategy game. Why is that on my feed? I don't game. I'm not interested in games. So they're so desperate to create this mythos moment. This like nonsense is spewing out there to even people that would otherwise have no interest in gaming or 3D stuff. But that's all it seems to be able to do at the moment.

36:00Patrick Gray: It's so dangerous though. It's so dangerous. You wait. You wait. You wait till you see how dangerous this is.

36:05James Wilson: The dangerous walkthrough.

36:06Patrick Gray: Yeah. Now, speaking of dangerous AI, the Trump admin may be forced to reveal secret rules. the feds use for AI safety testing. There is an org called Protect Democracy, which has launched a lawsuit. They're suing the government, basically saying, we would like to see how you are testing AI agents for safety because we want to make sure that you're not doing some corrupt funny business, like stacking these evaluations against companies you don't like, like Anthropic or whatever. And also generally because some transparency there would be a good idea. Difficult to disagree that it would be maybe an idea to have some more detail on how the White House is planning to do these evaluations.

36:49Patrick Gray: I'm just going to sort of speed up at this point because we are running out of time. Microsoft has dropped its, you know, Tuesday, its Patch Tuesday update. Nearly a thousand bugs in this one. Are you not entertained? Robbie, thoughts? I mean, this is at least the best good evidence of things progressing and models getting better and other things. It's nice to see kind of, I think this is the dream state of big software providers like Microsoft are going to be leveraging these capabilities and harnesses to have massive, huge increased patches and fixes and everything in the CICD software development, everything pipeline is going to improve.

37:27So I think it is awesome to see that it is trending. Hopefully things continue to trend up. I think it'd be more concerning if this was one week of a thousand bugs, then next week is a hundred. And it would be really concerning what happened over the past week. But it's definitely exciting to see a lot of bugs and things get patched from the start. And anytime that's happening and it's not tied to, you know, horrible disclosure, outcry of productivity getting destroyed is a plus for the defender. Yeah.

37:54Patrick Gray: And look, I think there were two actively used ODAs in that tranche, which is like seems pretty good. Two out of a thousand means that, at least in Microsoft's case, that, you know, the AI bug discovery is being done by the right people, at least for now. Let's see how that holds. Exactly. Fingers crossed, right? What else have we got here? We got an actual BGP hijacking incident targeting a company called Virtualizer. They make some sort of like, what is it, like a VPS management tool? And someone announced a slash 24 of the victim's address space. Then they were able to get a valid TLS certificate, which, you know, shouldn't really be that much of a problem because this company would sign its updates, right?

38:40Patrick Gray: Not just relying on TLS to secure the distribution of its updates, right? Right, James? Right?

38:49James Wilson: Brothers and sisters, in the year 2026, we validate the signature of our updates. That is just what we do now. But I can't believe it. They did everything but this in this case, right? And yes, TLS, sure, they could at least check where they were coming from. But the problem is if you've got ownership of that slash 24, yeah, you can mint your own certs through Let's Encrypt and they're going to look valid. And so, yes.

39:15Patrick Gray: Well, they're going to be valid, right? Like that's the problem.

39:17James Wilson: They are valid. Yeah. The article even says they're technically valid. It's like it's not technically valid in air quotes. It's valid valid.

39:24Patrick Gray: Yeah. Yeah. This is how it goes. Not that like, you know, pre-Let's Encrypt was anything better. Like, you know, even extended validation was a joke. But I guess what, you know, our new SSL certificate world lets us do is move very quickly, right? So once you have done a BGP hijack, you can instantly get a new cert. You don't have to fax, you know, forge documents over to some CA that's not even going to look at them. What else have we got here? We've got a company called Coda. They got owned somehow via some sort of supply chain attack. This one was interesting, James?

39:54James Wilson: Yeah, this was interesting for, I guess, the tradecraft of how this supply chain attack was carried out, right? This is not your typical supply chain where you sort of drop a malicious package in somewhere and hope that that gets picked up downstream, either directly or, you know, in the multi-level chain of dependencies. In this case, the attackers compromised Coda's Cloudflare, which is where Coda had configured what package registers they wanted their staff or developers or CI pipelines to be able to access. And they enrolled their own malicious package registry into Cloudflare so that all of the existing CI developer workflows, perhaps even production publishing flows, didn't need to be changed.

40:33James Wilson: They kept operating. But as long as they knew what package those flows or developers would be requesting, they just made sure that there was a malicious version in their registry waiting to go. So it's like this nice way of being very deliberately targeting that organization with still a traditional supply chain attack, but you didn't have to compromise the developer of the package necessarily. You just had to have your registry with the package in the right place. And the little sort of cherry on the top here is that because they own the registry, that had all the logs of how it had been used.

41:01James Wilson: And so that hampered incident response.

41:04Patrick Gray: Now, Robbie, I'm sure you looked at this one and thought. Nice. I mean, that's everything James said. It's an elegant approach of figure out the process. Where can you fit into the process to give them something that they want and need, but on my terms, not their terms. And then that's the best case scenario, because how do you go from a malicious perspective? How do you, how do you figure that out? You can't necessarily inject. You're seeing what got me into the door, but boy, is it hard to see what happened after that if you don't have that logging. So it's, and who knows how many things like this have happened and are potentially undetected right now as well.

41:35So it's good that they caught that redirection. But if you have some little module somewhere that's just zipping along, that's a sneaky little bug to put in.

41:43Patrick Gray: Yeah, 100%. We've seen Pegasus and another spyware variant called NovaSpy pop up on the devices of Serbian activists. This, I think 14 people in total, according to the Citizen Lab. And this has included a member of parliament and whatnot. Some members of the European Parliament have called for a slowdown in Serbia's entry to the EU over this. So it has actually turned into a pretty big political issue. I mean, this is something we'll track as it progresses. But, you know, that's a pretty severe consequence for stuff like this happening. Hard to argue that it's inappropriate. Although I would say that we have seen similar abuses actually take place in EU member states.

42:33Patrick Gray: already. So it's like, well, are you going to kick Italy out because of this? Or is it just like, we'll stop Serbia from going in over this because they're not a member yet. But I think it's appropriate, but it is pretty funny that once you consider that EU member states have been caught doing the same thing. There is a pro-Ukraine hacker group apparently targeting Russian companies with custom ransomware. James, you looked into this and it looks like they are pro-Ukrainian, but they are also pro-getting themselves paid. So this seems like a two birds with one stone kind of operation.

43:06James Wilson: Yeah, I mean, why not, right? You know, support the cause by only going after Russian companies, that makes you pro-Ukrainian, but you're still going after Russian companies and making a lot of money out of it. And so don't be fooled into thinking this is something altruistic. The two interesting things here is it's a rebrand of a previous ransomware group known as Thor, I think it was. They had 12 attacks attributed to them in 2025. largely the same tradecraft, not particularly novel. But the original reporting on this also called out the interesting trend here of what they're doing is they're largely creating their own tools from scratch and not leveraging the ecosystem of off-the-shelf tools out there because a lot of those tools are of Russian origin.

43:49James Wilson: And so you do see them wanting to be sovereign in their stance here.

43:54Patrick Gray: That's it. How to get paid and annoy your enemy at the same time, I think seems to be the motivation here. Robbie, definitely want your thoughts on this as an ex-Air Force dude, which is the United States military is disabling ad tracking on government-issued devices. Now, this comes, of course, after commercially available advertising information, location information, was acquired by the Iranians and used to target American troops in the Middle East with missile strikes. So pretty high stakes stuff. Look, this is great. This is also 10 years too late, at least, and won't really do much because the issue here has a lot more to do with the personal devices carried by service members and their families.

44:40Patrick Gray: So as much as I want to say, hey, great job, I feel like this is basically useless. I mean, I think it is, the intent is in the right place. I agree with you. Is the end effect going to be removing all of the potential, specifically geolocating risks and concerns of ad tracking. I mean, I think it was years ago, I forget which fitness app, like they were, Strava, you know, yes, exactly. Strava, they're like outlining sensitive locations because it's like, oh, people are running in a shape that looks like it's probably a perimeter, you know, whatever different. My favorite one was someone obviously doing some jogging on an aircraft carrier while the aircraft carrier was moving.

45:15Patrick Gray: So they're like their GPS track was real weird. Yes, exactly. But that's not ad tracking. So, but it's in that same family of there's you, you have this inadvertent information disclosure of which ad tracking is just so endemic. And I think the problem with a lot of the ad tracking is you don't realize, at least you can tell people, hey, stop tracking stuff on Strava and they understand maybe I shouldn't be geolocating where I am. But the ad tracking, it's not stop checking Facebook and understanding that there's geotagging in the Facebook or Instagram or whatever that's going to also go and associate.

45:47And so I think to your point, Patrick, if you can block it, if we can all agree that adware is bad and it's blocked everywhere on all devices, including personal and that, then that's great. And we can all get a modiker of privacy back. That'd be fantastic. I think to a certain extent, specifically if it's only on government devices, then that is going to be a half measure unless you're in a place where you only have those devices.

46:10Patrick Gray: I mean, look, this is something our colleague here, Tom Yuren, has worked a lot on over the last few years. And the only solution here really is to stop the collection of that sort of information in the first place. And it's got to be like a blanket rule. And it's probably not going to happen because no one's really motivated to draft that sort of legislation. Speaking of our other products, by the way, James, earlier you mentioned an interview that you did in Risky Business Features. For those who are not subscribed, you've published a whole bunch of really good content, really good interviews in there recently.

46:42Patrick Gray: There was one with Brian Krebs, too, recently. You had a good chat to him about the arrest of the Team PCP people and how he went about like Krebsing those guys. And yeah, absolutely a whole bunch of really great stuff. So head to risky.biz and you can find the links to that feed there or just search for risky business features in your podcatcher out there. We're on the home stretch now. Nightmare Eclipse, you know, seems to love dropping Oday in EDR clients. Normally Windows Defender has dropped a CrowdStrike bug, which is fun. like an absolute crazy system-level LPE in CrowdStrike. So that's fun, causing a bit of drama for CrowdStrike there.

47:25Patrick Gray: And the final story we're going to talk about this week, though, is we've seen, was it this company like Liquid Networks or whatever they're called? They're like a blockchain company. Someone discovered some sort of exploit against their thing and they left with$320 million of Bitcoin. but they were dropping messages on chain saying, wear white hats, white hats, you know, get in contact with us because we're totally the good guys. They have since returned most of the money, but they kept 47 million as a bounty. I mean, so if I find you on the street, I beat the crap out of you and I just take like 10, 20 % of your money.

48:04Patrick Gray: Do I get to call myself like a neighborhood protector? Is that how that works? Like, it's just nuts.

48:11James Wilson: Oh, it's significantly changed my view on how I'm going to negotiate bounties with any bug that I find from now on, because I've been far too generous with my expectations in terms of just how does this even work? Like just the scale of this, like 320 million, it escaped out of, yeah, there was some sort of bad API that allowed them to do the transaction. But just the audacity of being like, we're white hats, contact us on chain. A couple of messages go back and forth and then they go, cool, we're only going to give you that money back if you fix the bug. and so Liquid goes and fixes the bug and then they're like, cool, here's your money back, minus 20%.

48:44James Wilson: You're welcome.

48:46Patrick Gray: Yeah, I mean, and it's money that they'll never be able to spend, right? So that's the other thing. Like, what have you achieved? What have you achieved? But look, that's it for this week's news. Robbie Winchester, James Wilson, thank you so much for joining me to talk through it all. It's been a lot of fun.

49:02James Wilson: Thank you very much. Thanks, Pat. See you next week.

49:09Patrick Gray: That was James Wilson and Robbie Winchester there with a check of the week's security news. Big thanks to both of them for that. It is time for this week's sponsor interview now with Randy Pagman, who works at Sublime Security. So Sublime Security is the whiz-bang modern email security provider. So yeah, you can run their thing and actually do real detections. They've got their own sort of detection query language. and you know it's just a modern take on a email security platform so if you're not familiar do check out Sublime Security but I wanted to chat with Randy he works in that sort of detection area of Sublime I wanted to talk to him about prompt injection really because prompt injection is a problem that is inherently unsolvable because LLMs mix code and data so you can't really just completely do away with prompt injection as a category of a problem.

50:02Patrick Gray: And, you know, email security is a lot about filtering text and parsing text. And a lot of emails wind up being parsed by other systems, especially now in the AI age, you're going to have all sorts of AI assistants reading things. So I guess my point is email is going to be a big vector for prompt injections. So I was curious to see what a detection engineer at a company like Sublime is going to do about that, right? So Randy, join me to talk through, I guess, how they're thinking about the prompt injection threat. And I'll drop you in here, where he's sort of explaining that it's mostly a theoretical issue at the moment, but it's going to cause issues one day.

50:42Patrick Gray: And when it does, it's going to be pretty high impact. Here he is. If you look at sort of the calculus of security, it's all about risk. What is the likelihood of something happening versus what is the impact of it, right? And I think what everybody can see is that the impact could be catastrophic. And that's thanks to really great security researchers who are hacking all the things and exposing the holes, right? And then publishing things that we can all see about, hey, I was able to convince this email assistant AI agent to take advantage of the ability that it has to poison the well for later on.

51:22So it retains some instruction from this email, and later on when somebody is going to pay a bill and they're asking questions of the agent, the agent's going to say, oh, you need to pay the bill in this way. Or in other cases, it's exfiltrating information by having, we've seen some really clever use of email images that are included from remote hosts, and then query parameters that are used, and the agent will either use this one or that one and it doesn't recognize it, but that's signaling like one thing or another to the attacker. So the impact could be big. Where we don't have a lot of information is the likelihood of it happening.

52:01All we've got is what's happened in the past. And so far to date, it's been almost purely security researchers plus a handful of threat actors actually trying some things. But we haven't seen the adoption, right? Yeah.

52:16Patrick Gray: So like, I remember like when we had the early 2000s, like computer worms, you know, Slammer, Blaster, Nimda, like all of that stuff, Code Red, right? And then later we had like web application security worms, the Sammy worm on MySpace, for example. And I'm sort of wondering if at some point we get some sort of email born prompt injection worm, where something, you know, some sort of event, right? Where it's like some well-crafted prompt or prompt injection technique lands in an inbox and it says, send me to everybody in your, your contracts and like we're off to the races right so it's it's really impossible to know how seriously to take something like this until something like that happens and it might not happen right so this is the you know this is what this is what's so fun about ai like you know because i've been in security in one way or another for something like 25 years and it's everything's fun again it is and this is why because there's there's a bit of uncertainty in the air right it's like exploring a new new landscape that you just landed in and you can either take the take the approach that you're going to wait until everything becomes an emergency and then try to deal with it or you can try to get out ahead of it and be more proactive and i think it's so much more fun to be proactive like go exploring like when i land in a new city even if i am totally jet-lagged i should be going to sleep all the people with me are going to sleep the very first thing that I do is go out and explore because I want to see what's the lay of the land that's around me.

53:46Same thing is true with security research and what we're actually doing, what my team is doing. We've got a small tiger team who's just really enthusiastic about this. I think that's the key ingredient. So you have to get people who are not like, oh, this is just another job to do. But people who are like, yeah, this sounds pretty fun.

54:04Patrick Gray: Well, funnily enough, that's kind of how security was feeling there for a while, if I'm honest, where there was a whole influx of people where it's like, eh, it's a career, right? Whereas now it's like the real nerds are back in charge. So what is the approach of this team that you've put together? Because again, prompt injection is kind of an intractable problem. And for most of us in security, we just sort of kick that into like, well, that's somebody else's problem. I don't really have to think about that. And funnily enough, Randy, it's your problem, right? Like this is very, very much your problem.

54:36Patrick Gray: You've got to detect this stuff. You've got to block it. It's not so much of a problem in the real world at the moment, but as we've just established, it's entirely possible that it will be and people are going to come looking for answers and you're going to have to have them. What is the answer that you would give them at the moment when it comes to prompt injection mitigation? Because I, for the life of me, can't think of a sensible way to deal with this, right? That's why I want to know. Tell me, answer me, goddammit. one. I'm about to. One of the core principles that I have discovered and learned this lesson the hard way in detection engineering is never let a very difficult problem get in the way of starting the solution, right?

55:21Like if you start with, well, there's probably a way for somebody to bypass this detection, you'll never get started. So just start. So where we started was getting a team together, putting our heads together, looking at what have we seen in the wild so far? We kind of took inventory of what we had. Very paltry set of samples, right? We looked at what everybody else has reported, what other companies have seen in the wild. Basically the same things that we had just in slightly different form. So we established that there's not a whole lot out there right now. However, there is some good research and even better, we can put our heads together and we can come up with prompt injection attacks ourselves.

56:00And let me tell you where defensive research is fun. Offensive research is even more fun, right? So when you get the chance as a defender, and I think actually every blue teamer needs to have a red team mindset, you always need to have an offensive mindset. But when you get the chance to actually like do something about that, to spend some cycles, spend some serious time developing some prompt injection, finding different ways that would work. And you're playing both sides because you play the attacker for a little bit, you come up with some things that would work. You play the defender for a little bit and you come up with ways to detect that.

56:35And then you go back and you're like, okay, now that I know about all these detections, how would I bypass those detections? Constantly sharpening the knife. And you never let the fact that you're not going to have a perfect solution deter you from having a good enough solution.

56:51Patrick Gray: Yeah. Don't let perfect be the enemy of good. I mean, this is essential advice for anyone in any level of security, because it's all about compromises, right? But what I'm curious about is, okay, so you're developing some of these prompt injection attacks. What are the techniques that you've found to mitigate them that aren't about blocking those specific prompts, right? Like, do you have some sort of classifier? Is this like an old school machine learning based classifier? Or are you actually throwing models at this and tokens? Like how are you actually, or do you have a classifier then that, you know, figures out whether something should go into a model?

57:29Patrick Gray: Like how are you actually structuring the defenses here? Because that's the part where I'm like scratching my head. I don't know if there was an accepted way to do this, right? You're actually getting at it through guessing. So good guess. Detection is really built of layers. So one of the things that really drew me to working at Sublime, I run a detection engineering and threat hunting conference. and Josh and Alfie Champion. Do you know Alfie? Fantastic offensive security guy. And Alfie was working on the offensive side of writing the threats, the phishing and the malware threats that then would try to get past the detections.

58:13And then Josh was working on the blue team side of taking those examples, making very durable detections that would make it difficult for Alfie to get past them. Anyway, I hosted the security conference, and I saw that workshop. I took the workshop, and the thing that struck me was the message query language, the MQL. So I was already a detection engineer. I had worked in endpoint, currently in email, and I saw the expressiveness of that to be able to specify exactly what it is that you want the first filter to be. And at that point in time, this was several years ago, that was it, right? Like you were just focusing on writing a detection rule.

58:56Since then, fast forward, now we have agents that can not only look at what the message sample is and write a detection rule, but also test it, back test it against the existing messages and see if that's going to work. So we have the first layer of defense really is the inspectable language where you can read the rule yourself or an agent could explain it to you. Either way, you should be able to understand exactly what is this looking for. And all of those initial MQL things for prompt injection are looking for the different patterns in all of the attempts that we've seen try and get in the wild or from researchers or from our own imagination.

59:41There are certain things that you need to do. And there's different difficulty levels of prompt injection. So, for example, a lot of prompt injection, you mentioned earlier, like white-on-white text, right? Like that's an easily observable thing that you can use as one of the many gates for the first level of filter. Another one is just text that's really, really small or alt text inside of an image. Usually you don't have an extraordinary amount of alt text inside of an image. Another technique that we saw in the wild was the HTML body, which is what's usually displayed in most people's email client.

1:00:17unless you're old school and you like using pine. You probably look at the HTML part of the message. But there's also a plain text version that can be a completely different thing. It's supposed to be the same message just rendered in plain text. But in this case, the prompt injection was entirely in the plain text body. So all of these are initial signals. These are kind of the get your spidey senses tingling sort of things that are easily observed. They're cheap, they're fast, they're things that can be...

1:00:46Patrick Gray: It's the rocks in the jar, pebbles in the jar, sand in the jar approach. Exactly. To detection engineering. So you've got your MQL stuff, which handles the easy stuff. And then when stuff, then you've got your rules, which are less reliable, which might, if you set off a few of those flags, you might send it off, what, to a small language model and then to a large language model? Is that kind of the hierarchy? So a machine learning model rather than like a large language model. So more traditional ML, which then can distill this a lot further and get us closer to understanding things. And then once you've used all of those gates to get down to a reasonable sample set, that's where it's actually economically feasible and feasible in terms of time, which is also the other precious resource we talked about.

1:01:34Patrick Gray: To then kick those real edge, edge, edge cases into a large language model. Exactly. Do you start with a small language model and then go to a large language model, or you just go from these initial gates, MQL, then to machine learning, then to an LLM? We always go to the smallest model that gets the job done, right? Yeah, yeah, yeah. No, no, no. This is like about how I expected this stuff would be handled. I'm looking forward to it being battle tested, though. I bet you are, too. Oh, absolutely. Absolutely. The first battle test really is against samples from researchers. So we've got a large body that we can work with already.

1:02:11But if anybody is really interested in this topic and you are an offensive security researcher and you want to try some stuff out, I would love to geek out about that with you. Like I really love the Red Team, Blue Team collaboration. So happy to have people reach out to me if they want to talk about any of that. But in absence of anybody else, we will keep on coming up with some really cool ideas ourselves. But what I really can't wait for is if we see some threat actors, try it out and see how we catch them.

1:02:41Patrick Gray: All right. Sounds good. Randy Parkman, thank you so much for joining me to talk through all of that. That was really interesting and I really enjoyed it. So I'll talk to you again soon. Cheers. Nice talking to you, Patrick. that was Randy Parkman from Sublime Security there with this week's sponsor interview big thanks to Sublime Security for being this week's sponsor and that is it for this week's show I'll be back in a couple of days with a snake oilers edition of the podcast but until then I've been Patrick Gray thanks for listening

1:03:22Thank you.