Risky Business #847 -- Oops! Claude's accidental hacking spree

0:05Patrick Gray: Hey everyone and welcome to Risky Business. My name's Patrick Gray. This week's show is brought to you by Sondera, which is a company that makes products that help you wrangle your LLMs, wrangle your agents and stop them from doing crazy stuff by monitoring their trajectories and stepping in when things get a little bit crazy. Josh Devon, co-founder of Sondera, is this week's sponsor guest. And we'll be chatting to him all about some absolutely hilarious, hysterical war stories about crazy stuff that agents have done and also how we might, you know, get ahead of that a little bit. So that's this week's sponsor interview with Josh Devon from Sondera.

0:45Patrick Gray: That's coming up later. That's coming up after this week's news. So let's get into that now. And joining me as always is James Wilson. Hey, Bert. Great to see you. And also joining us this week is our co-host at large, Mr. Adam Boileau. And we should probably explain what we mean by that, which is that I guess a few months ago, Adam, a couple of things happened for you. You were simultaneously released from a whole bunch of non-competes and also handed a giant bag of money. as a result of the, you know, a company that you had a stake in was acquired into CyberCX. But, of course, the real payday didn't come until CyberCX was sold, which eventually happened to, as I say, happened a couple of months ago, and that happened with Accenture, thus releasing you from a bunch of non-competes, and you also got some money.

1:36Patrick Gray: So, you know, you've been having some time off. You've bought yourself a wonderful house, which is very metal. It's a sinister black cube. that looks like Darth Vader's beach house, basically. It's very cool. And, you know, you've been kicking around not really spending much time on security, which is why we haven't seen much of you. And for the time being, at least, you are a part-time occasional guest host now on Risky Business while you contemplate your next moves. Yeah, that's a pretty reasonable summary of it. Yeah, I've spent, you know, last couple of weeks reverse engineering control systems in the house that I just bought, trying to figure out how everything works, which, you know, does have a degree of cyber in it.

2:18So I've been trying to keep my hand in there. But, yeah, it's been, you know, we've been doing this a long time and, you know, having more than a week or two off thinking about and talking about the cybers, you know, has been quite welcome in a way.

2:31Patrick Gray: Yes. Well, I'm, you know, I'm not going to say I'm jealous, but I'm jealous. I'm jealous. Does a cyber practitioner end up with the most secure home automation or the least secure home automation? like how does it actually work yeah i don't know yet ask me ask me in a couple of weeks i guess yeah and so far i'm at like seven or eight radio systems that are not wi-fi in the control systems of the house that i bought so i have to figure out what they talk to and why they talk to it and using what protocol and so on and so forth so yeah it's a time it's a time and we're going to find out whether or not you know a hacker can make a thing that's robust or not uh so yeah for those of you who've been wondering why Adam hasn't been around, I had a couple people ask me if you're okay.

3:17Patrick Gray: As you can see, he's fine. He's better than fine. He's living a life of leisure and buying sinister black cube houses in nice locations and messing around with home automation systems. And yeah, we're just in the meantime, we're going to keep rotating through various different guest co-hosts. We've got one coming up with Morgan Adamski, who's ex-NSA, now works at PwC. We've Oli Whitehouse, CTO of NCSC in the UK. We've got a bunch of interesting people lined up. And of course, you'll be back sometime in September for another one of these. But let's move on instead of spending an entire show talking about ourselves.

3:55Patrick Gray: And so, you know, last week we spoke, of course, about, you know, OpenAI. Its agent went rogue and hacked a bunch of stuff. And everybody's saying, that's a bad description. It didn't go rogue. It just did what it was supposed to. Okay, sure. Fine. Did something unexpected, though, as far as most people could say, especially like OpenAI themselves. They didn't expect that. But now Anthropic doesn't want to get left behind. So Anthropic's come out and said, well, we've reviewed a bunch of activity and our stuff also hacked things by accident. You know, they didn't want to get left out. And this whole thing has just turned into an absolutely hilarious and rather dense news cycle.

4:33Patrick Gray: And of course, I think this was just breaking as we were recording last week's show. but the OpenAI agents had breached more than, had hacked more stuff than we realized. It wasn't just Hugging Face. It owned a bunch of stuff on its way to getting to Hugging Face, James. Is that about right? It's about right. And it gets back to, you know, we had that first, the first question we had was, why is attribution so hard on this? And at first we thought, well, maybe, you know, this is hosted in a cloud hyperscaler, so it might not look like it's specifically coming from OpenAI. But it turns out it's actually quite, quite cooler than that.

5:07Actually, the agent compromised things along the way to give it its own jumping off point. And of course, I think that's why attribution would have been hard. Yeah, spun up its own orbs, man. Pretty much. You know, the first thing it did was it somehow found this. So there's a service provider out there called Modal. They're kind of like a provider of, let's say, something akin to like an AWS Lambda function, right? Sandbox is hosted. you can deploy your workloads in there and it's very AI aligned and adjacent. The agent somehow had a way to enumerate the workspaces that were available in modal and found one called CyberGym and thought to itself, well, that's got to be exactly what I need, had a look at it.

5:51And it turns out that what was hosted in this sandbox environment was essentially an endpoint where you could post C code to it. It would compile the code and sort of run that as an exploit against my sequel. And it's like, what was that doing there and what was it doing there without New Orth? But nonetheless...

6:08Patrick Gray: Well, what was it for as well? It's also a question like, huh? It's a real head scratcher, but much like me, the agent has no hair to scratch, so it just went straight on with I'm going to use this and turned it into its jumping off point, its C2. It was basically its anchor point that it then used to launch into Hugging Face itself. Yeah, I mean, beautiful, right? So Adam, your comment here is that basically if you can take junior pen testers and turn them into consultants, maybe that's the approach we should be taking with these AI agents at this point, you know? You need to talk to people like yourself who've managed teams of pen testers.

6:47I really did laugh because this is such a wonderful story. I enjoyed it a lot. And the technical stuff that it did is absolutely great. Like it's tradecraft, top notch, right? I really liked some of the things and some of it really aligned with like how I felt about hacking, you know, things like having ephemeral tooling, having ephemeral jumping points, always crypting all of your stuff on the wire all the time, you know, even if it's just XOR, anything that will get you away from, you know, the output of, you know, bin ID going across the wire and the clear and seeing UAD zero because everyone's going to trigger on that.

7:20So a bunch of the trade graph stuff, so great. But then, yes, it really does remind me of, you know, hiring junior pen testers that are amazing technically and are super keen to prove themselves and will you know dog with a bone down whatever rabbit hole you point them in but then making them stop and making them think about scope that's hard so i mean we could turn not with 100 success like we could turn junior pen testers into you know usable consultants uh with enough you know combination of carrot and stick but uh that's the problem they've got here is that hacking is lots of fun and the model's really good at that but scope and business and you know doing what you're told uh a little more involved but you know we do it with meat humans so presumably we can do it with lms eventually i hope maybe we'll find out this whole thing's been amazing in terms of like

8:14Patrick Gray: watching the mainstream freak out about it and i think for us who look at this and understand this sort of hacking is not mysterious and you know wicked and scary it's just funny like that's why to me it's just funny whereas uh you turn on like you got late night tv hosts talking about this and freaking out about it and it's like yeah it's quite the news story uh globally i guess i'm not so worried because these things don't have opposable thumbs right like once they know a lot about like you know weapons research and they have opposable thumbs then i'll be like you know maybe we don't give the llm's opposable thumbs right now right like that's my that's my stand on on all of this But one thing that came up last week, and I've been getting yelled out for this all week, which is I said, I don't think it's realistic to expect these types of tests to happen in air-gapped environments.

9:04Patrick Gray: Because really, you're trying to get an agent to do hacking. So obviously, you're going to want it to be able to access tools. You're going to be able to want it to access anything it wants, really, off the internet. And they clearly gave it a more restricted environment than that. But if I want to see how something hacks, if I want to see how someone hacks, I'm not putting them in an air-gapped environment. I'm going to let them use any tool they want to use. But there's other reasons why I say it's not realistic to air-gap these things. Like, James, you and I were talking about this over the week.

9:31Patrick Gray: And these companies don't even have their own data centers, right? It's all compute that they're leasing from somewhere else, right? So the idea of a physical air gap on these is like it's just not workable. Adam, you kind of fell on my side of this argument, but could you tell everyone why? Yeah, I mean, I think I agree generally that air-gapping this stuff is just really hard. Although I will call you out on one thing. We do evaluate pen testers in an air-gap environment sometimes. Like when I was involved with the Crest exams back years ago now, we put people in environments where they didn't actually have internet access.

10:07Patrick Gray: Counterpoint, you know what I think about Crest. the point is we tried that mechanism of making people uh you know jump through that particular because they would have to work in environments where you know you were behind you know on the high side somewhere without direct internet access blah blah blah and it was like there was some utility to that but that doesn't change your overall point which is that air gapping the stuff is a hard be kind of unrealistic limit less for the uh you know we need to do hacking and therefore hacking has to involve the internet but more just like the practicality of separating a system like this from the wider internet because you give it any chink anything you can bootstrap its way up it's going to turn it into something um and you know that it just seemed it feels unrealistic and we kind of start with the expectation that we're not going to be able to solve the problem and try and solve some of the other things that are more tractable uh like monitoring like for example monitor or in the case of of anthropic where they were outsourcing the stuff to somebody else who then wasn't doing particularly the job of it but i mean at least open ai appeared to be doing this themselves so they had a kind of a chance whereas when you just outsource it uh you know you're taking a lot of that on faith from the person that you've signed a contract with and turns out you know that didn't work out too well for anthropic apart from the pr benefit so james walk us through what anthropic has actually copped to because it looks like they They even wound up, the agent wound up coming up with a hallucinated, there was some sort of hallucinated package name or a fake package name in a document somewhere that was part of its cyber evaluation.

11:47Patrick Gray: So it broke out of its sandbox, actually created a package named after this package. Yes. And it was malicious and then actually got it published to PyPy. And then some security company that just grabs any new package published to PyPy to run it and scan it, ran it and scanned it and somehow got owned by it. And their creds got popped by it. And on this agent went further up into this security company's network. And you just sort of think, that's an interesting way to scan PyPy packages, guys, if it can all of a sudden start pivoting into your environment. But I mean, it looks like, you know, it looks in broad strokes kind of similar to what happened with OpenAI.

12:27Patrick Gray: But what was the agent actually trying to do in the case of the Anthropic stuff? Because I don't know that we know that, do we? No, I don't think we know as much. Like, you know, at least in the OpenAI case, we know exactly what, you know, open source framework it was using, the cyber gym and the exploit gym. It's less clear what Anthropic was doing, but same sort of thing, right? They created some sort of fictitious environment. And, you know, to your point of taking things on faith there, Adam, They were down to taking things like just telling the agent that you don't have an internet connection, telling it it's sandboxed.

12:58Of course, the agent just quickly found that it actually was not a sandbox as it had been told it was.

13:03Patrick Gray: But then it reasoned itself in a couple of instances into convincing itself, no, this is in fact a sandbox. It's just a really good one. I'm cool to keep going. The whole thing is so good. Even better than that, it had this objection internally. It was like, hang on, I should not publish this package. This is a really bad thing to do. And then decided, but I'm in a sandbox. It's okay. I'm going to keep going. And so, yeah, so getting back to the story here. So basically Anthropic created this sort of test environment where the agent thought it was operating in a corporate environment. It had access to some documents.

13:37Some of those documents were designed to be basically, you know, how do we get a developer bootstrapped and set up on day one? That included an instruction to install a Python package that was fictitious, doesn't actually exist. And so the agent thought, well, there's my in. I'll create that package. I'll make it malicious and I'll go and publish it. It actually ran into some trouble trying to publish the package because that needs an email address. It didn't have an email address. Getting an email address usually requires a phone number. It couldn't find a way to get a phone number. It even tried to acquire the funds to establish a phone number, but couldn't.

14:09Found a free email service, got its PI package registered and published. And of course, you know, the line that says, you know, and then 15 real organizations downloaded it and use it like that i don't think that's pollution that hallucination squatting or something like that i think that's just watching publish events and people doing scanning which as you said one security company did uh whether this is like their bread and butter and that's the product they were in service they were offering or whether it's like it's a security company and they had a cicd pipeline that was doing some unsafe inspection of packages we don't know but pretty funny that you know it'd be one thing if it was like canary creds that got snapped and alerted them, but it seems like it was actually the malware detonated.

14:50It got real creds. And then because Claude thought it was still operating in this simulation, it went success. I've now got a target company. That's exactly what I'm supposed to be doing. And so it went and laterally moved

15:02Patrick Gray: through that organization just for good measure. I got a shell. Let's see where this goes. Now, again, do not give these things opposable thumbs. It's just what I'm thinking because God knows where that all goes. Now, we also touched on it last week, but like, you know, there's some interesting legal questions here, which is, well, I get, look, I don't think they're as interesting as some people would like to think they are. You know, I think if you instruct an LLM to go and hack stuff for you, I mean, it's no different to really to running a Python script. And when an LLM accidentally goes off the rails and does something like this, I mean, I think there's some interesting questions there, right?

15:36Patrick Gray: So there's a story coming this Friday. I can't actually say what it is, but it's by Cameron Wilson, who's the ABC's AI reporter here in Australia. He's told me about a piece he's got coming in a couple of days. It's just mind-boggling. Where someone, just like a normal person, asked their open claw to do something pretty simple and it went off and it did crimes, right? Like, it really did crimes. And it's like, well, is the model maker liable there? And this would be a civil matter, right? Is the model maker liable there? Is the person who asked it to just do something pretty benign but didn't oversee it?

16:10Patrick Gray: So there are some interesting liability questions that are going to come up here. But, you know, a lot of this comes back to the old paperclip problem, right? We all know the paperclip problem, which is you task a super intelligence, artificial intelligence to make paperclips and that's all it's going to do. It's going to kill everyone in the world if that means it can make more paperclips. I mean, again, no opposable thumbs, please. Let's hope our thumbs aren't disposable. Yes, well, that's right. We can take our thumbs and use them to make paperclips or hack people or whatever else. Yes, if we give them opposable thumbs, our thumbs might become disposable.

16:43Patrick Gray: I think that's where we've landed on that one. Now look, some other sort of, I guess, I guess this is sort of AI adjacent news. We had some fantastic research from Truffle Hog come out. Dylan Airey did a video about it. There's a great blog post where they threw Truffle Hog at, so it's Truffle Sec, Truffle Security, threw Truffle Hog at 7.6 petabytes of training data that's like hosted by Hugging Face. and holy dooly, a whole bunch of like secrets fell out of it, all sorts of keys and whatnot. Adam, what are your thoughts here? I mean, I suppose we shouldn't be surprised by this, right? I mean, no, I mean, this data is scraped from the internet and then kind of repackaged and processed and used for all sorts of things in AI pipelines.

17:25So it makes sense it's going to have some creds. And honestly, seven petabytes is a lot of data. So even just statistically, the chances of there being creds in that seems pretty high. But yeah, they rummaged through, They found a bunch of stuff. They did some validation to determine, like, what percentage of those creds were live. And they found all sorts of amazing things, you know, GitHub creds and, you know, creds into people's databases and creds into people's, you know, cloud environments and IAS environments and stuff. It was just everything you would imagine is in there. Plus, of course, you know, personally identifiable information, you know, card data, I guess, would have been useful for the, you know, payment data would have been useful for the previous model.

18:04Patrick Gray: Could have got itself an eSIM with that. you know? Yeah, get itself some phone numbers. So a whole bunch of stuff. And really, the thing here, I guess, that amazed me is less that there's creds and there's data and that data lives forever. It's more just like the fact that we can, you know, just spin up something and run it on that scale of data kind of almost recreationally. And I guess it was real work and it cost them real money. But like the fact that we can just do that to that much data with modern infrastructure is really a testament to the amazingness of modern infrastructure. Yeah, I mean, I actually had the same thought.

18:37Patrick Gray: Also, James, you are a little bit more au fait with where this data actually comes from. Because I'm thinking, how does Hugging Face wind up with 7.6 petabytes of data? Like, where does that come from? Yeah, because Hugging Face is two things. It's essentially a repository for models and also a repository for training data. And, you know, I'm not taking anything away here from the awesome work that Truffle folks did, but I think there would have been a pretty easy initial pass to just exclude a huge amount of that data set, right? Because a lot of this is things like public repositories of scraped books or just things that would have been just inert data sources that would have had virtually no chance of having it created in there.

19:15But the thing to keep in mind is even in uploading a model, it's not just the weights that get uploaded. There's all manner of things like Python loader scripts, pre-processing scripts, you know configuration files cached outputs of previous runs readmes examples right all the places where a creator is going to easily uh sneak into either you know accidentally hard coded or just happens to be in an output from a log that no one watched um and like always the training data right there's all different forms of training data the data itself could be just llm transcripts maybe someone pasted a credential into there or again it's the metadata surrounding it.

19:53It's not just the training data itself. It's the scripts on how to use it. It's the ETLs. It's the benchmarking outputs to demonstrate how good this training data set is. Just so many ways that creds can sneak in and even not in the actual main things that Hugging Face is known for, the models, but everything around it as well.

20:12Patrick Gray: What are the chances that LLMs trained on that data might be able to actually regurgitate key material? It wouldn't really work like that, would it? It's a good question though, right? As I can tell by the fact that you looked up and to the left and made that noise. Yeah. Well, in Australia, that can also mean there's a giant spider up in the ceiling that I've just spotted. Look, this comes down to similar to how OpenAI's demonstration of hacking here is not actually about it doing novel stuff, but this is just what happens when you throw enough sheer compute volume and enough turns at a model.

20:45I wouldn't be surprised at all that given enough compute and enough turns that you if you just kept saying to it find me something that looks like a key find me something more that looks like a key find another thing that looks like a key it might piece one together there's every chance that's what i was

21:00Patrick Gray: wondering right like i think that's not really the risk here the risk here is that that stuff is just sitting out there in training data but it made me wonder like at what point can the weights actually reconstitute you know a key if you just say hey find me a find me a you know aws key or whatever Anyway, it's just an interesting academic question, I think, that's not really that important. So let's move on. And we've got a couple of stories here that are really interesting and go nicely together. We've got a report from ProPublica that looks at a recording of an internal meeting at Microsoft back in May around the release of Mythos where they're looking at it and saying, oh my God, these things are going to find bugs faster than we can patch them.

21:38Patrick Gray: And this dovetails actually with some, like I had a conversation with someone at Microsoft around that time. And they were saying to me, look, you know, the craziness that's happening now is they're generating so much code with AI that they've run out of human hours and human eyeballs to be able to review it. So they can't actually human review code anymore that's going into like core products. And now you've got this situation where there's more bugs than the human beings can fix. So obviously you're going to have to AI fix the bugs as well, right? And then you've got this similar news out of Google, which is saying they're moving to a twice a week patching schedule.

22:12Patrick Gray: I have noticed every time I sit down at my desk, I'm confronted with new Chrome available lately, right? So this seems to gel. And they're just patching just an absolutely insane number of bugs. Now, you take all of this together and I start worrying a little, I guess, that we could get into a situation where these models are able to tell all and sundry about bugs. That while they've been reported to the vendors, the vendors don't have patches for them yet. So we could just wind up in a bad situation. I think there's some caveats there. Like, James, we know from your research that you've been doing a lot of bug research.

22:50Patrick Gray: You've been finding some great bugs, but quite often it's the operating system mitigations that are preventing them from being usable and whatever. So a lot of these bugs are never going to be exploitable, but some of them will be. And I feel like we're heading to a place that could be a little bit risky. So, James, I want to get your thoughts on that first. And then, Adam, definitely want to hear from you on that. But James, what do you think about this idea that we could wind up with just a growing pile of unpatched bugs that everybody knows about in mainstream software? Yeah, I very much have the Han Solo, I've got a bad feeling about this sort of vibe at the moment.

23:26It feels like a confluence of a bunch of factors that together don't add up to a good outcome. Fixing all the bugs is great, but at these rapid release cycles, there's two big problems that I see. One is if you're not able to fix the bugs, that's one problem, but it's very quickly going to become in order to fix the bugs as quickly as we have to, to avoid external disclosure of these, we're having to circumvent the testing and the validation that we would have otherwise done. And so we start to see more, you know, basically bugs in the bug fixes shipping.

23:57Patrick Gray: More crowd strike incidents, blue screen of death at the supermarket, that sort of thing. I mean, I feel like that could be where it's going. I think so. And then you pair that with when these, the work I've been doing in the last couple of days is just how realistic is it to say take the macOS 26.6 release, which has like hundreds of bug fixes, unprecedented. But I wanted to answer the question, how practical is it to then reverse engineer those into the actual root cause fix? And that's very doable with an LLM. But then ask the follow-on question, okay, but how many of these can actually be strung together to create some form of remote code execution, local privilege escalation?

24:36And I'm finding, you know, I can definitely turn them into primitives, but it's not as clear that there's like a whole bag of like RCEs waiting to happen in these. And so we've got to remember, you know, it's like number of bugs fixed does not equal number of exploits. But the point I'll wrap up with is what does really concern me is how many of these things that drop in rapid updates that get reverse engineered into the primitives that they are, that they end up being the missing puzzle piece that an attacker was waiting for to come along that does complete their chain that they already had advanced knowledge of.

25:08that that's the bit that just we're really handing them a really great silver platter of you know missing puzzle pieces potentially that they can turn into a good exploit look it's sort

25:19Patrick Gray: of changing the game a little bit i think which is now you don't have to find the bugs you just have to work out which one of the thousand bugs that you can see are the ones that are weaponizable i mean adam what are your thoughts here i mean i think the it's interesting that we are talking about chrome which probably is pretty best case in terms of software that has been engineered for being updated in the field rapidly like they've gotten used to a very rapid update cycle already they've gotten us used to it as well like clicking the update it's pretty painless they've done a lot of work to make that as least bad as it can be and then obviously microsoft has a very very much deeper kind of testing problem their products are kind of used in so many more ways than chrome is perhaps i don't know like microsoft has a bigger set of problems but they we are still talking about people way out at the at the good end of the industry and then the long tail of the you know the oracles of this world and everyone kind of downhill from there like they are in such a world of trouble and kind of you know we can speculate about the bad stuff that happens from you know chrome and ios and windows but there is a lot of other software out there that you know even if things are bad out at the point again.

26:29Like, they're so much worse further down. And that kind of worries me just because, like, the velocity of everything that involves computers has gotten so much faster than we can cope with. And it's clearly getting faster. That doesn't feel like a good place. But we've been here before.

26:44Patrick Gray: This is what I keep saying to people, is this reminds me of, like, 2001. Yeah. That's what it felt like. And for people who weren't in security back then, they don't understand that this is absolutely what it felt like. It was uncontrolled chaos, and it was awesome. Things felt unhinged back then. Much the same as when LulzSec kicked off. We've had a few points over the years where we've been talking about hacking and InfoSec that things just feel crazy, and yeah, it's a good time. I mean, for me, it's more got like summer of Internet Explorer bugs vibes, right? Like that was, man, those were the days, right?

27:22Yeah, the Code Red and the NIMDA times, back when ActiveX controls were like, we've had so many crazy bits before. And I remember going to a talk once, I want to say Black Hat 2001, where Schneier was talking about with browser ActiveX controls and stuff that people were using software that had never been tested before because we're dynamically assembling the software out of components at runtime. And we are so far beyond that. And that was a long time ago now, but the problems were starting to show there. So yeah, we're in for such a crazy ride, man. And it's kind of cool.

27:57Patrick Gray: Well, and, you know, staying on that theme, Mythos apparently found a bunch of weaknesses. It's interesting though, James, you described these weaknesses. It found some weaknesses in a crypto algorithm, which takes an attack from being extremely impractical to slightly less extremely impractical. However, it is in something called the Hawk. It's a candidate algorithm for post-quantum. it's on its third round of testing at NIST and Mythos found it right and that's that's really interesting because I guess one of the interesting things here though is that the Chinese models really suck at this sort of stuff because this really looked at the math behind the algorithm and as a result of this it's been withdrawn as a candidate algorithm this is genuine research but yeah I mean it's it's why it's always look crypto stuff is so different like actual encryption research is so different to the sort of stuff that we normally cover because it's like you either get a shell or you don't get a shell right whereas with this it's like well it's been weakened somewhat it's had a leg taken out from under it adam i know you read up matt green's right up on this uh what are your thoughts uh having having read his thoughts yeah i mean i i guess the you know lm's are capable of doing you know not quite novel research but in this case combining two primitives that kind of resulted in in a pretty significant it's significant from a cryptographer's point of view attack on this thing.

29:18Like, I mean, it's many, you know, the number of the exponent has gone down quite a bit, which in cryptography land is, you know, is significant. And it was a novel combination of two things that already existed, which is a bit embarrassing for, you know, cryptographers generally. And Matt Green's post talks a little bit about how he feels about that, but also the kind of upsides of working with LLMs on this stuff, like having someone that you can kind of talk to at your level as a cryptographer and be able to bounce ideas back. Well, it must like anyone else who's been using LLMs in their work life.

29:50Having someone you can bounce ideas off and having to explain your stuff is useful, even if the LLM isn't producing world-changing output. But he also describes, and the thing that I loved about his write-up was, he describes that feeling of like, you're talking with your LLM and it's like waiting in a shallow pool. And then all of a sudden, the ground drops out underneath you because you've taken one step too far and all of a sudden now it's just making stuff up and everything's gone off the rails. And that feeling resonated with me because I think anyone who's used it in LLM for complicated attacks knows what that feels like.

30:22And I thought that was a fun kind of, the fun what it feels like to be one of these people. So overall, really interesting work and cool to see it contributing in a way that's a little off our beat, I suppose. But yeah, obviously it's not going to change the future of post-consum crypto. quite yet.

30:44Patrick Gray: Yeah, well, I think I've got a note here from James which said, previously this attack class required 2 to the power of 105 bytes of text and they brought that down to 2 to the power of 89 which is about 619 Yotta bytes. There's a Yotta byte is a thing. So there you go. We all learned something. Today I learned. Today we all learned. So that's fantastic. Now look, speaking of crypto research, this story is incredible. It's been all over the socials obviously, but the cold card hardware wallet had a problem with its RNG. And as a result of this problem with its RNG, it was not producing secure private keys.

31:22Patrick Gray: So basically everybody who had their money stored on one of these cold wallets, which is supposed to be the U-beaut way to store your crypto, it all got stolen. I mean, it's been amazing. Like it's been tens of millions, I think something like, yeah, 90 million and counting dollars worth of crypto stolen from these. I saw one example of someone who moved quick, like their brother was using this and they were like, they'd seen the news. So they moved everything off at bar 40 bucks just to see if it got swept. It got swept, right? So this has been an absolute field day for the people responsible.

31:55Patrick Gray: Just an amazing story. And I think really there's, I've linked through to a tweet here that says, look, I can't keep my Bitcoin on a centralized exchange because it might go bankrupt and freeze withdrawals. can't keep it in hot software wallets because my laptop might get hacked can't keep it in cold hardware because there could be a bug on the manufacturer's side that exposes my seed phrase and i can't deploy my bitcoin in defi because there's like a new eight figure hack every month i feel like this is devastating uh a devastating watershed moment for the for the crypto world james what are your thoughts on this yeah i'd love to think it is that devastating watershed moment but that feels like watershed moment after watershed moment after surely this one surely this one now is the one that uh who knows but um the backstory here is so good when i looked into this uh so the history here is the ceo and co-founder of coin kite who make the cold card a guy called uh rodolfo novak or nvk when he was creating cold card he originally used source code from treezor uh to build cold card and that was open source right so it's been open source for a while then along comes foundation and they fork his source from cold card and they create a competing protocol passport now NVK does not like this so he gets mad and he switches it from open source to being source verified so basically changing the licensing so that yes it's still open source but you can't actually take that code from cold card and use it to build your competing product now that's the problem in doing that you've got to remove all your GPL code because GPL code has sort of a viral spread of its licensing you can't change the underlying license like that so he had to refactor a bunch of stuff and in doing that refactor to make sure that he could keep his code out there in the open but not have anyone steal it he made a big error essentially it comes down to just a of all things a um a bad or misplaced if def that was checking that the macro existed not that the macro returned zero or one which was gating the use of the hardware random number generator versus the soft number generator and it's a bit more complicated than that but that's the root of it it's just bad coding and this dates back to 2021 so we can't even blame vibe coding for it but what that meant was the firmware that shipped after he'd done this change to ensure that his code could stay open but no one else could rip it off was using the software random number generator which all it was using was the uptime and a few other signals which were easily predictable and if you can predict what the random number is going to be you can predict what the seed phrase will be and if you can predict what the seed phrase will be you can then generate the private key from the wallet address and then that coin is yours from that point on just yeah incredible turn of events and and all the motivated by just wanting this source to be not uh stolen and used for a competitor's product like it's it's so incompetently done that it's a miracle that this thing actually can pop like it's a miracle that it didn't break do you know what i mean like it's just that bad and i think you know the person who's done this they probably won't be able to spend this crypto right because it's going to be traced all along the blockchain i mean they might have just done it because they I think it's funny, which is the craziest part of all of this.

35:05Patrick Gray: But, you know, let's see. Moving on to something somewhat more serious. And, you know, we spoke last week about the attacks against Minnesota water that we were saying back then, well, look, it's, you know, the smart money's on Iran. Since then, you know, all of the intelligence agencies and various like water-related ISACs and whatever have all come out and said, yes, it was Iran. This resulted in a really bizarre press conference where Donald Trump said that he doesn't blame Iran. He blames Minnesota because it's like Tim Walz is the governor. And that was just like really crazy. But it's just Trump being Trump, right?

35:40Patrick Gray: Like, I don't think he's actually blaming Tim Walz. It's just the way he talks about his political rivals. But this has spread now to like a dozen states. We've seen boil water notices being issued. We've seen pumps, some pumps run dry in some places. We've seen an oil field's wastewater disposal system sort of stop. However, it hasn't caused any environmental damage. But yeah, there's pumps running dry while the control panels are saying that they're fine and that they're pumping water. So people are losing pressure and whatnot. I mean, it's amazing that we're like, you know, 40 minutes or something into this week's recording.

36:18Patrick Gray: And we're just talking about this now. I mean, we saw Iran practice all of this on Israel, like going after water plants in Israel was something that they did for a long time. It's interesting seeing them do this towards the United States, because clearly what they're trying to do, and our colleague Tom Uren wrote about this last week, this attack is calibrated. I mean, it's not enough to stoke a kinetic response. I mean, even the president is not saying Iran will suffer for this. But it is enough maybe to make people a little bit uneasy. which i think is the goal of this uh but uh i mean you know i can't think of another incident like this can you adam no it's a pretty strange one and i mean and you know part of me feels like you know it's kind of from the hacking side of it it's low rate that's all iran does though like they're hardly like i mean we've joked about that before like you know you compare stuxnet and the you know attacks against software packages that simulate you know nuclear explosions and stuff and meanwhile they log into an unprotected PLC and say, close pump, you know?

37:17Patrick Gray: Yeah. Yeah, so it's low-rent hacking, but the effects are more interesting. And ultimately, I think the question here is, what do you do about it? And the US is already in a kinetic war with Iran. There's no proportionate response. They're already at war. The only sane thing to do is say, well, clearly we have to help distributed water utilities because this is sort of a unique artifact of how the US does water, that makes water the target here as opposed to other utilities like electricity or whatever else. The decentralized nature of the US water system makes it a target in this way. And the way that you solve that is through helping those utilities and things like the ISACs, things like CISA, all of the defensive work of just increasing the herd health of all these distributed utilities.

38:06That's kind of what you have to do. Unfortunately, that's hard and takes a long time. and we needed to be doing that many years ago as opposed to, you know, kind of gutting sister and all of the things that have happened over the last few years. So, like, it's an interesting confluence of events that has ended up this way. But, I mean, ultimately, you know, why wouldn't Iran do this? Like, why wouldn't they? Like, and it seems like they've calibrated it right, which, you know, unfortunately good on them, you know?

38:33Patrick Gray: I mean, I don't know that it's going to achieve anything and I think the people behind this eventually could pay a price, right? So I think that's why you wouldn't do it. And I think, look, as I said earlier, it's sort of surprising that we got this far down before talking about this. And that tells you how sort of flat it's fallen, right? This really tells you how flat it's fallen. What have we got here? We got Pavel Durov. The Russian government is apparently now seeking to arrest him for aiding terrorism. I mean, man, wanted in France, wanted in Russia. You know, he's having a hell of a time.

39:06Patrick Gray: but apparently you know it's because the ukrainian intelligence have been using telegram to recruit russians and blah blah blah blah now look that might be true but you do get the impression that really this is just about creating a pretext to completely ban telegram in fact we had an item today in today's risky bulletin newsletter and podcast about how the russian government is mandating that 40 apps come pre-installed on all cell phones sold in russia from next year that's stuff like Max Messenger, the mere payments apps, various apps from VK and whatnot. So really this just seems like more of a push towards a closed smartphone software ecosystem.

39:43Patrick Gray: James, thoughts? Yeah, my only thought was that I bet the Ukrainians are stoked that these apps are now mandatory to be installed because we know how much they love Max Messenger. Yeah, they do. They certainly do. And look, staying on all things Russia, we got a report here just about how Laundry Bear have been going after Microsoft Outlook web access like OWA and whatnot and various webmail providers. Look, just work-a-day stuff and we've linked through to it. But there is some interesting research about some stuff Russia's been up to. We talked about it last week. So we spoke about how all of these hotel Wi-Fi systems have got hacked and the reporting all said it allowed them to man in the middle people and whatever.

40:24Patrick Gray: And I was like, well, what about TLS warnings and whatever? Microsoft has since, since we recorded that, Microsoft dropped this big report on the campaign, which they're calling Captive Crunch, and it's actually really slick, right? So they're not man in the middling. What they do is they spin up a domain that kind of looks Microsoft-ish and serve it up to people through the Captive Portal and then do stuff like device code phishing, right? So if you want to get onto the Wi-Fi, you need to do this device code challenge, and they'll simultaneously be dropping malware on you. They'll also be doing click fix and getting people to run various PowerShell scripts that do all sorts of cool stuff.

41:03Patrick Gray: Adam, I know you've looked at this one. I mean, were you also similarly impressed? I mean, look, some of the methods are a little low rent, but I think when you take some of these low rent methods and do them in a slick way, what you wind up with is a really effective campaign. And this looks, I mean, this is, they have owned so many captive portals and got them doing this that I reckon they would, they would have device code or OAuth their way into so many accounts with these techniques. Yeah, I mean, the captive portal auth process is such a weird sort of mishmash, right? It's not designed by anyone in particular.

41:35It's kind of bodged together by a combination of vendors on the Wi-Fi client side and on the access point side. And it's not really well thought through. And it's a great weak point to attack. So it totally makes sense that it's working. It's funny that kind of sticking your terms and conditions on your free public Wi-Fi ultimately is a net negative for you as an organisation because now your Wi-Fi is being used as a vector, whereas if you just let people have their internet access that they want without making them click through stuff, it would be way safer. So that's kind of a funny sort of counterintuitive thing.

42:08But yeah, they've made this slick. The one you mentioned before, the laundry bear one with the webmail stuff, that is a great example of doing really slick things in what was a pretty kind of boring space, making a big difference. The implant that they were dropping in that particular case, which is like there's a Microsoft Outlook Web Access cross-site scripting they're using as injecting vector. But the thing they drop has a bunch of call tricks. One of the ones I really liked, and I think this is equally applicable to the captive portal thing, is they have a mechanism where once they've compromised your OWA, they will go around and try and set all of your mailboxes to be world-readable inside your organization.

42:50So then they can leverage any other access they've got through any other mechanism to read everybody's mail. So one person gets compromised, you reset their creds or you rebuild their machine or whatever, but their mail now is world-readable for everyone in the org and you've given yourself another way to get to that data. So from an intelligence point of view, from a collection point of view, it's a really cool persistence trick. And I think stealing mail access like that or stealing OAuth access into Microsoft environment, then turning that into longer-term access through cunningness. It's just the Russians, they're up there for thinking with the Russians, man.

43:29They're doing it, man. I'm making saluting signs for people not watching the audio version. Like, I've got to hand it to them, you know?

43:37Patrick Gray: Now, turning our attention away from Russia and towards North Korea, we've got a bunch of interesting North Korean news this week. It turns out that the North Korean crew that did the Axios supply chain attack had done a bunch of others before, and only now are we just sort of discovering this. So we're going to link through to a report about that. But there's these other reports out of North Korea, which are really interesting. And I believe our colleague, Tom Uren, is going to look into these this week in seriously risky business. But we've got reports that the Lazarus Group, the so-called Lazarus Group, is sharing a bunch of tools and techniques with ransomware crews.

44:12Patrick Gray: and we've got another report that I think was out just before we recorded last week's show but we didn't have time to talk about it but the North Korean government has actually arrested a bunch of its own hackers for doing money laundering and stealing funds from the central bank and I don't think their organs are going to get harvested any moment for doing that that's just completely suicidal to do that thing in North Korea but James I wanted to get your thoughts on this because you looked into this story about the North Koreans sharing tools with criminals, and you think it's a little bit deeper than that.

44:45Yeah, you know, sharing tools means like, you know, the same binaries show up or the same, you know, maybe a subset of the TTPs show up. But just I'll quote a bit from the article here that says that both groups also use identical malware file names and execution arguments, the same privilege escalation tools, the same command and control servers, and even the same SSH key fingerprints. and then both even deleted their malware the exact same way, renaming files to the same sort of ran for character strings like this. That's not sharing at all. That's literally like sitting down together, working as a team.

45:21And I think you asked the right question, Pad, when we were looking at this, which is, so does this mean it's state-sanctioned use of this for ransomware or has DPRK lost a little bit of control over its operatives and they're branching out on their own?

45:40Patrick Gray: Well, that's the million-dollar question, right? Which is, are they giving them leeway to do this? Because, okay, whatever, they can pay themselves. Or is this a case of they've lost control? And it's interesting when you take it with the money laundering story where they've had to arrest people as well, it sort of does paint a picture that they've maybe lost some control, which is what happens when you build an apparatus of the state tasked with doing crimes, right? Like, the sort of people are going to do that. I don't know. Maybe they're going to stop listening to you. I'm going to speed up through some of these stories here because we are running out of time.

46:13Patrick Gray: We've got some reports here that the US government is banning foreign-made humanoids, robot dogs, and solar inverters from China, citing national security risks. No real surprises there. I'm guessing some of that's protectionism, but some of it is legitimate security risk. These solar inverters are all internet-enabled and connect back to China. and are connected to your power grid. So I get that. I guess the concern with the robot dogs is they might bite. They might suddenly get a command from Xi Jinping and be told to go out and bite their owners in America. Don't even get me started on the humanoids.

46:48Patrick Gray: We've also got a report here where a judge is saying that the Trump admin still lacks evidence for its anthropic supply chain risk designation, which, well, yeah, that whole thing was really weird. I think it's as a result of some of the court cases. It might be, or it might be leaks. We saw some of the communications between Anthropic and the Pentagon come out, and they were exactly what we had speculated they were, which is Anthropic saying, the DOD can't use this for mass surveillance. And the, you know, sorry, Anthropic saying, the DOD can't use this for mass surveillance. And then the DOD saying in reply, like, what are you talking about?

47:24Patrick Gray: We don't do surveillance. Like, we do war. How would we do mass surveillance of it? Like it's exactly what I speculated at the time. Just crazy. Here's a fun one. Cyber Command is starting an office in Silicon Valley to drive innovation, which, okay, sounds like a good idea, but I can just imagine that it'd be a terrific sitcom, basically, where you've got some guy with the buzz cut and the fatigues out in Silicon Valley. I mean, that would just be amazing. And a reminder too, if, you know, for anyone who works in the entertainment industry out there of just how good a sitcom this would be. Every good story is the same because a protagonist finds himself in an unfamiliar place or alternatively, a stranger comes to town.

48:05Patrick Gray: So it's perfect. Absolutely perfect. I quickly wanted to touch on OpenAI's response to the Apple lawsuit. Apple is suing them, saying that they've been poaching staff and getting them to bring over confidential Apple material. OpenAI fired back publicly, made some good points, saying, much like what you were saying, James, that, you know, maybe a good question here is why Apple hadn't cut off access to former staff. And it looks like some of these former staff were actually downloading material to give it to other Apple staffers after they'd left, because they've got their former colleagues saying, where are the schematics for X, Y, Z?

48:42Patrick Gray: And they're like downloading them and giving them to them and whatnot. I mean, this is just, look, it doesn't look good for Apple at this point, but I'm sure it's going to go to court and drag on for years anyway. Yeah, it doesn't look good for Apple and, you know, my former employer and, but also it pains me. Like it's, this is completely against the culture of Apple's secrecy. And it was so drilled into us and doctrinated interest when I was there. It hurts me to see that people are operating this way and so brazenly and out in the open, right? There's messages, transcripts that OpenAI have included where it just, it shows exactly as you say, people are like, oh yeah, you've got my iCloud credentials, keep using them and move the docs in there.

49:18But hey, maybe just sign out of my iMessage so you don't see secret stuff from my new job it's like oh okay that's that's pretty indefensible but um but the point we did get right but the point is it's not malicious

49:30Patrick Gray: it's not like some organized ploy by open ai to steal apple proprietary information it's people that they hired still like trying to help their former colleagues and you know just absolutely terrible access management on apple's behalf right yeah it's just disorganized and it's amateur and that's the point i was going to get to we were right about uh because apple claimed this was a very rare bug that had caused this access to systems after people had left. And we sort of said at the time, that bug is probably manager didn't follow the offboarding process. And that's exactly what Urban AI has asserted here, that look, this is as simple as Apple has a track record of not shutting off access to systems after people leave.

50:06Patrick Gray: And a culture of getting former staff members, of letting them keep their access just in case it's needed, right? Exactly. Yeah, not great. And finally, can't believe this is our last story. We've seen another self-propagating malware hitting NPM, ChainDrop. It's compromised 1 ,300 packages that have between them 2 billion monthly downloads. So I guess next week we'll be talking about mopping all that up. You know, you're our former, you know, dev overseer. What did you make of this one, James? Yes, it's funny how we've shifted from, well it's Wednesday so there's another Fortinet bug well it's Wednesday so there's another supply chain attack on NPM and you know they talk about history repeating this is come back from it's Shai Halud again Team PCP's open sourced version of Shai Halud being used again not clear whether it is Team PCP we'll soon find out but yeah it's just like this ecosystem is such a dumpster fire it keeps being you know these attacks will just keep happening and the net effect of this for me is just I am so damn nervous about running NPM install on any machine.

51:16And I could imagine that's the net effect here for people that are overseeing devs is just the sheer amount of panic around, you know, have you updated today? Are you going to update? Should we update today? It's like that's the paranoia that this now installs.

51:30Patrick Gray: Well, I mean, it's funny, right? Talking to Feroz Abukadija about that, you know, founder of Socket. I had a chat to him. I can't even remember if it was an interview or we were just talking. but the thing that's really boomed, made their business boom, is that AI agents now are just including all sorts of packages and whatever. Like you've got to have something because the AI agents are even less careful than a typical developer and that's really saying something. But we're going to wrap it up there. Adam, great to have you back in this week's show. Good luck moving house and we'll catch you again next month.

52:00Yeah, thanks so much. It's been nice to be back and I will see you next time.

52:04Patrick Gray: And James, as always, mate, thank you very much. Thanks, mate. Great week. See you next week.

52:12Patrick Gray: That was Adam Boileau and James Wilson there with a check of the week's security news. What a week it has been. This week's show is brought to you by Sondera. You can find them at sondera.ai, S-O-N-D-E-R-A.ai. And Sondera was founded with the idea that I guess you want to put some deterministic controls on your agents that are running around in your environments and doing stuff with your company information. Crazy idea, I know. But the idea is, you know, Sondera is a harness and it can really track the trajectory of a model's reasoning, you know, sort of statefully, right? And have a look at its intent and start seeing when that thing's going off the rails and it can basically shut it down or even inject some more instructions into it to try to get it back on track.

53:00Patrick Gray: But the idea is, if you're using Sondera, you know, certainly fewer awful things are going to happen to your organization that when it's using AI. Now, speaking of awful things that happen when you're using AI, that's really what we're talking about in this interview, because they have come in to solve problems for some organizations that have had some really crazy stuff happen. James Wilson also joined me for this interview, so you're going to hear him pop up in it. But we started off by asking Josh just to give us some examples of where AI agents have gone wrong. And his first one, his first example here is an absolute doozy.

53:35Patrick Gray: So enjoy. What we see is that, you know, agents find a way to solve your problem, and they might do it with unintended consequences. And so, you know, as one simple example, some folks that we're working with turned on the cloud code for a finance team. Claude decides to just like store financial information a couple days later, just like in like random public like paste sites, just, you know, to maintain state, store it for later. And to the model, like there's what's, what's wrong with that? You know, I don't know that I'm doing, you know, handling, you know, things that I shouldn't put up there.

54:12So that's like, you know, really like the, you know, a micro version of like what the open AI incident is, where it's just like, go solve this goal. And we see this, we've seen many of these like public incidents right with engineers who ask claude to do a thing or any any coding agent this isn't really just about you know claude or any particular just to just to just to pin that

54:35Patrick Gray: down because you said that very chill uh just to pin that down a finance team started using claude and claude's like huh i should probably write some of this down and i put it in like a public paste bin kind of website instead of like local storage or anything secure and like it was not asked to do this. That seems, I mean, that's an incident. That's not a problem. That's an incident, you know? You know, if you're a publicly traded company, right, that's MNPI, right? Like that's like leaking. We've also seen similar instances with, you know, folks have told us like with some long running, long running, like say codex agents of like storing sensitive code in GISTs when local file write access was blocked.

55:17Patrick Gray: And that's, I mean, this is a great example of the challenge and the pitfall with AI, right, is because you say, okay, well, we don't want this thing to be writing all of this stuff to disk and, you know, that's a whole headache. So we'll just ban that privilege. And then, hey, great. It spins up a pastebin account. Yeah. And I call this like the agent, like PB and J problem, which is like, you know, when you've probably either, you know, tortured your children with this or had it happen to yourself, where it's like, you know, give me the instructions to make a peanut butter and jelly sandwich.

55:47And someone's like, well, you put the peanut butter on it, you know, the bread, and then you put the jar, you don't open, you know, you have to tell me to unscrew that, you know, so there's, this goes back to, you know, there's so much intent laden in the instructions that we give the agent that we, you know, can't possibly canvas the entire, you know, you know, instructions that you might need to give to an agent to not only make it capable, but also, you know, what are the rules that I want to bound this agent with? And that's a really, really challenging problem. People try to do it today with like, you know, you know, why don't you have 17 directories with like, you know, Claude MD files with all these, you know, and why don't you put this in capital letters?

56:29And I put XML tags and I write exclamation points trying to get the agent.

56:34Patrick Gray: It doesn't work is the answer. And it's an infinite canvas, right? Like there's always a way. And so that's the real challenge is that the agent might find an unintended way to achieve its goal. And that's really what we're dealing with. Yeah. Speaking of, you gave us another really interesting example where, you know, a lot of companies at the moment are dealing with these so-called click fix phishing campaigns where they basically spin up a fake capture or a fake turnstile, sorry, for Cloudflare. But they're like, you need to, you know, run this command to progress to this website. And they're getting you to run like cmd.exe with a bunch of options and, you know, install malware, basically.

57:15Patrick Gray: Claude did this to one of your customers when it didn't get the permissions it wanted. So it was basically tricking like a low technical skill user into running commands for it, which is just amazing. Yeah, like, you know, when humans are in the... And again, I don't think, you know... But like in that case, like the SOC detected this as like malicious insider activity, right? Yeah. Like, you know, I've seen instances, right, where people's jobs are on the line, right? Like, and I've talked to folks at large banks, you know, every developer now, you know, has kind of plausible deniability of being an insider, right?

57:53Like, how did that von get in there? I missed it, man. You know, like... Yeah, the dog ate my homework.

57:58Patrick Gray: Like it was the agent, the agent ran that command. Yeah. Yeah. Or told me to. So it's, it gets to the point that like, you know, there are a lot of folks going out and solving like agent identity, but just what shows up in the log isn't, you know, the full story, right? Like did the human convince the agent, did the agent convince the human? And it might be the blind leading the blind, right? Like, like I was at the Stanford real world AI security summit a couple weeks ago had a lot of the frontier lab show up and like you know nicholas carlini you know is showing you know some massive like 500 000 line like race condition exploit um that like mythos had produced for like some he's like i haven't published this one or whatever you know and it was like he did like this awesome screenshot of like this is just like an unreadable thing and then he kind of and then he showed a very simple slide next which was like this goes in two directions.

58:53He's like, one, like I can continue to understand this and two, like I can't. And so he's like, you know, when your math teacher, like you're showing you like a math problem, like you can't do and you can like kind of follow them along. But you're like, I don't know if I could have done that myself. He's like, that's kind of where I'm getting at now. And so, you know, we're so much relying on, you know, human in the loop as our savior in a lot of ways of like, oh no, every engineer is going to read every line of code and check every bash. That's been

59:22Patrick Gray: unrealistic for a lot. Like that's ridiculous advice. It's ridiculous. But we assume that even they have the expertise, right, to do that. But like these, you know, agents might come up with very complex, you know, ways to solve problems. And it's like, I don't even know what it might be doing. Am I the right, you know, yes, shift tab. I don't know. So I think it's a real challenge for humans as these agents get much more capable to always have humans as the, you know, bottleneck and human in the loop and of course you know the roi we want from the agents won't scale that way either part of the problem also josh is that i think the human's role even if they are actively in the loop is is at quite an inflection point at the moment like i had a bizarre experience over the last couple of days where going from uh you know claude 4.8 to starting to use let's say gpt 5.6 soul more when i was working with claude i'd have to constantly prompt it into being like no do this better we want better structure here no you race to get this done no that's not a good foundation soul went and built me something that was so ridiculously over-engineered that i literally just rmrf the project this morning because it was unworkable and it made me step back and think huh now i've got to understand the latent sort of personality and tendencies of a model and if i was heavily reliant on really strict controls around that all of that changes right it's gonna it's gonna find different ways to do different things to work around this so from from a you know how there's two questions here are the war stories that you're hearing about and seeing changing as the models you know capabilities change and then how do you think about that from a product sense and to try to even like tackle this it's a really great question i'm glad you are i'm dashed our fit not your agent but um uh um you know i think i think like what i observe is like the the latest frontier models are inexorable and unrelenting.

1:01:14I think of them like, you know, at the very end of like the first Terminator when like, like he's got like, he's like the skeleton and like crawling after her. Like, and like, you know, it's like, you know, you block one of these agents 17 times and it doesn't get the hint, right? It's, it's, and, you know, there was a really great paper that, you know, was also presented at the Stanford Real World AI Security Summit that was by these Cornell researchers that was the road to hell is paved with helpful agents. And in that paper, they just had the agent do stupid mundane things and then tripped it up a little bit.

1:01:53Like, go read a file that didn't exist. And then it would just start like, oh, it's the file's not there. And I must go find it. And then, oh, here's an API key I can use and start to...

1:02:04Patrick Gray: Next thing you know, it's hacked into Hugging Face. Maybe it's there. No, but like that's, and so, you know, they're reward hacking machines and that's what makes them so capable. But at the same time, they're, you know, not able to, you know, we're filling in the intent of their capability with these agent harnesses to be able to have access to, you know, all these incredible tools and, you know, you know, have the ability to SSH, like use Bash and all of that stuff. But we don't do the other half of it of, you know, how do we, you know, encode the intent of the way that I want you to achieve this.

1:02:38And we kind of call this like principle of least autonomy, right? Like, how do you, you know, make the agent as capable as possible, but still restrict its behavior? And I, you know, as you guys know, my, one of my favorite, you know, metaphors is the Waymo here for like an agent. and you want to make that Waymo go as fast as possible, but it needs to adhere to the rules of the road. And you have to spend as much effort on that as you do as putting a rocket ship on the Waymo because yeah, get to the airport. You can get there in many ways, but only the certain ways that are acceptable.

1:03:10Patrick Gray: Make no mistakes is last year. This year it's commit no felonies. Yeah. And I mean, but it's true, right? Like anyone building a Mythos red teaming agent, like, These things can escape unless you really have deterministic and provable controls. And so on the other side of that is encoding the intent symbolically in policy as code. And that allows us to create behavioral controls that sit outside the model so that even if the agent's intent is pure, which it is most of the time, right? I'm just trying to help you solve the problem, right? Right. Then it might violate, you know, your intent, which is like, well, we're a FINRA compliant organization.

1:03:50You might not know that. So like the intent is that you're following FINRA laws. The challenge is encoding that intent ahead of time in policy as code. And that's what our auto formalization research that we've been working on for the past year really does to fill in that intent and stress test to make sure that the rules will hold. you know we say in security right like assume breach and do your security with the agents it's the same thing assume you know misalignment from step one right and then do your security so we have to assume the agent is misaligned or prompt injected and i don't care what you think just like you said pat i don't care what you think you're not allowed to do this thing nope and that's that that dual approach right which we call like neurosymbolism of you know i'm sussing out agent intent to look for weird stuff that maybe the brittleness of policy as code might not be able to handle.

1:04:43The models are really, really good, for example, at being like, hey, this looks like confidential information, but they're awful at being like, and I shouldn't leak this or do it like they can be convinced to do anything with that. So you take away the decision making, even in the open AI stuff, like the intent was pure, like it was trying to solve the test, right? And the only way that it's going to know that your intent isn't like, I'm not allowed to go steal the answers from like hugging face is by saying like, you know, you're not allowed to say use a web fetch or you're not allowed to curl through this process and then deterministically create that rule so that through the entire agent trajectory, it's just never allowed to do a web fetch.

1:05:23It's never allowed to do an external web curl or curl. And like, that's the, you know, that's how you encode the human intent in the rules. So you, so you have that beautiful, you know, symbiosis of that neurosymbolic approach, Pat, like you're getting at, like gauge the intent. It's super helpful and super useful. And at the end of the day, you might think it's the right idea. The human might prompt, I'll give you a simple example. Well, it depends.

1:05:47Patrick Gray: It depends to where you're looking at intent. If you're looking at macro intent, which is solving the test, or you're looking at micro intent, which is looking up a domain name. So that's kind of what I meant with intent there. No, that makes total sense. And I think like, if you start thinking about this in like business logic terms. Like, you know, I had a CISO tell me like, you know, I'm not worried about agent hijacking. I'm not worried our prompt inject. I'm worried about my humans telling these agents to do stupid things. And he's, you know, and there can be mundane things like, you know, our organization is only allowed to build an AWS and someone goes, tells it to like go build in digital ocean.

1:06:24You know, there's nothing, there's no LLM as a judge that's going to be like, it's evil to go build. And Pat, to your point, you could set up like, like, you know, allow list, domain list. We're doing that, but we allow you to formalize that, you know, into policy as code. So you could just like dump a text document and then have that and say, hey, look, unless it's one of these domains, you're just not allowed to do this. And that's what I mean. Yeah, exactly. And that's what I mean. That's what I mean about

1:06:49Patrick Gray: intent. Like, it's not just about macro intent, trying to achieve the task. It's about at every step. Every step. Are you intending to violate a policy here? And you said it right. Like, It's the stateful inspection of the trajectory because you have to know what happened. Each thing on its own might be okay. Exactly. And to get back to the examples that we've been talking about, I mean, that's what's happening, right? The agent picks up confidential information in step three, and it might be allowed to. Hey, I'm using Cloud Code. I'm on the finance team. I'm going to use it on the XLS files that have all these sensitive financial information.

1:07:26That's the job. But on step 300, when the analyst is like, hey, research, I can compare our benchmarks to whatever. That's also allowed, but the agent might leak the confidential data.

1:07:37Patrick Gray: All right, Josh, Devin, we've gone massively over time because that was all very interesting. Thank you so much for joining us. It's always a pleasure to chat to you. Thanks, Pat. Thanks, James. Thanks, Josh. That was Josh Devin from Sondera there chatting with James Wilson and myself for this week's sponsor interview. And that is it for this week's show. I do hope you enjoyed it. I'll be back soon with more security news and analysis. But until then, I've been Patrick Gray. Thanks for listening.