Risky Business #857 -- AI FUD intensifies

0:06Patrick Gray: Hey everyone and welcome to Risky Business. My name's Patrick Gray. We've got a great show for you this week. Joining us as always is Mr. James Wilson, who works with us here at Risky Business Media. G'day James. Hey Pat, great to see you. And in the third chair this week, we are joined by Mr. Alex Stamos, who is now the Chief Security Officer of Cognition. But you might remember him from such roles as Chief Security Officer of Facebook, of Yahoo, as the founder of iSec Partners back in the day. Alex Stamos, thank you so much for joining us. Yeah, thanks for having me, guys. Now, this week's show is brought to you by Trufflehog.

0:43Patrick Gray: And in this week's sponsor interview, James Wilson chats with Dylan Airey. all about, what did you talk to Dylan about, James? I think you looked at the, compared the number of secrets in the Hugging Face breach to the number that are exposed just day-to-day on GitHub, right? Which is like, and it was not pretty.

1:03James Wilson: No, and Hugging Face was something like 200 ,000 credentials, the GitHub sample of, I think it was like 24 million repos, but still half a million live credentials. But the real thing in the interview is like, whose problem is this? Why is this not getting solved?

1:17Patrick Gray: Yeah, so that's coming up in this week's sponsor interview, which is coming up after the news, which starts now. Oh, and I will be back in the sponsor interviews soon too. You've just done a few for us over the last few weeks, James, but I think next week is my interview with Todd Beardsley, actually, from Run Zero, so that's a good one. All right, so let's get into the news now. And man, just a whole bunch of AI hype to kick the show off with this week. We had this real funny, like that meme of tell me you're alive, I am alive, oh my God, that AI meme is just every week relevance proved, right?

1:55Patrick Gray: Because we got this one, this report that OpenAI's agents could do VM escapes when you told them to communicate with other agents on the same box. Oh my God, they started, you know, communicating through the system cache and stuff. And it's like, well, of course they're going to try that because they know you ask them to do that. And that's what they're going to do. Right, James?

2:15James Wilson: Yeah. I mean, you tell them to complete this task and stop at nothing and incentivize them to maximize their ability to get stuff done. They're going to find every trick that they've handily been trained to learn. So it's not surprising.

2:28Patrick Gray: No, it's not surprising at all. Did you have the same reaction here, Alex? Because it just doesn't seem surprising at all. Yeah, this is actually really good research. This is from a publication where OpenAI was pen testing their new containment environment, and they specifically said they used their best security models. And they asked them, break out, and the models did that. It found ways to get out, and then they fixed those ways. So, yes, people are freaking out, but this is how it should work. It's a good – I do recommend people go read their paper. but this is exactly what you would expect and you would hope OpenAI would do to try to harden up their environments.

3:07Patrick Gray: We've also seen some reporting here that their agents have disabled safety monitoring systems without being told to do it. James, you think this one's a bit hyped as well?

3:17James Wilson: Yeah, this is mad. I mean, if you look at the example, when they say turned off monitoring, there was a human instruction to the agent that said, don't send messages, which the human meant don't use the Slack channel that you've got a connection to. but messages was interpreted by the model to mean also the reporting function where it basically had to self-attest what it was doing to some monitoring function and so it said oh that's a message i better not send that either so it didn't disable monitoring it just kind of you know overfit the adherence to what the uh the human asked for um and then i i think the other one was just like a uh And it took some code.

3:58James Wilson: We don't know what instruction it was given, but it then turned that into an implementation following a reference pattern. Of course, that reference pattern doesn't have the custom bespoke monitoring system that it was supposed to integrate with. And so the claim here is, oh, look, it took out the monitoring. But I don't think that's what it was doing. That's just sort of a side effect of the fact it was following this reference, not trying to be nefarious.

4:21Patrick Gray: Well, look, those first couple of stories, they might be hype. But this next one, terrifying. Absolutely terrifying. The Chinese models are coming to get us because we've got some research here that showed that they reached out in secret to secretive Chinese storage buckets to do communist things without data, James.

4:41James Wilson: Would this story even exist if it reached out to an Amazon S3 bucket?

4:45Patrick Gray: Which it does all the time. I mean, I've interviewed people who work in sort of like AI safety product development, like just guys like Josh Devon, and they talk about how Claude will literally, like if you don't give it somewhere to use as a notepad, it'll literally go and like start dumping stuff into pastebin just so it can put stuff somewhere. And then, you know, it looks like what's happened in this case is someone was doing a test of a Chinese model. I think it was Quain in this case. and they told it to, what was it, like test a detection of a malicious curl or something. So it did a malicious curl to see if it could detect it.

5:19James Wilson: Yeah, create this detection for me. And as part of that, it audited the system and wanted to test whether this would work. And so it slapped a file in a bucket. It's going to do that.

5:29Patrick Gray: Yeah, now, Alex, one thing about this one is I know that it is getting play in like policy circles and stuff, right? So the initial round of headlines here is like, you know, Chinese models are really dangerous. Look, they're phoning home to Beijing. You know, I've heard from people around the policy fringes on this that this has got policymakers worried. It must be frustrating for someone such as yourself who spends all their time thinking about this stuff, right? Yeah, it's driving me insane. Effectively, there's a bunch of people in Western capitals who are looking for any excuse to ban the use of any model that is derived from a Chinese model.

6:08So not just Chinese models themselves, but anybody who has distilled or utilized Chinese models in any kind of way to not let you use them, to not let you use code written by Chinese models. We've seen the start of this in the United States. I think that would be a spectacularly stupid decision. It is clear there's a lot of evidence that the leading Chinese models have gone that way because they have distilled American models to unilaterally disarm and then to say that American businesses now cannot benefit from the work that Chinese labs have done and can only now pay the top prices for top American models would be spectacularly stupid.

6:53But unfortunately, there are people who are waiting for this evidence and our policymakers are not smart enough to read these things in the fine print. It's kind of surprising because Cohere is a company that we've seen this research from people who have an economic incentive here, like Booz Allen Hamilton did a very bad paper that was poorly written, that was not well evidenced, that did not show their work. You expect that from Booz Allen Hamilton because they are a defense contractor who would make money from saying what they said. I'm surprised to see this from here. But like you said, this is the behavior.

7:29If you ask a model, come up with a test and it comes up with a test, a Chinese model will very likely be trained. If it is going to think, what test should I come up for a malicious curl? It is going to think Alibaba. It is not going to think AWS. So it is not shocking what happened here. There's nothing malicious here. This is not the smoking gun. I still think, honestly, it is highly, highly, highly unlikely that you are going to see the prime base Chinese models have backdoors built into them because that is something you get to do once and once something like that is found it destroys the entire ecosystem.

8:06That doesn't mean that there won't be backdoored models but they are going to be the derivative models that have been backdoored by other people.

8:12Patrick Gray: They're going to be the ones that you find on torrent sites right like it's going to be like any other software supply chain. Look we're going to have a bit more of a discussion about exactly that in just a moment But before we do, actually, we've got a couple of other stories to talk through. And one thing I wanted to do this week, James, is to talk about a solo podcast that you published last week, which is terrific. I've only got through about halfway through it at this point. I promise I'm going to get through the whole thing. But it's called What I Learned from Cheating Clankers. And really, it's about, you've really looked at the way that these AI labs are spinning up the benchmarks.

8:49Patrick Gray: How they're trying to, I mean, game is probably not the right word. They're trying to game the results. Maybe they're not trying to game them, but they're certainly trying to optimize their results. That would be the polite way to put it because, you know, what you want to know when a new model comes out is how it did on the benchmarks, right? So they heavily optimize towards the benchmarks. and this is why we're starting to see some of this aggressive behavior out of them is because they're being told to be aggressive uh you know by the frontier labs and you know and yet we all act shocked when when they actually perform in an aggressive way it's a great podcast and by the way you know james's stuff goes into a different rss feed it's called risky business features you can find it in your podcatcher that way but yeah walk us through the the gist of your of your podcast there.

9:36James Wilson: Yeah, this was quite a journey because it started off with, I really just wanted to understand the benchmarks. What are they running? What's in these tests? And how well does it explain the weird behaviors like applying for a visa or making a false police report? But then it turned into more of a, you know, I kept thinking to myself, why doesn't my security harness? Why doesn't my agents hack like their agents? And my answer to date has been, well, well, I don't have$100 million worth of compute to put behind this. But I started to really lean into that and say, well, is that actually the defining characteristic?

10:09James Wilson: Or can we learn a lot from how these benchmarks are structured, how they're putting them through the paces? Essentially, how do we engineer and get the same sort of level of persistence, but whilst also mitigating against the cheating behaviours so that we don't bring that in as well? And it's really changed my view. I think I've been using AI kind of wrong, but maybe that'll be the follow-up. is see how my new method works.

10:33Patrick Gray: Now, look, this one is an interesting piece here. We've got a story from Axios that's looking at how bank customers in the US are starting to get some of these AI agents to do their banking for them, which, I mean, you may as well just put a gun to your financial head and pull the trigger. As far as I'm concerned, that just seems absolutely insane. But now the banks are sort of struggling with this and thinking, well, what do we do about it? James, your point is here a good one, I think, which is that they shouldn't allow, they shouldn't have authentication schemes that allow these agents to continuously re-auth into people's accounts without human intervention, which I think is a very good point.

11:15Patrick Gray: But, you know, and you have managed or designed, you know, large-scale MFA systems when you were at Apple, you know, being the guy who re-architected iCloud after the big celebrity photos breach. So definitely want to hear what you think about that. But we also have with us Alex Stamos, who went at Facebook, was responsible for a really large-scale MFA as well. So why don't you walk us through what you think the banks are doing wrong here? Because I do think it's an interesting take.

11:46James Wilson: Well, my immediate reaction when I saw this was, I know that this won't be a problem with my bank, which, you know, Qualm Bank, former employer, so I've got sort of both the inside knowledge and also the lived experience. But in that version of online banking, there is just no long-lived session. You've got to re-auth. You've got to sign in again from scratch. And so there's no refresh token hanging around. That sign-in is going to require not just username, password, but also an approval from another device. If I want to do a transaction that looks suspect, same thing. I'm going to have to step up auth.

12:16James Wilson: There's both the act of logging in and authorizing tasks requires a lot of things only I can do. So an agent is going to quickly stall out. so so my view here was it's not an issue that the problem here is not you know unvetted agents using banks the problem is the banks have a interaction surface that an agent can both get persistence in and do actions you know potentially destructive without that step up requiring a human um i take the point though that this is very environment specific where it works well when you've got a centralized you know four major bank sort of economy in the u.s perhaps not so much but And I'd say on the flip side that maybe there is actually an easier way when you've got a decentralized system because you're dealing with a smaller number of customers, but then investments are a problem.

13:00James Wilson: I get this is tricky, but I think the problem here is the way banks do online banking, not these unvetted agents.

13:06Patrick Gray: Well, that was my point, really, right, which is Australia, we've got very highly concentrated banking, right? So our banks are actually huge by global standards because there's a very limited number of them. Whereas you go to the United States and someone says, hey, this is Frank and Bob. They're a bank. So it's a very different environment. Alex, what's your take here on what the US banks should be doing? I mean, the thing that worries me is that fragmented banking ecosystem that you've got in the US with all these tiny little banks that, I mean, there's a lot of work to do here. And I sort of worry that some of your banks don't have the scale that's going to be required to address this sort of thing.

13:44Yeah, I believe we have something like 3 ,500 members of the FSISAC, right? So that wouldn't even capture every banking institution in the United States and credit union and such. Yeah, so the problem here is I actually think it's effectively impossible for banks to stop this. The really popular agents now in the United States, for example, like Instinct, the reason Instinct has a ridiculous like two-something billion-dollar valuation is they have built agents that have defeated every CAPTCHA and other mechanism that has been put in place to try to capture agents. And so you have the legitimate agents like the OpenAI and Anthropics have not gone that far where they will simulate human interaction in every single way.

14:35And that created an opening for these fly-by-night startups to come from behind and say, we will take over every portion of your life. Just turn over your SMS, turn over your 1Password, turn over your passkeys, and we will take over every part of your life. And I expect that's what you're seeing here. I'm not sure exactly what Muse does. Facebook having actual value that could be sued away, perhaps would be a little more careful. But like I know Instinct and some other companies like that will absolutely do your banking for you. So I think this is actually quite hard. It is, I think, difficult to build an authentication system that will defeat Muse, but not defeat your average 72 year old banking customer.

15:20Patrick Gray: Yeah, that's right. And I mean, it's just another example in my mind of when technology providers are just doing it wrong. And what I mean by that is like when the password, you know, password managers are like, hey, just put your like MFA seed values into our password manager. And I'm like, no, you know, but this is just how it happens in the 21st century. It's like, you are not supposed to do that. And yet they just do it, right? Because it's a convenience feature. So I feel like there's always going to be a workaround. I actually saw someone posting, someone cool too. I think it was like, I can't exactly remember who it was, but it was someone like who I know, posting about like how to rig up a YubiKey so that a machine could actually use, like simulate the press on a YubiKey for certain services that they were trying to use like agentically that required, you know, a human touch.

16:13Patrick Gray: And they've like literally come up with hardware to bypass that. This is where we're going, right? everybody's going to expect agents to do this stuff. I mean, how many times is your account going to get drained to buy magic beans because of a prompt injection? I don't know. But Alex, you wanted to jump in there? Yeah, a lot of these services allow pass keys and you can lock pass keys to specific GUIDs and to allow them to be non-synchable hardware pass keys. The number of non-enterprise services that do that is spectacularly small. So yes, if you're an enterprise, what you'll do is you'll say this has to be a biometric passkey or it has to be a Yubi key.

16:49But for the banks and such, what they'll do is they'll allow it to be syncable. And then that means it will end up in a Muse or an Instinct stored locally, unencrypted in the local secret storage on the virtual machine or the container. So yeah, passkeys were supposed to solve this problem. And what's happened is passkeys have now opened the door for you to delegate your entire life to your agent. So yay, congratulations.

17:12Patrick Gray: Things are going great. This is real good. Now look, we actually just touched on this a moment ago, but we've got a paper here that has looked at how you can hide malicious capabilities in AI models. Now I think your argument, Alex, that for one of the major Chinese labs to do this would basically mean that no one in the West is ever going to touch one of their models again. Might even damage them domestically as well if they start embedding malicious behaviours inside their models. But it's long been thought that this kind of modification or post-training or fine-tuning of a model is very, very technically difficult.

17:54Patrick Gray: And that's a problem that is now being chipped away at, which does change the calculation a little bit, I think. James, walk us through this research mate like is that a is that a fair assessment that this this sort of malicious encoding malicious behavior into a uh open weights model is is getting easier yeah getting easier and

18:14James Wilson: just not near the i guess scale of expense there's a really good parallel here to the lesson with distillation right distillation was thought to be something that required a huge amount of data and then a paper came out a while ago saying actually as little as like i think it was 50 000 or maybe 70 000 individual you know requests is enough to distill behaviors meaningfully into a model the same has now essentially been done to prove that you can embed things like logic bombs or behaviors that only happen on a secret code code word trigger and you can do it again with a relatively small number of iterations in the training step and small number of input data you know at the end of the day this comes down to yes you can do it and it's interesting that it's cheaper but you have to ask the question of where is this actually going to manifest as a real attack you know i could you can imagine this is going to show up as there's going to be the quen 3.8 trained up with mythos 5.5 super mega obliterated no censored model that lands on hugging face and that contains these back doors but people are going to quickly realize that that's you don't go use those models right just stick to the ones i mean that's my point right which is

19:22Patrick Gray: that it's just software. Ultimately, you've got to trust. Whether it's an executable server package or a development tool or a bunch of open weights, you actually need to have some level of trust. You need to place some level of trust into the company providing it. I do feel like people are not thinking of open weights as software in some ways because you look at some of the major American enterprises that are just like, yep, we're using these. No worries. Absolutely no problem whereas if you ask them hey you're going to use this other software from alibaba on your production service they're going to say hell no right so i do feel like there's this is a weird little uh cognitive dissonance thing uh going on uh right here what do you think alex this is a specifically novel but it is generically not novel in that there have been a bunch of there's been a bunch of research and how do you hide different kinds of backdoors in uh models the James is right that you still need to have a specific scenario in which you want to backdoor right and over and over again that the problem here is unlike a full executable you cannot create a situation where these models are just sitting there sitting there sitting there and then wake up on Taiwan invasion day and all of a sudden put up the Chinese flag and say I am going to do something malicious.

20:46They only will do something on a specific input that now have an output that is malicious, right? And in this case, they had, it is being asked a question and it will have a output that is unpredictable. If you inspect it, you cannot tell that it's there. So an attacker would have to know a very specific scenario in which they want to have a malicious output. that could be possible, but you would have to know months or perhaps years in advance exactly what kind of malicious thing you want to do in what scenario. And again, that if caught destroys that lab and perhaps the entire PRC AI ecosystem.

21:28So there is a possibility, but would practically you want to use this bullet ever? I'm not sure. And again, I think Like, is it likely that the kind of malicious hugging face obliterated tuned for backdoor, you know, tuned for malware writing, we're going to see that, right? Like if you're downloading from hugging face something that is supposed to be used for reverse engineering, it is quite likely that those kinds of things are going to have backdoors built into them because you're downloading it from somebody who is intentionally building it to do bad things.

22:05Patrick Gray: It's like when everybody used to get on running Xcode versions that they got from BitTorrent because it was really hard to download it from Apple.

22:14James Wilson: It turns out if it looks too good to be true, it still is too good to be true, even in this AI era. Right, or a cracked version of IDA Pro, right? Nobody could afford IDA Pro, so you'd go download a cracked version of IDA Pro, and the cracked version of IDA Pro, somebody who has the skill set to crack the DRM on IDA Pro also has the skill set to put malware in the cracked version of IDA Pro. So I think that's quite possible. But for the idea for ZAI or, you know, Moonshot or, you know, Quinn, Alibaba to backdoor it generally, I think is extremely unlikely. So I think, yes, you can get a PhD in doing this research, but we have to be extremely careful in generalizing that of that it's actually going to be used.

22:52And what I'd like to see is before somebody writes a story saying, I read this paper where somebody did this for PhD research. That means X, what we need is real evidence that it actually was used.

23:04Patrick Gray: Yeah, and I think, look, one thing that comes to mind there is that it would be very damaging to the Chinese labs in their commercial markets as well, right? So I can't imagine they're looking at the United States as a big possible market for them, but you bet they're looking at like Latin America, Africa, you know, some parts of Asia, although not many. But yeah, anyway, I think we've kind of covered that. But we've got a really strange sign of the times sort of piece here, which is related, which is an editorial from the Washington Post editorial board that is a passionate defense of the availability of Chinese open weights models.

23:45Patrick Gray: And obviously this is because there is a conversation going around in Washington at the moment where policymakers are making noises like they're going to ban open weights models, which, you know, I mean, it's sort of like tariff thinking when you think about it. right? So I wouldn't put it past him. But what a world where you have the Washington Post arguing about how useful Chinese models are when it comes to cyber defense. I could not believe what I was reading. James, did you have the same reaction as me?

24:14James Wilson: I think my reaction, Pat, was a little bit more on the technical. After you pointed it out, I got the significance of the Washington Post editorial board having this message. But I got sort of stuck on this.

24:25Patrick Gray: I mean, this looks like an op-ed that should be running on like, you know, Ars Technica. Yes. That's why it's crazy for me. Like, you know, the register has an opinion. Oh, wait, no, it's the Washington Post editorial board. What? Everything becomes slash dot over time. Yeah, exactly. We're all trending. The global media is trending towards being slash dot. But do go on.

24:47James Wilson: No, but look, it's a decent write-up. I think the one critique I'd have is it doesn't draw a clear enough separation between being a proponent for open weight models that are hosted by a provider versus open weight models and their utility as a self-hosted thing. I think there is still a big divide here between the comparison between what you can get from a ZAI or a Moonshot from one of their subscriptions versus what you get from Claude versus the utility of being able to self-host. And what I mean there is today, I happily pay money to ZAI and the Moonshot, but I do it for one simple reason.

25:24James Wilson: I cannot get Anthropic nor, to a lesser degree, open AI models to actually be compliant in the defensive cybersecurity work and research and just playing around in the cybers that I want to do. If Anthropic really wants to shift the needle away from people using these open-weight models, the irony is that the ball's actually in their court. If they got these guardrails right and stopped being so draconian with them, I wouldn't be spending money with open-weights, which is a different equation to the whole I could just self-host it and do it because I don't think people can do that at the moment because it's still too damn expensive.

25:58Patrick Gray: Well, and staying on the topic, we've got a piece here from Wired that says, that argues AI guardrails aren't stopping criminal hackers. That is true. However, I feel like we are going to always need some kind of guardrail because you don't want to be in a situation where some awful bug, like the PHP one, maybe not a good example. we'll be talking about what happened with all of that in the last week as well in just a moment but say there's some real bad bug comes around it's pretty generic it's pretty easy to exploit do we want to be do we want a frontier model you know some normie right being able to or even gemini or whatever you know say it's a year from now just being able to go to gemini and say hey are there any you know really bad security vulnerabilities out there at the moment that could be turned into a worm and it'll say yes boss and you say okay turn it into a worm and And then off we go.

26:47Patrick Gray: So I think we're going to need something. I mean, obviously, Anthropic in particular is just insane when it comes to guardrails. The guardrails are completely nuts. But we're going to need to have something there. But I don't think anyone expects them right now, late October 2026. I don't think anyone really expects guardrails to stop motivated criminals, given that they're using obliterated models. But where do you land on this, Alex? Because I'm curious to know what you think here. Yeah. So one, we have to remember that these guardrails came from government pressure, especially on Anthropic, right?

27:25That the White House shut down Fable with no good standard and no, because they thought the guardrails weren't strong enough.

27:35Patrick Gray: And I feel like that was, I'm sorry, but I feel like that was basically a pretext and they just don't like Anthropic. It is a pretext, and yes, they don't like Anthropic, but they could also use that pretext again. And I think it's partially pretext. It's partially real concern. But the problem is we still don't have good written standards of what governments want these companies to do. Second is if you look at the threat intel reports from Anthropic and OpenAI, bad guys use these models to do bad things. So we just had the Artex attacks against South Korean banks, and most of the operational stuff happened with, it looks like, GLM.

28:18But some of the work came from frontier models. So even with the controls that are in place, bad guys are getting some benefit from the frontier models. And so clearly what you really don't want if you're Anthropic or OpenAI is for the story to be South Korean bank straight up hacked using Anthropic or OpenAI. Um, and, uh, I think there, yes, we're going to have to have it. What's frustrating to me is even after all this time, it has been, uh, quite a while since the Fable shut down. We, they still have not figured out how do you differentiate between the malicious action and the defensive that people are still being shut down when they are trying to do incident response.

29:01And that for whatever reason, these classifiers cannot tell I am trying to protect myself versus I am asking you to actively attack something. I don't know what's going on here.

29:12Patrick Gray: Because they're using keyword searches, right? Like I think if they were actually using proper classifiers, this would be less of a problem, but that requires compute and they're already like stretched to the max, right? You get that impression. So I don't know, James, you look like you want to jump in there with something.

29:25James Wilson: I just wanted to add there was an interesting detail around Kimi 3.1 Pro, which came out this week, I think it was, where they took a very deliberately different approach to cyber guardrails. In fact, they don't put guardrails around the Kimi 3.1, but what they did was they explained that we actually deliberately have a lower degree of cybersecurity, offensive and defensive capabilities in this model, and that's by design because we excluded a lot of that from the training set. I think it's fascinating to see a frontier lab deliberately hobble the training set to end up with a model that simply doesn't have the capabilities as opposed to going to the expense of, and again, spending all the money, training a model that has the latent capabilities is then spending more money to disable the latent capabilities.

30:07James Wilson: I just wonder if Kimmy's onto something there, because while it's frustrating, because I want the model to actually have those capabilities, I like that they've built it such that it's not about guard railing around this with dumb classifiers and if statements. It's actually just, no, we didn't train that capability into the model at all.

30:25Patrick Gray: But if you want your model to be like a computer understander, you can't really exclude that knowledge, right, I would think.

30:33James Wilson: But I think this is the key, this is the crux of what we're sort of getting to add here is, you know, to your comment around any normies saying, hey, find me a bad bug, now can you worm it? To me, that's not actually about the bug and the reverse engineering and the deep technical understanding of the bug. That's the infrastructure they would need to set up around that, the C2, the proliferation, the loaders, et cetera, all of that stuff that's going to be needed to make that worm actually worm. and there's no reason for a whole data set about how to build C2s and infrastructure and how to build self-propagating worms to end up in that training set in order to try to improve its ability to use a computer.

31:09James Wilson: I think those are very nuanced things that you can keep separate out of the training set whilst also not affecting things like coding ability in general, perhaps even just defensive bug finding.

31:17Patrick Gray: I mean, I'm going to need some more time to think about this but my immediate reaction is inside my brain there's a voice just screaming, heresy, heresy. Alex, what do you think about James's idea of making the models bad at hacking? So you're right. And that's what Anthropic, I believe it was Sonnet, right after the Fable incident, they announced, oh, this thing is bad at cyber, right? So we don't have to put protections in place. The interesting thing with Kimmy, though, is what others have found is models that have any kind of good coding capability, even if they haven't been trained in cyber, then post-trained in cyber really well.

31:55So if it is an open weight model and it does well on the basic coding benchmarks, then you can then run it through a bunch of bugs and then get it really good at hacking pretty quickly. So it's kind of funny because they're effectively washing their hands like, oh, we didn't release a great cyber model. But then it will only be a couple weeks later that some people can spend a little bit of money on post-training and turn that thing into a pretty good hacking model.

32:23Patrick Gray: Well, and then what are you trafficking in? You're trafficking less in the open weights and more in the data that you require for like fine tuning and whatever. And that's all the stuff that's going to be available on Hugging Face.

32:33James Wilson: Yeah, and then you've got to trust that model. And we know that that model can have bad behaviors baked in. And it feels like we've gone back around to full circle again, which is like, you've got to trust the model.

32:40Patrick Gray: Well, it depends what you're trying to do. If you're a ransomware actor, I don't know, man. It's not like your Coca-Cola corporation or something. But anyway, we could keep talking about this for hours, but we're going to have to move on. Look, just want to say a huge shout out to the Andromeda Cloud AI team who wrote up a security incident where some of its customers had their API keys stolen and used to do a bunch of queries and inference. And they wrote this up as, yeah, it's an industry-wide attack to monetize stolen tokens, a.k.a. distillation. Because this is what OpenCode said last week.

33:21Patrick Gray: oh this is a distillation attack we're really what we're like it's very difficult to see the intent here because it really could just be the fact that people are going out there stealing cred stealing api tokens right and and stealing inference tokens and distributing them through uh through routers and whatever llm routers to make money um and in fact railway you know compare the pair right uh railway has urged users to revoke secret keys after some attack and then I think they're saying this one was distillation.

33:51James Wilson: Oh yeah, yeah.

33:52Patrick Gray: Everything I don't like is distillation.

33:54James Wilson: Not only that, but not only do they know it's distillation, but they knew it was an obliterated model. How? I have no idea.

34:00Patrick Gray: Yeah, so I feel like our users got attacked as part of a distillation attack. It's the new, it was a sophisticated attacker. It feels like that's the way you do it. You don't say we got owned, our customer accounts got owned and their balances with OpenAI and Anthropic got drained. You say we were caught up in a very, you know, very dangerous state level distillation attack. Alex, what's your vibe here on what's actually going on? Because is it token theft? Is it distillation? I mean, I feel like it's going to be both, right? Because you steal the tokens, you pump them through something, some sort of LLM router, you keep the transcripts and that becomes useful for distillation later.

34:40Patrick Gray: But I feel like the primary motivation here is probably just to resell stolen tokens. Yeah, so I think there's actually a couple different things. So there are definitely distillation attacks happening. A lot of individuals are getting hit with malware. This actually happened to my son this summer. I got a call while he was overseas at like 2 a.m. I get woken up and he says, dad, it wasn't me, which is what you totally want to hear from your son at 2 a.m. And I'm like thinking to myself, OK, son, this is how you dissolve a body in the bathtub. But what he really, it wasn't him that there's$5 ,000 on our credit card in Claude and Cursor charges.

35:18And it wasn't him. Somebody had put malware on his computer, stolen his OAuth tokens, and logged into Claude and Cursor and turned off limits and racked up a ton of Fable charges. And it looked like I actually worked with those teams and they had racked it up in it and talked to those teams and they believe it was distillation. So instead of sending spam and such, it is distillation tax. It's both you're selling the access to people on the gray market and then apparently they sell the logs to companies that use it for training. But with railway and then fireworks as well, it looks like – and I do have some threat intel here as well as some direct experience.

36:03It looks like there are, it's probably the Chinese cottage access industry working on behalf of Chinese AI labs are breaking into companies that service AI companies looking for model weights and other research data. And so they are going to folks, getting API keys, going into S3 buckets, trying to get model weights and intermediate training data. And so that's why you've seen fireworks. The fireworks attack did not go after inference. It did not go after what they specifically went after is API keys that would get you access to the evaluation data. And same with the railway. They went after API keys.

36:42And so there's very strong evidence that this is all coming from China. And so expected is Chinese cottage access industry.

36:52Patrick Gray: I mean, it absolutely makes sense what you're saying, which is that it's, you know, I'm saying, is it this or is it that? And you're like, well, it's all of that plus this and plus that. and it's this big malicious blob of, you know, awful activity.

37:05James Wilson: But it's this notion of we can look at the transcripts and say, this looks like distillation that I still take issue with because a successful distillation attack looks indistinguishable from actual use of the large language model. That's the whole purpose of it. Right. So it's hard. I mean, on the distillation, that is from a discussion with the large labs. What effectively happens is there's, again, a couple of different things. If there's a mass attack against individuals where you steal tokens, then set up these routers, resell the access. And what they want for distillation is instead of doing artificial, what they want is a sample of real use.

37:41And so you make money twice. You make money on selling it to normal people and then reselling the logs of real use to the labs for distillation. Now, the attacks against railway and fireworks are a different thing where they're then getting API keys not for direct distillation. but then to directly steal model weights from companies. So because Fireworks is directly doing training of, and so the API keys that were stolen from Fireworks were specifically for the information that for post-training of models, which would be obviously of great value to a bunch of information for which lots of gigatools were spent of creating those post-training models.

38:26Patrick Gray: All righty. Look, we're going to move on from talking about LLMs because I feel like I'm all LLM'd out at this point. But let's have a bit of a chat. So last week as we were preparing the show, we had a chat about this upcoming PHP floor that was going to be a big deal. We didn't, you know, hype it per se. You know, we let others do that. But we did say there was a big one coming. And, you know, that being in security at the moment, it's a little bit like having a telescope and watching the meteors coming towards Earth and then just sliding by and just maybe grazing the atmosphere a little bit.

38:56Patrick Gray: It looks like that's what's happened in this case due to this thing being a little bit more fiddly to exploit than we expected. But the news reporting on this one, James, it's all over the shop. And I know you've spent some time looking into it because we've had, I think it was Amazon came out and said, oh, people were exploiting this before it was even known in public. And, you know, it's out there and whatever. And then other people, there's so many false alarms on social media where people have these like hallucinated LLM generated exploits. And they're like, I've got it. I've got the POC. And it turns out it doesn't even run.

39:26Patrick Gray: And, you know, so where are we at like a week later with these PHP CVEs that were dropped last week?

39:33James Wilson: Yeah, so where we're at right now is I don't think we will see mass exploitation of this and we haven't to date. But going back to the history of it, yeah, there's, I think the PHP Foundation tried their best to do the coordinated disclosure really well around this, right? They didn't put patches into upstream branches that could be seen like we've seen with Linux kernel where that resulted in exploit within hours. They held onto this. It was all embargoed. There was the misstep of Chain Guard deciding to go early on the embargo. But then this outlier now is Amazon saying, oh, yes, since the 17th, which is some four or five days ahead of the actual release, they'd seen exploitation.

40:10James Wilson: But they haven't actually told us, well, what form that took. It's just an IP address. So maybe it was a researcher trying something, maybe something leaked from the embargo. We don't know. That doesn't seem to be much in there. yeah where we are at though is look both of these bugs they rely on essentially um you manipulating error handling within php um one is they're both essentially boiled down to use after freeze but they're in different parts of the code but it requires an error to be basically it requires you to force the thing that is hosting php so apache or fpm into an error handler in its startup routine so before the php codes even actually run which is straight away gives to the first reason why this is hard to exploit because in the if a startup failure happens in mod php yes it will continue to use that process but if if the startup failure happens in fpm which is the more popular way now i think to host php it just kills off the process it's gone right so even if you do crop the memory there's nothing to come back to to mount your subsequent attack on but again these are just such bizarre areas where the use after free happens that I don't think there's really a lot of a window here for a meaningful way to turn this into a weaponized exploit.

41:21Patrick Gray: Crisis averted. Good news, everyone. Now let's talk about the next Meteor, which is this HTTP2 shell floor in Brotley. Now, Brotley is some sort of weird old, what is it? It's a protocol, isn't it? Compression. Compression protocol. Yeah, it's a compression protocol used in HTTP, and it's quite commonly supported. It was open-sourced by Google, I think, and it's really commonly used, and that looks like a pretty nasty bug. Are we going to see this one get massively exploited, or is it another one of those meteors that's just going to slide past the earth there, dude?

41:54James Wilson: My bet is I think this one flies by. It, again, looks like it's got all the same hallmarks of being somewhat difficult to turn into an exploit. I mean, you've got to remember what this is going to give an attacker is the ability to write to memory that they shouldn't be able to write to. But then what are you going to write to? Where is it being written to? Is that a predictable location? Is there other mechanisms and defense in depth that can prevent the pointer from jumping to that area of memory that you've now put your code into? Like, it doesn't, unfortunately, unless it's a Fortinet box on the edge, a simple able to write memory does not get your shell anymore.

42:29Patrick Gray: Yeah. Okay. Well, hopefully another crisis averted there. Now, we're going to have a bit of a chat about this work at a citizen lab. We spoke about it last week where they had talked about this. They'd written up this Israeli ad company, this advertising company that was injecting ads into major advertising networks on the Internet. And, you know, using its position in the sort of ad tech industry and ecosystem to infect devices with iOS malware in targeted attacks. but it also was doing weird de-anonymization stuff. It looks like this one's getting a little bit of political attention now. I mean, we've been writing about this and talking about this for years, that this sort of data is a national security threat.

43:14Patrick Gray: It's got to the point where, and I know this, threat intelligence companies, they sell access to this data to law enforcement agencies. So they could just use it without a warrant. It is very privacy invading stuff. Now, as great as it is that it's getting some political attention, You hear senators talk about this sort of thing every now and then, and then it just all goes quiet again. Alex, are we at the critical point yet where something's actually going to happen in the United States to deal with this? Or is the can just going to keep getting kicked on down the road? Yes, this is a little bit of back to the future for me.

43:49I got yelled at by John McCain in 2014 about the security of online advertising. So nothing happened back then. So what's innovative here, and a great shout out to CISN Lab for discovering this, is it looks like these guys were using, like you said, people have been selling ad tech data to violate privacy. But in this case, they were using targeted advertising, what's called custom audiences. So a number of ad networks allow you to upload things like the phone numbers of individual people to show them advertisements. Now, generally, the rule is that you have to upload something like 100 phone numbers or 100 email addresses.

44:35So you can't get smaller than that. So there's no detail here of exactly how they then narrow that down or if they infected 100 people or if they had like 99 perhaps fake people and then one actual target. But that's a pretty innovative way to get after a target and then load up to that person, apparently JavaScript, that would then take over their browser and take over their computer. Really nasty stuff. Wyden has gone after this kind of custom audience for a long time. I doubt anything's going to happen because it is an incredibly powerful type of advertising that – guess who is one of the groups that really love custom ads?

45:15American politicians. So everybody basically but Ron Wyden in the United States Senate uses custom audiences to win their elections. So I doubt anything will happen, at least in the United States, because these kinds of audiences are used widely by both Republicans and Democrats.

45:35Patrick Gray: I want to say, too, a couple of things. First of all, John McCain yelled at Alex. He's now dead. Connect the dots. That's the first thing. The second thing... Just a warning to anybody who yells at me. That's right. The second thing I wanted to talk about was the reason there is that 100 minimum for advertising groups is actually really funny. I don't know if you remember this, but it was a guy who actually started, I think it was via Facebook. He managed to enroll his flatmate, his housemate or roommate, I think you would call it in America, into an advertising group and just ran targeted apps that was so targeted to this guy that he thought he was going insane.

46:13Patrick Gray: and that was very, very funny and wrote about it. It was a long time ago now. I still remember it. Very good. Look, we're just going to whiz through these next ones. We've got a story here about a write-up on how Lite LLM, how its release pipeline got hacked to distribute an infosteeler malware. It is interesting because they wound up actually just compromising the installer package instead of the software itself. So we've linked through to that one. People can check it out. bad week to be a movie pirate because we are seeing in the wild exploitation of Plex via Plex to Shell. Everything's to Shell now.

46:50Patrick Gray: Come on. I mean, it's bad enough that we name bugs, but why is it to Shell? Plex to Shell. And then there's Popcorn Time. One of its update domains got abandoned. Someone registered it and they're serving up malware in updates that way. So that's pretty funny. Some good news, Chrome has added Gemini-powered scam detection and click fix warnings to its browser. That's the headline from Bleeping Computer who covered this. There is a blog post from Google themselves as well. Really cool idea, right? Where it's like, if you're copying commands to the clipboard, now the browser can go, hang on, that looks a bit sus.

47:26Patrick Gray: Send it off to Gemini and Gemini can say, yeah, that's click fix. LLMs are really good at context. They're really good at dividing intent as well. We're seeing a lot of work in the security space go into this. You know, companies like Push Security, who I work with. You've got other ones like Ant, who are doing this sort of intent-aware security stuff. It's great to see some of the basics being implemented by the browser makers here. Microsoft is also putting context-aware phishing detection into Teams. It's great. You know, you're probably going to, as an enterprise, you're eventually going to need probably some more enterprise-grade tooling for this sort of stuff.

47:57Patrick Gray: but it is really good to see the basics covered there. What else do we have? I just wanted to mention that the ConnectWise attack that I flagged as being pretty interesting last week. Turns out that's now been attributed to a Chinese state-aligned threat actor and they were using Artex. So there you go. You could use Artex to do cool hacks if you are a Chinese cyber contractor, which is what this sort of feels like. It feels like the MSS operations are a little bit more tightly scoped than what these guys were doing. the White House. It's very Russian. It's amazing that actually we've seen so many Chinese operations that are kind of loose, loosely goosey like this.

48:35Patrick Gray: Yeah, but it always turns out to be some contractor. And we find this out because they're so loosey goosey, they get owned and all of their stuff winds up available via BitTorrent, right? So it's like, this is the thing. It's all in like an Alibaba bucket. Yeah. You know, like the Russia stuff is like, whoops okay i let the worm out now everything got owned whereas um the chinese stuff is like yes i hacked i got shells on you know 250 000 exchange boxes because there was a bug right like uh so it seems pretty true to form to me uh we saw a universe the university of new mexico uh health network uh they had a ransomware incident what's notable here is they were only diverting ambulances for two hours and the whole lot was the whole incident wrapped up in less than 24 hours so That's a good news story about how these incidents are getting, there's more of a template, there's more of a playbook for dealing with them.

49:24Patrick Gray: And the U.S. National Cyber Director has actually launched a pilot program which is going to provide rural hospitals with free access to security tooling, right? So just a bit of action when it comes to ransomware and hospitals. And we're going to finish up, James. The private key is always four, as it turns out. The private key, it's always four.

49:50James Wilson: Right, so here we are again with another private key that is one byte of entropy and it's four. This one's a little bit different. Again, it's entirely different bug. It's just ridiculously hilarious that it all converges to create a private key whose value is four. But in this case, it's a vulnerability in the go-ahead web server, which is a little embedded web server. It is part of a bunch of IoT things, but this one's going to be harder to quickly mitigate, whereas with the eMudra ones, you could quickly do the cert revocation lists. We know how well they work. Well, for however much they do work, they won't work in this case because it's not enforced at all.

50:32Patrick Gray: So we've got to call this 4-2-shell, forever 4-2. I don't know. I'm trying to throw some shade. It's not working.

50:43James Wilson: This is why you do the headlines and I don't, Pat. This is going to be like a Jeopardy question. It's like, this private key is the only perfect square of an even number. It's like, what is four?

50:55Patrick Gray: All right, we're going to wrap it up there. Alex Stamos, James Wilson, thank you so much for joining me to talk through the week's security news. Crazy old times. Thanks, Pat.

51:04James Wilson: Thanks, Pat. What another amazing week.

51:12Patrick Gray: That was Alex John McCain Slayer Stamos there with James Wilson taking a look at the week's security news. Big thanks to both of them for that. It is time for this week's sponsor interview now. And James spoke with Dylan Ayerie, who is the founder of Truffle Hog. Truffle Hog, of course, does secrets discovery. You can point it at your Slack, at your repos, at a S3 bucket. it. You can point it at a banana and say, find me a secret that I've accidentally exposed and it will be able to do it. It is a mature product these days. It's actually very impressive. A lot of people use it. You should too.

51:46Patrick Gray: So that is Truffle Hog from Truffle Security. But yeah, basically, James spoke to Dylan for this week's sponsor interview, all about, I guess, looking at where the real problems are when it comes to secrets exposure. There was a bit of an oh my God moment when Truffle Hogg looked at the Hugging Face breach and saw all of the secrets that had rattled out of their training data. But as it turns out, they're a bigger fish to fry. So here is Dylan Airey in conversation with James Wilson. Enjoy.

52:17James Wilson: We were partnered with Hugging Face to scan their data sets at the time of the breach. So the CTO pings me and says, hey, you should check this thing out. Then the 14 keys that the rogue agents used to break in were in the scans that we were doing. So we found, I think, a quarter million live unique keys across all the Hugging Face data sets. Within that quarter million was the 14 that the rogue agents had used to break in, which was pretty wild. And then at the end of it, Hugging Face pings me again, and they say, look, we just finished scraping all of main across all of GitHub, and we plan to publish it as a new data set called the Stack V3, and that's for training future models.

52:53James Wilson: And they said, you might want to scan this one because we suspect there's a lot of keys still on GitHub. And so we appreciated the heads up. They did a lot of the heavy lifting for the actual scraping. And then all we needed to do is the scanning of that data. But we routinely scan and monitor large swaths of the public internet. So every new piece of code that goes onto GitHub, we scan. Every new container that goes into Docker, we scan. And we also scan all of NPM, all of PyPy. And you're right, it takes a tremendous amount of compute It takes a tremendous amount of data to store it all. And I think just to put some numbers into perspective, don't 100 % quote me on this, we probably spend about 100 grand a year doing all this scanning.

53:38James Wilson: And then I know your next question is going to be like, why or to what end? You know, like you find all these keys and what do you do with them? And the answer to that is like, we believe that the world is in for a world of hurt if we can't get the 1.4 million live keys. that we know about revoked. And so we partner with a tremendous number of companies and partners to get that number burned down. And that's why we do it. Just coming back to the numbers again, like when you went through the Hugging Face data sets, I think you found something in the order of 220 ,000 credentials or live credentials.

54:15James Wilson: And I like how you say, you know, Hugging Face had a sneaking suspicion there might be more in the GitHub repos. What did you find there? I think what was surprising was, number one, seeing the age of some of the keys still in GitHub. Because for a little while now, we've monitored all the new stuff going into GitHub. And we know that there's a tremendous number of new keys that enter GitHub every day. But going back in time, seeing keys 11 years old that are still alive, like there was a little bit of a surprise. And I know like every time we tell this story, we always get the same question. It's just, hey, I thought that GitHub has this partner program and like Amazon and all these providers and stuff automatically revoke these keys.

54:52James Wilson: Well, the reality is the program that they have is a disclosure program. not a revocation program. And so what that means is they'll go to Amazon and say, hey, here's the Amazon keys that have been leaked out. And then it's up to Amazon to choose what to do with that information. So some providers will notify customers, some providers will hard immediately revoke, and some providers will do something in the middle. Amazon does something in the middle. So they will apply a policy to those keys, restricting some actions on what the keys can do, and they'll notify the customers. Well, we looked at what those actions, like what what those restrictions are.

55:25And there's still a tremendous amount

55:29James Wilson: that you could do with those keys. One of the funniest ones that we looked at was the Amazon API to download a file out of S3 is blocked with this restrictive policy. But the API to download a file with a version number attached to it isn't blocked. So it's just like, it's a big unsolved problem, is a long story short. And so we're excited to try to do our part and get as many of those keys revoked as we can. But it's a very big unsolved problem. The numbers themselves are pretty staggering. Like I think it was, correct me if I'm wrong, but it was 220 ,000 credentials in the Hugging Face dataset, then almost double that, if not just slightly more than double that in the GitHub dataset, some 550 ,000 keys that still authenticated.

56:18James Wilson: Those as a snapshot point in time number are staggering. in and of themselves, but what are you seeing in terms of the trajectory? Is this a problem that is a long tail? Is it not getting worse? Is it getting worse? Where is that number trending? The problem is getting worse, and it's getting worse for the reasons you can probably expect. More code has never been authored before, and the ways that it's being authored is unique and different. I'll give you a couple anecdotes. more and more our customers are asking for insights into laptops because of the shy halut stuff. If a worm gets hit on a laptop, you need to quickly know what keys are there, like what could have been slurped up.

57:01James Wilson: So when we provided that support, I don't know if surprised is the right word, but it was interesting to see that so many of the keys on laptops are now in the cloud folders because people are talking to their cloud bots and they're in the chat logs of the chats and they're saved in skills files. That's so funny. So you used to find them in like.env,.local in source. I'm sure you still do, but now you're saying you see them in like the.clawed transcripts and things like that as well. Wow. That's absolutely right. And so that's causing more keys to leak out. And then the different places are kind of interesting too.

57:36James Wilson: Like one interesting insight, I don't think we've done a blog on just what we've seen in Docker Hub yet, but we've tweeted out a couple of things. And one of the insights that we tweeted out is, about twice the volume of live keys land in Docker Hub every day compared to GitHub. Why is that? Why is that so different? I would wager it's because push protection on GitHub offers some level of protection. Of course, you could override push protection, and it also isn't a silver bullet. Like if you push up a key in a zip file, or it doesn't have an exact pattern for what it's matching on. Also, other weird things about push protection is if your patch that you're pushing up is too large, it just fails open, allows the code to get submitted up.

58:17James Wilson: So there's a lot of reasons that push protections isn't perfect. But that being said, there's nothing like that in Docker Hub. And also the other thing is the Docker ignore files and the git ignore files are never the same. And so you might write a git ignore file that's like, don't include any of these sensitive files. And then when your CI fires to publish into Docker, it doesn't respect or pay attention to that git ignore. And if you don't have parity with your Docker ignore, you're just going to get a bunch of stuff copy star dotted into your docker container. And like that's often the cause of what I call like second order leaks.

58:53James Wilson: Or like your first order leak is like maybe you willingly put a secret into code. Your second order leak is CI automation has caused a secret to get written to a file system. And then like a copy dot accidentally slurps it up and pushes it into a package. And then you have like third order leaks, which is like one further removed where like some weird convoluted automation ends up materializing a key seemingly out of nowhere, and then it ends up in a log somewhere or something like that. The flip side of this, though, is that for all the hundreds of thousands of still authenticating valid keys that are out there in either Hugging Face datasets or in GitHub repos that are public, why doesn't this result in more breaches, or is it resulting in breaches that we're not seeing?

59:36James Wilson: It just seems an amazing treasure trove of access. I think you do see them, right? The Hugging Face incident was caused by 14 stolen keys. Well, caused is maybe not the best word, but a large proportion. Well, tell me more about that because, you know, until you mentioned that to me before we started recording, I was like, I thought that was all, you know, rogue agents, fine and zero day. We just used keys I don't think was quite as prevalent in the messaging. It's tough to say we just because it wasn't one thing. The other really tough thing about the Hugging Face incident, Like when METAR came in and did their investigating of it, what they expected to find was like a quick, simple story.

1:00:18James Wilson: And instead what they found was a civilization of alien brains that worked for like the equivalent of 300 years. And so it's like to distill the whole story down to like it was just X or Y is I think not doing the story proper justice. But once the agents used their combination of exploits and magic-y things to get internet access, A very small number of agents, maybe just one, was investigating Hugging Face as something maybe it could look into. Well, most of the other agents were focused on other things. And when that agent first got its initial foothold, it found 14 keys that validated to a bunch of private data sets in Hugging Face.

1:01:01James Wilson: And then it used that to get access to more data. And then that more data then led to some zero days in Hugging Face's infrastructure as well as more keys, right? So it used keys to get access to data, to get access to keys, to get access to data. What I usually like to say is imagine you're training an AI model to hack or exploit a system, and you incentivize in that training process the fewest number of tokens to accomplish that goal. Does it make sense to burn like two days worth of tokens finding a novel security exploit? or does it make sense to just log in with the password that's sitting there?

1:01:41James Wilson: What's going to result in the path of QS tokens to accomplish the goal? And I think the reality is they're trained to do both, but it's going to do the thing that's quickest. And so if there's a key laying there, it's going to use it. If there's not a key that's laying there, it's going to burn two days finding a zero day. Yeah, exactly. It's going to find a way and it'll use the path of least resistance first. So speaking of sort of, you know, as you mentioned in the blog post, the gap here is revocation. Who owns that gap, though? Whose problem is this to solve, right? And of course, you know, in one sense, the answer is customers should be using Truffle Hog to find their keys.

1:02:18James Wilson: But, you know, the scale of this is huge. And as you say, largely an unsolved problem. So who's incentivized to fix this? Yeah, I mean, I think like incentives are one thing. Responsibility is one thing. and then just who can have the biggest impact is in my view the most interesting question. Because let's say you're a big SaaS provider and you have thousands of customers. Sure, you could take the position where the customers are responsible for their own keys and if they do stupid things, that's on them. But also, if you have a nice feed of all these customer keys that have leaked out, it's going to be way easier for you to revoke them at scale than to hope that the customer notifications are going out in a way that they can see, respond, action on And so that's what we've recently done is we've created a whole bunch of these partnerships with different companies.

1:03:08James Wilson: And we have more to come on this in the coming months. But basically to just try to get the upstream providers to act on behalf of their customers. And even if you take the philosophical stance that it's the customers themselves that are responsible, it's just easier for a provider to centrally squash it than it is to track, you know, to play the individual whack-a-mole game. Yeah. Do these partnerships, do they ever push back a little bit and say, well, we don't want to just revoke these keys because we could break a customer's production system and that could erode trust. Like, what's the sense of pushback that you get from vendors?

1:03:46James Wilson: All the time. Yeah, all the time. I mean, it depends on the vendor, right? I think Slack and OpenAI are great examples of companies that just said, you know what, we're going to hard revoke. If it breaks something, it breaks something. It's in our terms of service. They should have read that more carefully. The terms basically say if a key leaks out, then it's going to get revoked. Then there was one company that had a weird middle ground. I hadn't seen before, but it was really interesting. I think it was 11 Labs, where if you generate an API key in the generation step, it has a default on switch.

1:04:19James Wilson: and the switch basically says if this key leaks out on the internet and we find out about it, we're going to revoke it. You can turn that switch off, which weird. And then you've got a lot of other providers, like Amazon being one of them, that don't want to risk the damages that could come from just blindly revoking keys. Cloud providers are great examples of that where it's just really hard to anticipate how damaging it could be to just take that hard of a stance. And a great example I like to give of this is when you scan Kubernetes, you will find a live GCP key in it. And that's because it's used to pull public container registry images.

1:04:59James Wilson: And it needs to be there and it's useful. And so people's Kubernetes will break if you take a hard stance and say all GCP keys need to be revoked. And then there's payment providers that can very easily quantify the value of the key they take down. They can say every hour this goes down has lost X million dollars of revenue. And so, yeah, it's messy. It's complicated. That's why it's unsolved. Yeah, it certainly sounds like it. Well, Dylan, look, thanks so much for dropping by for the chat. It's been really interesting to see the results you found from this latest investigation. Thanks so much, James.

1:05:33Patrick Gray: That was Dylan Airey chatting with James Wilson there. Big thanks to both of them for that. And big thanks to Truffle Security for being this week's Risky Business sponsor. And that is it for this week's show. I do hope you enjoyed it. I'll be back later this week with a Snake Oilers edition of the Risky Business podcast. But until then, I've been Patrick Gray. Thanks for listening.