Between Two Nerds: Why frontier lab security will always suck

0:03Tom Uren: Hello, everyone. This is Tom Uren. I am here with The Grugq, another Between Two Nerds. G'day, Grugq. How are you? G'day, Tom. Finding yourself? I'm very well. This week's episode is brought to you by Nebulok. They make autonomous threat hunting agents that can continuously scour your network to make sure that there's no polls, vulnerabilities, what have you. Good stuff and very relevant to this episode. So as we speak, it seems like there's been a couple of vulnerabilities in PHP, which are starting to be widely exploited or attempted to be exploited. Right. Now, one of the things I wrote about in my newsletter this week was the security culture at OpenAI.

0:51Tom Uren: And my contention is that it'll never be a very good one because they've got such an incentive to move fast because of the competition with other frontier labs that that just overrides the incentive to be very secure. They've kind of been burnt by their agents basically breaking out. Now, the reason I mentioned that is that in the wake of the vulnerabilities being discovered, the PHP Foundation has actually produced this document, which talks about how we're strengthening PHP security. And my take on this document was that it really talks about a change in security culture of that organization.

1:37Tom Uren: Right. And so I thought it'd be interesting to compare and contrast, I guess, PHP and the drivers and what might happen. And why I'm kind of fundamentally optimistic, maybe. Maybe I'm just always an optimist. Maybe I'm an naive fool. I was going to say, I'm in agreement all the way until you got to that point. This is going to be a boring episode. Yes, you're right. Yes, you're right. Oh, hang on there. And then contrast, so there's reasons in that document that PHP talks about. And contrast that back again with OpenAI and why that's problematic. So tell me about the history of PHP, because we were talking about this, and you said that they had never had a good security culture in the past.

2:29Tom Uren: Yeah. So PHP has always been problematic, and there's two sides of that. One is there's, like Dave, I tell's contention that it is impossible to build secure software in PHP. So, like, you can't write code in PHP and have it be secure. Like, that's just, that's not happening. It's a non-starter. Do you know what those reasons are? That the language is just fundamentally, it's basically made of foot guns and bugs, I guess. Right. Right. So anyway, so that's sort of the one side. Like writing software with PHP is notoriously difficult to get it right. Right. But then PHP itself has had a colorful history of – Yes.

3:19They've had a – they haven't always been the best by a long shot. Right. So at one point, you know, Stephanessa was basically dropping bugs on them all the time. to the point where he started doing this thing called month of PHP bugs where every single day he just dropped another bug. And they would patch this one and then they would patch that one and then they would patch the other. But I don't think there was a fundamental shift in the like, oh, if we're having this many bugs, one a day for a month, maybe there's something fundamentally wrong with the way that we're approaching things that needs to change.

3:54It was just more of a...

3:56Tom Uren: Right. So how long has it been around for? so it's been around for 30 plus years basically it could have kids in it could have kids who are in grade school at this point right um it's been voting and it's starting to get more conservative that's that's how old it is yeah so that's the the history right so it's it's been around for a very long time In this document, which is titled Today's PHP Security Release and How We're Strengthening PHP Security, it talks about the history of these particular bugs. And it points out that in 2024, PHP underwent its first external security audit in years. and so that audit surfaced the the problem but it didn't identify an actual way to exploit it so that describes it as the first signs of the um it was an out of memory right so it's like the bug itself is complicated right like you you can create an out of memory condition which will leave some memory uninitialized that you can later on do stuff with, which is all tricky.

5:21And so you could see how at the time it would be like, it's not as straightforward as like, oh, this buffer is too small. This is too big. It's going to overwrite whatever. It's very much. Yeah.

5:32Tom Uren: A potential exploit was theorized, but no concrete vectors were found. And it goes on to say that there were other problems that had clear impact. so yeah it makes sense that they worked on something else like if you were doing a triage it does make sense that you go like we we have so many we have so many man hours of fixing bugs here's a very very theoretical possible thing that might happen and here's a concrete one that is is much more likely like let's do that yeah for sure not going to fault them they talk about trying to fix that problem but each of the approaches they came up with came with different trade-offs.

6:14Tom Uren: So backwards compatibility, potential denial of service, changing third-party extensions. And because no one could ever demonstrate real harm, you could never get to agreement. So one of the key things is the security team operated on consensus. So everyone had to agree. Which, as we all know, is absolutely the best way to move forward. I've always said that if it's not done by committee, it's not done correctly.

6:45Tom Uren: well yeah anyway i went i think you can have committees that work where you don't but the problem is requiring consensus right um for example i would never i would never make a security agreement between nations that ran entirely on on unanimous consensus that would be just unrelated of course just bringing that up but um so basically what happened is ai changed everything so part of that is that they got a whole lot of extra help from different programs where people are let's try and get ahead of ai mediated vulnerability discovery discovery in important software right and so they've got stuff from open ai um trail of bits with their what's it called is it daybreak there's some program right and and anthropic uh with someone else was also with had mythos so they had a lot of extra help and they were able to make the impact clear and like it's so serious it becomes the top priority the uh the prime minister of japan was tweeting at 5 a.m that you need to patch your php which is yes yes so i think by the time this podcast comes out it'll be pretty we'll be in the middle of the disaster it looks like a looming disaster anyway so yes now this was back as far as august early august and july a number of people working sort of semi-independently but with ai assistance make clear the impact so they've got an influx of resources because of AI.

8:29Tom Uren: They've got clear demonstrated impact. But I think what's really interesting is that they've decided to change how they deal with security. And one of the things they've done is now they don't need consensus. You just need a majority to move forward. Right. So I want to focus on that for a second, because I think that that's probably the the most important thing out of this is that, yes, on the one hand, this is a very bad bug that's now been found and fixed and will eventually get patched. Well, the patches will eventually get rolled out everywhere in a few years, no doubt. There is a lot of servers using PHP.

9:13Yeah, it's kind of popular, I think. So there's that. But I think, you know, you and I were speaking a bit earlier about this. And it's like security culture is probably the one thing that PHP can do here that will actually have an impact long term. Like by just changing the way that they view security. And it's a thing they can do because they're now, you know, 30 plus years old. They're established. They've got like fixed schedules for rolling out. They have processes, foundations. They've got full-time staff that are just keeping this thing going rather than trying to beat the other PHP alternatives to market.

10:02They're a mature technology that needs to be maintained and developed. Right. There's no exactly.

10:10Tom Uren: Right. Yeah, yeah. And so I would contrast that with, I mean, what would be a really hot market right now where there's a real... Just spitballing here, like maybe AI stuff, maybe the frontier labs. So before we get to that, they say start changing its processes so that decisions can be made by majority and compromises are encouraged. If I was PHP, I might be a bit cautious about using the word compromise and PHP together. So, and, well, decisions can be made my majority. And it just struck me as that feels like those discussions where everyone is so bloody minded that for various pedantic reasons, you can never get to agreement.

11:00Right. And particularly, from my understanding, this is quite a complicated bug. And fixing it is going to have, you know, as they say, there's some trade-offs. And as soon as that comes up, you're going to have a champion saying, we cannot break backwards compatibility. Someone's going to be like, this is going to be the end of the world if we let it go through. Someone else is going to be like, we can do it, but we'll introduce a lower risk bug instead, which is better, and then someone else will. basically everyone can find a thing to champion and never agree to sort of yield to someone else if there's a lot of options out there you can always find someone who's going to be absolutely convinced that they're right

11:44Tom Uren: and you need the kind of culture where someone is both empowered to make a decision and people accept that that decision well accept the decision even when it hurts their own interests support the fact that the decision is made and they're just going to go forward with it. Yeah, and I think that whole paragraph left me vaguely optimistic. It feels like to me that the foundation has been through a trial by fire and they've realized that there's a fundamental problem and they need to change. And they're making a fundamental change, right? Yep. That's the gravity of the situation. The problem is not this particular piece of software or this particular sort of thing.

12:28it's that we haven't had a good process for taking security culture as serious. Like we haven't had a way of dealing with this as a central pillar of what we do. It's always been a sort of side thing. And they're shifting that. And it's a fundamental patch to a fundamental problem that they have. And that is very encouraging.

12:49Tom Uren: And I think also writing about it, being cognizant that this is a thing that you need to change, and writing about it, it's like they're all good signals. Right, particularly for podcasters. We need things to talk about. That's right. So that one organization is able to change is optimistic on one hand. A key organization, PHP, very important software, is changing. That's great. But it makes you wonder about how long does it take to change the culture of every single organization across the planet? Yeah, one down, several millions to go. I was just going to say, not all of them are going to be as consequential as PHP.

13:33Tom Uren: Right, yeah. So in terms of a triage, this is probably one of the better ones to start with overall. Yeah, so that's kind of vaguely optimistic. The more important organizations will learn their lessons faster and there'll just be a long tail of increasingly inconsequential stuff. Now, you mentioned OpenAI. Which, by the way, does not apply to any of our listeners. You're all very consequential as far as... Yeah, so this is going to be the least listened to episode because everyone's going to be so busy patching and putting out fires. I've never been so glad not to actually work at the Coalface.

14:10Tom Uren: So now pivoting back to OpenAI, one of the things I noticed is that they've hired a number of really very excellent cybersecurity people. So Halvar Flake or Thomas Dullien? Thomas Dullien. Dullien. Dave Itell. So you mentioned Dave before. He used to work at NSA once upon a time, then founded a company. I think it was, was it Immunity? Is that it? Was it a different one? Yes. Yeah, yeah. So he had Immunity, then he finished with that. And he started a foundation for a while where he got really, really into using AI to find bugs. Right. Then he went to OpenAI where he developed Aardvark. It was their first bug-finding thing.

14:58I guess they're on something else now.

15:02Tom Uren: And then there's some other very well-known people whose names escape me. So the AI companies have sort of been on a hiring spree. And they've been pulling in a lot of talent. There's very good people at Anthropic. There's very good people at OpenAI. and there's the names we all know but they also have a deep roster of extremely competent people. Yep. So that said... Yep. Well, that's clearly been driven by problems that, well, self-inflicted problems, I guess, where they've allowed their agents to go hacking things that they probably shouldn't. Right. And I use those words very deliberately. I think they had a lot of choice in the matter And I think that goes back to the prioritizing speed over.

15:54Correct. Yeah. So there's a tweet a long time ago, weeks, possibly even a month. Yep. From a security guy at OpenAI. And sort of one of the things that really struck me about it was he was talking about the culture. And he was saying, look, this has been a research lab. It's a university research lab. It's not a Fortune 5 company managing a huge amount of money that has to really look at security as central to what they do to stay in business. It's a company that's been very much like open and just trying to do and improve things. And then security is a thing they worry about, but it's not a thing that sort of, it's not all encompassing.

16:41Tom Uren: Yeah, they worry about it, but they don't care about it. so my question to you they've hired all these excellent people they've got a lot of talent does that do anything at all to change the culture there's there's very much this feeling of like a band-aid is not going to stop a sucking chest wound right like you can get like the best talent available and if the culture is just we don't care then it doesn't matter like you can't you can't patch things that are fundamentally broken right and it's i think that's that's even possibly the wrong way of putting it's just that if the priority of the organization is speed development and innovation and all of these things if you bring in people who are like we can add some of the best friction like we can make the most exquisite top tier world-class friction they're they're great people they're very good at what they do but i think it as the saying goes culture eats strategy for breakfast and this isn't even strategy this is just like tactics so i i my view i kind of agree my view would be that the what you described as a sucking chest wound is that the their agents just keep on going and operating beyond the realm of the acceptable.

18:07Tom Uren: Like, so, you know, they're getting data from a website and they go just a bit too far in terms of, instead of making a normal web request, they're like trying to hack the website to get the data that should be there. Right. And so that's... Can I just say how incredibly frustrating it is that there's some guy in Sydney who has his agent hack a gym and put him at the top of the list. When I ask Claude to make a summary of everything I've been talking about for the last two hours with it, it will immediately boot me to Opus 4.8 and then sometimes even to Sonnet 4.6. Like I get dropped for talking about cyber and this guy's over there like hacking gym memberships.

18:53I just think that's very unfair.

18:55Tom Uren: Yeah, so I actually think they'll make progress on that sucking chest wound because that is a problem for the company. Like that's essentially a PR problem. But I think it'll be very narrowly defined and we don't really care about anything else. And it'll be, we want you to fix this problem and make it go away. Which I think because the people are very good, they may be able to do. So I was just going to say this. So we've got the, you can't put a band-aid on a sucking chest wound. But one of the other, one of the other things with that is a sucking chest wound is nature's way of telling you to slow down.

19:41So, you know, I agree. I think that they, they can fix these point problems, right? Like there's going to be the agents keep escaping. All right. You figure it out. You stop them from escaping. There's issues of like whatever, like our agents are being abused and people are using them to hack. They'll probably figure out something for that, right? I think you can do these points, like these little patches here and there, particularly if there's the strong incentive of doing that. But I think you're going to keep creating new ones all the time because the overall incentive is to move fast. And to do that, you really like all gas, no brakes.

20:25You have to go forward. Yeah. Yeah.

20:28Tom Uren: And we want people to fix problems. We don't want to stop the problems occurring in the first place. Right. Yeah. So like in the 80s, they were trying to. everyone was sort of trying to emulate the Japanese because the Japanese were like the best at business and everything. They had like the five whys and they had a thing where there's the big red button at a factory and anyone could go and hit the button and stop the entire assembly line so that a problem could be fixed. What's it called? A Yandon cord or something like that? Yeah, yeah. The idea that everyone has the ability to improve things at a fundamental level and they could do it by just stopping the whole company flat until it's fixed.

21:13I don't see that. I feel like that's not the culture that they would have. If you came up, you're like, everyone stop everything.

21:21Tom Uren: It's more of a Ford culture. Yeah. I remember the story. They go to the Ford lot and there's a whole lot of cars that have got defects immediately as they've come off the line and they then have to rework them to fix them. It seems like the frontier lab approach. Right, right. Exactly. And it's because it's a lot cheaper to do it that way. I mean. I don't know that it was cheaper in the long run, though, right? No, no, but it was. You didn't have to stop the assembly line, which is expensive. And you can measure how expensive that is because you know what it costs when it stops. But you can't measure some arbitrary number or some random number of things that need to be fixed afterwards.

22:04Like that's an unknowable. Right. And those are cultural things, right? Like to have a culture where people can sort of proactively go and try and fix problems before they become serious problems. That's one way of working. And then there's a different way of working which says, you know, we'll cross that bridge when we get to it. And so that's what we're like. That's the cultural thing. And I think AI is very much the we will burn that bridge. when we get to it.

22:41Tom Uren: Yeah, so I think the real contrast between PHP and the Frontier Labs is that PHP, like there's just, well, it seems to me there's no strong driver to move really, really fast. So if you get a lesson that says we must do better at security, you can actually do something about it. You can internalize it. There's no competing priority where you feel like. You can put that to committee. You can discuss it. You can create a subcommittee to come up with a bunch of options. And then you can unanimously agree to work by majority instead of by consensus. It's all good stuff. So I guess what we're getting at is that fundamentally the incentives don't exist for the frontier labs to be secure.

23:31Like it's just, it's not there. and regardless of who they have or how they go about trying to do things it'll they'll like they'll always be swimming against the tide uh yeah yeah trying to and i think when you say secure

23:47Tom Uren: there's some sort of overlap between security and safety right right yeah and it's and i think they'll do somewhat of a maybe more than half-assed job at both of those things like that's the optimistic they'll do a solid c plus effort so i mean i guess we both agree that the incentives for the companies the frontier labs are just wrong for them to end up with good security naturally or good safety i suppose that brings us to the question which we're not going to answer in this podcast i don't think about whether regulation would actually help. And it seems to me that regulation is also about fixing the problems you already know you have.

24:35Tom Uren: And it's not necessarily about... This is like everyone's conception. And it's not necessarily about preventing the problems that you don't know that you have yet. And I think a security culture can address that second thing, the new problems that are coming up, because you spend time and effort thinking about it and at least trying to mitigate it. Whereas when you've got regulation imposed from outside, you're like, well, I will do this and I'll be right and I can do, you know. You try and find a way to game the system to do exactly what you're legally required. You'll comply with the regulation and then you can hand on your heart full speed ahead.

25:13Best efforts. Yeah, like we did all the things we're legally required to do and we found a bunch of ways around them. so now we can go ahead and do what we wanted to do anyway. So there's a management paper called No One Gets Credit for Solving Problems That Never Happen. Right. And it's basically about this strategic culture sort of thing where if you anticipate there's going to be an issue and you correct it ahead of time so that it never happens, it's very hard to get promoted for that because you're going to have to explain well you see this was going to be a huge problem but i fixed it and they're going to go it wasn't a problem though so what are you talking about exactly like that never happened exactly i you know so yeah it's very like bureaucracy is tend not to reward people who do things that are invisible and that's sort of an invisible thing.

26:20And so it's, and this is even without the incentives of a race of trying to be first. Right. Like it's just fundamentally difficult to get an organization that rewards foresight of that sort.

26:32Tom Uren: So what I'm thinking is that the lack of security culture is actually going to pay off tremendously well for whoever gets hired by OpenAI and Anthropic to do cybersecurity. There's always going to be work. This is why you're optimistic. I get it. Thanks a lot, Tom.