Between Two Nerds: Finding bugs does not mean security

0:02Tom Uren: Hello everyone, this is Tom Uren. I am here with The Grugq for another Between Two Nerds discussion. G'day, Grugq, how are you? G'day, Tom. Finding yourself? Well, this week's edition is brought to you by Sublime Security, which makes a next generation email security platform. Find them at sublime.security. so there's a post from the venture capital firm A16Z as we Australians would pronounce it and it talks about the new security agenda that's the subtitle or the title what CISOs told us about AI budgets and the priorities shaping 2027. So it's by a couple of AC16 people, Steph Jang and Owen Colby.

0:54Tom Uren: And it seemed like good BTN fodder. Why are we tearing into this neutral objective third party making a... So I guess my starting point is that I am somewhat sceptical about how useful AI will be for defensive purposes. So I think it's really interesting to see what industry professionals and senior industry professionals are thinking. And so I thought we could just start off by looking at some of the charts here. So there'll be a link in the show notes. So the obvious one kicks off with just what the environment is like. And the number of vulnerabilities is up and to the right hugely. No surprise there.

1:39Bit of a hockey stick going on.

1:42Tom Uren: This point, not used to anyone. Expected change in cyber spend. And so this chart shows that 87%, nearly 90 % of ZYZOs expect that they'll get a bigger budget in 2027. In any previous year, I would have said bigger budgets, maybe. but this actually seems like the the stars are aligned i would say that right i'd say that it the hype around ai security is so big that board members and so on are going to be aware of it and therefore they will approve additional funding so it's not i would say this is not necessarily a reflection of cso's concern for ai as it is of boards and other people like it's it's the management of the company.

2:35Tom Uren: Yeah, yeah, yeah. You can't walk into a room and say you do cybersecurity without someone asking you about AI.

2:44Tom Uren: So yes, that seems fair. Head count, about half think that it might increase, half think that it might be flat. That kind of seems fair enough. Seems a track, I guess. Yeah. So I was going to say, I say that looking at this, you know, half and half graph, I would say to me, that seems like a fair assessment of like, yeah, we'll probably stay at the same size or maybe we'll add a few headcount. But the way that they seem to be spinning it is people are not expanding their headcount because they're adopting AI to do more with less. And I don't think you can get that from these two graphs. I see how you could interpret it that way, but I don't see the support for it.

3:31Tom Uren: I mean, I guess it's up to CISOs to a degree where they spend their money if they get bigger budgets. I guess it depends what the process is, whether they justify things with people attached or not. So it seems like if you expect more money, how are you going to spend it might be a different question. And some of them seem to be saying, I'll spend it on more people. at least some more people and others are like no more people for me i hate managing people i hate the weirdos in cyber security give me computers any day so the next section which i thought was pretty interesting is ai rewrites the priority list and so there's a list of different initiatives this particular chart looks at what what's in your top five things you want to work on so i'll just read the top five most popular ones there's more than that because not everyone has the same top five ai sock almost half of respondents put that in their top five ai code security very similar number ai vulnerability remediation also 50 % or just under internal AI agent security AI usage and governance and that's these are all very similar numbers and then finally AI offensive security which is a clear second behind all of the others the others are much of a muchness right and then it sort of collapses from there um yeah yeah so like what's your first so looking at this um a lot of ai yeah it's gonna say ai sock is the one that sort of stands out as different from the others because if you look at it you've got code security vulnerability remediation uh internal security and offensive security as like these are things where ai finds bugs right right and then one of them is it finds and it fixes bugs so these are just these are essentially coding or hacking things which we know ai is good at so i would say yep that's exactly like that's what i would experience of vulnerability research or hang off it or it related to it in yep some degree or another like it's it's very much uh problems with a piece of software and fixing those problems or the configuration of etc like it's it's to me that makes sense because we know ai is good at that ai sock i think is more aspirational where people are looking at like socks are overwhelmed and there's just more and more data coming in, AI is good at Datapro saying, I bet you there's some sort of magic thing out there that's going to make SOCs much more efficient and easier and better with AI.

6:30I haven't seen anything for that yet.

6:34Tom Uren: Right, right. It seems like to me that would be plausible that you could break down the job of a SOC analyst and have AI help in discrete parts and probably do pretty well, right? Yeah. I mean, I'd say that I would be optimistic whether that product exists right now in a way that's usable. I don't know. But I think it's a solvable problem because it's ingesting data, finding things which AI can ingest data and it can find things. And then surfacing those things to be remediated, which is like that's all within the capabilities of what you'd expect from an LLM. So I'm optimistic that that would be a thing that can be done.

7:17Tom Uren: I'm vaguely disappointed that we're not just ripping into the... I wanted to be more skeptical, but now that I've... Yeah, like, this is the thing, right? Like, the things that it's looking at are, like, doing code security. So I'm going to be a little bit controversial and say, like, all of these people doing, like, code security reviews for their internal applications probably don't actually need to. Like, it's not bad. Like, it's not a bad thing to do it. But the odds of someone being inside your network and finding your internal thing and, like, developing a specific bug for that are vanishingly small.

7:54Like, I don't think that that's going to be your problem. Your problem is going to be that you have, like, a flat AD network and someone can get, like, basic access, you know. So, like, it's not a bad thing. I'm not against it. And I just, I'm not like, I think these are good things to do, but I don't, I don't think that they help like a smaller business. Like if you're an SME, these, like you don't have a SOC, you don't develop your own custom software. You don't do pen testing. You don't run internal agent stuff. Like none of these things matter unless you're a very big company. Right.

8:32Tom Uren: Yeah, so that was going to be my next question is, do those things matter? So a SOC for a big company, having it run more effectively, that would be good. Sure. Is it going to solve everything? No, but it would be nice. It seems like a thing to... If nothing else, it could make the lives of SOC analysts easier. So sure, go for it. Yeah. Now, one of the other things that is kind of related to this is that there was this, or there is this discussion on Twitter, X, whatever it's called nowadays, that was kicked off by, let me just bring it up, Hafei Li. I don't know who Hafei is. Oh, I do. He's a Chinese, very, very good hacker.

9:19yeah he's he moved into an ai company quite early on like i think he's developing his own thing

9:25Tom Uren: anyway dude's amazing um yeah so his question is it's quite obvious now that the ai frontier labs have been pushing very hard on vulnerability discovery from acquiring world-class vulnerability researchers to training specific bond discovery models however i don't think from a business point of view vulnerability discovery could be a long-term big source of revenue and it's just a very small part of the cyber security business so why do they do that what am i missing so this is kind of the flip side of that chart we just looked at where there's ai sock but there's a whole lot of stuff related to i was going to say one of the one of the other things about all of this is that they're continuous and ongoing which means it right so like if you have ai do secure deployment, that's a sort of once every whenever.

10:17Like it's intermediate. If you have it doing a SOC, that thing's going to be burning tokens 24-7. Like that is the ideal customer for an AI lab, right? You want someone who's just buying constant usage. Similarly with code scanning and all this other vulnerability like detection or mediation, all of these are very very token intensive and they need to be run frequently or continuously so those are things that i would want people to be adopting ai for if i was selling ai like like i don't want to be selling like um it's a bit of that pharmacy thing of like they don't really like selling cures. They like selling management medications, right?

11:11Like if someone's got a chronic disease, they have to buy medications all the time. That's great. They don't necessarily want to fix it because then they only, you know, buy it once.

11:20Tom Uren: Yep. So one of the longer replies is from Jags, Guerrero Saad, Juan. Guerrero Saad. I don't know how to pronounce his name, so apologies. I don't either. I just, I call them JAGs. Okay, right. So one of the points he makes, vulnerability research appears easier to RL, which I presume means reinforcement, than the rest of cybersecurity. So we've talked about this many times in the discovering a vulnerability and writing an exploit, there's a clear yes, no answer. it's unlike defense you can provably demonstrate that a model is good at vr right if you find a bug you have found a bug right if you have uh if you've mediated between two different departments and helped find a compromise resolution that will allow you to adopt some amount of improved security without impacting business throughput that's very hard to do right right like well i suppose uh i was just thinking of if i was in a job interview i would do everything i could to prove it but it's very hard to prove that that was the best solution or that there were no other solutions or that you did better than someone else these are things that are going going to have a menu of options from like you hard enforce the rules because those are the rules and you have to do them to be secure to you give them a complete exception because you don't want to impact business and then anywhere in between.

13:01And it's not clear which of those is correct. Right.

13:04Tom Uren: Yes. Right. And you're going to find out if and when you get hacked that you didn't do the right thing. It also feels to me very much like the results that they've produced for maths and physics and what's the other one i think those are the main ones right yeah it's maths physics and that they're doing a bunch of stuff in science but it's hard sciences where like right the parts of biology where it's like if you mix these two things together does it produce a drug that does something right right yes well like it's it's very much the like produce a thing that can be tested true or false very easily and which there's a lot of literature on which is very yeah like this has this effect sort of thing so i think they all fall into the same category where the maths is you send off a whole lot of agents they work and then they come up with a result and my understanding is that they use a separate programming language called lean and they can code it in that language to verify that what they've said is always not true.

14:10Tom Uren: So then you're left with the question of, well, is Lean a true representation of the world? Right, so Lean is very good for mathematical proofs and I wouldn't, like, because that's what it was developed for, it's done by mathematicians, I'm sure it's fine. Yeah, I mean, I wouldn't know. but uh like we're not seeing a lot of ai in anthropology for example well i guess there is no lean for cyber security defenders right i suppose right yeah it's this is this is the the halvar thing that they said long ago where you know all offensive cyber security problems are technical and all defensive cyber security problems are political and in a way that's the ultimate explanation of like if you have technical problems you can throw ai at them and it can explore that problem space sort of like branch out hit everything and find stuff for you that's great right but it you can't that's not an approach that works with humans in general right right have i told you about my like million monkeys theory of ai so remind me yep in my head The way I'm thinking about it right now is that AI...

15:29Sorry, I just have this vision of you with a symbol clashing monkey.

15:35Tom Uren: So, for example, in the OpenAI hack of Hugging Face, they sent out thousands of agents inadvertently, and they did all sorts of things. And it feels like to me most of those agents did not succeed in a particular task, right? but they were searching this kind of space and it's like they're not doing the dumb stuff. They're not necessarily doing clever stuff that will work but they're just trying everything that's plausible in a way. Right. And eventually they get a success condition and then they can move on. And so it feels like to me a million monkeys typing except they're kind of not dumb monkeys.

16:15Right. Not necessarily smart monkeys but they're not dumb.

16:19Tom Uren: And when you're hacking something else, you can see how that would be very useful. I'm going to explore the entire space as best I can. But when it comes to defending a network, what would I do with a million not dumb monkeys? I don't know. I'm sure I could find something, right? Right. Well, for example, I would say if you've got a sock, maybe a million monkeys can read all the data coming in and announce if they see something suspicious. I can sort of see that's a good use case. Then it feels like a filtering use case, right? Right, right. It doesn't necessarily feel like redesign our architecture, get everyone to buy in, arrange.

17:09right like it's it's all sort of like there's all the easy stuff where it's like uh ai redesign our architecture so it's secure fine ai go and negotiate with all the other business units and get them to agree to restructure their architectures based on our new design

17:27Tom Uren: that seems like ai is not going to help very much so one thing i wrote about in the newsletter was that CISA has released a strategic plan and it made essentially practically no mention of AI. So there's a throwaway line right at the beginning where it talks about superintelligence having an impact on stuff. And then to me, it seems like if you're in any sort of leadership position in cybersecurity, you've got to say something about AI. Right. Right. Because it's like the meteor passing overhead. You can't ignore it. Like if you just said, I'm a CISO, we're just going to double down on doing the basics.

18:13Tom Uren: And you like, I think that would be the right move though. Right. Well, I think you could say that, but you would have to say because of all the attacks I've seen, the basics would have been effective. Right. You'd You'd have to frame it in the context of AI. Exactly. So you couldn't say, like, I think that we've been doing the right thing all along, and we're going to double down on making sure that MFA is deployed everywhere, that our configurations are secure, that we remove old accounts. It's like just the sort of standard security hygiene stuff. Like if you come up with that and you're trying to convince people who get their security information from ads in airports, they're going to be aware that AI is up and to the right.

19:02And so you can't do that. You can't say, yeah, we're doing the right thing. We should just keep doing that against the apocalypse. But if you say everything that we're worried about is stopped by these novel concepts of least privilege, And separation, like segmentation, you can sell it in that context, I think. Yeah.

19:27Tom Uren: So it felt to me very strange that CISA made no mention of it whatsoever. Right. Because I think you need to mention it as this is the background of what's going on. Right. It's changing the level of threat. It's sort of critical context, I think. Yeah, exactly. and so i think that operating as a sisal you've got to incorporate it into the language of your messaging right that's nothing else if nothing else yeah so one of the things that jags is sort of touching on with this focus on vr like the vulnerability research is not improving security in a direct way that's not exactly what he says but he's you know like this this over emphasis on Vong research is a very small part of security.

20:14That reminds me of a previous A16Z.

20:22It reminds me of a previous A16Z post, which was about how tokens are like the new measure of security or something along those lines. It was very much like, look at the number of bugs being found. This has changed everything. Cyber is completely different. the world is you know blah blah blah that like because there's more vr stuff and anyone can do it now it's the end times because vr is so easy and like vr has never been the thing so way back in like so in 1996 there was or maybe it was 1999 it was very very long time ago before some of our listeners were born. There was a paper by Crispin Cohen.

21:08He was proposing that buffer overflows are like a bad problem. And he posted to a security mailing list saying, I'm writing a paper where I think that the biggest RCE issue is buffer overflows. I'm looking for feedback. And then he He didn't post the actual feedback, but he posted the stats that he got. And what's surprising is his question was along the lines of, I say, buffer overflows or memory buffer overflows are the worst type of RCE exploit. And so this was in the era where there were exploits for CGI. People don't know what CGI is anymore.

21:54Tom Uren: Well, I was going to say, so recently some people have said that they listen, but they don't understand what's said. So RCE, remote code execution. So usually that's the goal for an adversary. If you can get remote code execution, basically you can do stuff to own the network. CGI, I don't know what that stands for, actually. No, common gateway interface, I think it might have been. Right. Like, it doesn't actually matter. It was just back in the day, websites used to be HTML, and then you had to use a CGI script, which would then process input and generate a new page or whatever. So, like, CGI was the way that you interacted with web pages.

22:36And they used to be really badly written, so you'd send them, like, semicolon shid, and it would basically just run system on whatever you gave it. And so these were a sort of RCE exploit. And so the question was along the lines of, I say memory buffer overflows are worse than all of these other ones. And the responses that he got back were 60 % agreed, but 30 % just said misconfiguration is the biggest issue. Right. Which is not an RCE thing. It's just like in general, like misconfiguration is a bigger problem. And to me, the fact that like AI agents are getting in through misconfiguration, not through memory buffer overflows, says a lot about like even back in 99, 30 % of people had it absolutely right.

23:27Like that's the problem you need to fix.

23:29Tom Uren: right so it seems like they're the opportunity for ai is the ai red teaming in in terms of improving your defense because you would at least theoretically find these misconfigurations because they spider out and at least in theory you then do something about them right so the thing to me is that even way back when when buffer overflows were like new and very very easy to find and anyone could get them. There were people saying, like, this is not the real issue. And again, I think we sort of reached that point where now people are saying buffer overflows are easy to find, or memory corruption is easy to find.

24:11Anyone could do them. This is the worst thing ever. And I'm going to be in that 30 % saying, like, it's misconfigurations. Like, that's not all of security. Like, that's a part. Like, I'm in complete agreement with Hiveli and JAGS that, like it's a myopic focus on things that are very easy to prove true because you have a you've got a system that's very good at proving things true or false well i mean i think the problem

24:39Tom Uren: like to me ai is very good at finding both vulnerabilities and weaknesses in networks like it's good at finding mistakes because yeah yeah and opportunities i guess you know in chinese the word for AI bug hunting is opportunity.

25:01Tom Uren: And so it does seem to me that if you've got a budget to use AI, one thing would be to say, here's my network, have at it, what do you find? Now, the problem is that it finds you a report that tells you all the things that are wrong, and it doesn't fix it. like it just gives you a work list right right and so it can prove that you're insecure i think it will probably do that very well for many organizations but it doesn't help you actually fix the problem it just gives you i don't know if it's more work yeah that's right maybe that's why everyone is talking about headcount increases so those 50 are in fact the the enlightened 50 50%.

25:53Probably our listeners. Yes, of course.

25:56Tom Uren: So my understanding is that the AI labs didn't actually expect them to become so good at hacking. And that it was just as they started training them, the models, that they just got better. And that kind of, I don't know that I would say it snuck up on them. But it was a kind of revelation that they became so good. So relatively quickly, because it seems like over the space of about three months, they went from not very good to, oh, this is actually pretty good. Right. And so I'm not sure that there is any grand strategic plan behind the focus on vulnerability research. And it feels like some of the discussion, as we've said, is a bit myopic on cybersecurity, where there are other fields that have the same properties, where they're also...

26:48Tom Uren: So it feels like to me they're mining the opportunity there to get good results. Publicity worth results that are good at demonstrating that, yes, AI really is a capability that can achieve things, even if we don't know what these mathematical proofs are good for. Or if we don't know what the vulnerabilities we're finding. Like, I mean, there's no positive value to the vulnerabilities other than we can do a lot of work to get rid of them. It's like, you know, there is a societal benefit to not having them. But if we didn't know they existed, we would just continue on oblivious. So it's, you know, we do a lot of work to stay very skilled.

27:36If no one knew they existed, it wouldn't matter.

27:37Tom Uren: Yes, that's right. Right. So AI is very good at taking a piece of code and saying, this has problems that can be exploited in this way. So I'd say like AI can find all the bugs eventually, sure. I don't know that AI will write the next great novel. Like I feel like that's sort of, it's not provably correct, right? Yep. So for example, just recently, I think there was a, I mean, there's been a few scandals now, but the one I'm thinking of, I think, was a Trinidadian author had written a book that was nominated, I think, or it was in the shortlist or something, and then people read it.

28:23And it was just littered with AI tells. And, you know, I guess... I'm going to have to have a look.

28:34Tom Uren: So here's a phrase from it. The girl smiled like sunrise over a sink. she had the kind of walking that made benches become men I don't even know what that means oh my god I like this one it's got basically it's got a list of three things followed by an m-dash

29:02Tom Uren: so i suppose one of the things about these scandals is it seems to revolve or devolve down to we use some tool that said it was very ai like some of it is looks like to my eye that's just bad right right i mean it's funny bad it's or some of those phrases are right there was right absolutely no way as an editor i would say yeah that's okay i wouldn't go with that right So looking at this, I can see how reading it, it reads like someone is trying to be like a high literary author of some sort. And I mean, honestly, failing miserably, but very much an attempt at evoking a type of style. Right. But some of the other stories I've read, it's, yeah, we use this AI detector.

29:56Tom Uren: It got a high score, but we can't prove that it was actually written. Like there's no provability there. And it feels like cyber defenders. Like you can get AI, but you can't. And you can get AI to help. But there's no yes, no test to make it clear that your network is better off. I guess there's like maybe you'll have metrics. So like for the AI sock, you'll be able to say, yeah, we cleared more cases, whatever. But there's no, the flip side of being able to hack a network and get a result is not, you can use AI in defense and stop it. Yeah, that's terrible. I was just thinking of the, like the opposite of love is not hate, it's indifference.

30:46Right. It's sort of the thing I'm getting at. It's like the opposite of vulnerability research is not security. It's politics. Thanks a lot, Guy. Thanks a lot, Tom.