Between Two Nerds: Why AI will mean infinite bugs

0:03Tom Uren: Hello everyone, this is Tom Uren. I'm here with The Grugq for another Between Two Nerds. G'day Grugq, how are you? G'day Tom, fine and yourself? I'm well. This week's episode is brought to you by Sondera. They make rules and like guardrails that work for AI agents so you can trust them to do what you want them to do and nothing more. Find them at sondera.ai. So AI is obviously having impacts upon cybersecurity, in case you hadn't noticed. And one of the things you pointed out to me is this tweet. And this podcast is going to break that absolutely wide open. No one has been on this, but we're going to be there first.

0:47Tom Uren: Yeah. So this tweet from Andreessen Horowitz, the venture capital firm. cyber is having a moment across 21 major software companies including apple aws microsoft and google reported critical vulnerabilities uh basically they've got a chart showing them up and to the right in massive amounts and so i suppose there's a question about whether this is good news or bad news is it good news that lots of vulnerabilities are being discovered is there like a clean out are we finding a whole lot of technical debt and removing it and so it's gone for good so heather atkins who's the i think it's the vp of security at google very very senior in google security looking at her tweets talking about this she is optimistic so one that we mentioned before reminder defense is going to win another one more recently she refers a couple of times to the vonpocalypse and she says the deep clean is upon us and the way that she talks it is the implication to me at least is that the deep clean will remove a whole craft and we'll be in a better place afterwards so i think that's one view and as of like a week or two ago i also would have been optimistic and the way i was thinking is that the problem with developing secure software is that finding vulnerabilities was somewhat of a specialist skill like there was it was not many people were good at finding vulnerabilities and so it was hard to incorporate that part that as a process into developing secure software like to to make secure software my thinking was you make software and then you figure out what's wrong with it to make it secure and so ai being very good at finding vulnerabilities yeah so maybe i'm my my mental model was entirely wrong in the first place.

3:07But my thinking was that... Microsoft spent a lot of time and money developing the secure development lifecycle, which is not really the write it and then find if anything's wrong with it approach. So that's a little bit antiquated. But you're not entirely wrong that there is the find everything wrong with it part of the process.

3:29Tom Uren: Yeah. And so my thinking was if that part, the find everything wrong is easier and cheaper and can be automated by machine, people will do more of it. And so software will be more secure in the long run. So I was optimistic that things will get better in the medium to long term. Now, I didn't know whether medium to long term is one year, five year, 10 year, 20 year. Right. But I was broadly optimistic. It's finite. There's an end in sight. And I always also, and we've discussed this before, I also thought that there was not an infinite number of bugs in current software just because current software is not infinite.

4:15Tom Uren: And I think because I'm fundamentally a positive person. And I concede that that is not a very good reason for having a belief about the way the universe is going to evolve. right so today we are going to talk about all the ways in which i was wrong and that i should have been a negative nelly from the get-go so i think that right now it's true that software is not infinite but i don't think that's always going to be true and the reason for that is the same reason as everything right now like i think llms like ai makes developing software so easy that people will not be like when i needed something in obsidian a couple weeks ago i didn't look through the the modules list to see if there's a plugin that already did it i just went to claude i said i want this feature as a plugin write it i went away 15 minutes later it's installed and working and does what I want.

5:22I think that...

5:23Tom Uren: Yeah. No, it worked. Like, we've had to iterate because there's been, like, things I hadn't specified that I've discovered, like, you know, but it gets fixed without me doing anything, just saying, I need this fixed. Like, this is the problem I have. I want it changed, and it changes it, and, like, it pushes an update, and I keep using it. So it's been absolutely great for me. I think this is... I know I'm in a special class right now of people who understand software and can ask for that. But I think that's going to change. I think what's going to happen is your mother will be going, I want a recipe keeper that I can use with my iPad when I'm cooking.

6:09And AI will just develop that for them. But I don't think they'll develop a secure version of it because that's something to be part of the spec.

6:18Tom Uren: Yes, that was exactly where I was going to go with that, right? It's a perfect example of you don't care about security. And maybe there's probably good reasons why you don't need to for your particular use case, right? But I think that my experience in the last couple of months has been that the software I use, there's many, many more updates that are shipped. And they're all vaguely kind of crappy. in that there's user interface things that used to work that don't work and they continue not working for some number of updates and then they get fixed and if you're not shipping software that is a marked improvement on what's there already like it's it's kind of maybe the trend is to have more features whether that makes it better software or not i don't know but if you're not like actually improving upon that every time like sometimes they're you know you're going backwards the chances that they're doing security right are zero that's absolutely fair right if you if you uh mess up a part that people can actually see the odds that you've messed up a part that people can see is significantly higher yes yeah that's like it's yeah and what i was thinking is that there were in the past at least frameworks or advice on how here's how you do secure software development was it the ssdf or something like that the secure software development framework there was yeah could be i mean that i think i think the thing was that historically you could only produce so much code that compiled and worked right like there was a you couldn't just like type and whatever you produced from that you could ship like you'd have to there was some minimal level of skill needed to make it not crash on its own immediately yep right so you had to have this like minimal level of stuff and a human could only produce so much and it was not easy but it was possible for another human to review that output in a reasonable time you could write 500 lines of software and i could read those 500 lines and it might take me a bit longer overall to figure out that it's not secure or whatever but it's a it's a realistic proposition but if you're producing 500 000 in a few hours right like there's just it's not possible there's just no way that a human can review it and then i did come across something just today where it was looking at the pace of software development and despite the rise of ai it seemed that in many i think it was large companies the pace had not really changed at all and it seemed like that had something to do with writing code is no longer if it ever was the actual bottleneck in terms of getting stuff shipped and so for example you would think that many of the types of software i'm talking about are relatively small software shops where you can imagine they implement AI, they don't have huge processes to review it and make sure that everything works properly.

9:41Tom Uren: And so you get this kind of incremental slop that... Yeah. What you get is someone who says to the same agent that's developing the code develops the tests. Right. And so what you get is like, it passes all the tests, but then the tests were never particularly good to begin with because they were written from the same spec by the same agent. Right, right. Yeah, so up until a few weeks ago, I was thinking, okay, so we've got a secure software development framework. I'll call it that. Maybe that's not the right exact term. And AI will help people follow this because the cost of doing stuff with code is lower.

10:26Tom Uren: Right. And that the phenomena right now I see where software gets better and worse, and it's kind of incrementally improving, but not in a linear way. It's going two steps forward, one step back, perhaps might be the way to describe it. And that over time, people will learn how to use AI so that it'll be more one step forward at a time rather than that sort of random drunken walk. Right. That is probably improved. And so I was optimistic that we were at a moment in time where everyone was learning how to use this technology in a way that it actually delivers what we want rather than just suck it and see with errors and mistakes and steps backwards and all that sort of stuff.

11:16Yeah. So the paper you were referring to, the companies are not necessarily more productive just because they have integrated AI into their development process. I'm a bit suspicious of it because I think that the quality of the AI has improved dramatically from when they did their data collection. Right, yeah. And I think that there's like basically there's step changes. Like there'll be these long, you know, for many months it'll be the same thing and then it'll be a dramatic improvement. Right. And I think that those happen fast enough that a study that was done earlier this year is no longer relevant to what's true right now.

12:03But I think what you're talking about, though, is a fundamental issue that isn't changed by how good the agents are. And you touched on this here, which is that it will make it easier to produce good code. but I think you didn't say it will make it just as easy to develop good code right like right right and that's the thing is that it's it will be cheaper to do it but not as cheap as not doing it right yep yeah so I what I was thinking before is you can write code

12:44Tom Uren: and ship products. And that's what most companies did most of the time. Or you can spend more time and effort and use a secure development process. And the key thing is you can spend more time and effort doing that and ship slower, but have, air quotes, better product from a security perspective. Right. And I think the... Historically, no one's ever done that. yeah and i think the people who do that are no longer around right yes and i think historically it was a rational choice right because it was the like you you you like you incur technical debt in exchange for moving fast right and moving fast allows you to stay alive which means that you can actually pay the debt down at some point in time at least that was the bet that was the bet and it That was true for a very long time.

13:40But I think, you know, now we're seeing that the cost of collecting technical debt has gotten very, very cheap. Right.

13:48Tom Uren: Yes. So, I mean, maybe that's optimistic. But so the problem I'm starting to see is that that was the situation before AI. And the situation after AI is you can very quickly write code and ship it. or you can spend more time and money making your code more secure before you ship it. So you can ship more secure code slowly. And so it changes the, maybe it changes the absolute cost, right? You can ship things faster, both secure things and insecure things. But the speed, if speed is important, it's always quicker and easier to ship relatively insecure code. And so if that dynamic hasn't changed, why would things get more secure?

14:39Tom Uren: We're just in a never-ending pumping out new products with vulnerabilities from the get-go. So I think part of the thing is that developing secure software starts at the very beginning. Like you need to have a secure architecture or at least an architecture that is easy to secure. Right. Right. And if you don't do that step, like if you don't have an easy to secure architecture, then basically it's a minefield. You're just going to always find new ways to shoot yourself in the foot. The thing is, I don't think that the improved software development agents do better security architecting than they did before.

15:29right like they don't because that that requires a level of context and planning and forward thinking that is i think it's very very hard to encode that in a written spec like to be able to

15:44Tom Uren: say like i was just thinking that if if they're trained on all the software that already exists all the software it's like okay okay my job is to create insecure code with vulnerabilities because that's what already exists. It's like doomed to repeat. Yeah. So I think that one of the ways of reading the deep, clean analogy that Heather Atkins was speaking about is that we're going through all of the existing software and we're just going to fix all the bugs now. We're going to scrub all the bad stuff out, replace it with good stuff, and then we'll be left with this pristine, wonderful, secure version of everything to build on.

16:35And I think that that would be a great outcome if it happened, but I'm not particularly optimistic. And the reason is, we've cited this before, but I think it's Martin's law, right? Where there's first you need the political will, then you need the resources, and then you do the technicalities. And the technicalities in this case is rewrite the software to be secure or fix the bugs. And the problem is like someone has to want to do that and then pay to do that with whatever the investment of resources is. And I think there's too much software that no one cares about anymore to do it. You simply don't have the people, like all of the OT software, like all the operational stuff, almost all of the things that have been in place for decades now, like replacing them requires getting them recertified.

17:38And it's actually more expensive to do that than it is to rewrite them. Right. So it's literally not worth it to fix it. Like it would cost more money to fix it than to just not fix it and accept the risk. And that's what I think the issue is. That it's just there's too much stuff out there that we rely on. So it's very easy to see like, oh yeah, like Chrome, like Google's Chrome. And it's probably going to get very, very secure. Right? Like they care about it. They invest in resources. They've got great people on it. They've got like lots and lots of compute. It's a thing that they can do because they push out updates all the time, et cetera, et cetera.

18:17Your washing machine that connects the internet, on the other hand, not so much. Yeah, yeah.

18:23Tom Uren: So I think it's interesting you mentioned Google because they've cared about security for a long time already. So they already have the political will. And so for Heather, maybe what she says is exactly right. They'll be in a better place. and they've published things like moving to Rust because they think or they know memory-safe languages are better, more secure. And they talk about how they've incorporated AI into their development processes to end up in a better place, right? Right. And so they're the perfect example of how you would use AI to end up in a better place. But then, to your point, they already cared about security beforehand.

19:13Tom Uren: It wasn't that AI came along and they went, oh, hang on, maybe now we've got the capacity to deal with this. I think I can mount an argument that AI will make all companies care more about security. And that is just the ability to exploit. AI, like ransomware, will make all companies care about security. And I think in a way, linking it to ransomware is right in that it makes it easier for malicious actors to take advantage of software flaws. And so if you've got more, I guess, in supply and demand terms, you've got more supply of malicious actors, your poor software or insecure software is more likely to be exploited.

20:04Tom Uren: therefore you've got more incentive to make it more secure like does i think that works doesn't that sort of makes sense um does i don't think the incentive shows up right the incentive shows up if it costs you right right so if if it i think it's an externality as a developer right it doesn't cost you when your stuff goes wrong it costs the people using your stuff and so in theory if your interests are aligned with theirs you would be incentivized to make it secure but in practice uh since you're indemnified and it's provided as is and it's sort of like yeah that sucks for you man well i mean the fundamental problem i think is that people don't care about security the vast majority of people don't care no one has ever cared about security right except for the the nerds of the nerds the people with mental problems that listen to this podcast and psychological problems i should say and like does i don't think that ai fundamentally changes that does it right and so no i don't think so i think but here's the thing I don't think it, like, I think it, I think it does change the number of threat actors who exist, but I don't think it changes the number of serious threat actors who exist.

21:43Tom Uren: Right. Right. Right. So I think you can now have more people who had no skills suddenly going, oh, I can hack a thing. And I think when they're done with that, they go, now what do I do? Yep. Which we've seen several times, right? And so I think that if I'm a company and I get hacked by someone, I don't really like that. But I don't like it a whole lot more if they immediately go on a breach forum and sell the data to other people. Whereas if they ask how to sell stolen data, how to sell stolen data, big money, right? Like dark web. I'm kind of okay with that because it means that it's not that the breach didn't happen, but it effectively didn't happen because there's no consequences.

22:32There's no sort of actual negative fallout. The debt is lost and it goes to someone who keeps it as safe as I did. Like it gets stolen from them.

22:42Tom Uren: Right, right. So if it's a novice actor who doesn't know what to do, that's fine. It's like kind of noise rather than impact. Right, and it's sort of, it's a benign version of a hacker, right? Like it's not great, but it's not necessarily bad. And at least, you know, to fix stuff and maybe you'll take things more seriously and you won't get a worse hacker. Okay, so I guess this is the optimistic view. Back to the optimistic view. We'll have a whole lot more inconsequential hacking that will result in people fixing things. right so I think this is the like we're going to have the vulnpocalypse but we're not going to have the deep clean but we are going to have the benign hackers so it all balances out in the end and so we'll have infinite bugs and we'll be balanced by infinite hacks that don't amount to anything so we're all back to square zero thanks a lot Tom Thanks Scott