#205 Randall Degges: The Biggest Risks of AI-Generated Code (What Developers Need to Know!)

0:00A year and a half ago or so there was just a ton of layoffs across the tech industry. things are pretty rough, but that wasn't really a result of AI or anything. That was more just a result of increased interest rates, less easy to get VC money, you know, the macroeconomic environment. Today, I would say, even though the macroeconomic environment is still fairly rough in that regard, there's a lot more opportunities because of this stuff. I'd actually say that if you're a developer today and you understand how to leverage AI tools to do your job faster, you have a lot of employment opportunities.

0:31I wouldn't be worrying about it in the near future at all. Could you go ahead then, Randall, and introduce yourself and tell us about Sneak? Yep. So hello, everyone. I'm Randall Daggs. I lead the developer and security relations team here at Sneak. If you've never heard of Sneak, we are a developer security company, and we really focus on two things, which is finding security vulnerabilities in your applications, but more importantly, fixing those vulnerabilities so you don't have to worry about them. And yeah, that's what we do in a nutshell. So myself and my background is I've been a software developer for over 20 years now.

1:09I got started doing operating system security work back in the day. Then I actually got into being a technical co-founder of developer services companies. And for the last 12, 13 years, something like that, I've been mainly focused on building developer security tools for people with the intention of just making security nicer for developers overall. And we were talking previously about code generation. How much is SNEAK's security service or detecting security vulnerabilities? How much of that is automated? Are you using AI in that process? Absolutely. Yeah, it's been a longstanding thing. So many years ago, we acquired a company called DeepCode.

2:02And the DeepCode team essentially brought the first version of Snyk's AI products into our portfolio. And the way that Snyk works, so I'll give you a really quick sort of breakdown there, is there's a couple different pieces. So first of all, if you think about detecting security vulnerabilities, there's a lot of different ways you can do that, right? So you can do static analysis, which is what we do, where you basically analyze the source code, you analyze the dependencies, and you break them into an AST. and you parse this big tree style structure and you try to figure out based on rules that security analysts have created, how these things match together, is something being like used in an improper or insecure way, what the potential results of that can be.

2:50And you essentially like find vulnerabilities by just doing static code analysis. However, since we now have these big generative AI models, you can also do this in a generative way where you can basically take a chunk of code or an application, you can say, hey, please analyze this and tell me what security issues there are. Now, doing security vulnerability detection using generative AI is kind of a risky thing to do because it's essentially just using big training sets of data and patterns to come up with these answers. And so there's no guarantee you're going to get really high quality results.

3:28So our point of view is that for security detection, it's actually better to use like classical AI, like using knowledge graphs, using symbolic artificial intelligence, which is the proper term for it, to essentially find these things with a very high degree of accuracy. And the difference between these two branches is you have symbolic AI, which is like extremely accurate, but very niche. And then you have generative AI, which is extremely inaccurate, but very broad and can answer lots of different things. So in our engine, we use symbolic AI for a lot of the detection stuff. Now, on the fixing side, it's actually the reverse.

4:08So if you imagine that you've successfully identified these security problems in your application, and you're looking at them and you have all this information there, you can't really use symbolic AI and existing training sets that humans created to say like, okay, well now we're going to go. And if it's this very particular line of code and this very particular thing, we're going to like write this very particular patch. It's just, it's way, there's too many variables basically. So what we do is we use generative AI to generate potential fixes. Then we run those fixes back through our symbolic detection model to make sure they're actually going to work.

4:41And so we have a lot of effort that goes into both sides of those things. So we use this hybrid approach. We use symbolic AI for detection, generative AI for fixing, and we use a combination of these things together to make the most robust possible developer security tools. And that symbolic AI, that's rule-based. Where did the rules come from? Is there some compendium of security rules that has been compiled over time? Exactly. Yeah, you nailed it. So I mean, we literally hire security analysts and we even built this massive internal web application. So the way it works is if you're a security analyst at sneak, part of your job is you actually log into this web application.

5:28You take a look at examples of known vulnerabilities that we've fully detected and fleshed out and you look at them and you analyze them and then you look at other chunks of code that might be similar. And you're like, okay, well, I can tell with my expertise that this vulnerability is also present in this thing that wasn't detected. And so then we have a whole custom rules-based language that they write to help detect and build these heuristics out essentially for the detection engine. So at the end of the day, we get like a massive database of these things so that we can then apply these rules at scale.

6:01Yeah. And on the generative side, when you're generating fixes, uh is there uh you i mean you as you said uh you you have to run it back through the symbolic side to make sure the the generated fixes aren't hallucinations or or inaccurate in some way what what's the percentage of of false positives let's say uh in the generated code i mean do you do you find that uh the the generated code 80 of the time it passes the symbolic ai uh test or or what what's that percentage uh that's a very good question so i don't know the exact percentages off the top of my head what i will tell you is that when we started designing this functionality um one of the core things that we built it for is this like trial and error model essentially.

7:04So imagine you're a developer, right? So you're building an application and you're writing your code inside of your IDE. So you're probably using like VS code or Vim or Emacs or whatever, right? But you have your tool set up in your environment. If you're using something like VS code or one of those editors and you have the sneak plugin installed into it, then what happens is we'll actually provide some linting warning. So we'll like, as you're writing code or importing open source tools or whatever, we'll underscore things that have security issues in them that we've detected using that symbolic engine.

7:37And if it's something that we feel confident we can fix based on the type of vulnerability, the programming language, the frameworks being used and all of these other variables, then we'll give people the option in the editor to actually say like, fix this with AI. And if they click that fix this with AI button, what we do is we take the context, the code, the application, all that, it gets parsed by Snyk's engine. And then what we do is we use generative AI to then generate potential fixes. And we generate multiple versions of it. So I believe by default, we generate three for every single fix.

8:11And what we do is we actually run them back through our thing. If they're not suitable, we actually try again until we have three unique solutions. And then we present those to the end user. so we actually say like by default you'll use number one out of three but like you can cycle through them and see the differences yourself before choosing which one to apply if you want to do it that way so we we're trying to take this approach where like you know like yes gender AI is going to hallucinate a lot we're able to reduce the occurrences of that happening substantially by feeding it back through our symbolic engine but at the end of the day there's still multiple ways to solve a problem.

8:50There's not always like 100 % guaranteed way to fix something. There's always going to be multiple ways to do it. So we try to expose that optionality back to the end developer so they can then use their best judgment to see what fixes the best for their particular style or code base and things like that. Yeah. What's the model that you're using on the backend to generate code? But do you guys have your own large language model or are you using OpenAI or Anthropic? So to answer your question, we're not using any of the big model providers. We're not using OpenAI or Anthropic or any of those. We have our own stuff, which is based on open source models, but is then customized on top of that with our own training set information.

9:40um i'm not sure i'm allowed to provide these very specific details so i'm not going to say that but uh but yeah it's it's a big joint effort and we have a very large dedicated ai and machine learning team that that's their their primary job is building that yeah yeah and it's uh it's an open source model that then's then is fine-tuned or or yeah or do you do you have a vector database of millions of code examples that use RAD. Good question. Yeah, so it's actually both. So it might also be a good distinction to just talk about those two approaches because you mentioned like the biggest two. So when you're trying to make AI responses smarter, essentially, right?

10:30You really have a couple different choices. The two most popular are the ones you mentioned. So there's basically taking examples of things that are nice and breaking them into embeddings and storing those into something like a vector database. And what happens then is when you're asking AI queries, you're first going to talk to your database and say, hey, here's the question being asked. What are some chunks of context that are very relevant to this question? So it's going to pull those out and then send those along with the actual question to the AI model so that it can then have that context to provide a better quality answer.

11:10And that pattern is sometimes referred to as retrieval augmented generation, like RAG, RAG. The second option is you pre-train these models by fine-tuning them with tons of examples beforehand. Now, that takes a lot of pre-planning and forethought. And so if you have a lot of structured data that you can go through and carefully vet, then fine-tuning is a really good option. But ideally, you actually want to do both, which is what we do. So we'll actually take all of the stuff that our security analysts and ML team works on and basically fine-tune these models ourselves. But then in addition to that, we also have context that's provided both by the very specific code base that you're using and other heuristics to help improve the results even further.

11:55yeah yeah that's interesting and and i'm going to ask a dogleg question uh before i come back to to code generation i was talking to boston consulting group uh a week ago and they were showing me a conversational agent that they built and instead of using rag they're using the context window and and they load you know like if if if they want the agent to be able to talk about bcg research they load i think they can load up to 250 pages of text into the the the main prompt the system prompt uh and and just the the model then will will answer out of that Is that, and as these context windows, when people are talking about sort of infinite context windows, is that going to replace RAG that kind of, where you just put everything in the system prompt or the prime prompt or whatever it's called, instead of having this external database?

13:09Good question. And I think the answer is definitely not. So we'll not replace it, but we can talk about why. So right now there's like a battle, if you will, going on in the AI space around how big you can make your token limit, your context window essentially, right? And there's always a direct trade-off when you're taking that approach. So let me explain why it matters, especially for developers. So imagine that you want to ask AI a simple question like, please, maybe you create a prompt that says something like this. please generate a secure bookmarks web application for me. It should be written in Python using the Flask web framework, and it should be no more than 500 lines of code, and it should be perfectly secure.

13:54So it should be production ready so I can deploy to AWS. So this is your prompt, right? Well, if you just send that prompt with no other context to a model like OpenAI, we'll just use OpenAI since everyone knows about them, send it to OpenAI GPT-4 model. it's going to take it it's going to give you a response now how do you improve that response well you can improve it by providing additional context now in the case of this question what you could do is you could say okay well we're talking about flask we're talking about python we're talking about aws so what if we just built a tool that goes to the flask website the python website the aws website and downloads the entire websites for each of those and then dumps all of that right before the question says hey by the way everything in the next trillion pages of context is like relevant documentation you might need here's my question that's essentially what you're doing when you're inputting it all into the the context window for a question and the the fundamental problem with that is that first of all a lot of that information is already taken into account by these models during the training phase right so like flash documentation aws documentation like those things have surely been incorporated already.

15:07So you might be doing redundant work. But that's not the only problem. The other problem is that your request is going to be way slower because you're taking a ton of information and you're sending it in HTTP requests to a website that is then processing it. So there's a fundamental transfer time over HTTP. There's also the problem of like, this is now a much more computationally expensive operation for the model to compute. So instead of just chunking through like a limited amount of tokens, it's now chunking through a lot of stuff. So your response time will be much slower and it's much more expensive to actually run those larger queries.

15:44So it's a great question, but I think fundamentally retrieval augmented generation is like, it's always going to be around in some form or another because it makes things time and cost effective where there may not otherwise, it may not otherwise be possible for that. Yeah, that's interesting. So back on code generation, you and I have had a long conversation before about code generation. I'm very interested in it because I'm not a coder and I would love to be able to code things using natural language.

16:26The problem is the hallucination. And for somebody like me who isn't qualified to make a judgment on generated code, how accurate do you think code generation can be? And one of the problems causing the hallucination is that there's a lot of bad code in the training data. uh if if you trained if you curated your training set uh so that it's all very clean accurate code would that create a model uh that's uh that's more accurate in code generation than say co-pilot or something that's built on top of a general purpose pre-trained transformer model so yes and let's i'm going to break this down a little bit more because it's a very interesting question and there's a lot of uh there's a lot of potential impacts of this in the long run for our industry.

17:42So I'm going to go a little more in depth. So first question I think we should answer that's very interesting is if you have a model training set of perfectly curated data, first of all, is it possible to get that? And second of all, if you have that, is the model when you ask the questions going to generate perfectly generated things? So those are the first two fundamental questions. So I would argue that for software development in particular, it's not possible to have a perfect training set. There's no way to get there. And the reason why is sort of interesting, right? So if you are open AI, and you are building the GPT-5 model that is yet to be released, right?

18:27The first thing you care about is where do we get training data from? And the obvious answer is, okay, well, let's go to Stack Overflow. Let's go to GitHub. Let's pull in all these developer-y sort of places so we have good code examples for people, right? Well, there's a problem with that. The first problem is what you alluded to earlier, which is that there's a lot of bad code examples out there. Sure, that's a given. But a lot of those could be mitigated by having people go through these examples and prune out bad stuff or do all sorts of things, right? However, even with that being done, which is a huge assumption that like there's enough human resources to do that type of thing.

19:06The next problem is that what does it mean to actually have perfectly secure code? Like that question in and of itself is extremely difficult to answer because I can create a Python application right now. And that application I can design to be flawless from a security perspective. Like I can build a web app that properly authenticates people and properly authorizes them and does all this stuff. However, there's a lot that I just don't know that I don't know. So there could be new attack techniques discovered tomorrow, which would then make my thing vulnerable. And so it's not vulnerable now when I'm assembling the training set, but in the future, it will be.

19:44There's also a lot of context related things. So if you're building, here's a good example of this. you're building a secure web app to store and manage bookmarks or something, right? So you're storing and managing these bookmarks. You have authentication, you have authorization, you have all these things. Even if you're doing that properly, if you were to take that same context and then try to apply it to a banking scenario and say, okay, well, it's the same thing. We're just logging in, we're logging out, we're doing these things. It would no longer be a secure application because I guarantee you that the differences between a secure banking application, it's going to have shorter live session cookies.

20:20It's going to have different types of token based authorization. It's going to have all these very nuanced changes. So in your training set, there's no way to properly reflect all of that without just an, an infinite amount of information and curation. So yeah, I guess the, what I'm saying is I don't know if it's feasible for software development to have a perfect training set. And so because of that, you'll always have some form of hallucination or uncertainty in model responses for sure it's just a given i think and go ahead well well then and that's where the hybrid approach comes in where you have uh you know rule-based symbolic ai sort of checking the work of the generative AI.

21:08Yep. And we talked earlier about as these generative AI tools, coding tools, proliferate. And my understanding is more than half of the world's developers are now using some form of coding assistant. Is there a risk that the internet or the global code base fills up with generated code and that that generated code may not be as clean even if it's executable as human written code and and pretty soon you have a world where these code bases are nearly unreadable by humans that you're going to need you know another AI assistant just to check the code or read the code or find bugs and that sort of thing Yeah.

22:26So, I mean, that's a trillion dollar question you're asking potentially. If I knew the answer to that, I would be betting money hard into certain companies in the stock market today. I think the answer is nobody really knows. There's a lot of potential for this to go in either direction. so let's play it out for a second let's say that a lot of ai generated tools are generating software and documentation and other things that go onto the internet and that those same pieces of information are then fed back into the training sets to inform future models the main question is is that enough quality and differentiation to improve models or is it going to hurt models right um i think that as of right now the technology is at a point where it would absolutely hurt the models for sure however there's a potential to change that which is having human review uh carefully analyze new new things and curation really at the end of the day are you able to successfully curate this massive amount of information in in such a way that you're going to improve quality.

23:39And I think OpenAI, Anthropic, Facebook, a bunch of other companies are all actively working on solving this problem. There's a lot of tools and techniques and papers coming out all the time to address this. But all the latest ones I've seen at this point in time basically say, it's to be determined. We're just not sure how it's going to go. And so there's a lot of money and time being invested in that question right now. Yeah. And you mentioned earlier that already these large models have ingested most of the open source public code available out there. And so, you know, how do you go beyond what the models know today?

24:30Yeah, I mean, it's it's it's a very difficult thing to answer like if you assume and i think it's very easy to assume for sure because it's it's absolutely the case that all of the models have been trained on all available open source code that's like a given at this point for sure so where are they getting new information from the future i mean it's basically just coming from two places really it's either people publishing net new stuff into these databases maybe three places right so people publishing that new stuff, right? If you're a developer and you create a new web framework, for example, you build a website for it, you put it online, that will be included in future things.

25:07And so there's going to be that information eventually. Second thing, maybe there's not publicly available sources that are being utilized. And there's a lot of like security and governance and policy issues around that. But are places like OpenAI and other model providers, are they pulling in proprietary information from someplace? Did they buy it? Are they getting it from different places like we don't necessarily know? And then the third option is, are they just accidentally almost slurping in AI generated stuff without realizing it? And those things are gonna be derivative works. And so it's a lot of like, yeah, there's three potential sources of this.

25:48And the uncomfortable truth is that without a lot of human curated stuff, there's gonna be problems in one way or another. And so that's the biggest issue I would say right now is like, how do you do curation? Is there a way to do it more scalably? Is there a way to use AI to help with curation, right? Like how can you apply all these things to just make sure that progress can continue to be achieved? And then furthermore, what is progress gonna look like? You know, we've seen since 2022, right? Like the rate of AI improvement has been like exponential hockey stick growth. However, how long can that last?

26:24Is it gonna go up like 200%, 1 ,000 % in the next year? is it going to level off are we going to start going down those things are going to make a huge impact on the world and future generations for sure yeah and and as um i mean i've i've talked to uh to researchers about this not specifically about code but as generative models are trained on generated content, there's a reversion to the mean. You end up losing the variation. And I would think with regard to code, you would end up with sort of standard ways to do things that may be in that they're generated or not the best way to do things for a human engineer if he has to read the code or fix a code or that that's what I was referring to mean is it possible that we'll get to a day where there's so much code out there that's been generated by AI that that humans can no longer really parse through it without depending on AI that has been trained on all this generated code to begin with.

27:52And so you get into this loop where you can't break out of. Hey, it's time to make online shopping personal. Bloomreach personalizes the e-commerce experience, unifying real-time customer and product data to understand what customers really want. By connecting all that understanding to every channel, the e-commerce experience becomes limitless. Amplified by Loomy, BloomReach's AI for e-commerce, this creates endless new paths to purchase, greater profitability, and fast business growth. So give BloomReach a try. Yeah, and that's a very real fear, I would say. I don't think it's like unfounded to worry about the future of some of those topics.

28:42Like, you know, the approach we're taking at sneak for this is we're basically saying, okay, there are very specific use cases where generative AI is useful and helpful and can improve things. There's also very specific cases where it's very risky to use it. And so how do we mitigate the risk? In our particular industry, we're able to mitigate a lot of the risk by just using symbolic engines and models with human curated rule sets that can guarantee accuracy of certain things, which is great because that's going to take you from, let's say like a 10 % confidence level to like a 95 % confidence level, which is a very big jump.

29:19Getting those last 5 % are extremely difficult to do. And there's a lot of work going into that here internally, but what does it mean for things outside of our domain? You know, like we're very specifically focused on software security. And so we have the leisure really of being able to hyper-focus on these very specific problems. There's so many, there's an infinite amount of domains though, where AI is helping and being applied to problems. And not every domain has the luxury of being able to do the exact same type of workflow that we're doing to improve confidence there. So yeah, it's a very difficult thing to speculate on, I would say.

29:57Sorry for the wishy-washy answer, but Yeah, that's fine. Well, let's talk about how Snyk integrates into IDEs. You have this deep code AI fix. Is that right? Am I remembering that? But how does somebody work with this? And then can you talk about, presumably you've been referring to the research, what sort of research initiatives is SNEK working on to further enhance AI-driven code security? Great question. So let's start by talking about how it works real quick. So the way it works is really, really simple. So fundamentally, from a developer point of view, you have your IDE, maybe it's Visual Studio, IntelliJ, Vim, Emacs, whatever.

30:53What you want to do is install the Snyk extension for the IDE as like the very first thing. And when you do that, you can usually just go into like your, your IDEs marketplace and search for Snyk and say install, right? You then authenticate with Snyk, and it sets up all your API tokens and stuff in the background. So everything just works. from there what you do is when you're writing code sneak is going to analyze in real time as you're working on the code all of the different things happening in your project so analyze your custom code that you're writing it'll analyze all of your open source dependencies that you're pulling into your project to utilize for different things it'll look at all of your container definitions your infrastructure's code definitions there's even a runtime agent you can deploy but fundamentally, it looks at all these different things.