NVIDIA’s Agentic AI for Container Security with Amanda Saunders and Allan Enemark

0:00Docker container vulnerability analysis involves identifying and mitigating security risks within container images. This is done to ensure that containerized applications can be securely deployed. Vulnerability analysis can often be time-intensive, which has motivated the use of AI and ML to accelerate the process. NVIDIA blueprints are reference workflows for agentic and generative AI use cases. One of the most prominent blueprints is focused on vulnerability analysis for container security. Amanda Saunders is the Director of Enterprise Generative AI Software at NVIDIA, and Alan Ennemark works on NVIDIA's Morpheus Cybersecurity SDK team.

0:40Amanda and Alan join the podcast with Gregor Vand to talk about Blueprints and their application to vulnerability and container security. Gregor Vand is a security-focused technologist and is the founder and CTO of MailPass. Previously, Gregor was a CTO across cybersecurity, cyber insurance, and general software engineering companies. He has been based in Asia-Pacific for almost a decade and can be found via his profile at vand.hk.

1:21Hi Amanda and Alan, welcome to Software Engineering Daily. Great to be here. Yeah, great to have you both here today. We're super privileged to have you from NVIDIA and we're going to be talking all about blueprints and obviously we'll get into what they even are. and especially in the context of security today. So as we often kick off, I think we don't often get to hear from people from NVIDIA. So I'd love to hear from both of you, kind of your path to NVIDIA and what you're doing there now. Absolutely, I'll kick off. My name is Amanda Saunders and I'm the Director of Enterprise Generative AI Software here at NVIDIA.

1:58So my focus is on NIM microservices, which are packaged AI models that are optimized to run on our infrastructure and blueprints. I've been at NVIDIA 10 years. So I've had roles from everything when we started with graphics through machine learning and data science. And now obviously into one of the most exciting topics that we get to see, which is generative AI. So it's been a really fun ride. Awesome. And Al? Yeah, I'm Alan Anamark. I'm part of the Morpheus team here at NVIDIA. And this is kind of the team working around cybersecurity, cybersecurity issues. and yeah been it feels a bit unreal to say it but at nvidia for about seven years now joined on as a data viz person on rapids which is we're still working very closely with but yeah actually my original background is in design industrial design actually and then kind of meandered my way to data viz and then to more product and now cyber security awesome yeah i mean just before we kind of dive in i guess both of you being at nvidia relatively long time now I mean, can you just describe maybe very briefly what it felt like to kind of, I guess, I think you said 10 years ago, you've been there for 10 years.

3:08Yeah. How does that feel? Because I mean, I think from the outside looking in, it's sort of, I mean, I was quite aware of NVIDIA given I like gaming and just keeping up with technological advancements generally. And then I think for most of the world, sort of NVIDIA has only become a word that you even understand maybe one year ago, maybe two. Yeah. How's it kind of felt from the inside? Yeah, I think, I mean, from the inside, you know, when I first joined, the only questions I got were about gaming, even though I've been focused on enterprise since I joined. So there was a lot of, oh, so tell me about your gaming GPUs and things like that.

3:41And now the questions are obviously very different. It's all about AI and it's all about what we see the future in. So I think that's a very cool evolution. You know, the other thing I would say is as somebody who's sort of watched the work that this company has done grow from graphics and particularly gaming graphics into AI, I think what's just really fascinating is that these building blocks, it's not like we started something totally new. It's all building on top of that original mathematics of graphics that then transitioned seamlessly into AI. And again, now this generative AI wave that we're seeing.

4:18So it's really fun to watch people as they see gaming, go to graphics, go to machine learning, data science, to where we are now, to the point where we're actually bringing that AI back into gaming. And so it's kind of a full circle exercise, which I think is pretty cool. Yeah, that's a fun loop. So let's dive into blueprints. So it's interesting, blueprints had crossed my radar from someone else in Singapore. But admittedly, I hadn't really gone in depth with them sort of until until we were going to do this episode. And the more I've got into them, I'm just more and more impressed and kind of fascinated by kind of what they are.

4:55So maybe can we just talk about what are NVIDIA blueprints? And I guess why this initiative? Yeah, I'll jump in here on, you know, really, this was, again, a journey, like everything we do here at NVIDIA, we've been building software, we've been building libraries, SDKs, now microservices. All of this is to package up software that can run on our accelerated infrastructure, on our GPUs, on our networking, so that, you know, customers can get the most out of it. Developers can build these applications. And as we built these building blocks, customers, developers, they were really excited. But what they started to say is, well, can you show us how to use them?

5:33Can you give us recipes? Can you give us guidelines that help us as a starting point? And so we didn't want to build end applications. We wanted to build reference workflows. And that's what NVIDIA Blueprints are. They take all the libraries, the SDKs, the microservices, even things from the open source and from the ecosystem around AI and simulation and everything like that. And it packages it up into something that can be taken, composed, customized to meet the needs of whatever company is taking them on. So I think that's really the why. And then they're continuing to evolve to solve some of the most common use cases that we're seeing out there and bringing on new blueprints from not only NVIDIA, but also our partners to just really help people take advantage of the software as quickly as they can.

6:21Yeah. And, you know, we're going to be diving into sort of one of those blueprints specifically today around vulnerability scanning. Could you also maybe just before we do that, just yeah, what kind of maybe some other blueprints topics that sort of blueprints cover that come to mind? Yeah, I think, you know, as of recording, we have about 14 that are available. As I said, I think we introduced five at CES a couple weeks ago. So we're continually adding to them. There's sort of four main categories that we're really building these out today. The first one is AI, very common, lots of things that we see there, whether it's customer service agents or digital humans.

6:57Obviously, as you said, the security blueprint we're going to get to dive into fits in there. Omniverse, which is our digital twin platform, lots of blueprints around simulation and how we can advance those use cases. Bionemo, the healthcare marketplace is a really important one for us. And we want to help those digital biology developers take advantage of all the new techniques in AI. And then the latest ones that are going to be coming out are for Isaac Group, which is our robotics platform. So as we see that start to grow, again, how can we make it faster? How can we get developers started more easily?

7:32That's where we're going to start adding blueprints. Nice. And yeah, I mean, just to kind of, I guess, describe if I'm looking at just the build.nvidia.com site and like what does that blueprint kind of look like you know i've got kind of three tabs here i've got experience so sort of actually being able to just try out what results that might come back if i was to use this blueprint i've got the blueprint card which is kind of giving me architecture and then i've also got nim nim did you say nim or nim how do you say Yep. NIM. So these are NIM microservices and they are just packaged, optimized AI models.

8:10Awesome. Again, making that run really well on our software and really easy. Yeah. I think we'll be sort of talking a little bit more about NIM and which NIMs, I guess, are being used in this specific blueprint. So I think let's now just sort of dive into the one that we're talking about today, which is vulnerability scanning. Again, we've had a few episodes on in the past just about this topic in general, usually focused on maybe a particular product. So, I mean, just kind of for listeners, maybe, I don't know, Alan, maybe we could just kick off with just simply talking very high level. One is vulnerability scanning in the first place for those that are not familiar.

8:45Yeah, absolutely. Cybersecurity, broad, broad topic, lots of different areas of interest. This one in particular is around CVEs or, you know, vulnerabilities and exposures. And that's basically this kind of registry, right? Where if there's a known security vulnerability, it's kind of registered there and available for everybody to look up and kind of mitigate on their own, right? Right. Now, the issue is there are so many CVEs that are registered. I think there was 40 ,000 last year. It's just going up every year. And, you know, this is in our case, if you're trying to release secure software, it's very difficult to kind of manage all that and kind of keep track of it.

9:28and you know in our case this particular blueprint is for container security right so if we have a container image you're trying to release a container you need to make sure that the cves it's scanned for are somehow mitigated or not actually a problem and then you have this many it kind of you could see how it doesn't scale very well right so if you're releasing thousands of containers and then you have tens of thousands of cves and each one is a real difficult thing to mitigate, it's hard to put out secure software. Yeah. So this uses the microservices. It also uses, I believe, NVIDIA Morpheus.

10:03And I think that's a product or a sort of, could you maybe just speak to what is Morpheus in the first place? Yeah. Yeah. Like we call it cybersecurity AI SDK, right? Which is sort of novel in itself. Usually when you think of cybersecurity products, it's like part of some sort of service or some company that's providing services. But what we've kind of found to be most useful is to release this capability out as SDK, which means that you can build the tools yourself with this framework, right? So this blueprint, like we said, is very composable. It's sort of built using the Morpheus SDK, and that sort of is an event-driven pipeline.

10:40And so the reason we kind of developed this framework in the first place is ultimately, like cybersecurity is a data problem. And, you know, there's so much data that it's hard to get grips on, you can't store it all. And if you have stored it all, it's hard to kind of munch through it. And so really, the way we're trying to approach that problem is to use it as like streaming in line event driven. So you can kind of manage things line speed, right. And so having GPUs enables that kind of for the first time, because you're able to process data rapidly and quickly on line speed and do ML on it, you can do ETL on it.

11:17That's a very modular kind of pipeline system. And so, as you can see, we're, you know, all kinds of capabilities through the Morpheus SDK. And a while ago, back to CVEs, right, it was sort of our own pain point where it was our own Morpheus container. And this was like during the holidays or something. And we got like a CVE, it was like critical. And nobody was around because they're all, you know, on break. And so whenever kind of like leads was, all right, I'll handle this myself. And he's going through it and munging through it. And like to do a good CVE mitigation, you have to kind of open up the container.

11:49You have to see if the CVE is valid. You have to go and see, all right, well, in this case, it's like a version is, is that the versions that we're using? Sure. Okay. And then is this an actual function call? And whatever, long story short, like hours and hours later, it turned out to be like not actually a vulnerability, right? Because in this case, there was something with some Java runtime, and we're not even using the Java runtime. But to even get to that point, you have to dig through everything. You have to kind of be an expert in code base. And at that point, he's like, there has to be a better way.

12:19So make something. You have a bunch of energy. Just figure it out. So that's how this whole thing started, to use this Morpheus SDK. And then we were like, OK, these LLMs at the time were starting to get really big. So we were like, well, how can we take these LLM capabilities and push it further in the cybersecurity stack, right? And so that sort of started the prototype of what eventually became this blueprint, and which was released out to the wild world, basically. Yeah, the CVE problem, as you call it, it's not solved by any means in the sense that, okay, it's great that we have CVEs that, you know, people report something that is in theory, a problem with a particular piece of software or hardware.

12:56But the critical problem is knowing what is actually critical or sort of even moderate. And, you know, there are some not exactly false positives, but there are also some there that are kind of not really a problem for anyone, but they're still there because, yes, you have to, in theory, you have to report things. It is a very messy space because, sure, you can have like tools that do the security scans for you, right? But the issue is like they don't have access to the full context of what your software is and what you're actually using it for, right? It's like a pretty naive scan, right? It's like, is this version and is this software there?

13:28And so that's sort of what, you know, traditionally would have this security analyst or application analyst do. And oftentimes it's, you know, not their full-time gig. That's like something they do on the side as part of some group, right? Ever like they meet every Tuesday. And so it's not something that, you know, is terribly pleasant to do because it's not even the software that you're writing yourself, right? You're sort of just, you know, you just want to deliver a secure software. And so you have to go through all this. You have to ask the right people. So like I said, it's a very kind of often tedious process where you're aggregating tons of information from different sources.

14:01You have to even validate that it's a CVE. You have to ask, like, how is this stuff used? So you have to be an expert in a lot of different domains, but then you also have to summarize a lot of information and then go out and find it. Right. And so that's sort of how we approach this problem in the first place is, hey, this kind of workflow that this person is running for is like, it seems like it's something that can be kind of automated in a sense. of like creating a workflow using this Morpheus and these LLMs, right? Kind of that plan, execute style LLM pipelines that we can then use to really like, you know, accelerate the timeline, right?

14:38Because some of these, it could take a long time to kind of figure out and like just collect all the information. And also like, you know, you just don't have enough people to do it, do it well. Yeah. I think that's what makes this such a great use case for AI. Humans do it. We don't like doing it. We're not particularly good at it because we get bored. We get distracted. We have other things that are on our plate. And so by leveraging, you know, an LLM to take not all the work out, the security analyst still has to come in at the end of the day and they have to have their expertise, but taking out the tedium that they don't need to do and doing that a lot faster and probably a lot better, to be honest, because it is an LLM.

15:18It doesn't get board. I think that's what really sort of helps us identify a great use case for AI. I completely agree. I was in a role a couple of years ago, where we were building, it was, you know, as a tax service management. So we were looking at CVs more from the outside. But exactly, we had a person who would be tasked with sort of trying to translate, okay, take the CV, what's reported from the CV, but let's try and change this into effectively natural language for the end recipient, you know, it's got to be very contextual, like this person might not be technical, so that can't be technical language.

15:51And I think that's, as you've just said, Amanda, that's exactly what LLMs are fantastic for being able to be applied to. Yeah, and it's sort of the magic sauce for these LLMs too, is also they're able to ingest so many different types of information modalities, right? If you look at our architecture diagram, it's like we are ingesting stuff from the actual code repository, we're ingesting stuff from the Sbomb, web searches, like general documentation. I mean, it does this whole stuff. Morpheus is really good at that. But you're able to kind of feed that all into, you know, using VDBs and kind of good prompting and stuff like that into something that LMs can just digest.

16:29Usually you'd have to even do the pre-processing back to more data analytics days to get it in a way that you could do traditional ML, for instance, is like, that's a huge project. It's going to always break, right? And so the fluidity that these LMs enable you to do just kind of widens the amount of context you can feed them. And you don't need that level of API specificity to kind of make it be useful still. That's sort of its superpower. And so from there, you can then more naturally describe tasks for it. And again, something that mimics how person-analyst-driven workflow would work. right like in our system you have one lm call that's setting it up to do a task list right the first thing you do is like all right well i have the cve like what are the tasks do i need to do to figure this out okay well first i'll check the version numbers and then second after that the version numbers match is like i'll do what as a deep function calls for this right and after that it's like all right is this actual function is you know the vulnerable aspect you know that is described in cve yes or no and then it kind of to the benefit of our systems like we can do all that in parallel, right?

17:37Instead of sincerely like a person would do. And then we kind of make an ultimate judgment on it with like kind of a summarization, right? So like based on all these individual tasks that these agents have gone out and like concluded on and like reasoned through through serial reasoning, are you ultimately vulnerable or not? And then kind of present that to the security analyst as like kind of a summarized view, which is useful in an outright saying like, yeah, you are vulnerable, you aren't vulnerable for ABC reasons. But we're also found that's super useful is kind of generating a very big report that is the full kind of reasoning process.

18:11And it's kind of marked down nicely so that they can go and verify the sources or like indicate that, oh yeah, this reasoning is accurate or maybe it's like a bit off here. Let's go tweak that. Yeah. And I'd like to maybe just step back one piece, which, I mean, we're just talking about sort of actually the architecture and sort of some choices around what's being used here. because I think that's where a blueprint to me comes into its own where some of these quite hard choices have been made for me in a really good way. You talked about plan and execute, LLM pipeline. Could you just describe that architecture a little bit and actually why that was applied to this blueprint?

18:49Yeah, absolutely. Essentially, something that best matched how the analysts actually approached the problem, right? And kind of mimics how you can break up a task into something that you can parallelize and then you can kind of connect this like reasoning to. So that's sort of the main structure of it, as well as what we're finding out is while this is a cyber specific use case, like the benefit of the blueprint is like, it's not, you know, it's extensible, right? And so a lot of people are saying, hey, you know what, this kind of general architecture, if I ingest different types of data, I do different prompting, you know, I can apply it to different problem spaces as well.

19:28Right. And so we having a lot of like, solution architects, which are folks that run out and kind of interact directly with partners, they're doing a lot of very interesting stuff with this as well. That is like cybersecurity adjacent, right? But they're using the similar sort of agentic architecture that this blueprint, like you said, we sort of got a lot of the complicated stuff sorted out and laid out for you. And like, hey, how do you connect these pieces together? What framework should I use and say, well, hey, here's a really good reference. You know, it's much better than starting with like a blank page right exactly i mean even the choice of models and both you know from the sort of pure lm side and the embedding side i believe is lama 3.1 7tb was the model of choice could you yeah just speak to sort of why that model and how do you even go about choosing a model like for this kind of thing yeah that's been a fun process we've gone through the gamut of so many different models here, which again shows the ability of Morpheus and Blueprints to be very extensible.

20:28We started with like GPT 3.5. We were experimenting with some Lora tuning to make it more cyber specific. And then essentially as we're prototyping this out, NIMS became available. And that just made our life so much easier because then it kind of handled all that stuff for us. And it's like, look, you just hit an API and we're like, awesome. We don't even have to spin something up. So we just ran with that And then we found that the specifically 70B was sort of the sweet spot with number of parameters. If you do more, like we have some 405s out there, it doesn't do that much more improvement.

21:03And then if you do smaller, the agentic actions are as good. It's not as good as a tool usage. So it's a little bit of trial and error. But then we kind of, with that, we sort of dialed it into 70B being the most robust. And in the case of a Blueprint, like you could mix and match models if you need to. if you want to go like just a little bit of extra performance or, you know, tweaking it for your use case. But in the case of a blueprint, it was just a little less complexity and easier to deploy if we just kept it to one model, which is pretty much good at everything. Yeah, I think one of the cool things about this, I mean, and you know this, Gregor, it's like models, it's almost weekly that we're seeing new models come out as well.

21:42So I think that was the great thing is Llama 3.1 70b is an incredible model. It's so powerful. And Meta continues to innovate on what they're putting out there. So as new things come out, we can put it in and we can test it. And because it's this containerized model, it's really easy to swap them in and out and then say, okay, well, the prompting has to be adjusted a little bit or things like that to get the same performance. But it does really simplify this down. So yeah, the blueprint today built on Llama 3.1 70B because we think that That works really well. But three months down the line, six months down the line, maybe we'll update it with some of the later models, depending on what new features come out in the models themselves.

22:23Yeah. I mean, I'm going through that process at the moment with a product that I'm working on, which is it's both that model as well as the embedding model. So I guess, yeah, just also speaking about the embedding side, again, I never want to assume anything from our listener base, sort of what they perhaps know and don't know. Could you maybe just speak to very briefly what is embedding and why is there such a specific model for embedding as well? Yeah, sure. I mean, the embedding space is sort of how you translate a document to something that the model can understand, right? There's a couple of different ways you can give the model context, and embedding is sort of the best way to do that.

22:57There's a lot of performance stuff you can do by creating an embedding. So, like, this is a really good example of, for this blueprint, we've made the embedding process and the VDB process, like, inline, right? and there's a lot of different libraries, rapids associated that are good for VDBs and utilizing them in very accelerated ways. Definitely check those out. In our case, it made sense to do it inline because again, for ease of deployment and when you're turning a code base into embedding, you kind of, all right, is this the latest version of the code base? Is this up to date? Yes. If we do it live, then that's not a problem.

23:34If you were to take this into like full actual production, maybe not the best methodology. you want to like separate that out, right? You want to have it. So that's like another process. I mean, still could use Morpheus. It's still going to use the same embeddings, but it's running like a nightly basis because you could have, you know, again, your thousands of different of code bases and that you don't want to do every time. And it's up to you to define your kind of process of like how often you want to update it and mimic that. But piping it into the LM would be the same way, right? So that's sort of like, you know, where we draw the line between this is a blueprint versus like what somebody would take into production, right?

24:07and kind of being mindful of it. And again, taking the ability of the fact that the blueprint is pretty extensible. It's not too hard to decompose. Yeah, absolutely. Talk a bit about sort of how all these processes are sort of how the data sort of makes its way into the system, if you want to call it that, and sort of at a higher level, how that's going to be queried. You know, there's a repo with all the information, incredibly well documented. So really - Thank you, we're working very hard on the documentation. I'm a little bit of a nerd when it comes to like good docs. So thank you. We work very hard to like look at the index.

24:41It's a huge, right? It's because we want to. Yes. Yeah. I mean, yeah, I was going to say it from such a large company, I was not expecting such good documentation. So I'm within a repo itself. You know, sometimes you can go find it somewhere else. But yeah, this is awesome. And, you know, arc diagrams in there, et cetera. And if I kind of look at the main one that sort of talks about the key components, you know, and, you know, on the left side, we've got sort of the ingestion, you know, things, you know, S-BOM, et cetera. And then on the kind of right-hand side, that's kind of all the bits that are then doing the figuring out, I guess, is mostly NIM.

25:16You know, we've got NIM checklist generation, we've got NIM task agent, NIM summarization, NIM justification. So I guess a question I've got is NIM walkthrough, like, are these sort of API endpoints or are these things that can be self-hosted or like, how do these work? Yeah, I'll jump in on that one. So yeah, NIMM is essentially a containerized, optimized model. So NVIDIA, as I said, has loads of libraries and SDKs that we've been working on for years. And rather than asking developers out there to go package this up themselves, we thought, why don't we do it for you, put it into a container that either you or your IT person can go deploy.