Cybersecurity in the GenAI age

0:28Welcome to Practical AI. near your users. Learn more at fly.io. Well, our friends over at Speakeasy have the complete platform for API developer experience. They can generate SDKs, Terraform providers, API testing, docs, and more. And they just released a new version of their Python SDK generation that's optimized for anyone building an AI API. Every Python SDK comes with Pydantic models for requests and response objects and HTTPX client for async and synchronous method calls and support for server sent events as well. Speakeasy is everything you need to give your Python users an amazing experience integrating with your API.

1:16Learn more at speakeasy.com slash Python. Again, speakeasy.com slash Python.

1:35Welcome to another episode of Practical AI. This is Daniel Whitenack. I am the founder and CEO at Prediction Guard. And I am joined as always by my co-host, Chris Benson, who is a principal AI research engineer at Lockheed Martin. It's good to record one together, Chris. There have been a couple where we've been apart. That's right. Um, yeah, it's, it's good to, good to have the band back together, I guess. Absolutely. I took a brief respite. I had some, uh, lower back surgery. And since I'm always thinking about technology and AI, I was trying to imagine what, uh, what the surgery might be like today versus if I had done this, if we were in a slight time warp and gone to the future.

2:19Did they have a robot arm? Exactly. If, if they did, you know, reinforcement learning combined with, you know, kind of a chat interface and the whole thing. So it was kind of a novel mind experiment there that I had about what would it be if I was, this was a little bit different time. But yep, all good for here. Yeah, well, I would hope that in all of our futures, as we have medical procedures done increasingly by AI or AI-assisted and other things like that, that those things are very secure. as they happen, that would be very important, I think. Yes, absolutely. And on that front, I'm really excited because, Chris, on this show, actually, a couple times, I referenced the OWASP top 10, Gen AI top 10 sort of risk white paper breaks down security and privacy-related risks into sort of different categories and helps people think about it.

3:21This is a collaborative thing with multiple organizations involved. But today we've got with us Dennis Cruz, who is a founder at the Cyber Boardroom, but also has been involved in OWASP and in various capacities over the years and is aware of all of this that's going on and contributing to it. So we're just super excited to have you with us. Dennis, this is one I've been really looking forward to. Thanks for inviting me. And this is a topic that I'm very, very interested in, because I think there's a lot of potential. There's a lot of dangers, too. But I think it's very exciting times for us as an industry in all sorts of levels.

4:00Yeah. Well, like I mentioned, you've been involved in OWASP in different capacities. How did the kind of, I guess, give us a little bit of background and how did, first off, what OWASP is and what that means for those that aren't familiar. but also as you kind of started seeing this Gen.AI stuff come about, how did it strike you from that standpoint of being involved in OWASP over the years? Okay, so OWASP is the Open Web Application Security Project. And you kind of can say that OWASP is the people around the world that kind of cared about application security, that found that sort of intersection of the world moving to apps versus networks and then the security elements of it, right?

4:44And I think it's definitely one of those organizations has grown from nobody really cared about it to, you know, some more people carrying it. So now it's been referenced left, right and center. Right. And and has really, you know, changed the world in very, very nice ways. And it's always attracted, I think, the people that want to do better. I think it's a lovely community. It's very open, you know, by its nature. But it's also I think it's been a great hotbed of innovation. in terms of the first OS top 10, the testing guide, a lot of amazing tools come out of it, OSJAP, you know, dependency checker.

5:17There's billions of tools. I develop a couple too. And it's a great red community, right? And I think Gen AI, it's an interesting evolution on our technology, even from a security point of view, because I think for the first time, we have a technology that kind of understands intent, right? And I think that's quite different. Now, I want to say, ah, the bad, I think what Gen.AI is doing, ChatGPT and everything, is actually making most of what we always talked about in OWASP even more important, right? Because if you think about it, Gen.AI is an API, right? It's fundamentally an API that you send some data in, you get some outputs out, and in these days you can connect internal systems.

5:59And in a way, it's your most vulnerable and your most dangerous API at the same time. And I would say that the most thing that is quite unique with Gen.AI is the fact that you're now sending English or Portuguese, right? Or French or whatever language or Klingon, right? Whatever language you want to talk to, that is now code, right? And that's quite different. You know, for the ones who've been doing security for a while, we always talked about securing code and data, right? Like if you can separate code and data, we know what code is. We know what data is. You actually can secure things quite well.

6:33Now you're in a world where data is code. So that introduces a huge amount of interesting challenges. And I think it's one of those technologies that on one end, it's going to create a lot of problems. But on the other end, I think it has the potential to solve some of the biggest challenges of scale that we always had in security and in engineering, right? And in development with that technology too. So I think it's going to add up to both sides of the universe there. It's interesting. I actually want to go back to something you said right as you were starting off a second ago and build on it a little bit that, like you kind of alluded to, for a long time, you know, over the years, security was always kind of a, you know, the redheaded stepchild of the application developer.

7:17You know, there was a lot of folks that just didn't care too much. And obviously, we've progressed along and we're talking about Gen.AI and things like that, what we're doing now. But going back, I'm just curious as we're talking about security in general in app dev, even before we get to AI being introduced into it, what has motivated people to get into the security side in your view as you've watched this? Because I too have seen that the security concern go from being kind of a backseat thing to being very front and center and very important in a lot of organizations. and that's been an evolution.

7:51And as you've been in the middle of this ecosystem, how have you seen that happen and why? And where's that progression come to? Well, so I would say from personal experience and I think from most of my peers, there's a thing of it's nice to make the world safer, right? There's definitely a shag of like, it's nice to do something that the outcome is to make people safe. And I've always liked to stay on the sort of defense, you know, yes, a bit offensive, but like on making things systems secure, right? So I think intellectual is really cool. I think from a technology point, it's really cool. There is a thing of hacking and breaking into stuff, which is also quite interesting.

8:28There's a little bit of a James Bond hacking into a big system and getting there and all that jazz, which I think is part of the evolution. So I think it's a really cool industry. It's very open. It's very welcoming. It's also very pragmatic. In the beginning, it's like, can you exploit it or not? Can you do this? Can you do that? right? And I have to say, we need to also be pragmatic to say for a long time, it didn't really matter, right? Like in a way, there was a period where you knew who's being attacked because they were the companies who had the best security teams. Literally, there was a one-to-one correlation.

9:01That team does security very well, cool. They've been attacked, right? They're on the evolution, right? And I've now been a CISO. I've worked a lot of companies. And there's definitely, I tend to break into three different worlds, right? There is the nation state, you The high level of spionage, there is the ideology attacks, right? And there's the commercial attacks. The reality is that this one is a completely different level. If you're dealing with nation state, you know about it, right? You have hell of a team. And we can do it, right? But it's a very different game. The ideology, it depends on the industry you're in, right?

9:31Some industries are much more toxic. They tend to attract that. Most of it is to do with the business model of the attackers. So what has happened in the last, you know, 10, 15 years is that the more we move our society into digital, the more the business model of the attackers evolve. So in the past, you can get away with a lot of things. You could get away with crazy insecure applications out there and the probability of being attacked was quite low, right? Now you kind of can't, right? So I think there's been that evolution of, in a way, the stakes are much higher, right? Now, you know, even the recent CrowdStrike problem, right?

10:05You know, which I still think is a security problem, right? We can talk about it if you want, but that just shows how one problem in technologies can bring down, right? Large chunks of society, right? And in the old days, it was the NIMDA, right? The ISS worms that brought down a chunk of technology. But at the time, the impact was limited. Now, it really makes a difference, right? Like now a cybersecurity problem in an organization of a medium size can break the company, can really severe disruption, right? Even a ransomware incident in one of our suppliers can actually cause a lot of problem in your world.

10:37So I think, you know, the application security and OWASP and everything, we have in a way matured with the kind of evolution of the market, right? Although I have to say that I still feel that most companies, it's still a marketing exercise. It's a controversial view, but I think we still don't have a good way to measure the cybersecurity preparedness of companies. So there's still a lot of companies that kind of roll the dice and go, well, I hope it goes okay. Although much less than before, but I still feel there's a lot of maturity that we need in our own industry to do that. And just a final point, the thing I also learned a lot was that cybersecurity is not an isolation.

11:21Cybersecurity is a side effect of engineering and business practices. Right? So there's a moment where if you want to fix cybersecurity problems, you want to fix application security problems, you have to fix engineering problems. Right? And that leads to that. Right? If you do really good engineering, if you do really good development, really good practices, half of what we talk about in application security is not needed. Because you just do it. Right? It's just good practices. The problem is a lot of companies, a lot of teams don't do that. So the side effect is the security problems. I'm wondering when you're talking about this, this sort of business practices, the engineering, where these security vulnerabilities pop up and that sort of thing.

12:02Part of what comes to my mind on the Gen AI side is the fact that it brings the domain experts a lot closer to the actual technology in the sense that like often there are tools or there are interfaces that people are using to create sort of chains of reasoning just with prompts and other things, you know, just with natural language. So they're not, to your point, I think now this sort of natural language is kind of like the code of programming these models. How do you think that that shifts the dynamic between like the engineering side and the business side? Because from my perspective, a lot of these domain experts and business people are coming much closer to the actual kind of core functionality of an application.

12:46And I think that's the insane opportunity, right? So first of all, I don't view that the real power of ChatGPT is that it can create things. The real power of CTPT is that it can understand context, it can understand data, it can create mappings and relationships. And what is in practice, the way I visualized what you described is that in the past, everything you wanted to do had to be coded. It had to be solidified in bits of code, which is also where vulnerabilities were created, but also where in a way we were locking down the business logic into code, microservices, structures, et cetera. And then we got bogged down by the complexity of that, right?

13:26And then we couldn't understand the side effects and then things ground to a halt. And then again, particularly vulnerabilities appear in those gaps, right? It's those gaps, it's those misunderstanding of how APIs work or an API that was created over here in a secure state that is now plugged to the internet, you know, maybe not the best move, right? But I think where we're now entering a phase is that you can have a layer where the business logic is now described in prompts, right? And what happens when you describe the prompt, you start to describe the intent of what you want to see happening.

13:59And instead of that intent being locked in code, which is really hard to change, it has a huge amount of things, you know, side effects, you can have that in prompts. Now, we need to be better understanding the prompts. I'm doing a lot of work on provenance, deterministic AI to make sure that we actually have AI outputs that are deterministic, but it's very powerful to be able to start to describe that, right? Even I'll give an example. If you go to any organization, like when we do threat modeling, so one of the practices in security is threat modeling, right? Threat modeling is just social engineer the other side to give us an architecture diagram, right?

14:34That is up to date. Because once you have that, you'll find vulnerabilities, right? Because you start asking questions. What about this? What about that? What What about that? What about that? In a way, the hidden elephant is most organizations have no idea how the app works. They have no idea how actually things connected. The guys who develop it have gone. Now imagine a world where we start to use Gen.AI to explain how it works. Imagine a code commit that goes to a Gen.AI layer that I can ask the question, did this change the attack surface? Did the attack surface of my organization change because of this?

15:08and they say, oh, if you did, then we need to do a review, right? If you don't, cool, go all the way to production, right? Now, these questions in the past were impossible to ask, right? Or you had ridiculous static analysis that, you know, just about didn't work. Now we can start to codify a lot of those business logic questions, a lot of those intents of what I want to do, right, in, you know, basically in simple language instead of code. Because imagine, if you say, I want to do a change that allow anybody from the internet to access every record from this company, right? You probably go, ooh, hold on.

15:44I'm not sure. That's a good idea, right? Or I want this to allow an attacker to run and control JavaScript on my clients, right? Ooh, maybe not, right? I want my secrets to be put on a source code, right? Maybe not. But at the moment, all those things get locked in to code, right? And it's hard to understand the side effects because nobody arrives one day and says, I'm going to create a security vulnerability, right? Unless they're malicious, right? But apart from those, right? It's just a lot of times it's just genuine mistakes, right? So I think what Gen AI gives us is the ability to start thinking a lot more like three-dimensional, right?

16:23In our questions, but also in our applications where we start to describe the intent of what we want to do. And that can be analyzed. In fact, that can be analyzed by other Gen AIs, so we can start to get a much better sense of actually what is happening.

17:00For those who don't know much about Fly, what's special about building on Fly? Fly gives you a lot of flexibility, like a lot of flexibility on multiple fronts. And on top of that, you get so I've talked a lot about the networking and that's obviously one thing. But there's various data stores that we partner with that are really easy to use. Actually, one of my favorite partners is Tigress. I can't say enough good things about them when it comes to object storage. I never in my life thought I would have so many opinions about object storage, but I do now. Tigris is a partner of Fly, and it's S3 compatible object storage that basically seems like it's a CDN, but is not.

17:42It's basically object storage that's globally distributed without needing to actually set up a CDN at all. It's like automatically distributed around the world. And it's also incredibly easy to use and set up. Like creating a bucket is literally one command. So it's partners like that that I think are this sort of extra icing on top of Fly that really makes it sort of the platform that has everything that you need. So we use Tigris here at Changelog. Are they built on top of Fly? Is this one of those examples of being able to build on Fly? Yeah. So Tigris is built on top of Fly's infrastructure, and that's what allows it to be globally distributed.

18:18did. I do have a video on this, but basically the way it works is whenever, like, let's say a user uploads an asset to a particular bucket. Well, that gets uploaded directly to the region closest to the user. Whereas with a CDN, there's sort of like a centralized place where assets need to get copied to. And then eventually they get sort of trickled out to all of the different global locations. Whereas with Tigris, the moment you upload something, it's available in that region instantly. And then it's eventually cached in all the other regions as well as it's requested. In fact, with Tigris, you don't even have to select which regions things are stored in.

18:52You just get these regions for free. And then on top of that, it is so much easier to work with. I feel like the way they manage permissions, the way they handle bucket creation, making things public or private is just so much simpler than other solutions. And the good news is that you don't actually need to change your code if you're already using S3. It's S3 compatible. So like whatever SDK you're using is probably just fine. it all you got to do is update the credentials so it's super easy very cool thanks annie so fly has everything you need over three million applications including ours here at changelog multiple applications have launched on fly boosted by global anycast load balancing zero configuration private networking hardware isolation instant wire guard vpn connections push button deployments that scale to thousands of instances, it's all there for you right now.

19:45To pull your app in five minutes, go to fly.io. Again, fly.io.

20:12So, Dennis, you said something that was pretty intriguing to me, which I think is maybe a kind of distinction that is maybe interesting to draw out and get your thoughts on, which is this idea that when you think of the intersection of cybersecurity and AI, you could come at it from two perspectives. So you could come at it from how can we use AI to help us in our cybersecurity tasks or to create new tools for cybersecurity. And then the other side would be, well, how do we operate AI systems in a secure way? So there's probably some interaction between these two things, but could you give us a sense from your perspective as an expert in this field and also seeing a lot of things so far, how do you see the kind of maturity of these two sides of that coin?

21:07Anything you'd want to highlight on either side of that in terms of how both things are progressing, at least at the state of where we are now? So you're saying that the difference between using AI to sort of build systems and do things and then using one of those outputs is the cybersecurity analysis, right, of what you have? Yeah, I could imagine there's ways I could use AI to fight cybercrime, for example, or to prevent malware, or like you just said, to help explain applications. So that's using AI to help you create more secure systems, whereas there also could be just your AI system is insecure in and of itself, right, in how you've deployed it and run it.

21:51Yeah. And just on the second one, I think if you're not careful, most AI deployments are ridiculously insecure, right? In fact, we have to take into account that we still don't have a good understanding for how the models work. So the reality is there's nobody today that can tell us that these models don't have ridiculous back doors in there. Even non-intentional, right? Even maybe just the way it works. When we started this, people thought that a string copy was okay, right? People thought that a little catch between a memory copy in the OS was okay, and then realized that you can drive birth overflows, ridiculous exploits through it, right?

22:29So I think we're in a nation state at the moment now, like in early days of understanding everything you can do with a model. So my kind of view in this is that models that you want to use on that how to use models secure should be read-only, should not learn. You don't want them to almost bring any contents. You want to give you the content. You run them in complete isolation and you assume that whatever you put on it is already exposed and you verify the hell out of what comes out of it. Right? And I think there's a lot of companies who are rushing into pushing models. The problem is that they not take into account that the models themselves are ridiculously powerful.

23:11And this is where you want to imagine that somebody can put a payload that is then executed by a model. And that model sits now in the middle of your organization, in your cloud, in your environment, who probably has access to APIs or other assets, right? That is ridiculously dangerous, right? And that's what we're doing, right? So I think in one hand, I think we need to be very careful in putting models in line in how we actually validate the inputs and the outputs, which is kind of why I view them I mean, multi-tier sort of flows. And on the other hand, when we use them in a safe way, they're ridiculously powerful because going to your first form of how to use them for cybersecurity, what I really like is that I always felt that the model for cybersecurity is a model based on the attacker making a mistake.

23:58It's not about you protecting everything. It's about you want the attacker to make a mistake, i.e. make a call that was not supposed to happen, and make a download, make a connect with connection, access the application in ways that no user will access it, call web services that are completely out of sequence. In the past, again, it was impossible to model this. We tried increasing technologies. Even people that created ridiculous installations of seams and technology, et cetera, they really struggled at. But I think we now have a good chance of doing that. So that means that we can now create much more, I would say, hostile environments for attackers because we force them to follow the paths of the users, which, by the way, they don't know what those paths are unless they're already in your system.

24:45So I think we have a chance of using that, but what we need is we need models that are really, really reliable. So OWASP has an amazing top 10 for applications, has a really good top 10 for Gen AI models, right, etc., LLMs. What I think about is most of that is trying to deal with the fact that the models can learn and the models can actually be, you know, don't have deterministic outputs. And I like the idea of actually turning the tables around and say, hey, I don't want my model to learn. I want the data that my model has access to be completely determined by the session and the state that that request comes in, which is normal AppSec, right?

25:29And ideally, I don't even want the model to have knowledge, right? I want to give the model the knowledge that it's going to use so I control hallucinations, right? Like, you know, Chris, you know, you talk about your operation, right? Like, you don't want the Gen AI doing the operation on your back to slowly go off-piste, right? And start doing an operation on your leg, right? So in this theory, it would be that you want, for example, the Gen.ai model that is facilitating your operation to only know about back stuff, or maybe to know general things about the body, understand that, but the domain knowledge that it has should be laser sharp focus to the situation that you're in.

26:08And then that's how you control hallucinations, right? So I think the fundamental problem that we actually went backwards in security is that we now don't have a separation between code and data, right? And And I don't think we speak enough about this because for me, that's a massive problem, right? It's a massive problem because we really need to be able to distinguish what is code, what is data, what's an input, what's a command. So I'm doing a lot of stuff where I go from JSON to JSON and the latest model to this better where I almost want an API coming in. I give that API, which I can form nicely with data validation and stuff to the model.