#256 Stephen Schmidt: Inside Amazon’s AI-Powered Cybersecurity Strategy

0:00Security teams, as we talked earlier, can use Generative AI as one of a group of tools to address challenges. But from the security perspective, it's the use case that defines the relative risk. If you're using LLMs to generate custom code, is that code well written? Is it free of vulnerabilities? And does it follow the best practices that you've implemented for your company? And associated with that is, do you have the proper security controls and threat models in place for your particular use case? There are two major classes of vulnerabilities that our sensor infrastructure emulates. The first is sort of the oldies and the goodies.

0:35The big ones out there that we know our adversaries have exploited in the past, maybe an old version of a database engine or a vulnerable web server component or things like that. And that's to look for the kind of people who are doing mass exploitation of infrastructure around the world. Hey, Craig, Steve Schmidt here. I'm the chief security officer for Amazon. I've been with Amazon about 18 years, which in Amazon terms is a long time. And I'm responsible for information security, the security of our customers' data, wherever it comes from, as well as the physical security of our facilities and our personnel.

1:10Prior to that, I was in the FBI where I concentrated on counterintelligence activities of primarily the Russian and Chinese intelligence services. That's fascinating. Was that obviously electronic intelligence? And were you running systems similar to what Amazon does for its cybersecurity? I mean, was that directly portable, that experience? So what we spent most of our time doing was understanding how our adversaries exploited systems, whether they are a national laboratory computer system or the defense department, et cetera, what are the tools that they use, the techniques? How could we figure out what they were after?

2:00And whether a particular activity was just something with straight network exploitation or might have been human assisted, because obviously there's a different set of work you have to do in those situations. Yeah. And actually, that leads me to my first question before you jump into what Amazon does. I heard you speak at a conference called HumanX in Las Vegas, and you talked about running this massive network of nodes that you described as MadPot. It was HoneyPot, as they say in the cybersecurity industry, that I presume exhibited vulnerabilities that would attract cyber criminals. and you were talking about how you had built AI tools that could track the initial type of exploit and all the way back to the probable actor according to some database.

3:08Can you talk about that first? That was really interesting. So Madpot is a threat intelligence sensor network that we operate. It's tens of thousands of nodes that operate across the world. And their job, the individual node's job, is to be a honeypot. A honeypot is a system made to look like a customer system so that our adversaries out there try and connect to it, try and exploit it. And we get to gather intelligence about who they are, where they are, the kind of tools that they're using, what they're after, et cetera. And the nice thing about Amazon's scale is it really does give us visibility into the entirety of the internet so we can see what's going on everywhere around the world.

3:52Now, some interesting facts about the MadPod infrastructure. It takes about 90 seconds, 9-0 seconds, for one of our honeypots between when it comes online to when our adversary first tries to discover it. Wow. It's incredibly quickly. That gives you an idea. When people are talking about security on the internet, they're like, well, I'll just put this thing up there. It won't be up for long. People won't find it. No, you've got 90 seconds before an adversary discovers something. And the pivot from when they discover it to when they try and exploit it is incredibly sure as well. It's less than three minutes.

4:29So discovery to exploitation, three minutes. what we needed to do was to build a set of software and services that allowed us to digest all of the signals from these honeypots and to turn it into actionable intelligence. What I mean by that is you have to take all the logs from all these honeypots, put them together, build a model of what our adversaries are doing. And then in our case, we use some of our foundation models, our AI models, to take that raw information and turn it into indications of who is doing what using what tools, where in the world, focusing on what customers. Now, AI is not perfect.

5:08AI is not the solution to everything. But what it can do is it can dramatically reduce the amount of sort of grunt work that our security engineers have to do and to give them much more polished intelligence out the other end so that they can make good decisions about how we're going to protect people. To close that circle all the way, you've got sensors, you understand what's going on, you produce actionable intelligence out the other end of it. The question is, what do you do with that intelligence? What we've chosen to do is we use it internally, for sure, to protect ourselves, but we also build that into the services that our customers consume.

5:44So if you're using, for example, Amazon GuardDuty and you're an AWS customer, you get to take advantage of the threat intelligence that that MadPod infrastructure produces and is refined using those AI tools. Yeah. Okay. Can we drill down a little bit on that for listeners? Because, you know, I've had many episodes on cybersecurity and it's usually at a 40 ,000 foot level and, or it's deep in the, you know, the data stock and whether you're, you know, guarding what goes in and out of a database and things. But in this case, can you describe what kind of a vulnerability, as much as you can, you would put out there?

6:32And what kind of an exploit? Just give sort of a use case or an example so that people can really visualize it. Certainly. So there are two major classes of vulnerabilities that our sensor infrastructure emulates. The first is sort of the oldies and the goodies. The big ones out there that we know are adversaries that exploited in the past, maybe an old version of a database engine or a vulnerable web server component or things like that. And that's to look for the kind of people who are doing mass exploitation of infrastructure around the world. Those, of course, we build into rules in the firewall services that we've done so that customers can automatically just say, throw that traffic away.

7:16We don't need it. It's garbage. The other piece that we do, the second major tranche, is emerging things, new things that are coming out. So if a very interesting new vulnerability was just discovered in a pipeline engine that takes raw text and turns it into something valuable out the other end, we'll build an emulator for that, install it in the sensors, so that we can look for people who are trying to exploit the newest stuff, the most interesting things out there. And then lastly, there are situations where we'll build sensors which have no vulnerabilities that we know of, and we will look for people trying to exploit them because the people who are building the zero days who know something about the backend engines are always interesting to anybody who's collecting threat intelligence.

8:05Yeah, and these vulnerabilities are on servers, not necessarily on public-facing websites that were tied to public-facing websites. So is that right? I mean, these cyber criminals are scanning servers. They're not looking at Amazon webpages for vulnerabilities. Is that right? Lots of people try and look at Amazon webpages for vulnerabilities. It would be foolish of us to think that didn't happen. We know that it happens periodically because we see people attempting things. But yes, for what we're talking about here with MadPod, these are servers that we put out on the internet with the express intent of having them attempted to be exploited by an adversary.

8:54Yeah. In that 90 seconds, the adversaries are using AI tools to scan servers. Is that right? Looking for opportunities. of these? What we've seen with AI tooling is that there isn't a great deal of tooling focused on the exploitation of servers yet. What we do see quite a bit in terms of the use of AI by adversaries tends to be focused on the things where it used to require a human to take an action. So for example, building a very convincing phishing site or building an email that you can use to target somebody for something. Quite often, that took a lot of skill to make the language sound right, to make the imaging layouts work well, etc.

9:40AI tools can help adversaries succeed more quickly in that space. Similarly, AI tooling can help less skilled adversaries take certain actions more readily or more quickly. What we don't see a lot of, however, is the really higher-end adversaries taking advantage of AI to do things differently than they have before. Because remember, AI is not necessarily inventing something new. It's repeating a pattern that it's learned previously, largely, in training material. So it's hard to completely invent something new in that circumstance. But it is much easier for AI to build things more rapidly for adversaries than they could have before right the uh this relates to an ongoing conversation in the community the debate uh about open source and closed source and and with regards to cyber security closed source closed source models uh one of the drawbacks is that uh their security profile is not transparent uh so you have to trust the vendor but then again the vendor has a lot invested and presumably they're spending a lot of time on protecting the system on on the open source side people say well you've given away uh in some cases the source code uh and and that's going to give an advantage to bad actors but there's this worldwide community that's focused on the source code that can quickly find and fix vulnerabilities how do you see that debate or is that too close to what amazon does to talk about we've seen both the use of closed source and open source software writ large for many decades it's something that we do internally we use closed source software we use open source software it's all a question of how you implement it, what you're using it for, and how you monitor it.

11:56When you look at Amazon Bedrock, which is our service that allows customers to gain access to foundation models, we offer closed source models and we offer open source models because customers want choice. They want the ability to choose something that's right for their job, for their particular activity. I mean, some of the models are better at generating code while other models may be better at generating images, for example. Some are better when I can tune it myself because it's open source, but some are better because it's just so well tuned from the start by the creator of the model. So my opinion is use RITs right for your job.

12:34Pick the one that's appropriate for you, then build guardrails around it so that you know that your use of that model is safe and appropriate. If you're in an area, for example, that has high regulatory oversight, you had better build in visibility so that you can ensure when a regulator comes to you and says, hey, prove you're using this thing appropriately and safely, that you've got the logs that allow you to do that. And those are the areas where customers tend to trip is in, oh gosh, I didn't think that once I build this into my workflow, I'm going to have to prove what it did out the other end.

13:08Yeah.

13:12All of this data that you're collecting through your Madpot is fed into GuardDuty. Can you talk about GuardDuty? That's a product that you use internally, but is also available to Amazon customers. Certainly. So Amazon GuardDuty is a service that allows customers to build protections around their use of AWS and the apps that run on top of AWS. Think of it as a combination of a firewall orchestration, an alerting system, a SIM, et cetera, all put together in one place that allows customers to say, hey, is there anything weird going on with my use of AWS infrastructures and the apps that run on it?

13:55It alarms, and more importantly, it offers recommendations on how to fix the potential problem. So it'll say, for example, that hey you've got this particular database running on aws did you intend to expose it to the internet was that what you meant to do or should you lock this down that sort of thing yeah does that pick up for example a few years ago i was working with a guy that writing an article about uh i can't remember the name of the system but it's a medical uh imaging uh system that stores and shares medical images. And a lot of these places had left their S3 buckets open, and this guy was showing how he could scan the Internet for this particular opportunity and then go in and look at the images stored in that system.

14:57And a lot of it was lung scans of COVID patients, and you could get their PII there. You could identify who they were. I'm not sure that anyone was actually exploiting it, but it was pretty fascinating. Is that the sort of thing that GuardDuty would immediately flag, hey, you've left the door open? It will. So it's important to note that every Amazon service is built secure from the start. So for example, when you use S3, there is no public access to the content by default. That is the way it comes out of the box, as it were. Customers can choose to open that bucket to the internet if they want, but it is an intentional choice that they make.

15:42More importantly, though, there is a switch, a configuration that customers can set, which says block public access. And that means no matter what I do, don't allow this stuff to go out on the public internet. And that gives customers a sort of a hard fail safe in there. Now, Now, GuardDuty does detect open S3 buckets, and it will alert customers to that if they make a configuration change. It'll say, do you mean to do this? And customers then get an alert to do that. Lastly, though, and this is something that we haven't talked about a lot, is we have an internal program that we call Active Defense.

16:19Say again, Active? Active Defense. And so, for example, we look for people who are like the person you're talking about scanning for open S3 buckets, and we will intentionally mislead them. So we will allow them to scan our infrastructure. We will provide them with answers which are not correct about which buckets are open and which buckets aren't. and that has removed it's about 99.96 99.97 depends at the the day of the week the month etc of the scanners on the internet access to s3 buckets so basically what we do is we lie back to the the people who are doing the bucket scanning saying nope not there sorry go away or oh yeah there's an open bucket then they try and access and it doesn't work So it's a way to mess with them.

17:10Wow. You know, I jumped right in on this. I want to talk about alarm dashboards and agents, which you talked about at the conference. But I want to hear, before I do that, can you go back and just talk about, on a broader level, what would you do to protect Amazon infrastructure and how that then flows to the protection of customers? Certainly. So our job on my team is the protection of customer data, wherever that data comes from. And to do that, we have to protect our infrastructure. And protecting our infrastructure is what a lot of people think in designing secure systems, deploying them appropriately, testing them regularly, ensuring we got the configurations right.

18:03But it's also the process of making sure the access that our authorized humans have, our engineers, is appropriate for their job and confined to the minimum set of access as possible. That's an area that a lot of people in business often fail to think about adequately. You start off saying, I'll just give everybody root access to this because it's an easy way to get things done. But then you have a large number of people who have access to data, which is very sensitive. Now, our adversaries are intelligent. If they can't break in the front door, they're going to try and compromise one of your staff who have authorized access to that information.

18:40If you look at the Chinese intelligence services as an example, that is one of their favorite things to do. If you look at the way the Russians attacked some of the other big service providers out there recently and succeeded in doing so, they were going after that authorized access of people to data. So we have some very aggressive programs that we've run for a long time on scoping that access down as much as we possibly can. And it's something that I recommend strongly to our customers out there that they do the same thing. You know, if Steve doesn't need access to all of this data, just remove it because he can't be exploited for the purposes of access at that point.

19:19And deciding, first of all, scanning who has access and deciding whether or not to close access to something. Do you use AI tools for that? Because, again, in a large organization, that's a massive undertaking for human actors. It is something where we use tooling to look at who has access. What we do, for example, is we'll build models of team or human behavior. So here is Steve. He is a software development engineer. He is working on this team on this particular project. What do his peers have access to? Is his access similar to his peers or is it different? It may be completely appropriate that it's different.

20:01But what I want to do is have a high-judgment individual look at that difference and say, yes, it is appropriate in this circumstance. Now, if you're a small organization, you can do that kind of work using human eyeballs. At our size, you can't. You have to do it using tools. Yeah. You talked about agents. I think at the point when you gave the talk, you were citing an accuracy or dependability figure of 65 % for agents, which was too low to deploy them on sensitive actions or workflows. But that is changing quickly. I think it's up to 80 % now. Now, how long do you think before agents are reliable enough to give them autonomy or increase their autonomy in, for example, closing loopholes in a security system?

21:12I think it's going to be very situationally specific, meaning the individual agent, the individual action, and the situation around that in terms of when the accuracy will be good enough. What I expect to happen is a progression. So you get agents that are getting better, more accurate, more complete, et cetera. And you have to look at both the accuracy of the action and the false negative rate. Is it doing what it needs to when it's appropriate to do so? Is that action correct for the circumstances? When both of those are at the correct level of completeness, we'll see us move to a okay double check situation, meaning a human looks at the actions, you know, the AI proposes, let's do all these things because of these problems with this outcome set expected, etc.

22:00And the human says, yeah, yeah, yeah, yeah, that looks good, fire. And we'll see that work for a while, probably a couple of years at least. Now, there will be basic things that we can peel off of that and make more frequently automated without human involvement over time because they're lower risk. So as you determine, you know, this particular action has a low failure probability and the outcome of a failure is not catastrophic. Therefore, allow the agent to take the action because we can always go and undo it later if we have to. You'll also see people build tooling, which allows them to check the efficacy of agents' actions and automatically roll back if there is a problem in the action.

22:47And as that relates to alarms, alarm dashboards, this has been a problem for a very long time that cybersecurity systems have all these alarms. Yeah. And the engineers are constantly, they have all these blinking lights or beeping. In the worst case, they're getting calls in the middle of the night. How do you guys reduce that alarm exhaustion right now? And do you see agents playing a role? I mean, I can imagine a day where an agent is monitoring the alarms and deciding which ones are false and which ones are easily resolvable and doing that autonomously. I think actually that'll be the first set of stuff that we see accomplished completely using agents is the initial triage process.

23:50Meaning, all right, this one's sort of the more important stuff that I got to go elevate to a human right now. This set of stuff can wait for a minute. it. This set of stuff I'll just take care of as an agent down at the bottom. That's going to be a very logical thing for agents to do as agents become more and more accurate and people can implement them more easily. Then they'll move further up the stack of just saying that that middle tier will let the agent handle it completely. I don't foresee a situation anywhere in the near future, meaning five years, 10 years, where we will let agents run completely alone in an infrastructure like ours the consequences are simply too high to do that if something goes wrong now there are a lot of different kinds of businesses out there there are a lot of different ways to do things it's quite possible that at some point for example um there may be situations where people are saying you know what this is my coffee shop this is my bookstore this is whatever i'm completely okay letting an ai agent run security for me why because it's better than anything else I can do myself because I don't have security staff.

24:57That's where we're going to get a lot of big wins is on empowering small and medium businesses to have more skilled security staff in the form of an AI agent than they could ever have otherwise. Yeah. And is that being, or do you expect that to be productized by Amazon? So a small business, instead of having to hire a cybersecurity engineer, can implement an agent to handle that. I mean, they're not particularly high-stake enterprises, I mean, in the broader scheme, but they are for the individual business owner. Is that something that Amazon is providing or will provide? Yeah, I would say that it's super high stakes, actually, for an individual business owner.

25:52You know, it is literally their livelihood in a lot of circumstances. And yes, we build into our services the ability to take actions so that we can help you protect yourself. We can help you scale your business as you need to. We can help you save money by de-scaling your business when you don't need to be spending money. And all of those can be driven by agents. We will see more and more of them be driven by agents as the agents get more accurate and more complete. Yeah. What tools are you guys using to build agents, presumably their Amazon tools? We use a lot of our own stuff internally. The Amazon Q stuff and the Nova models behind are pieces that we choose to use because it gives us a lot of control over how we build things, what we build.

26:39That being said, we'll use a lot of different kinds of models as appropriate, given the circumstances that we're working in. Yeah. More generally, generative AI has increased the attack surface, as they say. I mean, people don't yet quite understand even how generative AI works internally. internally uh can you talk about that attack surface and and how much you guys have covered that attack surface at least with within amazon sure and let's talk about what our customers can do whether it's our internal customers who are amazonians or it's somebody on the outside i think that the questions that they need to ask as part of the development process in using ai are the same Number one, where is our data?

27:39Now, business teams are sending data to an LLM for processing, whether it's for training to help build and customize the model or through queries of that model. So how is that data handled throughout that workflow? How is it secured? That kind of info is really critical to understand. Anybody who's a user, a company, ourselves, et cetera, need to be confident that their data is secured, that it remains confidential, and they understand if the model provider will be able to access or use that data for any other purposes. Number two, what happens with my query and any associated data? So training data isn't the only sensitive material you need to be concerned about.

28:19When users start to embrace generative AI, and I'm sure you've done this yourself, they quickly learn what makes an effective query. They start adding more details and specific requirements because that leads to better results. if your user queries an ai engine is the output of that query and the user's reaction to the results used to train the model further what about that file that the user submitted as part of the query you need to be able to question answer questions about how the lm provider is going to use that data and if you're comfortable with it the query itself can be sensitive and should be part of the data protection plan however you implement it there's a lot that you can infer from a question that a user asks.

29:02And then going back to something we touched on earlier, is the output of these models accurate enough? The quality of the output from models is steadily improving. And security teams, as we talked earlier, can use generative AI as one of a group of tools to address challenges. But from the security perspective, it's the use case that defines the relative risk. If you're using LLMs to generate custom code, is that code well-written? Is it free of vulnerability? and does it follow the best practices that you've implemented for your company? Do you have the proper double checks in place before that code goes into production?

29:41And associated with that is, do you have the proper security controls and threat models in place for your particular use case? This is crucial and often overlooked, frankly. You have to start with a strong foundational security posture for anything you want to use for actual business. While novel generative AI-specific security risks like prompt injection get highlights and headlines, most of the security issues that we're likely to see are going to be because more standard security best practices were either hurried, rushed, or not even followed at all. So, for example, implement robust authentication and comprehensive logging is so important, whether you're doing something using AI or not, and associated with that is effective vulnerability management.

30:32These basics are essential, as most of the issues companies face with AI are really similar to traditional security challenges, just with different tools and in often different names for things. After establishing those foundational elements, then you can focus on threat modeling specific to generative AI. This involves identifying some unique risks, like prompt injection, where attackers can manipulate AI responses, and model poisoning, where training data is tampered with. So consider advanced AI-specific threats that are relevant to your particular use case. That might include model inversion attacks that attempt to reconstruct training data, or adversarial examples designed to fool AI systems.

31:21People have to develop and implement targeted strategies to mitigate these risks, like using differential privacy techniques. Or, of course, my all-favorite is conduct adversarial testing. Yeah, I want to talk about your program on adversarial testing and see how that's going. But first, you talked about where is your data going. how does i mean you know i i'm not uh concerned about my data particularly but but i'm uploading documents all the time to chat gpt or claude or perplexity one of the models available through written asking for a summary and yeah where does that document that i uploaded go does anyone really know.