Pt.1 Beyond Phishing: Cyber Threats in the Age of AI with Four Flynn

0:06The goal of the project is nothing short of helping improve the security for the whole industry by finding vulnerabilities and helping the open source community get them resolved for everybody to benefit from around the world.

0:23Welcome to Google DeepMind, the podcast. I'm Professor Hannah Fry. Now, cyber attacks have never been easier. From deep fakes that are so convincing they can fool your own family to phishing emails that look just like the real thing. AI has allowed these attacks to scale at a dizzying pace. But there is some hope that the same technology that's fueling these attacks could also be the key to preventing them. And few people know this battle better than my guest today. Paul Flynn is VP of Security at Google DeepMind and a cybersecurity legend. He was in the room during Operation Aurora back in 2009 when a massive attack on Gmail rewrote the rules of cybersecurity.

1:05Today, he is on the front lines again, taking on a new wave of AI-powered cyber attacks. And in fact, For had so much to say, so many totally fascinating insights to share, that we decided to make this into a podcast of two halves. Next time, we're going to be talking about the human side of cybercrime, how we can be manipulated and tricked by bad actors, and how all of that is changing with the era of agentic AI. But for this episode, we wanted to focus on the battle itself, the ways into systems that attackers seek to exploit and what we can do to defend them. Well, thank you so much for joining me, Thor.

1:43It's a pleasure. Now, I thought I might start by talking about one of the most notable security incidents in Google's history, Operation Aurora. Sure. How do you fit into that story? Yeah, well, so Operation Aurora was a huge moment in the history of cybersecurity writ large, really, for the industry as a whole. I think the idea that a nation state would compromise a private company was quite a shock to really all of us. We had essentially a case in which China was compromising Google or attempting to compromise Google. And, you know, as part of that campaign, they actually attempted to compromise a number of other companies.

2:22And it was part of a long-running process that they had to conduct espionage across a great deal of various institutions in the West. And specifically, they were looking for people who had been vocal against human rights abuses in China. Yeah, that's what we believe at this point. Of course, at the time, whenever you're dealing with these situations, it's very hard to figure out who the actors are or what they're attempting to gain access to. you. And so way back in the early days, when we first detected the attack, which was something my team was responsible for, is sort of finding that attack and responding to it.

3:00There's many, many, many people across Google that were contributing to figuring out what happened. And then, of course, after we figured out what happened, to evict the attacker from our environment. And then, you know, maybe even more importantly, over the following years, to harden our environment based on the lessons that we learned. But, you know, definitely in the moment, you have this thing called the fog of war, where you really have no idea what's going on. You really have no idea even what the bits are of the attack, and you're doing forensics to try to figure that out. And so there's still many people I work with here at Google that were instrumental in sort of the determining of that, Heather Adkins and others that are just absolute unbelievable experts in the subject matter that I'm lucky to work with.

3:41Just take me back to that time then. When did you first realized that something was up? When was the first moment of detection? Back then, it was sort of famous. Maybe it still is. You'd almost guarantee that when you have a big Christmas vacation planned, most likely that's when the cyber attack is going to come to light. And so it was in December, I remember when all the details started to come out. And many of us worked tirelessly, really over the break, but also for months, trying to ascertain what happened. and trying to put the puzzle pieces together when you don't even really know what the puzzle looks like.

4:17And so you're faced with this picture of just bits and pieces of technical data. I mean, presumably you didn't get Christmas holiday that year. No, no, none of us did. Were you just sat in a room with Sergey Brin for like 15 hours a day? There was a lot of us in that room for a lot of hours in the day. Yeah. And, you know, when we think about it, we look back on it. You know, I still get a pit in my stomach, I think. Well, how stressful was it, though? I am intrigued by this. I mean, because the thing is, on the one level, this feels like quite a technical challenge. But I mean, there is a human element to this too, right?

4:47Well, I mean, look, I'll say those of us that have pledged our life to defending people, as I have, and I know all the people I work with have, it feels like a failure, right? And that's where, at least for me, the source of stress comes from, is you feel like you've let people down. I mean, I spent my entire life to protect people's data, protect their accounts, protect companies' networks, all in service to helping people's daily lives be as great as it can be. You know, that intersects, of course, with Google services in a bunch of different ways, whether you're using a phone or a browser or even search.

5:25And we take that super seriously. How did they get in? Do you know now? Yes, we do know now. You know, this was an earlier era in security, but in some sense, some things have never changed. This was back when Internet Explorer, I don't know if any of the listeners remember that, but that was a big browser back then by Microsoft. And it was a vulnerability in the browser that was conducted via a phishing attack against somebody who was employed at Google. Someone clicked on something, basically. Yeah, exactly. And so phishing is still very much, if not a bigger threat today than it has ever been.

5:58But the browser exploitation was still in its earlier days back then. We called it client-side attacks, basically people leveraging weaknesses. Let me just say one brief thing as a digression. In the earliest days of security, most of the ways that attackers broke into systems were through server attacks. They would be on the internet, you would have a bank or something like that, and you would have a big mainframe or some big website. And people would attack through the front door of that website. And so it was really this Aurora attack as an example of that evolution that really has never changed back to what we call client-side attacks.

6:34And so the attacks shifted to be against the weakest link of organizations, which are often the users. And so they would exploit the users through social engineering, do phishing attacks against their passwords, for example, something I'm sure you've had to deal with in rotating your passwords in your own life. but also taking advantage of things that were running on the laptop or on the desktop computer, not on the server side. And it was part of a huge change in the industry where we had sort of historically built this sort of moat and drawbridge model for security, where we'd built these big class of walls with these big firewalls and had all the people and the servers inside.

7:09And that was essentially the common wisdom for how the security of companies would work. And there's a whole bunch of weaknesses in that model that evolves. For example, we realized that employees weren't always sitting in the same building anymore. The mobility became more and more pervasive, right? First, the rise of the laptop and then the rise of the personal phone. The smartphone broke that one axis of the mold. But then the other axis of the mold that got broken was client-side attacks because people were no longer trying to attack these very well-defended servers, such as the castle behind the big castle wall.

7:43Instead, they were attacking the client, which had several weaknesses. One, all the client-side software had not been hardened like that we'd done on the server side. So there was a big attack surface there. But also, the human element was much more easy to exploit using social engineering on the client. And so that also led to an evolution post-Aurora that we created called BeyondCorp. That's also known in the industry as zero trust, which is a whole new way of rethinking how enterprise security should work. away from this moat and drawbridge model and instead sort of acknowledging the importance of the client and the user as the supreme thing to defend.

8:21Like you've almost assumed that a perpetrator has infiltrated the network and it's sort of how to stop them, mitigate against potential damage that they might do once they have. Yeah, that's actually another element of it that you raised, which is this assumed breach is what we call it in security. And that also was another innovation because I think what we realized is that as good as the detection systems we had started to build to find these attacks were, and some of them were pretty good, including the one that we built to detect the Aurora attack, that said, you know you're not going to get everything.

8:51And that as attackers, and especially nation states, had evolved more stealthy and below-the-radar techniques that even the best detection systems on the market couldn't detect, then you had to sort of take a two-pronged approach. So no longer could you just rely on your detection system to flag these things to your analysts. You had to also take a separate step, which is assume that all those things failed and make sure that you were doing what we call assume breach. And so that means that you do things like threat hunts, for example, where you assume that you have the bad guys already on your systems internally, that you didn't catch them by any of the systems you've deployed, and that you're going to go and scour the entire systems to find the people that had already penetrated your defenses.

9:38And so again, that was another novelty that grew up in this post-Aurora era. And many other things, you know, multi-factor authentication tokens that Google does better still today than any other company that I'm aware of using our Titan security keys, right? So it's an unfishable multi-factor credential. As in like, it's not just a text message, which you could divert to another phone very easily. Right, exactly. That's one of the things we were able to do at Google is invent our own hardware keys in partnership with some other companies we were working with such that it's not just about sending you a message, like you say, that somebody can find in your email or in your text message and replay that attack.

10:17That's a pretty good start. A lot of companies have deployed that, and it's better than nothing. But there's a lot of weaknesses of that system. If somebody compromises the phone system, if somebody compromises your email, then you can still have your account taken over. Well, at Google, again, in the aftermath of this era, one of the things we invented was a non-fishable multi-factor hardware token, which connects directly to the browser and authenticates as a second factor without having some string of characters that could be stolen by an attacker. There's one other element of this that made it a really historic moment, which is Google's decision to go public with what had happened.

10:54I mean, why did they do that? There's a couple other points of context here that I think are useful. These types of attacks had been going on outside of private industry for some time. We'd seen these attacks happening in the Department of Defense and espionage happening with cyber attacks in the military industrial complex, you know, the various companies that make that up. And that had all been precursors to the attack on Google. And so I think part of the calculus for that decision was bringing awareness to this thing that had been going on for some time. And in this case, it led to a whole bunch of really positive changes in the industry.

11:38You know, data breach disclosure laws that are now on the books in a lot of places. I think it led to what then became responsible vulnerability disclosure, best practices, and generally just bringing transparency to security overall is something Google's, I think, really brought to the table across the board for many years. In the immediate aftermath, so this is 2009, right, did other companies learn quickly from what had happened at Google? I think the awareness came quickly, but what happened very slowly was the adaptation of our security best practices to confront this risk. In general, across the industry, it's been incredibly slow to adopt the more modern approaches to security.

12:22Like multi-factor authentication. Multi-factor authentication is now getting fairly pervasive, but it took 15 years from that event till now for it to really become something enterprises were able to deploy in effective ways. Zero Trust is another example. I know peers of mine at various companies are still struggling, if you can believe it, with the modernization of their environments to this new reality. And especially, I would say, governments, as you can imagine, that built a legacy environment are struggling to make that pivot. And so I guess the lesson I've learned is that unfortunately, it's hard to change things that are entrenched.

12:59But then as a result of that, there are sort of quite big, dramatic issues that arise. I mean, I'm thinking here about CelebGate in 2014, where celebrities' photos were leaked. That's right. That wouldn't have happened had they been using multi-factor authentication, right? That's right. Yeah. And yet that was five years after Operation Aurora. Operation Aurora, yeah. So you're highlighting another point, which is the difference between enterprise security and consumer. security. The other piece is consumer security often does lag behind enterprise security. And enterprise security itself sometimes lags behind the best practices.

13:33And so that's a good example where I think Apple even supported multi-factor authentication as an opt-in for iCloud at the time, if I recall, but very few people had adopted it. And really what we see in consumer security is the thing that really moves the needle is not asking consumers to change their behavior, but changing the defaults. And making the defaults more secure is the real needle mover to make people in their daily lives more secure. Because the public are resistant to change. I think that's part of it, or they're just not educated on what they should be doing. And so credit where it's due, I think Android and Chrome OS and the Chrome browser and Apple and a bunch of other players in the ecosystem have actually done a lot to increase the sort of default level of security on consumer devices and consumer applications to a pretty high degree.

14:28I think especially consumer mobile devices is a really brilliant case study in this. I think the defaults that you find now versus 10 years ago on, say, Android or iOS is just dramatically different. A lot of people take that for granted, but it's a lot of hard work. Well, if there are lots of positives to be sort of thankful for, I think I also want to understand the scale of the potential problem here. So before we get on to talking about how large language models have kind of changed the game and the era of generative AI have changed things, let me ask you about the different ways that we are potentially vulnerable.

15:00So there's social engineering, sure, but what are the other ways in? What are the more technical ways into a system? I think of it in terms of three categories of security failures, let's say. So as you mentioned, one of them, and probably the most frequently abused one, is social engineering. And one of the interesting quirks of LLMs is that they have somewhat human-like behaviors. And so, in fact, you can cause an LLM to get confused through similar types of things that humans do. But I think the other two categories are issues with configuration and issues of integrity. So basically, the way to think about protecting a system is you have to configure it so that it's secure.

15:40And then you have to make sure there's not a way to bypass that configuration. And I think pretty much everything in security falls into those two categories, everything in terms of security prevention, at least. And so let's just pick any particular example, access control, right? So if you have a company where you're sharing a Google Doc or something like that, you might have it shared with everybody at the whole company. And that means any one particular person that has an account that's hacked at the company could view that document. And so that's an issue of configuration. Getting the configuration right means having only the right number of people with access to that document that really should.

16:17Pursuant to the level of sensitivity to the content of the document, right? Okay, we all know this. This is normal security 101. Now, where does integrity come in? So let's say you've done a good job of getting that document locked down to the right number of people. But what could also happen is that there's a vulnerability like a patch missing on the server that's hosting that document. And so somebody could potentially compromise the server, bypassing the access control situation altogether. And then the third issue is that somebody who does have access to the document's password is stolen, and then that account gets access to the document.

16:49So you see we have integrity, configuration, and people as the sort of three classes of issues that I think really is every kind of security problem. But then sometimes that manifests in like slightly strange ways. I mean, I read one story a little while ago about a fish tank in a Las Vegas casino that had a smart thermometer on it. Yeah, I think I remember reading about this. It was basically somebody that was trying to cause financial fraud, I believe, or some sort of abuse of that casino. And they had had a fish tank that was on their internal network. And the system that was running the fish tank, of course, like everything these days, has an IP address and is connected to the internet, right?

17:30My toaster probably does have an IP address at this point, I'm sure. And so what that allowed the attackers to do is to gain a foothold on that network and then use that as a pivot point to attack the more sensitive systems that were undefended behind the scenes. This is a classic issue with IoT systems, where the problem with IoT is that they often don't have enough CPU, memory, and power budget in order to do a lot of the security best practices. And so you end up seeing companies that skimp on those things. And so therefore, you end up with these IoT systems that are deployed somewhat pervasively.

18:09And then if that's compounded with a system that's behind the scenes, like a server, like we talked about in that old model of the mode and drawbridge, that is poorly defended and relies on network trust, which I think is an anti-pattern now in security. but if you have something that has like by virtue of being on the network gains you some amount of privilege to being able to interact with that system then generally speaking that's a recipe for disaster. But I mean I think this does just demonstrate the number of different potential ways that you can be vulnerable. Yeah it's a great point I mean this we call this the defender's dilemma is the term we use for it in the industry and it's essentially this asymmetry between the the folks that have to protect against all potential ways to compromise your people or your company, and an attacker that really only has to find one avenue in.

19:00I mean, I suppose in a way, you almost have to assume that you have some vulnerabilities that you don't yet know about on your system. Yes. Everybody that does security defense has that assumption. Those have a name, right? What is zero-day vulnerability? So zero-day vulnerabilities are definitely a type of vulnerability that is very difficult to control for because those are vulnerabilities where even if you've done everything right, patching your system, putting your access controls in the right place and so on and so forth, a zero-day vulnerability is something that can compromise a fully patched, secure system.

19:34And so those are the class of vulnerabilities that most of us are laying awake at night. Those are the ones that we're often most scared of because historically it's challenging to defend against those. The vulnerabilities you don't know are there. Yeah, exactly. And code is complex, right? It's important to remember these systems that we all depend on in our daily lives are built on millions of lines of very complicated code. And so while the number of vulnerabilities is probably finite, it's also a very big number. And many of them have never been discovered before. And so there's always this latent risk of code that might have a vulnerability in it.

20:11That is never seen before. You know, just to make you feel slightly better, you know, we have this concept in security called defense in depth. The idea is that you build systems such that any one particular flaw, hypothetical or otherwise, doesn't lead to a catastrophic failure of the whole system. So let's talk about a zero-day vulnerability in a system. And there's all these layers of defense in modern operating systems. And so, you know, even if there's a vulnerability discovered, these days it's very hard to exploit those vulnerabilities because of all these added security protections in the underlying operating system that make it very difficult to, even if there's a vulnerability you can discover, to actually exploit it in real life.

20:53And, you know, there's all this trickery you have to learn how to do, such as, like, you know, exploiting the vulnerability, landing a certain amount of bytes in the memory, and then causing the operating system to go to jump over to that memory. And, you know, these are different kinds of overflows and so on. And there's all these memory safety features that have been built into modern kernels and modern operating systems to try to protect against these things. And moreover, when you zoom out and you think about a larger company, a zero-day vulnerability could perhaps compromise your phone or your browser.

21:22But the other thing that we talk about, and hopefully I'm not introducing too many concepts that are novel, but we talk about this concept called the kill chain. And the kill chain is a concept we borrowed from the military and now is a sort of fixture of cybersecurity. security. And basically the idea was, you know, we had been struggling against this defenders of dilemma that we have to stop every possible avenue for an attacker, but they only have to find one way in. And what the kill chain allowed us to think about is think about the problem differently. Like, yes, that's true. But even though they have to only find one way in, they still have to go through this series of stages that you understand what they're going to be.

21:59And it's things like reconnaissance, delivery of the exploit, post-exploit, sort of moving around the environment in the network and so on and so forth. And so this actually, I think, re-empowered the defenders and sort of re-tilted the scales and balanced them a little bit more. Because when you zoom out and you look at a whole company, obviously, if they just fish one employee, that's not good enough because typically they have a target that's deep inside a system that that employee might not necessarily have access to. And so there's this whole series of stages the attack has to go through.

22:31After they've phished that employee, they've gotten code execution on their laptop. Now they're trying to spread out through the environment. Now they're trying to figure out what server they're trying to access or what code base they're trying to access. And so there's all these opportunities to detect, to set tripwires, to have defense in depth try to block those additional stages. and the whole entire company now is your field of battle at which you can deploy prevention and detection technologies that inevitably can stop and detect and slow the attacker down. So even though they may have multiple areas of attack, you've now got multiple areas of defense.

23:06Exactly. The whole company now can become your field of defense. So, okay, how do large language models change this situation? A number of ways. I mean, are there new vulnerabilities? Are there new ways in which the systems can fail? Yeah, so there's a whole bunch of interesting new things that come out of the advent of large language models. I think both for defenders and for attackers. But before I get to that, there's one other thing I'd like to start as a foundation. I think one of the things that we're still wrestling with as an industry is that ultimately traditional computing systems are deterministic.

23:40I'd say the fundamental thing from a security point of view about LLMs that's different is that they're generally non-deterministic. Oftentimes you can give the same prompt to a large language model and it will give you different answers depending on random things, right? As it's tracing those paths of those tokens and producing those tokens through the model's sort of brain, if you will, you'll get non-deterministic answers. And so we'll get into the details, but I just think it's worth starting with that as that's a pretty big break from the past for us defenders and security. And I think some of us are still kind of wrestling with that difference.

24:14Now, in terms of attacks, I think large language models are still new in some ways. I mean, obviously, they've been around for a number of years. But I think from a security point of view, I think we're all still trying to learn what the risk landscape is going to look like. Could attackers use large language models to create malware? So there's initial signs that attackers are starting to figure out how to use large language models to create malware. We work, I should say, closely with our threat intelligence teams to carefully examine what the bad actors are doing. And we put out periodic reports.