Pt.2 Beyond Phishing: Cyber Threats in the Age of AI with Four Flynn

0:04Who do you think wins in the long term? Is it cyber criminals or security? I think the defenders ultimately will win. We won't necessarily win every battle, but I hope that we can win the war. Welcome to Google DeepMind, the podcast with me, your host, Hannah Fry. We are bringing you part two now of our conversation with Paul Flynn, VP of security at Google DeepMind. In the previous episode, we talked about the terrifyingly large numbers of ways that our digital systems are vulnerable to attack, as well as the ongoing fight to defend them. And 4 is completely full of stories and incredible insights, having spent the last two decades at the very forefront of international cybersecurity.

0:45So if you haven't watched that one yet, press pause on this, go and have a listen, come back here later, we'll still be waiting for you. But for the rest of you, enjoy the episode. Well, up until now, we've been talking about the real technical challenges that come with protecting our systems. But as you mentioned at the end of the last podcast, that's really half of the problem in a lot of ways because there is also this very human side too. Who's creating the attacks, the tactics that they're using to trick us as people and also how all of that is changing in the era of AI. So I wanted to pivot to talk a bit more about that if I may.

1:20And maybe it's worth us starting off by talking about these bad actors. Like who actually are they? What is their motivation? Well, I mean, bad actors come in all shapes and sizes, I guess. There's a number of different classes of these. I think in general insecurity, we break them down to nation state actors that I think are largely focused on geopolitical aims, often espionage or offensive cyber attacks that are in support of warfare operations on the ground. I think there's also a recent concern about pre-positioning. Even if there's not a hot war going on between two powers, oftentimes there's cyber offensive operations happening as a pre-positioning.

1:58Like maneuvering in case a hot war starts. Exactly. So we've seen in some areas signs of this happening in places like power grids or critical infrastructure. Right. So knowing that there's a potential real world conflict, you infiltrate the power grid system so that you can act on it later. That's right. And maintain your presence and validate that you continue to have command and control of those environments periodically. Every few months you'll see somebody come back, make sure the lights are on, make sure that their systems still work, make sure they can disrupt things and then they go away.

2:30So that's unfortunately fairly more common than most people would like to admit. Yes. And then you sort of see sub-nation state quite a bit. And some nation state activity blurs into this, but a lot of that is financially motivated, right? And so you'll see, for example, ransomware, which I'm sure you have heard quite a bit about over the last sort of five years, which is very unfortunately quite common. And usually that's an attack that basically causes a company or a person to have their data basically stolen and encrypted and held for ransom. The more modern incarnation of this is that you as a company would have a core database that you rely on as well as a backup to serve your customers.

3:15And your whole business depends on it. They would slowly and quietly compromise your company, unbeknownst to you. and then all of a sudden one day they would lock out your ability to get access to your backups and your database and your whole company is offline. You would get a demand for a certain number of Bitcoin or what have you, cryptocurrency, to pay them to get the data unencrypted in your business back and running. And when you say sub-nation state, is this, I don't know, for example, a country that has a number of sanctions on them, it's sort of a way to generate funds? Yeah, so you do see nation states actually using it as a way to raise funds.

3:50But you also see sort of independent attack groups loosely affiliated or unaffiliated with nation states conducting these sorts of attacks sometimes as well. Is there a bit of a grey area in terms of whether someone is bad or good? I'm thinking here, we were talking about how there are these zero-day vulnerabilities all over the place. We don't know where they are, but finding them can be worth a lot of money. I mean, if you were an individual searching and found a zero-day vulnerability, you presumably have the choice to sell it to a bad actor. But you could also sell it back to the company itself, no?

4:23Yeah. So, in fact, there's a whole active black market, grey market even, for vulnerabilities. And there's a number of different buyers of those vulnerabilities, right? So some buyers are companies that are trying to equip law enforcement with tools to access the devices of people who have committed crimes. Various people have different views on that, but that's definitely one component of the ecosystem. I think also governments are one of the biggest buyers of these things. And so, again, it depends on what government you have allegiance to. It depends on how you feel about that, I feel like.

5:00but it's worth noting that these vulnerabilities are very much worth their weight in gold. They're very expensive, some of them. You know, it can be like we talked about many millions of dollars, in fact, sometimes. And there's a varied number of people buying these things out there. That is quite an interesting thing, a way to imagine it then, that it's like treasure hunting, as it were. And Google's kind of coming in and saying, we're going to hunt for all the treasure and then fix it so it no longer has any value. I mean, that's right, because I see a risk that AI, if we were to take no action, I think it could by itself disrupt the marketplace, right?

5:41Because we know bad actors will do the same things we're doing eventually, which is to find these vulnerabilities and make them, unfortunately, potentially more accessible to more bad actors. I mean, that's the concern, is that there's this wave coming of people using AI that previously was very exquisite, very expensive to find these zero days and these vulnerabilities and use them to cause damage. That's the risk. I'm not saying that's going to happen or it's not going to happen, but that's one thing we worry about. And therefore, again, that's why it's important for Google and others to invest in these systems to be the best there is at this and to use that for good.

6:24And to help the ecosystem heal itself and to improve itself and get more resistance instead of just waiting around and letting the bad actors do it. I think there's another really fascinating aspect to this, which is about the decision to go public, right? Like the act of transparency as a security measure in itself. I mean, tell me about the motivation behind Project Zero. How did that come about? Yeah, well, I have to give credit to the absolute incredible folks at Google over the years that have innovated Project Zero. But Project Zero is really an absolute world-class effort combining, I think, some of the most elite hackers that have ever lived.

7:01Their job day in and day out was to find novel vulnerabilities in systems. And then having a vulnerability disclosure and transparency approach that was designed to help the ecosystem become better. I mean, one of the things that Project Zero introduced to the ecosystem that at the time was unbelievably profound was this idea that there's a 90-day disclosure timeline on all companies. You have 90 days from when you learn about a vulnerability to patch it or we're telling the whole world what it is. Wow. At which point all bad actors can exploit it. That's right. For free. For free. Not even for sale for$50 million on the dark web.

7:39Right. And at the time, I think everybody was like, like you are, thinking that this was crazy, right? But what it did is it completely changed the way all people in the industry prioritize security. Because the problem was that the good guys would find a vulnerability and disclose it to a company. And then those companies often wouldn't prioritize them. And say, yeah, yeah, yeah, we'll put it out in the next release of Windows or whatever it is. And then basically six months, a year down the line, nobody fixed it. And then the bad guys separately and independently found the same vulnerability and were exploiting poor consumers everywhere.

8:15And so Project Zero said, no, that's not how this works. Now it's just normal for everybody to get a vulnerability through a vulnerability disclosure bug bounty program. You know, a lot of times people pay out and reward, you know, folks for these. And then, yeah, I mean, I'd say more often than not, these are often fixed within the 90-day window. Again, all thanks to Project Zero. So that was a huge impact on the industry. Have there been instances in the past then where people have known about really serious vulnerabilities and deliberately sat on them? Well, I think there's cases of governments doing that because they want to use them to exploit other countries.

8:49Give me an example. Well, you know, I think probably one of the most famous examples of that in security history is called Eternal Blue. Eternal Blue was widely believed to be attributed to America's agencies that were using these tools as part of national security for defensive purposes and offensive purposes. And, you know, I mean, I don't want to call out any particular country or group. But, you know, in the case of Eternal Blue, that was somehow leaked externally as part of a toolkit. And then I think some of the vulnerabilities that were novel in that toolkit were actually weaponized as part of a worm that was called WannaCry.

9:32Which was the one which targeted the British National Health Service. But the thing is about that one that was interesting as well was that there was a patch that existed for it. Is that right? There was a way that the vulnerability didn't need to necessarily be in those systems. Yeah, so that's right. A patch was produced somewhere along the timeline by Microsoft, and I believe it was prior to the worm taking hold at NHS. And so I think it's worth an interesting digression to discuss briefly why NHS was so vulnerable to this worm when indeed a patch was available. What happens is often that you go and you try to apply the greatest patches, and you're trying to do the right thing as a security person.

10:18And if you go into a hospital, you'll find you'll be surprised. You know, those things that are next to the patient's bed are like running windows. Yes. Or at least the computers that you go and you interact with with your bracelet, you know, or whatever to figure out what drugs dosages to give and take the notes from the doctor. Like these things are running operating systems that we know. And so as a consequence of that, I'm pretty sure what must have happened is the poor security team came along and was like, we're going to patch these systems. And they tried to apply the patches. And one day it broke half the hospital.

10:56Right. And then some administrators said, look, our job is saving lives here. We can't have these systems not working. We have a defibrillator and all this stuff. We can't mess around with this. And then they become institutionally risk-averse to change in their IT systems. And the side effect of that is that you have short-term prioritization of human health, whereas you end up with these long-term vulnerabilities because you don't have the ability to safely affect change. Absolutely. In the environment. You see this also in other critical environments like critical infrastructure, like power grids or hydroelectric dams that also often are running based on Windows or Linux systems or what have you.

11:40And again, it's the same situation where they're so critical. And ironically, the systems that are the most critical then have a side consequence of having the least acceptance of change. And that lack of acceptance of change means that you have the least ability to affect security improvements. to the environment. So this is unfortunately the way the world is. Absolutely. But this, I think, does really illustrate the depth of this problem here, that it isn't just a technical challenge. You have got the human element of this. That's right. So let me go on to the idea of social engineering here as a different potential point of failure, of getting in through the human.

12:17Yeah. How has AI changed that landscape of social engineering? Yeah. You know, I do think this is one of the things about AI that is concerning from a security point of view is the use of AI to cause deep fakes, basically being able to clone somebody and have a video, live video experience. You know, so imagine the CFO calls you up in a video conference. It looks like them, you know, they interact and have the same idioms and the same voice as them. And, you know, if you're a middle person in the finance team and you get the CFO on the line, you know, you might well be convinced that it's truly them and be asked to transfer money through a wire transfer.

12:57And this has literally happened. And it's happened multiple times already to this date. And that's one example of a deep fake attack. I think there's other examples that we've seen of, you know, where somebody will call up and have cloned their daughter's voice and call them up on the phone and pretend that they're being held for ransom and that the mother needs to send money. You can imagine any of us would struggle to have our wits about us in those types of situations, especially if it sounds like your daughter or your son, somebody you've known for so many years. And so there's a whole bunch of different types of these social engineering type of attacks that are really novel, that are enabled by AI.

13:33But in a way, it sort of enabled bad actors to make their phishing attacks more directly tailored to the individual they're targeting. I think that's right. You know, phishing attacks had already been pretty successful for some time. We had this concept in security called spear phishing. I'm not sure if you've come across that one. And in fact, a fun side note, there's also whaling. So spear phishing is targeted phishing that's sort of bespoke for that individual. And then there's whaling, which is a term for basically going after CEOs or really high profile people. So I think the problem with LLMs is that there's a whole landscape of new social engineering risks that are introduced by them.

14:15I try to always give a bit of good news along with the bad news. I think the good news here is that a lot of the best practices historically that we've invented in security are still coming to the rescue. So imagine you work at a company, right? We talked about that CFO scenario. Well, if you have strong multi-factor authentication that requires not just a face and a video that looks right, but in order to be able to call you in the first place with such a video system, they would need to be able to have a strong form of authentication that isn't fishable, that's only connected to your finger on your laptop or whatever, then that is pretty strong defense against that sort of attack, right?

14:56Do you think that we're going to be living in the era of passwords forever? Or do you think that from the consumer side that AI will make that a thing of the past at some point? Well, before we get all the way to AI, there's actually encouraging progress even well before that. So probably the thing I'd turn your attention to most is something called passkeys. Passkeys are actually a really exciting new innovation that's coming to a login near you. And you might have seen it before. It's essentially when you log in and instead of typing in a password, it'll show a QR code that you can take a picture of with your phone.

15:27And because your phone is already logged in on your behalf, it automatically logs you into that service on the web. And I'm a huge fan of Pasky's because if you just have a phone, you take a picture of your screen, what have you, you're just logged in. It's easier and it's more secure. I think another thing that you've probably seen is that if you have multiple touch points, multiple surfaces with a given company like Google, you can do push to log in or push to approve. And so like if you're trying to log into a new Google property like YouTube, you know, if you have the Google app already logged in on your phone, you can just go to that and say yes or, you know, on your Gmail account.

16:01So those are two examples, I think, of things where we're moving already past passwords as it is, which is great. Because I think passwords were important innovation at the time. But clearly, we all know that human brains are not compatible with passwords. No, 1, 2, 3 passwords. Exactly. And so I think it's really good that we're nearing the post-password era. Of course, AI can play a role too. And so there's a concept in the industry called risk-based authentication. And this has been around for a long time, long before LLMs. And so what this would be is that you basically are connecting to a sensitive system like a bank or something like that.

16:41And it basically looks at a mass as a number of different signals about your behavior. How's your mouse moving on the screen? What are you clicking on? Does that look organic? You know, you've probably seen like click and say you're not a robot. All of that is tracking your mouse and things like that oftentimes. times. So in other words, it gains a whole lot of different signals that you're not even conscious of that it's using to ascertain whether you're a real human and whether or not you're really you. And so those particular way you type. Yeah. And so this is pretty common in financial services, but it's really common across the industry that you obviously want consumers to have the least friction as possible to engage with your service.

17:18But you can step up the friction level, pursuant to the risk level. And so as you're like looking at what somebody is doing and gathering all these signals and behavioral features, you can actually choose to say, well, you know, normally I would just ask for this person's password because we already trust their browser and we've seen them log in from this IP address before, but their mouse is kind of moving weirdly and their keystrokes don't line up to how they used to line up. And so we're actually going to ask them to do a multi-factor authentication in addition to a password this time around.

17:47That's so interesting. And so that's actually something that, you know, is only getting improved with AI, but has actually been around for quite some time. So it's like the system has a belief in whether you are who you say you are and that changes depending on the circle. Yeah, it's literally a trust score that it actually calculates based on a huge array of different signals that you're unconscious of. The thing is, all of this stuff that we're describing is still in the space of like, you need to prove that you're the human so you can do the action. But I mean, the conversations I have with your colleagues, the Google DeepMind people, is that we are moving more and more into an era where agents will be doing stuff on our behalf.

18:19That's right. So, I mean, if we are moving into that era, if we want these agents to have autonomy, I mean, that changes everything again, doesn't it? It does. Yeah, I think a lot of us are still contending with this new reality. I mean, I don't claim to have all the answers here, but I do think that historically we had a fairly straightforward, you know, it was hard enough, but we had a straightforward concept between people engaging on the Internet. He was either a human or he was a bot, right? And now we have this sort of third thing, which is a bot acting on behalf of a human, which we call an agent.

18:51And so I think the question becomes, first, how do you identify that as being distinct from just a bot that's not tethered to a person? And what permission and access does that agent get? You know, and obviously you can imagine agents doing all kinds of good and bad things both, right? I mean, you'd want an agent to act on your behalf to do manage your bank account or to do healthy and helpful things. on their behalf on the internet. But I think at the same time, some bad actor could easily cause an agent to do negative things on their behalf. So just by virtue of having an agent doesn't necessarily mean it's a good or bad thing.

19:25I think we have to understand the sort of providence of that agent. You know, how much trust should we apply to that agent? What identity is that tethered to? And in other words, on whose behalf is it operating? And how do we trust that? And a lot of that plumbing to make the internet really understand that deeply into the foundation of the internet is still yet, I think, to be built. And then on top of that, you know, Google does have a paper on sort of agent security best practices. When you're deploying an LLM as just a simple chatbot with no other bells and whistles, a lot of these risks are fairly minimal.

19:57But the two problems start to come in when you have untrusted potential input, such as emails or websites, and then its ability to take action. In other words, like making changes to your home, your stove, your microwave. And so when you have those things all together, that's when you start to have challenges. And then moreover, and the last point I guess I would make, is you also have some restrictions about what tools that it can call and what it can do. So it can't just do any kind of random thing. You have some constraints and some rhyme or reason around what tools that it can call and what it can use.

20:30But there's also the question of privacy as well here, right? Because, I mean, a lot of the time the work that you're doing in security is about protecting people's data. but you've got these autonomous agents who are acting on your behalf. That's right. I mean, you will want them to give up your data at certain moments in time. So this is a super challenging problem. It's a problem we call contextual integrity. But the idea is how do you, exactly as you say, how do you train these AI agents that we know we will want to act on our behalf to know these things that we don't even know how to articulate in our own mind, which is obviously we want to give our social security number, to the tax IRS, but we don't want to give it to our social media friends.

21:14Well, we know that, but how do you actually articulate that in a way that an agent can be imbued with that behavior? And so that's the challenge of contextual integrity and something my team works on as well, is teaching Gemini these basic privacy norms that you and I take for granted about what data, both as a consumer, but also on behalf of an enterprise. If you have a company and you want to run these agents and you wanted them to carry out some commercial benefit on your behalf, you clearly don't want it to disclose the intellectual property of your company to everybody on the internet. And so how does it know what versions of that are right and what are wrong?

21:53And so right now we're taking baby steps. I can't claim to have solved the whole problem. But the starting point is essentially knowing when to ask for help and when to ask for permission. And I think you'll see that as the sort of next step at Google and probably across the industry where people start to give more and more trust to these agents, but they're still coming back and saying, okay, you know, here's what I want to do. What do you think? Is this okay? I'm about to make a financial transaction or I'm about to share this data. And knowing intuitively when to come up for help in a way that isn't annoying people too much.

22:28And so that, I think, is the first step of a long journey, right? A very long journey. Of teaching AI these privacy norms that even, you know, within our own society somewhat varies across geography and other things, right? I mean, it really does feel like, on the one hand, you're talking about multi-million pound breaches, but really you learned to defend against 15 years ago. And then on the other hand, we're talking about right around the corner, this potential future where you have agents interacting with agents, no humans involved, and the whole set of norms hasn't even been written. I mean, it feels like you've got quite a lot of work to do.

23:01There's quite a lot of work to do, both, you know, sort of building trusted, trustworthy agents against the attacks that you and I have discussed, such as prompt injection and jailbreaks, and then making it so that, you know, once you have these trustworthy agents, being able to run in a sort of trustworthy and repeatable fashion, imbuing it with the norms that we all take for granted in terms of privacy. Who do you think wins in the long term? In the long run, which way does the seesaw swing? Is it cyber criminals or security? Well, I have to say that I think the defenders ultimately will win to some extent.

23:32I mean, we won't necessarily win every battle, but I hope that we can win the war. The day job of competing between all these different AI labs is all well and good. But I do want to say that I really am proud of the fact that we all are finding ways to sort of defend our customers together. And so there's a lot of great papers coming out. I think we're having conversations appropriately about how to co-invent these solutions together. So that's been a really good part of the story is that it's not just one lab trying to do this, but it's really all of us are linking arms and trying to defend the future of agents together.

24:10Four, thank you so much. That was fascinating. Yeah, thanks so much. Really appreciate the time. Really fascinating.