AI, agents and quantum: The biggest cyber threats ahead

0:01Hello, I'm Arjun Kharpal and welcome to The Tech Download. And on today's episode, my guest talks about the growing cyber threats from AI. Suddenly, now you can rent computer intelligence and let it go berserk across the entire infrastructure. The AI safety and pacing debate. I think this is unrealistic. I think not everybody is going to pace themselves. And the threats to modern day encryption from quantum computers. We have seen activity where certain nation states are already harvesting data saying, OK, I don't know what's inside that data, but I can already grab it. And when I have my quantum computer, I'll crack the code and decrypt it.

0:41One of the most talked about topics in tech are the cyber threats arising from ever more advanced AI. And that debate was supercharged by an extraordinary incident involving OpenAI and HuggingFace, which you may have heard of. Hugging Face is a company which hosts a massive repository of AI models and data sets. Now, OpenAI was testing an AI system's hacking abilities in a supposedly sealed environment. The models found a way out, got onto the internet, and ended up compromising Hugging Face systems in pursuit of their assigned objective, even though no human had told them to target Hugging Face.

1:18You know who would be great to discuss the evolving cyber threat from AI? probably the boss of one of the world's biggest cybersecurity companies. So I traveled to an event called HumanX in Amsterdam and tracked down Nikesh Aurora, the CEO of Palo Alto Networks. The OpenAI hugging face incident underscored two growing concerns, AI being used by bad actors to carry out cyber attacks and AI systems themselves behaving in unexpected ways. That second problem becomes particularly challenging as companies talk up the use of agents or agentic AI, which we've spoken about a lot on this podcast. These are AI systems that can carry out tasks autonomously, and there could eventually be thousands of them operating inside a single company.

2:02So what happens when an agent does something unexpected? And then there's the way AI could make hackers themselves more sophisticated. Take phishing. That's phishing with a PH. That's when a hacker sends you a legitimate looking email or message designed to trick you into clicking a malicious link or handing over sensitive information. AI can make those scams far more realistic and personalized. All of this is what we get into with Nikesh Arora.

2:32So, Nikesh, let's just kick off. There's so much to talk about. First, top of mind, AI, cybersecurity, and the intersection there. And, of course, it's been top of mind since we had that open AI hugging face incident as well. Well, it's been top of mind since mid-toss, to be fair. For sure. And we'll get into all of the details about this. But what did that moment do for kind of the industry and the broader conversation around, you know, the cybersecurity threats that are coming from ever advancing AI? Well, it all started with Mythos. I think Mythos proved to people that if you've been training these models to do better coding, they'll be able to figure out what bad code looks like.

3:07It's kind of a bit of a revelation to the industry. Oh, my God, we have AI, which can actually understand code and will tell you where bad code is. And that kind of exposed the dark underbelly of enterprises because they've been sitting on vulnerabilities, on bugs, or misconfigurations for a long time. And we find them. Everybody tries to make an effort to find them. But AI was able to show how it can go in one fell swoop and it can apply a model like Mythos against your infrastructure and find out the whole host of vulnerabilities you haven't seen for tens of years. And I think that was a bit of a wake-up call, to be honest.

3:39Like a wake-up call to our industry, a wake-up call to the IT infrastructure industry. And honestly, management started paying attention. I mean, we've been trying to get people excited about cybersecurity for a while. Suddenly, your CEO is calling and saying, hey, what about this Mithos thing? Is this real? That kind of woke up everyone in the industry. Yeah, I was going to say, you must have got a lot of calls after that from C-suite across the board. Yes. We did about 2 ,000 individual briefings with CEOs, CIOs. 2 ,000? 2 ,000. Wow. Across the world, yes. Look, that was an part of the story here is effectively AI models that escaped what was supposed to be a controlled environment.

4:15That's one issue. But are you seeing these ever advancing models being harnessed by bad actors right now? And in what way? Well, I think what you've been seeing is the capability, sort of the exponential capability increase, because historically, bad actors don't have as many resources. So you have to have lots of different people try different techniques, try and insert themselves through some vulnerability into the enterprise infrastructure. Suddenly, now you can, let's just say, rent, compute, and intelligence and let it go berserk across the entire infrastructure. That kind of got people worried.

4:49Oh, my God, this is going to get harnessed in a different way. And what you're beginning to notice is that we'd made a prediction that when Mythos came about that in three to six months, open source models will get equally good. And you can see now there are open source models, which are almost as good as Mythos. And you expect that over time, they're going to get better. Mythos is already on its next generation. So what you're basically going to see is that if you don't sort of go ahead and strengthen your perimeter as an enterprise, you are going to be attacked. And there's a high probability attackers will get through.

5:21And that's kind of been a bit of a moment where people have stepped back and say, am I good enough? Is my cybersecurity capable? Am I good enough? Am I able to stop these bad things from happening? So actually, because of the amount of open source models and the capabilities of these open source models, it makes it easier for bad actors to be able to obviously use these models that are freely available. Yes, totally. Look, like you said, bad actors can now rent this on the fly, point it towards somebody's infrastructure, and just go hunting. And you know what? I find nothing, no problem. I'll move to the next target and go hunting again.

5:53And when you look there, there could be bad code you could have written as an enterprise. There could be one of your technology vendors, which has a vulnerability, which could be exploited. Or you could be using a lot of open source code yourself, and that could be exploited. Or you could just misconfigure something. So there's many different ways where these vulnerabilities manifest themselves. And I think the onus is upon all enterprises out there to make sure they robustify or strengthen their perimeter. What are the kind of main threats right now, as you see? Because, of course, you could use this to create malware or malicious software.

6:24but also there's the phishing side of this, right, in terms of being able to create ever-realistic emails or correspondence or images or phone calls even, right? Where are the main threats right now? Look, the main threats are, like I said, most breaches historically don't happen because of sophisticated sort of attacks. They happen because of simple, somebody got a hold of your credentials on the dark web, entered your infrastructure, made their way laterally across the infrastructure, a bit of a game of hunting. It's kind of a sport for bad actors or hackers. to say, how do I get in? How do I navigate through this stuff?

6:57All that has become easier because AI models know how to go out and do this stuff, how to go understand where your vulnerabilities are and attack your enterprise. I think equally, like you mentioned, on the other side, when bad actors are targeting individuals, they do that through phishing attacks, through smishing attacks, through fake video. So it's a lot of different ways these bad actors do it. And it's kind of interesting because it's a tale of two cities. One is on the enterprise side, the risk is it'll find your vulnerabilities. On the consumer side or individual side, you know, billions of dollars gets scammed from human beings because bad actors sort of convince them in some way, shape or form to let go of their banking credentials or, you know, surrender some private information.

7:38And that's going to get worse because things are going to get so much more realistic and so much more customized. It's like they'll start understanding based on your persona, what are you most likely to fall for? And they can now send, instead of sending a blocked email to 10 ,000 people, you can customize emails to everyone, basically understanding your circumstances. So the probability of you clicking on it goes up. Yeah. Did you say smishing? Yes. What's that? It's SMS-based phishing. Oh, right. Okay. That's a new word. That's a new word, yeah. There's phishing, there's smishing, there's phishing.

8:09We've spoken about kind of the bad actors and the offense side. What about on the defense side? How do companies protect against this? And what are the increasingly sophisticated ways you're having to think in order to defend against these kind of more advanced attacks from AI? That's a great question. Look, this morning we announced a new service. The new service basically allows us to bring Mythos class models to enterprises to help them build their defense up. So we can go on behalf of the company and say, come, let me take a look at it. let's go look at it from the outside and from inside to see where bad actors could possibly see your vulnerabilities.

8:46That's kind of do-ray. Do-ray is make sure you use AI to anticipate bad actors, do what they would be doing, and strengthen what you have from an infrastructure. That's one. Two, I think there's many categories where we're looking at attacks, URLs, files, and saying, is this malicious or not? Now, that requires a process of machine learning. That requires a bunch of experience to be deployed. Today, you can start using classifier models to see how do you classify it, yes or no. So you can effectively use AI in the classification process of bad things or threats and use that. That's two. Three, what you can do is let's assume that despite every effort to robustify or strengthen your perimeter, some bad actors got through.

9:26And the question is, how quickly can I find them and stop them? So there's a whole art and science of how do you look within an enterprise infrastructure, say there's malicious activity going on, understand the behavior, defend against it, and effectively throw them out. And that can be also accelerated by using AI. And that's where you might even use the frontier models to make sure that you're acting at the speed of compute so that bad actors can be deflected or kept out of universe. On this podcast, we've spoken a lot in various episodes about agents, AI agents, agentic AI. We love agents, yes.

9:59And everyone is talking about implementing agents within their organization, right? And so when you get into this idea of agentic workloads, agents potentially carrying out tasks within an organization, workflows within an organization, touching the sensitive information within an organization, what new challenges does that bring from protecting an organization from a cyber perspective? That's a great question, Arjun, because that is one of the areas where you will see a lot of effort over the next few years. Because I think the definition of agents and their capabilities, like you rightly said, is evolving, right?

10:32You mentioned the hugging face example when agents went riot. Those are agents who had a very simple objective, capture the flag. You have no rules, no governance, no controls. Go, capture the flag. And you saw the agents used every resource that they had access to and were able to go navigate themselves across the infrastructure and capture the flag. So, you know, that clearly has become a bit of a sort of wake-up call again to the industry saying, be careful how you deploy agents. So hopefully our customers on the enterprise side will deploy agents in a careful manner where you have to understand who's the agent, who created it, what rights does it have, what is it trying to do, and then be able to intercept the agent while it's executing.

11:13It starts to say, is the agent behaving as per plan? So there's a whole, I'd say, a whole new area of security. I expect there's not a 500 startups to be funded by all these VCs because they're chasing the gravy train saying, go figure out how to solve agent security. It's kind of an evolving field. It's happening on both sides. Enterprises are being very cautious about how they deploy agents. At the same time, the industry is trying to run as fast as we can to build the harnesses required to protect against bad behavior from agents. Yeah, I mean, one of the open AI hugging face, one of the big talking points was the fact that actually these agents try to almost cover their tracks to some extent, right?

11:50So that makes it, from a defense point of view, very difficult. There needs to be continuous monitoring, right, and figuring out where are these things. We've trained the AI agents based on human behavior, and we haven't been able to give it a moral backbone. We haven't been able to say, this is right, this is wrong. So the agent is just going to learn behavior and saying, I read on the internet that if you don't go this way, you go the other way. and agents are just trying to be unscrupulous and trying to figure out how to solve the problems. Yes, you're seeing that. I think the idea that we start putting governance in place, we start understanding.

12:22So there are two parts to this. One part is, as every frontier LLM has recently said in the last two weeks, that we need to be cautious. We need to make sure we pace ourselves. We need to go out and make sure these agents can be safely deployed in any setting. There's a whole different conversation we have around pacing, but I think this is the right idea. they have to make sure their models are safer in how they act. At the same time, our job is to work with them to build security into place. Today, many of their tools don't have the hooks required to intercept, to analyze, and to stop agent behavior.

12:54So we're working with them to build those hooks into their infrastructure. We're working with them also to build tools and capabilities to understand the intent of these agents. Because, as you know, the biggest change that has happened in software is software used to be deterministic. It was input-output. You asked to do something, it did it, and you were able to make sure that it did that. Agents are non-determinant, right? You can't actually expect. They could change their behavior next time around and their behavior the next time around based on circumstances. And that's that non-deterministic nature of AI and agents that makes it a much harder problem to solve from a security.

13:31Executive Decisions is the new podcast from CNBC, where I ask powerful leaders about their decisions that changed everything. I'm Steve Sedgwick and here's the CEO of Siemens Energy, Christian Bruch. Business leaders should not stay quiet in a world which is super complex. We sit in a privileged position and we have to use that to ensure that society remains prosperous, stable, successful. That's Executive Decisions with me, Steve Sedgwick. Get it wherever you're listening to this. Prospective.

14:06Before we get into this next part of the interview, I just want to step away and lay the context. You may have heard about existential threats to humanity from AI and the debate around AI safety. Well, it all kicked off after a researcher at Anthropic, the creator of Claude, said there was a more than 10 % chance that AI could kill all humans. The comments caused a firestorm. Days later, Anthropics CEO Dario Amadei published an essay arguing that Frontier AI labs must slow the pace at which we improve the capabilities of AI models. This is an idea known as pacing, effectively deliberately slowing the development of the most advanced AI models to give safety research and safeguards more time to catch up.

14:49Amadei said two things had changed with his thinking. AI was becoming much more capable, including helping to develop the next generation of AI. And incidents like the OpenAI hugging face hack had demonstrated how advanced AI agents could behave in unexpected and potentially dangerous ways. And Amadei's call was backed by some unlikely allies, Elon Musk and OpenAI CEO Sam Altman, both of which are rivals of Anthropik. And Altman said he agreed that the industry needed to pace the frontier. But not everyone in AI was on the same page. NVIDIA CEO Jensen Huang argued that speed and safety don't have to be mutually exclusive.

15:28And Nikesh Arora also weighed in on the debate.

15:34And Nikesh, you brought up all the different issues around pacing. And let's get into all of those. Because you had, firstly, you had since the Hugging Face incident, you've had multiple instances where other companies, Anthropic and Gemini have said, well, actually, we've had instances where models. Nobody wants to be left behind. Yeah, nobody wants to be left behind. Well, there you go. You also had, you know, OpenAI more recently come out and said they've seen more instances of concerning model behavior. Separately, you've had researchers at Anthropic saying, you know, there's a more than 10 percent chance that AI can kill all humans was the exact quote there.

16:05And then you've had the CEOs of these frontier labs at OpenAI and Anthropic have called for model pacing, solving alignment, regulation, all these different things. And we'll get into all of this first. Can we start with the existential threat to humanity from AI, which has obviously been a big talking point around this discussion. How real is it? How much do you believe it? How much should we fear for our lives? Look, every innovation comes with its pros and cons. And yes, can AI be used to the detriment of society? Of course, but that's no different. We've seen technology in the past that could have been used for bad things, but they've been used for good things, right?

16:45Take nuclear, right? So I think, yes, AI could be used for bad things, But you can't take the downside scenario and build towards it. You have to look at the upside scenario. AI has so many amazing things. It can cure cancer. It can solve problems that have never been solved before. It can advance innovation. It can advance in a whole bunch of capabilities, life science, et cetera. So if you look at it from a positive perspective, AI has these capabilities and powers that if evolved right, it has the opportunity of benefiting us tremendously. So I'm going to be an optimist and say, yes, it has the possibility of being used in the bad sense, but highly unlikely that those scenarios have a 10 % probability as has been touted.

17:26I think it is a small probability, which is extremely small. And it's our job to make sure we build in the guardrails, the safety and security required so these models are used for the right thing. And then on the idea of pacing, which is another, this kind of term to suggest maybe we need to slow down the development of the frontier a little bit. There's been a lot of discussions from AI CEOs, right? Dario Amadei, Anthropik, Sam Altman, of course. You've been posting on X about these discussions as well. And you said pacing is a ninja move. Yes, but I changed my mind after that. You changed your mind after that.

18:00So just unpack kind of your thinking and where you're at right now and what you believe around these discussions. When I saw it come out, I figured that clearly the AI labs understand this is a problem. And I think we all do. It is a possibility that if not managed right, we should not be in a haste to release these models. So I think that's a good idea to make sure you get it right. I think the notion of a ninja move, I thought was, oh, maybe they're going to try and get the sympathy of the regulators and sympathy of lawmakers saying, hey, wait a minute, this thing's going to be dangerous. Let's make sure we jump in and we put some guardrails.

18:36And in the interest of innovation, in the interest of the positive side of AI, we kind of give them a free pass, if you will, and on the liability side. But I think over time, as I've spent more time thinking about it, talking to more people, I think this is unrealistic. I think not everybody's going to pace themselves. You'd have to pace yourselves. You have to get agreement for every country in the world, or even for anybody who's doing any work on AI. I think that's highly unlikely. I think more than likely that some people will jump the gun and you'll have still people developing at the frontier.

19:08which means we shouldn't try and stop the frontier because they're the most responsible people. If you can let them keep pace with what's going on in the world, make sure they start thinking about safety and security in a more robust way, that'll be the right answer. So from my perspective, and I think many lawmakers have since said, look, it's your job when you release a product to make sure it's right. And if you don't believe it's right, you should not release it. That's true for any product company in the world, right? Don't release a product if you believe it's not gonna work or it does not have the safety attributes that you require for them, whether it's like drugs or it's anything, like cars.

19:42You can't release a car if it's going to kill people. So from that perspective, the idea is don't release your product if you don't believe it's going to be safe or you're comfortable putting it out there. And if somebody believes that they are, they should go out and do it. But the idea that all of us have to get together in a room and agree at what pace, at what capability do we launch this, I think it's just a lot of work, and it's not clear how are you going to test for it? And I didn't see them espouse, well, we'll stop pacing when X happens. So we don't actually even know how long we're going to be in this pacing rhythm for, and when can we break the pacing rhythm?

20:17Is that okay? So I think a lot of things are undefined in the idea of pacing. And I think where we'll probably end up is that we'll get more regulation because now we've got the attention of the lawmakers. I think where we end up is that some people will jump the gun and we'll go out there and produce models which have tremendous capability. I think the focus should be on safety and security. And we need the frontier labs to take that even more seriously than they have. And hopefully they are. Because I mean, with this discussion, a lot of part of it was what is the motivation for these companies?

20:47And what do you think is the motivation? I can't have an opinion on this, but the debates that are out there is, is this genuine concern around the dangers around AI? Is this regulatory capture, i.e. trying to get better regulation favors the big companies, which may push out some of the competition, maybe even from China, which we'll talk about as well. Is this a need for these companies to justify their valuation? Oh, we need to slow down because maybe the revenue is not there. I don't know. But are you concerned about the regulatory capture side in any way? I think part of it, like you just articulated, there are so many stakeholders in this.

21:25And I think we just invited a whole new set of stakeholders by the pacing conversation. We already had AI labs and competitors who were trying to figure out how to evolve models, what's the right state of the art, should they be better at coding models, should they be flash models, should they be deep thinking models. So there's so much around this technology, which is fascinating and evolving, that you only had all these stakeholders trying to figure out what their answer is. And as we recently saw JEV get launched, it's a classifier model. So there's constant innovation happening just in the field, period.

21:55Now you're saying, wait a minute, let's go get the attention of lawmakers to see if they would like to get involved. Traditionally, getting a lot of lawmakers involved in any topic has not resulted in better outcomes or more efficient outcomes or faster outcomes. I think sometimes you get what you wish for. So there's possibly going to be more attention from a regulatory perspective. And it's unclear if it results in regulatory chaos or regulatory capture. Right. Because there's these competing factors. Do we want to maintain our lead as frontier LLMs or in the US to go out there and make sure that we constantly stay on the bleeding edge of this innovation and we use this as leverage to improve humanity, improve business, et cetera?

22:36Or do we end up with the risk of slowing down and having somebody else overtake us, which we have to go chase down thereafter? So I've invited a whole set of stakeholders from a regulatory perspective, which I'm not sure we had to stand on the rooftops and go ask them to get involved. But I think they are going to get involved. And then the third question you asked is on the stakeholders around the financial markets. I think the financial markets, in the end, are going to look at promise, your potential for growth, and how quickly your business is going to grow. And they're going to do it, look at it from a purely financial lens and say, is this going to be as successful as I hope and wish it's going to be?