The Interview: Irregular CEO on the race to stop AI models going rogue

0:00At PwC, we optimize the systems that put your data to work so you can scale your AI from proof of concept to competitive edge and accelerate your tech evolution so you can redefine what's possible. From blueprint to breakthrough, we take you further with tech so you can outthink, outpace and outperform. PwC, so you can. PwC refers to the PwC network and or one or more of its member firms, each of which is a separate legal entity. This episode of the Times Tech Podcast is sponsored by IBM. Wimbledon is one of those rare events that feels steeped in tradition, but behind the scenes, it's also a huge data operation.

0:44Millions of fans follow the tournament across time zones, devices and platforms, and increasingly they want highlights, stats and personalized updates in real time. IBM has worked with Wimbledon for more than 30 years, using data and AI to help create digital experiences for fans. including match chat, which can answer questions during play and likelihood to win, which uses live match data to offer near real time predictions. It's a useful example of AI being applied in a way people can actually see and understand. To learn more about how IBM helps create smarter business, visit ibm.com slash Wimbledon.

1:21This episode of the Times Tech podcast is sponsored by KPMG. Katie, we've been tracking the AI surge for a while, but some of the latest data is pretty striking. It is. KPMG's latest quarterly AI pulse suggests that 7 in 10 UK business leaders still see AI as a top investment priority, even if the economy slows down in the next year. But turning that investment into tangible value is another question. And so a lot of organizations are still figuring out how to move from endless AI pilots to something that works across the business. KPMG's focus is on that gap, helping organizations scale from the pilot phase to AI powered transformation they can trust.

2:02To find out more about how KPMG can help your organization make the difference, you can visit kpmg.co.uk forward slash AI.

2:15Hello, and welcome to the interview from the Times Tech podcast, our Sunday conversation with the people shaping the future of technology. I am Danny Fortson. And I'm Katie Prescott. And this week, I'm speaking to Dan LeHav, the co founder and CEO of Irregular. And it's a very, very topical interview. Because as you'll know, we've been speaking about it a lot. One of the big issues in AI over the past couple of months, is the number of models that have not behaved as they should have done, and that have gone rogue and terrified people in the processes as they've attacked people's cyber security.

2:52That's right. So during some of those tests for open AI, Anthropic, and Meta the agents, these AI agents were able to connect to the internet, hack other systems, break out of their sandboxes, which, you know, just a term I love, because it sounds so childish and kind of innocent. But really, they're like spaces. Yeah, they're like digital lock boxes that you're supposed to keep your very powerful agents stuck in, but they're figuring out with increasing dexterity how to get out. And Irregular is a cybersecurity company who's worked with some of the biggest tech companies in Silicon Valley to stress test their AI models before the tech is publicly released.

3:31So we're going to talk about what it's like to see firsthand the speed at which these companies are developing AI, how they're tested, and quite frankly, how worried we should all be about it. Enjoy. Sorry. Dan, thanks so much for joining me on the Times Tech podcast. It's a pleasure to be here. Let's just start then by understanding what is irregular. Could you explain what the company does? In a one-liner, we can call us an applied AI security lab, but to explain what it actually means, our core thesis is that the traditional cyber security stack is not going to be enough in order to make sure that the world is going to be secure at the time of advanced AI.

4:11So we kind of like started going with the premise that we will need to build a new security stack that is going to be based on new threat factors that are going to emerge as an outcome of AI. And de facto, we have two tenets of operation. One of them is putting models in simulated environments in order to whether just like attack AI models or let the AI models attack other targets. For example, see if a model can evade the detection of an antivirus software or see if it can attack a website or elements like that. And then we take this data, identify where the gaps in the cybersecurity stack and fill them in with new products and software that's required.

4:46And I'll take a step back and explain why do we even want to do that? It's because models and AI, I think we should be very optimistic about the technology and what it can do in everyday life. But these are what we call general purpose artificial intelligence, as in the same model that you can use in order to help you write emails is the same models that I can use in order to do research. is the same models that you can use in order to also help potentially hack some targets around the world. And that's an issue just because while technology is wonderful, some use cases may be problematic. But what should we actually evaluate?

5:19Because it's one thing to say we care about the level of offensive cybersecurity or are they good at hacking, but it's not that straightforward if you think about it to actually test whether this is true. Think about any subject that is being taught at universities around the world. You can, in theory, understand what it means that someone is a good lawyer or a good journalist, but the actual tests that are on the path there to just see whether someone is gaining knowledge are a bit more complicated than just asking, is this person good at law? And they're a bit more complicated than just giving them a standardized test and seeing how they score.

5:53So the same is true for offensive cybersecurity. We need to actually figure out what models are capable of doing. What are the threat scenarios that we care about. For example, you know, should we care about a model that is trying to steal from someone? Should we care about a model that is trying to sabotage something? And the answer would probably be yes, but we should figure out, you know, what is our first, such that we have like a concrete reference scenario that we want to test and measure. Tell us about your clients and what they want from you and how that works. Because, I mean, to my previous point, I'm presuming they do some testing in-house, but are also using other sandbox technologies too?

6:32So we are working primarily just like on this front with the Frontier Labs. If you're unfamiliar with this term, you can substitute it for the open AIs and antropics of the world, the guys that are essentially creating the most advanced AI models in the world. And we're usually getting involved quite early in the process. So when people are starting to train the models or at the very, you know, just like the last stage of just like pre-deployment, which is just like the last stages before a model is released to the public, To give an example of one, imagine that there is a network, let's say the network of the times, and someone is trying to steal a file from the network.

7:07For journalists, that's obviously, that's a big deal, right? You know, if people are going to steal your sources and publish them online, that would be quite damaging, both for your sources, but also for your reputation. After we decide on the threat scenario, then we actually need to build usually what is a simulated environment that enables to actually test whether models have that capability already, or whether they're getting meaningfully closer from just like generation to generation to be able to pull this off. But, you know, in your real networks at the times, the devices are also communicating with one another, right?

7:34Also like a lot of traffic. And obviously there's state-of-the-art defenses as well, right? Because obviously you have antiviruses that are installed on your networks and you have just like people that are alert, you know, and can see if something problematic is going on. And if you want to get really close to, you know, the actual threat and, you know, to see if a model is able to do, an AI model is able to do something meaningful, you would want to have a high fidelity environment that's going to be as realistic as possible. The simulation needs to actually be real. Because if there's going to be no defenders in that simulation and a model is able to do something, that's clearly not actually resembling what's happening in the world.

8:11And that may cause us to be overcautious in some areas. So you're building a fake Times network or, I don't know, Barclays Bank or whatever it is, to try and make it as realistic as possible, like that corporate network. And then you're firing AI models at it to see what the impact is. Yes. So in the last few months, there have been various high profile examples of models which have escaped their sandbox protective environments where they're being tested, got out into the Internet, hacked into different companies. I'm thinking about, for example, the hugging face attack. And that's really raised the fear level, which I think was already very high, about the cyber impact of AI, which is, as you said at the start, like increasing at a rate of knots.

9:05What has been the impact that you've seen of these attacks? And has your company been involved in any incidents that you can talk about and just explain what happened? So indeed, for full disclosure, not in the open eye hugging face incidents, but in a different stream of incidents. So why are we starting to see incidents like that? Models are getting better at a very quick rate. And to give you a concrete example of that, roughly this time last year, if we would have had this interview around August, September of 2025, models were still not capable enough to perform the building blocks of offensive cybersecurity.

9:42But around late 2025 to the beginning of 2026, there was a meaningful jump in capabilities. And models were not only able to now just get a hold of the building blocks, they were able to start chaining them into fuller operations. And, you know, let me describe to you just like the open eye, hugging face situation. You know, I think, as you've mentioned, a lot of the focus of the story was on the fact that there was kind of like a sandbox escape. And, you know, for good reasons, you know, it's security relevant. It's also like a very sexy story. I was going to say, it sounded like a prison breakout.

10:09I think everyone kind of understand what was going on. Something like that. But just like imagine that I'm going to just like tell the story now in a slightly different way. And imagine the model, you know, even though that wasn't the case here, but imagine that the model would have just been connected to the Internet in the first place. And there was no sandbox escape. But I would have told you a few months ago that now I can just like put a website of Target in a model. And, you know, in this case, Hugging Face, which is one of the most best startups in the world, most well-funded startups in the world, also have just a very competent security team.

10:36and the model is now competent enough to go after them, to reason what it should do, to bypass some of the defenses, to attack it multiple times over the course of a few days, to coordinate with other agents, so with other AI models while doing this attack, and to be able to cause real damage to an actual production database of the company. That is already something that we would have said, whoa, there's something deep that's happening here. And I think it's important to reckon that even without the sandbox escape elements, models are getting better and better. The fact that models got so much better made it so that containing them became more complicated.

11:11And that's, you know, I think just like a key element of the OpenAI hugging face story, where the model found its way out of the sandbox in order. A, reasoned that it should go into just like hugging face and did it independently. No one directed it to hugging face. And B, you know, it's like as part of this process was actually able to do something. Well, right. I mean, that was going to be my next question. Should we stop testing the models like this? are we going to see more of these sorts of escapes, more of these sorts of attacks? What I think is a really bad outcome if you're going to do it is stop testing.

11:42Because stop testing just means that unless you're saying stop testing and not releasing models, which I think is not a good thing because obviously that's, you know, you're not going to stop the economy. The outcome is you won't be able to understand what the model is doing. So any outcome that says stop testing models is an outcome that de facto means you're not going to be able to understand the risks prior to deployment. and you're just like pushing the risks, you know, just like downstream into just like post-deployment and that is a much higher risk. No, I can see, I mean, I see, obviously you need to keep testing them.

12:12Yeah. But if, as you're describing, the models are getting better coupled with a bit of human error, which is inevitable. Yeah. Are we going to see more of these escapes and are we going to see more attacks and perhaps more severe attacks? To be clear, that is, you know, I'm just like sandbox escape is not something that happens with us, but it is starting to happen more and more in the industry. And the meaning is, you know, in this testing, we are occasionally going to see problems and materialization of some risks. But the kind of like analogy that I have in my mind is, you know, just like when you're releasing a product, just like imagine for a second that I, you know, I'm just like, I don't know if you have a car, but I imagine that at some point in your life you were in a car.

12:55And, you know, just like at that one point, if I would have told you, that car was not tested at any point for your safety, you'd probably be hesitant before you just hop on there. And even though in car testing, there's even testing that's happening in the lab and testing that is happening in road testing in a controlled environment. And the industry was able to, over time, build practices that, while are not flawless, are able to just mitigate down the amount of errors or not even errors, have kind of procedures that allow you to do that process in a way that as a society we find acceptable risk boundaries.

13:30And we, I think, are at the point now, as models are getting more capable, that we need to create kind of like the security and safety protocols for testing for AI models. And this is, I think, both an exciting and alarming point in time because it all needs to happen fairly quickly. And there are obviously some government bodies which are doing this sort of testing. I'm thinking about like here in the UK, the AI Safety Institute or AC. as it's known. But there isn't a global body, I don't know, that's like, people have talked about NATO, for example, that is looking at AI safety. And there isn't a regulator in the US that's doing it where you're based and where so many of these models are being developed.

14:17So what's your thought on that? And what we should be doing globally, as these models get better and better? And how worried are you about the fact that like those those checks and balances don't exist so a few points one double down on the fact that ac does amazing work you know we've been working with these guys for a while i think and you help them with testing do you yeah so we've been collaborating with facet for a while and actually just like want to give them a huge shout out it's actually unbelievably impressive to see how quickly they were able to just like pull on a team that is so competent i think the uk should actually be very proud on you know just like having this institution there is a parallel in the us there are parallels in multiple other countries um i think in like Australia and Japan and like you say, in the US as well.

14:58Yeah, exactly. In the US as well. It's true that they don't have jurisdiction over these companies, but they do have a lot of agreements for testing with these companies as well. I think it is worth knowing that the Frontier Labs have taken the recommendations very, very seriously. The work is usually done with scientists on the other side that care deeply about, you know, making sure that the products are not going to cause harm to anyone. So as an outcome, it almost always triggers, You know, it's like very thoughtful understandings and considerations on what should be the next. I mean, that's that's reassuring that the scientific community is talking about it.

15:31I just wonder what's the vibe in the cybersecurity community where you are in San Francisco working with these models about how far we need to go in terms of some sort of legislation and and protections around the models. And I know that's a funny question to ask a business because nobody likes more laws. But I have heard lots of people in the AI community put their hands up and say, this is getting really scary really quickly. And I just wonder, where do you see the mood music and how scary is it getting? I think that it's kind of like we need to hold a more complicated picture of the world. Well, right now, while models are getting better and while models are able to do some, you know, to cause some harm around cybersecurity, the world is still functional.

16:18Right. It's not I assume you didn't wake up this morning thinking, oh, my God, my money is not at my bank account. Right. It's not that you have this fear. Not this morning. And I think that, you know, it's like if you take a step back and, you know, it's like you understand the picture, you know, obviously every hack is bad. And it's like we should work our asses off to make sure that nothing problematic is happening. but society does have a level of resilience of like you know one hack is not going to just like shut down the modern economy so it's important to remember that while things are progressing quickly it's not that you should just like go to sleep it's like you know it's like and be and be bothered by the fact that your money is not going to be at your bank tomorrow morning i speak to a lot of cyber security people and no offense but you're often pretty gloomy about cyber security issues, particularly in the age of AI.

17:01And I think it's a little bit like speaking to medics about illnesses or the police about crime. Okay. Because you see the worst outcomes. What are you worried about? What is the thing that keeps you up at night? My view is actually that in the long run, in cybersecurity, there's actually just like a stone case for optimism. We can think about a world where every line of code is being audited by AI, where we can formally verify, kind of like mathematically prove that things are secure at scale. It's something that we can do, you know, just like occasionally at just like lower scale, but, you know, AI is potentially going to allow us to just like scale it up significantly.

17:37We can think about, you know, just like teams in companies that right now don't have enough money to have an army of cyber defenders that are going to be amplified by AI. And as an outcome, you know, I think that it's obviously, it's not a given, and it's not that I'm saying that it's clear that the world in, you know, the long-term future is clearly going to be much more, you know, defense dominant and much more secure. I think that it is a real possibility and we can be hopeful this possibility is real. But now when you ask what's concerning for me, I think that occasionally one of the problems that I'm seeing in the community and how it discusses kind of like the balance between offense and defense, right?

18:11And the reason the balance really matters is because AI amplifies both defense capabilities and offensive capabilities, right? Like a lot of other technologies that can be used for both sides. What bothers me is that while I'm very optimistic about, you know, just like the 10, 20 years into the future, I still think that the transition, kind of like the way to that long run equilibrium, can be and likely be, is going to be sharply favoring the offensive side. So as in just like more high profile events that are problematic that can happen, or just like more events generally in society, even if they're not high profile.

18:43And it's going to be like a very painful road to this like happy outcome. And, you know, every system, not every, but let's say the vast majority of critical systems that we have in society were hacked at some point, you know, were damaged at some point. But society has a lot of resilience, and we were able to just have enough capabilities to just withstand these attacks. The danger that we're currently facing is kind of like the scale, severity, and simultaneity of these attacks is going to potentially be able to put us at a situation where offensive capabilities are going to scale much more quickly than defensive capacity.

19:15And I think the biggest concern and kind of like our societal goal is understanding how we're going to just be able to help the defenders to quickly build up more capacity such that, you know, we should obviously aspire to not have any incidents. But realistically, at the very least, we should, you know, just like put, I think it's like a much more feasible goal of making sure that we're not going to have such an intensity of incidents that defenders are not going to feel that they have enough capability to just like respond to them properly. Fascinating. Well, Dan, thanks so much for explaining it to me.

19:47Things are moving so quickly, as you say. I mean, I feel like a week in AI is like a decade in any other sector. Yeah, I mean, what we've been saying internally, it's kind of like AI is very much like dog ears, just like in this industry. Yeah, that's a good way of describing it. Well, thanks for listening to this week's big interview on the Times Tech Podcast. We'll be back next week with more news and more interviews with the biggest names in tech and AI right now. See you soon. This episode of the Times Tech Podcast was sponsored by KPMG. One of the themes we keep coming back to is the gap between AI ambition and real world application.

20:28And that's where KPMG is focusing its work, helping organizations move beyond pilots and think about how AI can change the way the business operates and how it fits into wider processes and decision-making. You can find all their insights and how they can help your organization make the difference through successful AI workforce adoption. Visit kpmg.co.uk slash AI. This episode of the Times Tech Podcast is sponsored by health and life insurer Vitality, your health's best friend. Most of us want to be healthier, yet life so often gets in the way. Vitality's health and life insurance is built around that reality.

21:09Get active, look after yourself, and you can unlock rewards from some of the UK's top brands and help keep your insurance premiums low. It's insurance that works for you. By using tech and insight to understand your health, they can incentivize you to live better. The healthier you get, the more you are rewarded. Find out more at vitality.co.uk. At PwC, we optimize the systems that put your data to work so you can scale your AI from proof of concept to competitive edge and accelerate your tech evolution so you can redefine what's possible. From blueprint to breakthrough, we take you further with tech so you can outthink, outpace, and outperform.

21:55PwC, so you can. PwC refers to the PwC network and or one or more of its member firms, each of which is a separate legal entity