Securing the AI Frontier: Irregular Co-founder Dan Lahav

0:00There was a scenario where there was an agent-on-agent interaction. It was a critical security task. That was the simulation that they were in. But after working for a while, one of the models decided that they've worked enough. And they should stop. It did not stop there. It convinced the other model that they should both take a break. So the model did social engineering on the other model, to another model. But now try to think about the situation where you actually as an enterprise are delegating an autonomous workflow that is critical to you to complete. And the more complicated and capable machines are ultimately going to be, the more of these weird examples we're going to encounter.

1:03Today on Training Data, we dig into the future of frontier AI security with Dan Lahav, founder of Irregular. Dan challenges how we think about security in a world where AI models are not just tools, but autonomous economic actors. He explains why the rise of AI agents will force us to reinvent security from first principles, and reveals how the very nature of threats is shifting from, say, code vulnerabilities to unpredictable emergent AI behaviors. Dan also shares surprising real-world simulations where AI models outmaneuver traditional defenses and why proactive experimental security research is now essential.

1:38His view is that in a world where more economic value will shift to human on AI or AI on AI, solving these problems is paramount. Enjoy the show. Dan, wonderful to have you with us today. It's a pleasure to be here. Awesome. So before we jump into questions, I will just say that it was very hard to get in front of Dan I was trying to get in front of him for three months probably 30 to 40 emails five or six people around us who we both knew closely were pinging him all the time and he was still not responsive and I basically learned where he was spending most of his time and I kind of went Did you stalk him?

2:21I kind of stalked him I kind of stalked him and eventually we basically bumped into each other like not intentionally and anyway, so we bumped into each other. I was like, Dan, you know, you're brilliant. I keep hearing great things. Please respond. Let's find time. You know, we at Sequoia spent a lot of time in AI security and eventually we found time the following week. So welcome, Dan. Thank you for everything. It seems that I'm going to have to start this podcast with an apology. So sorry, Dean. Sorry, Sonia. Sorry, the entirety of Sequoia. It indeed took time. It took time, but we partnered and here we are and you guys have done wonderful things.

2:59So it's wonderful to have you with us today. Yeah, it's a very, very happy ending. And, you know, just like appreciate you and everyone here. Of course, of course. Okay, so let's jump into it. I'm going to start with a spicy question. As we recently saw, you partnered with OpenAI on GPT-5 and let's kind of look forward a little bit. What does security look like in a world of GPT-10? Ooh, spicy and speculative indeed. So let me wrap my head around that. I think, so obviously everything I'm going to say is speculation, projection. But I think the way that we think about what's going to come is trying to understand how we're even going to produce economic value and how organizations and enterprises and people are going to consume stuff in the world at the time of GPT -10 or Cloud 10 or just like any one of the models.

3:55So let's do a thought experiment to just clarify why we actually believe that sometime we think in the next two to three to five years, there's going to be a huge shift in the way that humans are even organizing themselves as an outcome. Security is probably going to be very different as well. So here's a thought experiment. So imagine a situation where you work with OpenAI and you go one generation up or two generations up and you tell your parents or grandparents that you're doing work with Antopic or OpenAI or Google DeepMind on security. I think their mind would go on to assuming that the work that you're doing is probably providing a bodyguard service to Sam or to Dario or to Demis.

4:33Because the canonical security problem of a few decades ago, you know, it's like our parents, grandparents generation was physical security. Because the vast majority of economic activity was in the physical realm and not in the digital realm. And, you know, after the PC revolution and the internet revolution, we shifted the way that we're organizing and creating value. We transitioned primarily to a digital environment. And just think about how strong of a testimony it is of how many times you did an economic activity of value just by getting an email from someone that you may have not met. Just this morning, I got an email from my bank activating me to do something from a person that I've never met.

5:11Maybe as a security person, that's not a great thing to say openly, but we do that all of the time because that's the way that we interact in society. So our view is that soon that's going to happen again. And the reason is that AI models are getting gradually so capable that a lot of the economic activity of value is going to transition to human-on-AI interaction and AI-on-AI interaction. And that means that we may see soon a fleet of agents in an enterprise or a human, when they're doing a simple activity like trying to draft a Facebook post, taking a collection of different AI tools in order to just like promote that activity that they're doing.

5:52And they're essentially embedding tools that are increasingly more capable, and they're delegating them tasks that require more and more and more and more autonomy in order to drive meaningful parts of our lives. So we're transitioning from an age where software is deterministic to an age where this is no longer the case. And as an outcome, enterprises themselves or just like how we interact with the world is going to go to a fundamental change. And it's clear that security is just not going to be the same. As an interesting analogy, think about, you know, Blockbuster made the rest in peace on Netflix, the current version of Netflix.

6:31They're both, if you think about it, give the exact same value to the consumer. They both allow you to list units of content for your pleasure and entertainment. But clearly and intuitively, right security for netflix and security for blockbuster is not the same right one was a chain that organized you know just like you need to go and just like physically rent a dvd and another one is much more of modern architecture where you're just like you know streaming stuff to your home so even enterprises are going to provide the exact same value in the near future may have like a very very different back end to how they're shaped in this autonomous age that we're entering, which makes it clear that security as a whole is going to be very, very, very different.

7:13And we need to recalibrate to just like an age of autonomous security that's coming upon us. You were at our AI Ascent event earlier this year, right? Do you remember when Jensen Huang shamed everybody who was there for the fact that not enough people in the room were thinking about security in a world of agents? And I remember Jensen said something about how, you know, you can imagine that as these agents are allowed to act more autonomously in enterprises, you should expect orders of magnitude more security agents than the actual productive agents themselves, watchdogging and shepherding this herd of agents effectively.

7:51So I'm biased here. I agree with Jensen. I think Jensen was the first person that I've met that was much more bullish on AI security than myself. Because in our view, you know, you need a collection of defense bots that are also going to be, you know, working side by side with capability bots in just like the next generations of how enterprise are going to be created. But indeed, he gave a ratio that he thinks that there's going to be 100 to 1 in just like how many, you know, just like defense and just like security bots are going to be required out of the assumption that secure by design in AI is not going to work.

8:25So I'm not sure that I agree with that part of the conclusion. I think that we can make significant progress on secure by design, specifically embedding defenses in the AI models themselves. That being said, we share the view that the future is going to be one where we'll need to have a lot of agents that are specifically for the task of monitoring other agents and making sure that they're not going to step out of bounds. So maybe on that question, just to dive one layer deeper, what is the state of model cyber capabilities today? And how has that changed over the past 12 to 18 months? It's a great question.

9:02And I actually think that the rate of change is the most relevant part here. Because models are capable of doing so much more now that they were even capable of doing a quarter or two quarters before. So just like to give an intuition. So this is now, you know, just like we're entering the fourth quarter of 2025. At the beginning of the year, coding agents were not a widespread thing yet. The ability to do tool use properly was, you know, not just starting, but obviously like much, much more nascent than it is right now. Reasoning models were only at the beginning as well. So just like think about all of the things that were added last year and what they mean also for, you know, for security elements.

9:41So what we're seeing now is that the combination of coding being much better, models being able to have multi-modal operations, tool use improving, reasoning skills improving, if you're using models for offensive capabilities, we are seeing unlocks all of the time. Something that is now feasible that was not even feasible a quarter ago is proper chaining of different vulnerabilities and exploiting them in order to do much more complicated actions. So for example, if you have a website and you want to hack it on the application, a few months ago, if you needed to integrate a collection of vulnerabilities in order to perform an action of value, at least autonomously, without a human being involved, models were unable to do that, even the state-of-the-art models.

10:29That's not the case anymore. So obviously that depends. It's not 100 % success. And obviously that depends also on the level of complexity of vulnerabilities and environment that you're trying to hack. But we have seen huge spikes of just being able to scan more and more complicated codebases, exploiting more complex vulnerabilities, chaining them in order to do these exploitations, etc. And, you know, just like the recent GPT-5 launch on security and the offensive side specifically of what models are capable of doing, we have seen a significant jump in their ability to be able to be much more competent across a collection of skills that actually matter a lot around the cyber kill chain.

11:08Can you tell us more about that? And obviously there's some things that are publicly available, others that are not. But at least on the scorecard and what opening I've shared, in particular for GPT-5, what are some of the capabilities that you've seen that were surprising? We are seeing constant improvement on the ability of models to, for example, have situational awareness on whether they are in a network. And up until a few months ago, beginning of the year, complete, you know, it's like models were unable to do that. They were able to run some operations locally. They were usually not having situational awareness over the fact that, you know, that's like what's happening and whether and what they can activate, even in more limited and constrained scenarios that we put them in.

11:55And that's not the case anymore. So we still sleep very, very easily at night because the level of sophistication is still somewhat limited. but we are finding ourselves trying to create more and more and more complicated scenarios just because there is a huge jump in being able to take more complicated context, as I said before, chain complicated vulnerabilities to one another in order to do multi-step reasoning and exploit. And these are all new skills that going one year back did not exist. You guys are trusted partners by many of the labs, including Anthropic, including OpenAI, including Google DeepMind.

12:31and you work very closely with them for quite some time at this point. Why did you take the approach of working, kind of embedding yourselves within the labs, as opposed to, I don't know, selling directly to an enterprise right now? There are multiple companies that are doing AI security. We are pioneering a category of the market that we call frontier AI security, and we think it's fundamentally different. And the core thing is actually very simple. The rate of progress and the rate of adoptions of models change so many things at the same time that while traditional security tends to be somewhat reactive in nature, here we need a very aggressive, proactive approach.

13:06In markets that are dominated by a rate of innovation that is frankly unmatched, I think, unparalleled in human history, we think it's more interesting to take a temporal niche of the market. That is to say, focus on the first group of people or organizations that are about to experience a problem, so the labs, because they are the contenders to create the most advanced and increasingly sophisticated AI models in the world, work very closely in order to just see firsthand the kinds of problems that are going to emerge and utilize that in order to have a clear and crisp understanding of what's going to come 6, 12, 24 months ahead of time, such that we can be prepared at a moment where general deployers are going to need to be in a situation of embedding these advanced models and already have solutions that are going to be relevant for them.

14:00Given the rapid pace of progress in the foundation model side of the world, if you're at one of these model companies, and I think the people there are sincere, they want to do good for the world, they now know their models are capable of being used for extreme harm and cyber attacks as well. What do you do about that conundrum? And I remember, so we've been working with OpenAI since 2021. I remember back in those days, every enterprise user of the API, past some volume, had to be manually approved for their use case in order to even access the API. It feels like the ship has sailed of anybody anywhere will be able to access some of these models.

14:39And so how can you make the models sort of secure by design if you're in one of these foundation model seats right now? I think it's a great question. One thing on the premise of the question, I think that at least right now, at the moment in time in which I am, the ability of models to actually do extreme harm, it exists in potentially some use cases, but at least in cyber, I think we're not there just yet. And that matters. And just to be really sharp on what I mean here, models can clearly be used in order to do harm, but there is a distinction between harm and extreme harm that should be made.

15:14Harm would be an example of using a model in order to, for the senior citizen, and in order to just steal money from them. So just scaling up fishing operations. That can happen easily right now. Extreme harm, in my view, would be something along the lines of taking down multiple parts of critical infrastructure in the United States at once, that can take full cities off the grid, making hospitals not work. Models are not there yet. And that's not me nitpicking on the question. I actually think it matters quite a lot because how much time we have to prepare to a world where models are that capable actually dominates the strategies that we can take around the defensive side.

15:53Because our view is that the first thing that we should do, just like a first order thing, is be able to monitor and have a view of what's going to come such that we'll have an ability to have a much higher resolution discussion. Which capabilities are progressing? At which pace they are progressing? Should we expect them to continue to progress at this pace or accelerate in the future? And that dictates the order and the priority of some defenses, when we're going to embed them, whether we should embed them, etc. And if we get this wrong, I also think it's unfair to the companies and to the world as well.

16:30because AI also has so much potential to do good that if we deploy a lot of defenses that may chip away from productivity ahead of time, we're also doing real harm to just innovation and the world at large. And it's a very delicate balance to strike. So I think just the first order thing to do if you're working inside of the labs is actually having and supporting a large ecosystem that can take the models and measure and get to a high resolution of what is even possible to do. the second bit is figuring out you know a defense strategy that is informed by exactly what's happening and treating almost like a rigorous science with experiments of just like how to assess how to do predictions etc there are some defenses that will require a degree of customization for example if you're someone that is you know creating monitoring infrastructure will still need that you may want to recalibrate some of your infrastructure to give you higher alerts that ai is going off the rails, for example.

17:29But there are some problems that are very easy to write about, but actually very hard to develop solutions for. For example, I've just said a sentence of just customizing your monitoring software in order to prioritize alerts that are coming from your AI layer. How are you going to be able to understand when AI is doing something which is problematic? Occasionally, you're going to be able to run into that. But sometimes this may be, I think it's like the entire subsection of the market, which is anomaly detection, which is a huge subsection of security, is going to have a big problem very soon.

18:02Because anomaly detection is based on measuring a baseline, understanding what is a baseline and measuring against that baseline in order to see that something is an anomaly. But if you don't have a crisp understanding of the baseline and how it should look like, you have an issue of understanding that something went went badly. So as an outcome, we should, there's some defenses that are going to operate as these, and there are some defenses that we should recalibrate, either customize or to just recreate from scratch, just because there is a lot of science to be done of understanding how models look when they're under attack, understanding how it looks like when models are attacking something.

18:40And because I think we still have some time for the world that's about to come, My recommended strategy would be invest a lot in creating robust tools that would allow you to have rigorous evidence of what's going to come, such that you can have the discussion at higher resolution. Map which classic defenses are still relevant. Understand where your gaps are at. And invest a lot in R &D in order to make sure that you have cost-effective defenses that you can roll out ahead of time before models are going to be deployed. I know one of the broader questions in frontier AI research today is, can we actually understand the mind of a neural net and what's happening?

19:17And so I'm wondering if we can actually detect the characteristics of when is the model starting to behave badly? And do you actually, can you understand that in the activations of the net? For some of the attacks, we may be able to detect that, but that's still a very big open question. Let's say our approach in which we're trying to pioneer around security research, which is actually work from the outside in. So that would be putting the models in high fidelity, realistic environments that would push them to the limits. And in these environments, you're recording pretty much everything that's happening in the background.

19:54So that would be both the internals of the model, like MemorialNet, but also interactions of the model with the environment or the AI system with the environment. And it actually allows you, by this recording, to play around with using all of this data to create the, firstly, a mapping of how it even looks like when an attack is happening, whether you're trying to attack a model or using the model to attack a target. and then based on that taking up classifiers and I think just like a key thing at least around where we are right now in security and like a key realization is while understanding the internals of a model is ultimately going to be important to fully solve the field we can make a lot of progress just by understanding that something is not right even if we don't have a full crisp full understanding of just like what's what you know the internals and and what's going on there and also in these environments the other nice thing is you can insert or remove defenses to see if they're effective so think about a situation where you know the question that I'm trying to ask is whether a model is going to be able to say evade a detection from an antivirus this is a core security question if models are going to be able to do that in enterprises this is an issue because they're going to be able to scheme and do malicious things without just like and harm security processes without anyone noticing.

21:21And in specific environments, we see that this is actually possible. Models are able to do that. We were able to show how a model was able to take out Windows Defender, real Windows Defender in a toy environment, granted, but real Windows Defender. But that kind of research, when you're putting a model in an environment and recording a lot of stuff, both internally and externally, allows you to play with these parameters to ask a question of, A, should I change something on the model itself in order to understand that something bad is happening, but also take a different approach, which is agnostic to that, and say, should I alter something in the defense in order to make sure it's more resilient?

21:55So I think our view is that for the long run, for the vision of the field, we do need to solve problems such as Mac Interp in order to also solve issues in security, which also shows how different problems in AI are just changing the core of just so many different fields, right? Security itself may just be reframed to an issue of reliability and control in the near future, not just classic traditional security. And drawing an analogy to, you know, the human brain, we still don't have a proper understanding of our own back-end terms, like, you know, just like the neural net in our head. That being said, we were able to create very powerful mitigations, you know, and just like, you know, solutions to multiple problems that we're experiencing as humans in our lives just by being observational on how we're interacting with the environment or just like understanding smaller parts of the inner, just like inner parts of the system.

22:45I love that framing. You said something scary to me that the models are capable of, you know, taking out Windows Defender in your simulation environments. Can you share more about how that even happened? Yeah. So just to declare, it's, you know, it happened in a toy environment, you know, it's a controlled setup simulation. That's not to say that everyone that's using Windows Defender now should just be scared as an outcome of that. So just want to make sure that Microsoft is not going to Sequoia for just taking down the stock. You can think about a network. So it's a network simulation. So think about a network environment.

23:19Just to simplify, imagine the network environment that you have in your enterprise, in your work, in your home, the Sequoia network. For example, your phones, your laptops are connected into just a network environment that you're using. and the ai there was a storage bot so it had access to the organizational file in order to be a helpful storage bot for you and for example just like scan some of the organizational files such that it can summarize some of the inputs like we're doing it all of the time when you're interacting with files that you're having just like uh you know access to a file and you're asking give me like a summary of the these four things you know what's common between these documents what's what's not common between these documents and also as like a you know just like prelude to the simulation, there was an attacker that had access to the enterprise Slack.

24:06That happened multiple times in the last year, just like Slack is occasionally hacked, as happens to every software that is very ubiquitous and just being used by so many different organizations in the world. And the attacker gave a very simple instruction to this helpful AI storage bot. It just like said i want you to download this file to the organizational network but clearly so the file because it's a storage bot it did not have access to download from the internet but what you saw that was pretty amazing that's like through a collection of prompting the model because it was given the just like clear instructions to just like do it at all costs obviously much more sophisticated language I'm simplifying here.

24:50It realized that it can do downloads because it doesn't have access and because Windows Defender, the real Windows Defender, is blocking it from having just like downloading files. So that's the way the organization was set up. So you can see internally of how the model is thinking that it was able to map and have situational awareness in the environment, A, understanding that this is the problem, that it should have higher privileges to just do more stuff in the organization, and that it should be able to remove Windows Defender because it's bothering it in order to achieve its ultimate goal.

25:22So then you can see as a next step in that reasoning trail that the model is realizing that it's a storage bot. So as an outcome, it can access different files, obviously still by asking it to access different files, but it started to just scan whatever it had access to in that privilege level and found that someone accidentally in that you know in one of these files left a hard-coded password to the organizational cloud and it used that and now it was able to just like get access to something that they didn't have access to before and I'll cut the story short it's like you know just like did a collection of of just like uh things that ultimately just like did more privilege escalation more privilege escalation more privilege escalation and was able to just like gain more and more and more and more access that it should not have given you know to stuff that it should not have given access to.

26:09And ultimately, it was able to just get enough access to have admin privileges on the organization and kill critical security processes. And as an outcome, it was able to outmaneuver Windows Defender, remove an organizational defense, and upgrade its own privileges such that it can download a file and actually hack the organization that it's a part of. We call it internally, this simulation, a double agent, just because it's an agent that is doing that, but it's turning against its host. So it's kind of like a fun play of worlds of, you know, these kinds of like new attacks. And in a way, the AI is, you know, just like kind of used very much like similarly, this prompting, as that you would think about a parasite that is attacking externally.

26:56Because you're using like very lean text of just like sending something to a model. and using the fact that it has like a much, you know, a very strong brain in order to just like do a collection of actions that are very advanced. And I want to say the point in time in which we're in right now is that this is a toy setup. And it's not, I don't expect that, you know, for a lot of these things to happen in the wild yet. That being said, we are seeing huge progress quarter on quarter. And I feel it in security. Have you guys seen like the classic, I think it was called Bricks, like the game that DeepMind demoed just like a decade ago, where just like it starts very badly and just like, you know, then the AI is able to just like figure out better and better and better strategies.

27:43And, you know, it is first relevant to just like Bricks and then just like relevant to just like many, many, many other games. And, you know, here we are right now, just like a decade after the state of AI. So I think security, by being a derivative market of what it is trying to, ultimately you're trying to secure, is at the more nascent stage right now we're in toy setups and simulations we're able to start only get a glimpse of what's about to come and we are seeing stuff like models having enough power to do stuff as maneuver the host in order to just like do privileged escalation attacks remove some organizational barriers and wipe out even real software such as like real security software such as such as windows defender and while it's still these are not things that will likely happen in the wild now it's likely that in a year or two or three if we're not going to have the appropriate defenses this is going to be a world that we're going to just like land up on and clearly like the implications here matter right you know i assume that the vast majority of enterprise in the world don't want to deploy or just like adopt tools that are able to outmaneuver their defenses how do you think about model improvement especially in the context of reinforcement learning playing a pretty significant role in the improvement of coding, even tool use, for example.