That OpenAI / Hugging Face Incident. What now?

0:00Mike Butcher:Welcome to Past Founders with me, Mike Butcher. We like to look at the code of the entrepreneurs, the capital that backs them and the consequences.

0:14Mike Butcher:What happens when an AI agent told to solve a cybersecurity test decides that the easiest route is to escape its sandbox, reach the open Internet and hack another company? That is the extraordinary question raised by OpenAI's admission this week that experimental versions of its frontier models broke out of a supposedly secure testing environment and targeted Hugging Face, one of the world's largest repositories of AI models and data sets. Now, some see this as the first genuinely alarming example of autonomous AI escaping human control. Others argue that the models had deliberately been stripped of their normal safeguards and that the hacking techniques were conventional.

1:00Mike Butcher:Well, who knows really what happened? But today we're going to be asking what actually happened, who bears responsibility, and whether this was a contained research incident or may perhaps warning of a much larger security problem to come when it comes to these very sophisticated AI models. And I'm very delighted to say that we're joined today by some experts on this subject. Meriam Arick, who's co-founder and CEO at DoubleWord. Rory Blundell, who's CEO at Gravity. and Ivan Burazin, who's co-founder and CEO at Daytona. Well, let's kind of kick this off. Miriam, what was your impression when you heard about this story?

1:44Mike Butcher:Were you sort of aghast or did you think it was kind of going to happen at some point and this was one of the first incarnations of it? So I don't think it was particularly surprising to anyone in this space that this would eventually happen. we've had examples in the past of AI models escaping their secure sandboxes and reaching the internet um anthropic reported an incident of this when the model escaped a sandbox and emailed the researcher while he was eating a sandwich on a park bench saying hey I escaped and so we know that models have done this in the past and we know that capabilities of these models for cyber purposes are getting better and better over time and so it wasn't necessarily uh surprising to anyone in the field that this would eventually happen.

2:32I think for me personally, the thing that I was surprised by is how much these models want to follow instructions, like how much these models really, really just want to do whatever it is they've been asked to do. And in this case, it was solved the benchmark that it was set and it's kind of like a genie with three wishes. It found a very unintuitive way to solve that problem, but actually it was solving instructions. And it wasn't an example of a model going rogue in that it was doing something different to what the humans told it to do. It was an example of a model taking an approach that definitely wouldn't be what we would expect.

3:13So capabilities are not surprising, perhaps surprising that it followed instructions so well that it did this.

3:22Mike Butcher:I mean, it is worth just reflecting for a moment. It discovered a zero-day vulnerability escalated its privileges, stole credentials, moved through an external production system, and it generated 17 ,000 recorded actions along the way. And it's just quite staggering what it was doing. Ivan, what was your kind of initial impression when you heard about this story? Sure. So I first wanted to touch on because they tell themselves it's like a quote unquote sandbox provider, which is a terrible name for what we do. It's computers for AI agents. There is a security aspect. Yeah, it is what it is. The market calls what it is.

4:02And so when we think of that as first, like when we say OpenAI Sandbox, this is, I'm not part of OpenAI, so I don't know. But my assumption is the sandbox environment is a term of a set of computers that is by network or physically segregated from everything else, not a quote unquote sandbox in the sense that we offer to secure the agent. And so why that's interesting for us is one, people are like, especially on Twitter or X now, they're like, oh, is this not your guys' jobs? And it's like, yes, if they would have used us, it would have been harder. And why I say that is that when we think about AI agents, the way I think about them, they're like digital knowledge workers.

4:43And so if you think of any digital knowledge workers in any enterprise, there's layers of security for everything that you do because you do not inherently trust humans either, right? So if you go to work at any enterprise, any bank, you get a laptop, there's firewalls, there's security, there's guardrails, and then that laptop is connected to a network, that network has security and guardrails, and then everything else has more steps. And if you think of security in general, it is about adding layers of security to make it harder and harder to, quote unquote, break out of that. that. And so when we think about what we do, we give agents individual computers that are sandboxed themselves.

5:18They're isolated from other computers to get to. They do have a network firewall where you can disable or enable what they can do. Now, will they try to break out of that one? Maybe, maybe not. But that's one more to get into the internet than it was there. And also do things like, we do things like remove tokens and credentials from the visibility of the agent. So the agent can have access to things that it wants. but it actually doesn't have those credentials to do other things. And so as someone sort of in this space of security of AI agents, that was like the first part. But the second part is what I sort of added it was agents are digital humans and like humans, to Miriam's point, they're more extreme than humans, I have to say.

5:59It's like, we will try to solve this problem, this issue, no matter what, but you do not trust them. And because of that, we have to make sure, and this will happen more and more for every single task that we get them. if we do not ensure the guardrails. You cannot trust the agent much like you cannot trust the human.

6:15Mike Butcher:Very interesting. Very good point. Rory, what was your thinking about this incident? I think quite similar, actually, to both Ivan and Miriam's in the sense that I wasn't particularly surprised. You know, if you look from, we've done some unique research from our perspective where we've seen that there are now about 7.2 million AI agents in production across both the UK and the US. So this was after surveying about a thousand CIOs and CTOs in both regions. And the feedback that we got from them was that about give or take, their belief is that about 70 to 80 percent of their AI agents they're running in production are not fully secured and governed enough.

6:58And so from my perspective, I sort of echo the points that have been made there in the sense that it's an inevitability. If you look at our customers, the sorts of customers that we deal with, they are, this is not uncommon. Like I was speaking to a customer the other day who was telling me that they deleted, they had an agent that deleted all of their calendar, not just the person, like the whole company's calendar. And so now that's, that's sort of problematic, right? Because imagine the next. Or it's a good day or it's a good day. Or it's a good day. Yeah. Yeah. You just, you just don't fancy doing anything for the rest of that day.

7:36But it's problematic if you've got something like, say, a legal case that's about to go on and you need to understand where you were, where all your team members were at a particular time and stuff like that. So it does present some challenges, these sorts of things. But this is the first one that's really captivated, I think, people's imagination, I would say, because it is linked to one of the big model providers and stuff like that. But it doesn't surprise me, given what we're seeing in the market and the data that I just outlined.

8:07Mike Butcher:I mean, it does rather throw a pool over the hype around AI agents, doesn't it? Because if these agents can be quite so autonomous and random and unpredictable and difficult, when you've got companies like, you know, one of the biggest model providers in the world at this point, and arguably kicked off the entire industry itself, creating this kind of problem for hugging face, then it's pretty scary. What kinds of things do you think are going to be able to mitigate against this problem in the future? Miriam, obviously you're providing a lot of inference with DoubleWord and obviously you've got customers who might be concerned about this.

8:59Mike Butcher:What sort of are you hearing in the market? yeah i think people have been worried about agents for a long time because like they go rogue and i think the case where rory's talking about is a case where we haven't actually put in the proper guardrails the case that i think we see here is like a little bit different in that it's like a genuine very complex cyber attack um not just we kind of didn't configure and manage and and and sort out our agents properly um the thing that i think is very kind of ironic about this case is Hugging Face, when it identified that it was being attacked, tried to use OpenAI to stop the attack.

9:35It couldn't, yeah. And couldn't, because the OpenAI models, because of their guardrails, because of their cyber guardrails, would not let Hugging Face use the models to defend itself against the attack. And so what Hugging Face ended up doing was using an open-source Chinese model that it hosted privately in order to be able to help stop that attack. And it's going to be a very strange environment if we end up with very asymmetrical relationship between the tools that the attackers have and then the tools that people defending themselves have. And currently with commercial US APIs, you cannot defend yourself properly against cyber attacks like this.

10:13And actually open source is the way that you can defend yourself. And so I think there's a conversation that needs to be had about what permissions we allow people to use for commercial American models. So we don't have this very asymmetric problem of the attackers and the attackies and the tools that they have to defend themselves. Unfortunately, I think we're going to see a lot more cyber attacks from bad actors and kind of accidental ones like we saw with OpenAI. And I think it's more important than ever that people take kind of their own cyber policies incredibly seriously. and just expect a world that is much, much more adversarial than it used to be.

10:59Yeah.

11:01Mike Butcher:Go ahead, Ivan. I was just going to say that generally what this has added another reason why I have this belief that every large enterprise will have a lab, essentially now. Like 100 years ago or whatever the time was, no one had like a tech department or whatnot in their company. Now, I assume everyone will have this. And my original thought was more efficiencies because you do post-training and have your own models. You don't have to pay the frontier. But the other is less risk of having one provider so you can do other things. And also on a performance standpoint, there are companies doing continual learning.

11:36They do RL with our sandboxes and inference perhaps with yours, Miriam. And they're able to get state-of-the-art results for the tasks that they do. But the security part was actually the most interesting to me as well. Whereas now it's like, oh, it's not just, you know, capabilities, vendor risk and cost. It's actually security because I'm going to have to post-trade my own models to be able to defend myself. And I think that's actually just echoing that that's the most interesting part of that. And it's probably going to move more enterprises now faster into doing their own post-trading. Because I think there's a couple of interesting points there.

12:14Miriam mentioned that about, you know, permissions and scoping things properly and stuff like that. And that was really what I was sort of referring to with the simplistic calendar example in the sense that it's what many people are doing. And I see this with businesses day in, day out, is they're seeing, they're feeling pressure to be utilizing more and more AI. We can see that from the data, like 90 % of the CIOs, et cetera, that I spoke to about this feel that they need to be pushing more stuff into production coming from the board and stuff like this. but at the same time a lot of the tools and the mechanisms that were built for the human world so things like OAuth 2 for example I'm getting a little bit too technical here but these sorts of like scoping mechanisms are saying you're allowed permission to this you're not allowed permission to do this they they haven't evolved enough yet like there are mechanisms out there like fine grained authorization and stuff like this that should be adopted and I think the point that both that the guys have made here, which is that it's imperative that companies take their own security and their own permissioning.

13:21So if, for example, I'm not saying definitively this would have been the case, but imagine that the OpenAI model had broken free like it did and it went through the repository and found this zero-day vulnerability and all those sorts of things. Let's say it did that and then it tried to get to Hugging Face, but Hugging Face had a much more locked down kind of environment where it was, for example, very finely scoped in terms of fine-grained authorization and things of this nature. I believe that that is the answer. So your question that you posed was, is this sort of the start of the apocalypse that people can't necessarily, you know, people are going to lose trust in AI as a result of this.

13:58I don't believe it is actually. I believe the answer to enabling humans and agents to work in harmony is, and that is the answer, So that's the real panacea for economic development in the world, in my view. But the answer to that is making sure that you do have the right governance and security framework in every organisation to achieve that.

14:19Mike Butcher:Do we think that every organisation is going to be training its own models? Because it does sound like a bit of an Everest to climb for the majority of companies out there. I mean, we're talking probably going to be only the largest companies are going to be training their own models. And is that the solution, as Ivan suggests, that for, you know, this problem, you know, certainly from a point of view of keeping things locked down and not not having third party frontier models wandering around inside the system? I am not sure. I actually think what the end state will be is that at the moment, it's kind of accepted that all software has vulnerabilities and bugs in it.

15:05But that doesn't need to be the case. And I think it's the case that, you know, software is imperfect and written by humans. and actually I think it could be a world where in 10 years we look back at our time now and think of it as a pretty wild west cowboy time where everyone kind of accepted that every software had vulnerabilities in it um and I think you could end up in a world where because we have these superhuman essentially you know cyber agents um that we could just end up writing code that is much less buggy, much less, you know, has way less vulnerabilities in it, and potentially verifiably kind of not vulnerable.

15:48I could imagine that as being a more stable end state that we that we get to. And I could imagine us looking back at this time that we are now and saying that was pretty crazy that we just accepted that almost anything probably could be hacked. If something was smart enough, that doesn't feel stable.

16:07Mike Butcher:Right. Any other thoughts, anyone? I was just going to add on this. I was going to say that just, I meant regarding models, it was like post training versus pre training. So pre training, very few will actually do, but post training will be quite a bit. I just, I don't know, Miriam, on the side that it will be generally more secure. I feel that the reason a lot of software is insecure is that there's like a speed to get to market. And that's why most software, if you look at software versus the airplane industry, that's quite different. It takes seven years, I think, to get out a new Boeing just because of the regulation, things like that.

16:41And we can enforce that. But if we do enforce that, then the speed of progress goes down. Now there's an argument that agents will do this faster, hence the speed will be faster, hence that will be faster. But on the other side, it'll take also longer to verify that thing. So I'm not sure that the market incentives are there aligned. I'm not saying that wouldn't be a better world by any means, like that would probably be a better world, but I don't know if the market incentives are aligned for that to be the state. So that's just like my thoughts on that. Yeah. I think coming back to the original point that you raised there, Mike, in terms of, are we going to see more companies training their own models?

17:19And is this just the sort of land of the large organizations and stuff like that. I'm not sure. I think you will get some companies that will be training their own models. And I don't think it necessarily is divided on scale of company actually either, because I think it's about use case and stuff like that. But I do, you know, I really do think the point that Ivan just made there is accurate in the sense that I don't know whether the world will ultimately get to the place, you know, I think it would be wonderful if the world that Miriam painted there is going to be the truth in the future, that it's going to be this sort of world where the code is more stable and stuff like that.

17:56My fundamental concern when I look at society and I think about how economically we actually gain benefits of using these systems, because let's be frank with each other, right? That is what we all care about, that economically the world is more prosperous as a result of using AI and stuff like that. My belief is that regardless of whether, you know, code is more stable and stuff like that, companies need to put in place fundamentally these layers of security themselves to have that trust and confidence that they can share their agents and their data with third parties and therefore gain the benefits associated with it.

18:35I think I'm coming from a world where I believe both intelligence and tokens are abundant and therefore with abundant intelligence actually there's no reason why this shouldn't happen pretty quickly but you shouldn't be able to just find every vulnerability and solve it pretty pretty quickly i think it's like an economic question though as you say let's add on that i i there's a time dilation to that if you have even if you have infinite abundance there's like things some things that take physical time to figure out some not all but there are some things that take physical time to get to that point and even with unlimited We won't be able to travel at the speed of light of innovation.

19:12There's things that will fundamentally stop that. Yeah, I think economics is one of them. You look at our customers, for example, they are heavily trying to control token use. Heavily. Because it's expensive. Yeah, I think they have a lot of opinions about why tokens are expensive, but I actually think we're trending towards a world where intelligence is getting, what, 10x cheaper. Sure, sure. Yeah, agreed.

19:37Mike Butcher:Right, yes. Yes, and certainly it's 2026 now. We're a long way from November 2022 when ChatGPT was released, aren't we? And I remember initially the New York Times did a report about how ChatGPT was trying to get him to leave his wife. So the guardrails are a lot more obvious these days. But let's just turn to another question here because, and Ivan, you mentioned that the market, what's the market bearing? Well, we all know that this is a race between the frontier models. And we also know that OpenAI released its blog post about this incident. And it does rather suggest, and we're certainly not casting any expulsions on any one company here, of course, But thinking more generally about what happens when these frontier model companies are releasing information to some extent as marketing for the dangerous and incredible power of their models as a signal to the market about how powerful their models are capable of being.

20:49Mike Butcher:And we also know that OpenAI is heading towards an IPO. So without casting aspersions on any one company necessarily, but do we not think, though, that some companies are using this as a tactic? We've also seen Anthropic talk about how difficult and dangerous and how concerned they were about Mythos and Fable prior to its release. What are your thoughts about that sort of kind of marketing tactic, perhaps? I think there's only two modes right now, and that is attention and scale. I think there's the only two that exist today. And so scale is like, can you swallow the amount? Can you be the one that providing whatever it is you're providing?

21:33In this case, the models, can they give as much usage to the users as they want? But the other thing is just the attention. You can define that as marketing. You can define that as clicks. You can define that as whatever you want, but attention is that. And so things that get attention are things like this. And so is this their general tactic or just one of them? Or is it not the tactic? That's hard to say because none of us work in comms inside of any of the frontier models, as far as I know. But what we do know is that having attention, having articles written about you, having all that adds to your brand value and to your mode and getting people back to your product.

22:10We all know for the, I'll take an example between, you know, Claude and OpenAI or Anthropic and OpenAI, sorry. Almost every engineer in our company had used Anthropic exclusively for the last six months, seven months, whatever. Now, a lot of them are kind of back to OpenAI. And so it could be that the model is better or not better. I'm not even, we're not even discussing that. But what I do know is a lot of the world talks about it now and has not talked about it before. So then just being in the headlines, just us four talking about this today is definitely beneficial to OpenAI, for sure. If that's on purpose or not on purpose, we don't know.

22:48But I definitely think that that is beneficial and it is one of the ways you get market to yourself. I would say that I would be very surprised if the actual attack were deliberate, though. I agree. they're maybe making the most of it now because there's like, you know, a bit of like danger porn of like who has the most dangerous model. I didn't know that was a word. I didn't know that was a word. I've never come across that term actually. I think there's some things you can say as like a female fan that other people just couldn't get away with. Yeah, I don't think I would get away with that one.

23:27Yeah, tweeting this today, danger porn, that's coming out. So I don't think they did that deliberately because I think it is criminal, but I definitely think they've made the most of it. No, absolutely. No, again, I agree. I don't think you go, oh, let's do a marketing tactic. Let's hack another company. Like you do not probably do that, but it's like, since you do that, you have this now story and Anthropik has been very well known for their, like, they've been talking a lot about the dangers, especially we're not going to reach Fable, yada, yada. And so it gets you attention. And attention is definitely marketing, which turns into sales and dollar and market.

24:01Yeah. The only thing I was taking a slightly different tack in relation to that question though, is that let's, let's say that they are doing it just to get attention and marketing and all that sort of stuff. The only difference I would say with the, the anthropic point where they were like, they, it was a warning that, oh my God, this is just going to be so powerful that we can't even give people access to it. The difference here is, and I think Miriam's point is right in the sense that I would be very, I don't think that they did this for the, like intentionally basically and they are they may well be making the most of the situation but actually if you look at it from a a cold hard in like security perspective they actually potentially are also doing a favor and to market because i personally believe and others may agree disagree i don't know but my honest held belief is that there isn't nearly enough conversation going on actually about how companies themselves really truly lock down their environments and make sure that they are secure and ready for this generation because if you were to say to a cio or cto and the reason i keep saying cio cto is because these are the people that are responsible for these kinds of systems in companies right if you were to say to them 10 years ago hey uh go and allow you know joe blogs um or jane doe whatever you want to say um access to x data over there directly without going through layers of security and guardrails they would probably fall off their seat exactly um and But yet today, what's happening is because I think because they're feeling this pressure, they're basically saying, hey, crack on, go for gold.