OpenAI's Bots Hack Hugging Face Autonomously — With Alex Stamos

24:45I mean, remember the model here was doing what it was asked, right? It was not scheming against its bosses at OpenAI. they asked it to take the test and they didn't they i don't know what exactly what the prompt was but apparently they did not tell it not to cheat right so who knows like this this is also what open ai needs to be more transparent about is exactly what their prompt was exactly what the constraints were it did they tell it explicitly like it is a much bigger alignment problem if they explicitly said do not try to break out of the network do not try to get the test answers Now, if they told it all those things, then they have a much more significant alignment problem, right, than if they were less explicit.

25:35But in any case, yeah, I mean, that will – if these models get trained to be more evasive from a network intrusion perspective, that will be very dangerous, yes.

25:47Big Technology Podcast:and what I would argue is for the legitimate companies I would not do that I don't think I think there is a if you're open AI and you're building 5.6 cyber what you should be training it to do is find bugs you should be training it to write proof of concepts you should be training it to do all the defensive stuff you should not be training it to hide it to hide all those things like if If the US government wants to build a model that does that stuff for the NSA, then you can let them do that. Or you can let Lockheed Martin do that. But if I was open AI or anthropic at this point, I probably would not do that.

26:29I think I would leave that for somebody else.

26:32Big Technology Podcast:But that's scary, though, because it could then take actions that, you know, I think one of the things that is. So the question is, like, should we be concerned with the AI, you know, sort of doing things on its own? and should we be concerned with you know or is the bigger concern that humans direct this ai to do bad things so we've definitely covered the fact that humans will be more should be you know humans who direct this ai to do bad things can do a lot of damage but if you create an ai that can reward hack because this is all coming from reinforcement learning where like these ais are given rewards and they are basically like maniacally focused on achieving that goal and if you it's almost like sort of gain-of-function research on a virus to a degree, right?

27:14Big Technology Podcast:Because if anybody builds AI that doesn't leave a trace and it goes out and reward hacks its way into hacking something else and maybe isn't so fully going with the prompt, then that's where you can get into a real problem. I know that's a more out-there possibility, but I don't know if it should be completely discounted.

27:37Yeah, I mean, I guess as they get more and more complicated, the question is is like what at what point are they is it their own motivations versus just doing what you've asked it to do you know i mean so far again i

27:55i don't think we should still think of these things having their own desires or wants they are still doing what they're asked to do. It's just, like you said, there's a lot of inputs of what they were asked. It is not just the initial box, right? There's all, there's the system prompt and all the training and all of the rewards and everything that's gone in. And so the question is like, what is the humongous history of all of the different things it's been trained to do when you've asked it, take this task. um right and especially if you've removed all the protections um and so in a situation where these things have all the protections removed that is very dangerous and as we talked about like with the open weight models either there are no protections or the protections are trivially eliminated right like a bunch of open weight models have been trained with protections but you can obliterate those out and you can go on hugging face and look for obliterate and you will find a zillion models where people have removed the protections but this goes basically

28:58Big Technology Podcast:back to that like long held thought experiment of, you know, the AI can follow your goal and achieve your goal, but it might have a different idea about what it takes to get there than you do. So in this case, you know, so exactly. So I was, I was going right there. So, you know, this is, and it's funny because I am speaking with Nick Bostrom later today to, you know, for an episode that's coming up, but basically he's this Oxford philosopher who came up with this idea that if you ask an AI to make paperclips, eventually it can seize so much on this goal that it can find humans as an impediment to its objective to maximize paperclips and sort of kill us all and turn everything in the world into paperclips.

29:40Big Technology Podcast:So the fact that it was on tasks, this is kind of a Twitter user said this, once out of their sandbox, the models did not scheme, engage in behavior that had nothing to do with their instructions, like hacking the NSA or launching a cyber attack on Russia or stealing secrets from a rival AI lab. But like, you know, sort of, if the model found it suitable to go out and hack open, hack hugging face in this situation, who's to say that, you know, maybe a less careful model doesn't do this. And then maybe an even less, like doesn't go and hack the NSA. And then even less careful model turns us all into paperclips.

30:15Big Technology Podcast:I mean, there's a continuum there. Yeah, I mean, it's why you have to be very careful what tools you attach to them and it's why you need to have they have to be supervised by different things i think like you just can't um you can't have models that have no protections on them that have have connections to tools right like that's why these models then you have dumb classifiers or dumber models watching them you don't just take the smart thing and then hook it up to everything and you're like give it a task you have the smart thing and there's a bunch of dumber things watching it. And those dumber things can either kill it or they can call a human that can kill it.

30:54Right? That's the idea. It's like, there's supposed to be cyber classifiers and there's supposed to be mechanisms that can stop it. And those mechanisms should be either deterministic or dumb and undefeatable by the model. And they removed all those things so that the eval would work. So I think what OpenAI is basically hinting at, they haven't been explicit, is like, if we're removing those protections, this thing is going to be in an absolute physical jail. It will be physically separated. It will not be hooked up to the internet anymore. And that seems like that should be the standard. That is fine for open AI.

31:26My point here is that doesn't matter. This situation is good that this happened because this has pointed to us where we might be in six, nine months, a year from now, no matter what, because other people, unless we can get a international agreement to just stop development, which is what other people are talking about, right? you've got this, I forgot what, like Project 2030, or like you've got people talking about international treaties or whatever. I don't think any of that's going to happen. I don't know what my position is on that, but I just don't think it's going to happen. I just, I think there's no way, this is just math and silicon.

32:04And so I just don't think there's any way you get like a, this is not like nuclear weapons, where the major input, like the reason our species is alive is the major input to nuclear weapons is uranium, plutonium. Plutonium does not occur naturally in our planet and uranium is incredibly rare. And to turn raw uranium into uranium that can go into nuclear weapons is a massive industrial process. If uranium was something you could just dig out of the ground anywhere, our species would be dead, right? Like that's just the truth. Because the knowledge to build a nuclear bomb is in the hands of anybody who gets a physics PhD, unfortunately.

32:41So in this case, these chips are not something you can really control. We have found that in that the Biden era controls on silicon have created a massive industry in China. And the knowledge on how to build large language models is something that there is a undergraduate class at Stanford where you get that knowledge, right? You know.

33:06Big Technology Podcast:You can find it from like a Karpathy interview, YouTube video. Yes. Right. So we, we cannot control that knowledge. Um, and, uh, so like the idea that we can just have like a bunch of people agree in a room to stop all development of this is just silly. So from my perspective, being a little bit of a pessimist here, we just have to get ready for this level of capability to be in the hands of an unfortunately large number of people. Yep. All right. So I want to go a little bit deeper into the potential solutions here. and also I want to ask you the age-old question of is some of this all this none of this just good marketing for open AI given some of the statements they've been making we have to address that one here on the show but I'm going to let you have an answer I'm going to try to at least you know illustrate those the case of those who might be saying it so we can have a discussion about that let's do that when we come back right after this hi everyone Alex Cantrowitz here I want I want to tell you about a documentary I've made with Gravity to explore the future of AI agent security.

34:07Big Technology Podcast:To find out if we're truly ready for autonomous agents, I sat down with MIT professor Ramesh Raskar, former White House CIO Theresa Payton, Michelin's Group Chief Data and AI officer, Ambika Rajagopal, and Sharon Guy, a former executive at Alibaba. They each offer unique insights into this evolving landscape. We conclude with Rory Blundell, CEO of Gravity, to discuss the path forward. With Gravity leading the way, join us on this journey. You can watch the full documentary at the link in the show notes.

34:50Big Technology Podcast:This episode is brought to you by DeepL. When I sat down with DeepL's founder, Yarek Kutlyovsky, on YouTube recently, we got into the case for specialized AI. DeepL Voice is what it looks like when the stakes are real-time conversation, and honestly, it's something I wish I'd had for my own cross-border interviews, turning a language barrier into a non-issue. DeepL Voice delivers live translation in over 40 languages for virtual meetings and in-person conversations, helping people speak in their preferred language without losing flow or nuance. Whether you're meeting with a customer, negotiating with a supplier, or collaborating with global colleagues, it keeps pace with you in real time, easily handling the technical terms, acronyms, and product names specific to your business, so what you actually mean never gets lost in translation.

35:33Big Technology Podcast:And for the builders listening, DeepL's voice API lets you embed real-time speech transcription and translation directly into your products. So go check it out for yourself. You can try DeepL voice for free at deepl.com slash try voice. That's deepl.com slash try voice. This episode is brought to you by Google Chrome. You think you know a browser. But Gemini and Chrome, that's new. It can help you with practically anything on the web, like restoring a vintage motorcycle from a 50-page restoration block, or finally break down that long article you've had open for weeks. Gemini and Chrome is here for it.

36:08Ready to make anything online make sense? There's no place like Chrome. Check responses set up required compatibility and availability varies 18+.

36:16Big Technology Podcast:And we're back here on Big Technology Podcast with Alex Damos, the Chief Product Officer at Corridor. You sort of answered the question before the break, but I'm going to ask it anyway, whether part of this is open AI marketing. Let me at least read some of the statements here and give you at least the argument that people have made for why some of this is marketing for open AI. The first part is, you know, the Anthropic started to be declared as the company that was in the lead once that anecdote came out about Mythos breaking containment and emailing somebody when it wasn't supposed to have internet access and emailing an anthropic employee while they were out in the park having a sandwich.

36:58Big Technology Podcast:So this could be potentially, you know, OpenAI's attempt to like one up that. Then there's also the language that you see. OpenAI says in its tweet about this, we are partnering with HuggingFace to investigate an unprecedented security incident. You don't usually have the attacker and the attackie partnering together in these situations. They also said, we consider in their blog post, We consider this incident to be an unprecedented cyber incident involving state-of-the-art capabilities and are responding accordingly. You know, it's sort of like, oh, look at this terrible thing that happened, but a moment to share how good our cyber capabilities are.

37:37Big Technology Podcast:That's the argument. What is your response to the notion that this might be some marketing from OpenAI? I know lots of people at OpenAI. Every single one of them absolutely hated Anthropics marketing around Mythos and thought it put the entire industry at risk. This incident has put OpenAI at risk of regulation from the White House, regulation from the EU. It is also an admission of the violation of the Computer Fraud and Abuse Act, as well as multiple European laws. It would be absolutely insane for them to use this as a marketing moment. What you're seeing is them being very, very careful and defensive in their language.

38:20They're also very lucky that Hugging Face is being super cool and chill about this. So that is why they are saying these things. Because Hugging Face initially comes out saying, we've been attacked. We don't know who it is. But it does not look like the model was being subtle. I don't know where it was running. It's quite possible it's like Azure or something. It was probably not covering its tracks. And so I expect Hugging Face got their American lawyers involved, was working with the FBI, was probably issuing subpoenas, and was very, very close to finding out it was just open AI. So, like, or did find out.

38:54I do not know the timeline here. But, like, the legal issues here are very fascinating and interesting. And because they're all working together, I expect nobody goes to jail. Nobody gets sued. Everybody's going to hold hands and hug. And if there is tokens being exchanged or whatever, I don't know. But there's absolutely positively no way this was a intentional marketing move. And OpenAI is doing the best they can, I am sure, right now to use this to forestall any kind of massive government overreaction, either from the United States or the European Union.

39:31Big Technology Podcast:When you were at our summit, you said that, you know, speaking of the sort of release of Mythos and Fable, that a lot of people were very concerned about the bug finding that those models could do. But you said basically, listen, this is not very different from what you could get with Opus 4.7 or 4.8, I believe. Is this, is what we're seeing from OpenAI very different? is this a step up? Yeah. So this is what I don't know if I said on stage here, but I've said in other places, there's a difference between the short term and long term. And Anthropic to their credit, and I think OpenAI has in other places.

40:11I think we talked about how in the Fable model card, they talk about short horizon versus long horizon cyber tasks. And what I've talked about is we need to not focus on the short horizon tasks because those are dual use. Finding bugs is dual use. everybody needs to find bugs right um that is something that defenders need to do all the time and that's what's driving people insane right now in the defensive industry is that because of the white house american models are refusing to help fix code they are refusing to help us find our bugs and fix them thanks to the white house's actions that is not this problem this problem is go run an entire attack chain for me.

40:57That is the long horizon tasks. And that is where we need to continue to have appropriate classifiers that are like, bro, I am not going to break into a bank for you, or I'm not going to plot out or run a C2 mod for you or any of that. So yes, this is what, you know, explicitly Anthropic said, we will allow Fable to do short horizon stuff, but we will not allow it to do the long horizon stuff that Mythos does. Right.

41:26Big Technology Podcast:And so Mythos, just to confirm, what Mythos can do the long horizon planning and what we're seeing in this instance from OpenAI, that is the step up. That is the step. And I can't, obviously, I don't have access to this, whatever this thing is. And so I can't say whether or not where they are. AISI has done these assessments. And so who knows how good this is versus, but this seems beyond even mythos capability and long horizon who knows right but like yes this is this is what when people talk about mythos is long horizon this is what they're concerned about okay so so let's end here what what happens next like where what should be the the approach from the government and the companies uh developing this stuff to ensure that we can sort of move forward as a species and safely.

42:16Big Technology Podcast:So let me give you like a couple of solutions and have you comment on them. Let's go back to Harlan Stewart. He's the spokesperson for MIRI, which is the sort of rationalist organization that thinks that AI will kill us, run by Eliezer Yudkowsky. Harlan says, this should go without saying, but it would be insane for open AI to now proceed with building a new model that's 2x or 4x the size of this one. Doing that should be deeply taboo it should be illegal preventing it should be a top priority around the globe your thoughts

42:50um i mean if if we realistically could get everybody to pause ai development or slow it down and have reasonable safeguards i'd be fine with that i just don't think that's reasonable i think there's absolutely no way you get china to agree to anything like that i think there's It's impossible at this point. And I think a enforcement of anything like that would effectively be impossible, right? Like a start treaty for AI, I don't know how you'd possibly make something like that work. So what, we're having like satellites see if people are building data centers. We're measuring power usage. Sorry, I shouldn't laugh.

43:37Big Technology Podcast:But yeah, you're right. It's really, it does not seem like a feasible thing. Yeah.

43:45So, I mean, you know, it's effectively, we'd have to invent the Turing police out of Neuromancer.

43:55And, you know, I think more realistically, what we need to do is we need to build controls for, we need to say is like, as AI gets smarter, it has to have controls in place. um ai systems that don't have the control have to be air gapped right so like if you're going to do these kinds of evaluations they absolutely have to be air gapped um the problem is is like we've we've lost there was a process in place to create standards for this kind of stuff that process was stopped by the current administration um my recommendation to the companies i just is that they need to move forward with building these standards themselves without waiting for the admin.

44:33There's a foundation model forum that's talked about doing that. They should just move forward with like, okay, great. If we're building models and we do not have restrictions on them, these are the controls in place. So what I like to see is opening Ionanthropic say, great, if we're building cyber models and they don't have restrictions, these are the standards of what air gapping looks like and such. For any cyber models, these are standards of who gets access to them. These are the capabilities that the cyber models have. This is what we define as cyber models having versus an open model. This is our definition of a short horizon versus long horizon.

45:07Like those are the kinds of things that people have not written down. They have to be written down now. Right. Um, and I think the industry needs to move forward with that without waiting for Cassie, like this is all just taking way too long. Um, and the focus ever since the fable freak out has only been on one tiny little part of all of these risks. And it's just, as we see, like we've been frozen in this tiny little discussion and all of these things are moving forward too fast. Like we, we just can't wait for the White House politics here. We need to, to move much more quickly. Yeah.

45:37Big Technology Podcast:And while we've had that, there's been, sorry, go ahead. No, no, you go ahead. Go ahead. And, and then while we've had this tiny little discussion in the U S GLM five, two shipped, Kimmy K three is shipped. Like the, the, the Chinese ecosystem has caught up really quickly. So sure. I mean, it would be great to just hit pause, but like it's, I just don't see that as realistic. So like, I just don't see how that possibly happens. So the open AI suggestion is basically, you know, kind of, it's almost like to solve this problem generated by AI, you need more AI. This is their statement. We believe advanced cyber capable models need to help security teams find weaknesses before attackers do.

46:21I mean, right now, I think that is probably the only way. Like if we're not going to be able to hit pause, then we really quickly have to find bugs and fix them. And we have to put AI enabled protections in place because the only way you can respond to attacks at that speed is using AI. Unfortunately, that's the truth. Yeah, again, like if we could pause for a year to figure this all out, that would be great. I just don't see that as realistic.

46:48Big Technology Podcast:Alex, does your gut tell you that we're screwed or that we'll figure this out?

46:56um i wouldn't say we're screwed but i think we're going to go through a couple of years of craziness um like we have 20 we're all living using 20 something years of really important software that was written mostly in non-type safe non-memory safe languages we're using you know for the software that is written in those kinds of languages it was not written with formal methods or appropriate security protections or reasonable uh you know secure development life cycles or architectures and these things were have tons and tons of bugs that we can only use safely because there's just not enough attackers now with ai you can spin up any individual can spin up dozens or hundreds of qualified attackers uh at a moment's notice and it used to be that those then six months ago those attackers had to be in the cloud and soon enough they'll be able to run on local hardware in the new M5 Ultra Max that'll be shipping soon, right?

47:58And so that, I mean we're just going to have a couple years of total chaos from a cyber perspective in the long run software is going to be much better because AI is going to be paired up with humans to make it more secure and more trustworthy but it's going to take us years to do that and to clear out the decades of mistakes we made.

48:24And yeah, it's just going to be pretty rough. It's going to be pretty rough going for a little bit.

48:29Big Technology Podcast:Yeah. I just want to close with this. This was a tweet from Kevin Ruse that kind of made me laugh. I thought I would read it here just so we could enjoy it. He writes, opens the portal to the godlike superintelligence that solves 87 year old math problems and carries out autonomous cyber attacks and asks how long peanut butter good in fridge it's it is amazing that this technology is um you know at once so capable and we're we do seem to be like more and more turning to it for the most mundane of all things which is sort of it's the wild thing about you know the generality of these systems they can do so much interesting time yeah alex you're gonna be busy i think over the next couple of years as this stuff gets sorted out i was hoping to retire man i guess not yeah well either way i do hope that you uh join us again to help us sort through uh this stuff i mean your thoughts on fable mythos last month and and now talking through this situation with open has really been and valuable for the show.

49:36Big Technology Podcast:So really appreciate you being here. I feel like there's going to be plenty of emergency podcasts. I think so. We should have you on speed dial. And I know you're coming at us from like the middle of an offsite in an exotic location. Now I know I will always take a podcast microphone and a different shirt with me wherever I go. No. Sound good, look good. Alex, thank you so much. Really appreciate you coming on. Okay, thanks, man. Talk to you later. All right. Thanks, everybody, for watching and listening. And we'll see you next time on Big Technology Podcast.

50:08Athletic Brewing Company Crafts award-winning non-alcoholic beers For those who want to be part of every round With over 185 flavor awards

50:16Big Technology Podcast:They're exceptional NA beers That fit your lifestyle and any social occasion Summer's full of good times And Athletic fits right in Go to athleticbrewing.com To have brews delivered to your door Or find them at a bar, restaurant, or store near you Near beer Athletic Brewing Company Fit for all times You have the AI strategy, you bought the tools, but your teams aren't using them effectively. Pluralsight AI Academy closes that gap. Hands-on upskilling, trusted by major brands around the world. Learn more at pluralsight.com slash AI Academy.