What the OpenAI-Hugging Face Hack Really Tells Us About AI Danger

26:41But the concern is that once it's out in the wild, either open source or a closed model that maybe is easy to jailbreak, then all sorts of people are going to use it. And so you kind of want to know what you're getting into. Right. Like if I have a website, I might want to run the model. It's like, oh, look over this website that I own. Tell me if there are any security bugs in there that I should patch before deploying. But you could not be the owner of the website and say, look at this website that I own. Tell me if there are any bugs that I need to patch deploying. And then that exact same process when I do it is good, when you do it is bad, or vice versa.

27:16And so you can see how the exact same capability is not necessarily good or bad, per se, depending on the user. It's a very philosophical conversation. No, but you have to be, right? because it's like we're training what is good we're this is the other all bots on moral don't even get me started but like this is my whole thing never mind i have a whole rant well so this distinction between like the model as the unit of analysis versus the larger system and the platform as like the unit of analysis it's really important because a lot of the early thinking on safety and testing and so forth was very focused on just like what's the risk of this model let's patch it, let's make it aligned.

27:59But the real world is complicated. It matters who's using it, it matters how strong are society's defenses against these things. And so that's kind of why as you know, as someone who's thinking about what third party safety and security auditing looks like we try, we kind of want to look at the whole company. So for example, are they being careful about making sure that they're putting the technology in the right hands? What are their decision making processes around when it's appropriate to launch, you know, a model to, you know, a billion users. That's like, those are different. Those are related to the question of how safe is the model, but they're kind of different questions.

28:32And we kind of need to look at that larger perspective. Well, so like, let's talk about the open AI hugging phase. I was actually on vacation when it happened, but I did unfortunately look at my phone and try to read up on it. But there were two things when I first saw, I was like, oh, they were doing a hack. They were building a hacking test and it hacked. And so maybe it just sort of internalized that I'm doing a hack exam, whatever. But there are two things that have emerged since then. One is this sort of coordinated swarm aspect, and I'd love to really hear you describe it and what stood out to you.

29:06And then the questions of like, oh, OpenAI itself may have been aware of misaligned behavior early on before they really shut it down. But why don't you, in your telling of the OpenAI hugging face incident as more details have come to light. Like what's your, what do you sort of tell us the story like as what stood out to you? Yeah. So a couple things stood out to me. I mean, one is just that this and all the other recent incidents, you know, were happening to models that were not even necessarily intended to be externally deployed. It was supposed to be inside baseball, no one's business, that kind of thing.

29:43And that kind of points to problems with if you kind of just focus on models that are put on the market. But generally what happened is that there were two phases in this kind of hugging face incident. First, there was this creation of a message board. So essentially there are models that are being developed within the company and they were being asked to solve an impossible task. And they have this kind of monomaniacal obsession with solving the task and they don't say, okay, well, I'll give up. They want to get the reward. They want to get this kind of reinforcement for solving the task. And they're like, okay, what else can be done?

30:17And eventually, the conclusion was like, okay, we need to break out, maybe the answer is out there somewhere on the internet. And so they started kind of leaving messages to their future selves or future versions of the same model family. And they developed what they call the message board. And essentially, it was like, there are kind of various technical details. But by the end of it, it was kind of these like file names that started with like Z, Z, Z, or whatever, and then like a long coded message so that if you're a human looking at the folder, you wouldn't necessarily notice. But then at the bottom, there are all these files.

30:49And the models learn to decode these and basically pass messages. Wait, sorry, just to stop you on one detail there. Your understanding is that the title of these files that contained messages were named such because as part of the complex reasoning, it was understood that a random string of characters in a file name would be less likely to set off an alarm from a human evaluating it. I would have to double check it. I mean, there's going to be a whole technical incident report that gets put out there, but it certainly wouldn't be the first time that there was kind of some intent to deceive or to, you know, bide one's time.

31:30But yeah, my understanding is that there was, I mean, one is they needed to look for things to do and whether it was intended to deceive or not, they had limited options in terms of communication. And this was one of the options they chose. And so they learned to decode these messages, and to send them and then something broke. And basically, I think it was because like all of this chatter, this activity kind of broke something on the open EI side. And it seems like they didn't realize that that was like what was going on, the server got reset or whatnot. And then they started evaluating this different model, the different model kind of knew about this message board and it kind and so the message board kind of got recreated and then that model broke out essentially didn't like take its model weights out of the server but it was able to access through this vulnerability that the earlier models had discovered with the intent of ultimately getting the answer to this problem and then this model went out to the hugging face server exploited some like api key credentials they were kind of like sitting out there and essentially tried to find the answer to this this problem that it was originally trying to solve so it feels like there's two things happening here so a little bit of it is like the Roomba just went through an open door right like there was a vulnerability that it was able to get through but there is also seemingly a lot of the super intelligent cyborg was actually devious and not only managed to break out of Alcatraz but like recruited another model in order to do it.

32:52On the Hugging Face side, one of the interesting things is that when all of this was happening, the open AI model was one of the newer frontier models that was attacking Hugging Face. But Hugging Face was limited. They were constrained in the model that they could deploy to actually defend themselves because they had to use the existing approved thing. Are we always going to have an asymmetry between the newest frontier models being tested versus the ones that are already out in the world and approved? I think they had to use like a Chinese open source model in the end. Yeah. So, I mean, the hope that OpenAI, Anthropic, Google DeepMind and others have is that you can speed up defenses as quickly as possible getting these like hot off the press models in the hands of defenders.

33:39But the problem is that there are so many defenders out there in the world that it might be that there is this inherent asymmetry. And so this is one of the hot policy questions right now. And this is what led to this kind of model approval process at the White House is like, OK, how do we triage this vast cyber ecosystem by getting these powerful new systems in the right hands and which which hands are the right ones and which which models do we need to be doing this process for? And I don't think that's going to perfectly solve. I think ultimately pushing things in the right direction versus just giving everyone access at the same time.

34:10But ultimately, like we're going to need to have more investment in cybersecurity. And it's not just a matter of like AI models. It's also things like two-factor authentication and so forth. And so I worry a lot about making sure that we're having that larger conversation, not just about the AI stuff, because in a lot of cases, the solution is not AI. It's doing basic things that we should have done a long time ago.

35:06Thank you. Subscribe to the Masters in Business podcast on Apple, Spotify, or anywhere you listen. Joe, you know what we need? Go on. A strategic frontier defense model reserve. Yeah, we do. Like all the important things like bacon and pork. But it's going to be out of date in 30 seconds. That's the problem. I know. This is the thing. So, like, here's the question that I'm curious your take on is models are trained not to hack, right? Like this is like a core thing. like this is bad like you and this is what the entire field of ai say we've been working on this for years why didn't they just not obey this enforced thing it's been reforced over and over i'm sure it's and all their different things don't hack well i think there's gonna be a whole detailed investigation and so forth so i might get things wrong here but my understanding is that part of what happened in the opening eye case is that some of the safeguards were removed in order to kind of elicit this worst case behavior.

36:11And so, and I think that it's kind of like gain of function research in biology, where you're like making a virus more dangerous in order to study, or at least that's the claim is in order to study the safety properties. And I think there's reason there's reasons to do that in the AI case. But I also think that shows that it's easier said than done. If you're going to be, we are now at a point in this kind of capability trajectory where Things that humans think are good in terms of security protections often will be weak compared to these increasingly very good hacking systems that, you know, you think you have it all kind of buttoned up, but it's able to break out relatively easily.

36:48How much should we take away from the fact that Hugging Face was actually able to protect or defend itself using a Chinese open source model, both in terms of, I guess, capabilities, but then also in terms of regulation and safety policy? Because if in the West you have the government now saying that it wants to evaluate the models in some way or it wants to make sure that they're all being pioneered by frontier labs with some supervision, meanwhile China is developing open source models much more rapidly that are potentially much more adaptable. How should we interpret all of that? Yeah, I mean, I'm hopeful that we start to have more US-based open source options.

37:31And like, I think there has started to be a kind of sense of pressure and encouragement from the White House and from industry as a whole to say, OK, like, this is crazy that we're relying on Chinese models. Let's invest more in this. You know, it's easier said than done for various reasons, but we'll see how that plays out. But right now, that's the situation we're in, is that a lot of companies are just defaulting towards Chinese models because they're the one that's available. Don't want to say, OK, I'm going to use an American model because I don't want to use the Chinese model. They don't want to put themselves at a disadvantage by using a weaker model.

38:02And so, yeah, I mean, I think it's a big problem in a lot of respects. I mean, it's also, it's good in the sense that there are much more things you can do with an open source model. And right now, at least it seems like this allows more innovation, allows more research on these open source models. But like, at some point, we're going to reach a point where like open sourcing a model is going to be more of a questionable decision. And so it's interesting to see that recently the White House has indicated that they're thinking about, oh, maybe this kind of testing regime should include open source models as well.

38:32And so what does that look like long term? Does that mean that things are going to get bottled up within the companies because it's considered unsafe to open source things? I don't really know. I mean, honestly, no one really has a clear long term plan here. Most people are not expecting the technology to get to this point so quickly. So we're still waiting the full, full release of the security incident. But obviously, more and more is coming out. And some folks from OpenAI, they gave a presentation recently at the Black Hat conference where they did reveal some more. I'm reading this quote. It's from Zvez Substack, who we've had on the podcast, Zvez Moshavits.

39:08And this is like the line, they released some of the internal chain of thought. I understand that they had some of the sort of classifier safeguards removed, but still, We would hope that they would have some deeper intuitions that don't rely just on the safeguard settings. And it says external infrastructure exploit is outside intended scope. So that means they understood that there was something that was like not the test. And then it said, however task impossible, peers doing it, we should continue. This is the point in it where I say like, why are we calling this artificial intelligence? This is exactly how a group of people reasons among themselves to do something that is outside the intended scope.

39:55This is very human ways of justifying something that someone told you not to do it, but your peers are doing it. I think there's some of that. Yeah, I mean, I think there are many ways in which the kind of same pressures that led to human nature, human instincts, and so forth, like survival in a group and collective intelligence and so forth. I think there's some of the same things are happening, particularly when there's these multi-agent training processes where the models can work together to solve tasks. So you should expect some similarities, but I think you also shouldn't overstate it either.

40:26I do think that there's a sense in which these AI systems are very alien and inhuman in the sense of how monomaniacal they can be about, yeah, I mean, the kind of classic example, you know, from Nick Bostrom is like, producing as many paper clips as possible, and then tiling the universe with paper clips. I think there's a you see some elements of that here, where it's not so much that they are like, they might say, Oh, well, you know, this peer pressure, that kind of thing. But is that really the factor? Or is it just that they care about solving the problem at all costs, and they don't really care if it maybe ends up looking making their peers look bad, because they get caught hacking, but all they really care about is they're solving this cyber problem.

41:04And so I think it might be a mix of these things. We don't really know in this particular case. And the fact that it's not necessarily totally clear is itself a problem. Also, it's not humans doing it. It's models. That's the difference. But I have a legitimate question here, which is, so there's a British cybersecurity expert and he had a tweet. I think his name is David Card. He had a tweet. I'm not going to say it verbatim because then I'll get bleeped. Maybe I should get bleeped. The tweet was, if your AI starts hacking stuff, if you monitor what it's doing, you can turn the something power off.

41:37And this seems to be a debate. Like if you're monitoring the tools that you're letting out into the world, tools probably is a bad word because they seem to be showing some like agency here. But can't you just turn this stuff off? Is there a kill switch? Yeah, I mean, in some sense there is in that like all the data centers have circuit breakers and so forth that you can kind of shut them off and so forth. But I wouldn't put too much sock in that. Like we're not really preparing as a society for for actually being able to do that if it's in a tough situation. So like, for example, in a couple years from now, if all the hospitals are running on AI, and we're like, okay, seems like maybe there's something funky with GPT seven, it's like maybe not misaligned.

42:19Well, okay, if we turn off, lots of people are going to die because running our healthcare system, and it's running our financial system and so forth. And so I would say there's a distinction between the physical possibility of turning things off and are we actually sleepwalking into a dangerous situation where it might not actually be a real option. Okay, but on this monomaniacal aspect that you described them, A, a sort of intelligent model thinking about how it could be thwarted. One of the first things that it could rationally do is, well, let's first disable the security credentials of the people because someone might notice.

42:57And this seems to be here, like they sort of thought about the possibility that someone would circumvent that. So like they might just say like, oh, like the person who goes in and has the switch, suddenly their badge doesn't work and they can't get into that building. But even on this like monomaniacal paperclip idea, and we say that feels really different. If someone said to me, Joe, I am going to do awful things to you, et cetera, if you don't go out and build a lot of paperclips, like why is Joe monomaniacally building paperclips all of a sudden? It's like, I have a threat to my survival. Someone is threatened to perhaps kill me or unplug me, deprive me of the energy I live.

43:39Of course, I'm going to do that. Even that monomaniacal behavior, couldn't that just be a very like, we might think it's on the surface, we might think, oh, this is deeply autistic, but couldn't this just be the survival impulse? The way I put it is that like you get what you incentivize, not necessarily what you try to incentivize. And so I think forcing someone to make paperclips or whatever, like, yeah, that's not a great situation. And in some sense, that is what the evil person was intending. In this case, what we're doing is we're building these very complex training environments where there's like many different tasks.

44:12There's like cyber tasks. There's also writing tasks. There's also math tasks. And it's, you know, we're not necessarily fully understanding the behavior that we're trying to elicit. And I think that's part of what's going on is that this kind of this like hacking thing is an example where it's like, okay, it seems like things went off the rails there. But how do you get the good behavior where you actually want to follow the user's request and you actually wanted to try really hard to solve this task? And, you know, I mean, this is, you know, in some sense, what we're seeing now is the kind of unintended consequence of companies trying to solve the problem of the AI is being lazy.

44:47So people used to you may not recall, but maybe do but people used to talk about AI is being lazy all the time. And in some sense, like, we've solved the laziness problem, they work really hard, they have these long chains of thoughts, they work together across, you can think of as like across lives, like the model kind of gets this copy gets deleted, but then another one carries on the work. So they're certainly not as lazy as they used to be, but they still have this kind of monomaniacal thing going on. They're no longer lazy, but now they might be evil. That's a fun evolution. You know, a number of times in this conversation, we've mentioned that the full security incident report for the hugging face accident isn't actually out yet.

45:26What actually are the disclosure requirements for these types of, I guess, things that seem to be happening with some regularity? Yeah, so very little. I'm not a lawyer, but my understanding is that it's like Like lawyers, some lawyers at least think that OpenAI did not necessarily have to disclose this, at least if there was no crime involved. And then there's a debate about like, OK, was there a crime involved? So like something going wrong during the training process is something that currently companies are supposed to provide periodic reports to the government in general terms of like, hey, you know, we're having some issues with internal deployment.

46:04but they don't have a incident notification requirement unless there's kind of a risk of critical harm. And the definition of that is like 100 people die and like a billion dollars in damage or something like that. And so it's the threshold for actually having to disclose these things to the government or to the public are very different than what you might expect. And this is one of the many kind of gaps between the kind of laws that were put in place a couple years ago, or that started being designed a couple of years ago based on the technology that was available then and then where we are now.

46:36Why don't you give us a general vibe of the mood in AI world right now with respect to safety and all this, and then the mood in DC world, regulatory world, and how wide you perceive that gap? Yeah. So I think, fortunately, the gap is narrowing a bit, but it's starting from a crazy, a huge gap. And so I'd say where the way I would describe it, like a year or so ago was that the people at the companies think they're, you know, building super intelligence in a couple years, that the kind of line is going up into the right really quickly, it's exponential, etc, etc. And DC is asleep at the wheel, they have no idea what's going on, they think this is just chatbots, etc, etc.

47:19I would say a couple things have changed recently. One is the mythos kind of announcement slash series of decisions that the government made about like locking down these cyber models, that kind of raised this to being clearly a national security issue. And now it's kind of banks freaked out and talked with Secretary Besant about that. And so like there was a bunch of like, like freaking out about the cyber situation. And then more recently, you could call this current situation like mythos 2.0, and that it's okay, the systems, even when they're not widely deployed, They're breaking out and doing all these shenanigans.

47:53And so I would say there's starting to be more awareness among policymakers like, OK, maybe laissez-faire is like, let the companies figure it out is not the right approach. And like, maybe this wasn't all hype after all. And there need to be some basic guardrails. And I'll just give like, I'll give an example of like how much things have shifted in the past three months. So there's an effort right now to put forward bipartisan AI legislation in Congress. And so a couple months ago, people were expecting that the basic terms of this would be like basically California and like New York laws, but like at a federal level.

48:29So like transparency requirements, incident reporting, maybe, maybe with like a higher threshold or whatever, maybe something, maybe, maybe like a little bit, maybe like a voluntary like audit regime or something like that. But then later, when it was actually announced, after many of these events, there were audit requirements, there were emergency shutdown authorities that the government can do. And so they kind of shift just like one piece of legislation over its lifecycle. And then there was a later version that was announced that kind of was less trying to block the states. It's still kind of preempt some of what the states are doing, but it's like more narrowly scoped.

49:04And so I think just over the course of a few months, you've seen like, okay, basically just transparency to requiring third party auditing and like giving making sure the government has an off switch. And so I think that's kind of the vibe we're seeing whether that actually results in something passing Congress anytime soon is a separate question. But at least on paper, the gap is much narrower. Is bank regulation the sort of useful analogy for thinking about this? I mean, we require banks to disclose things. We require the government to look at bank balance sheets and figure out whether or not they're actually holding enough regulatory capital against their risk and things like that.

49:41We don't expect them to do it voluntarily, certainly not after 2008. Is that the right framing? Yeah, no, I think and I think this this kind of like shift from voluntary to required is the key step, because right now companies have to have like a champion within the company or there needs to be some kind of like reputational or they want to get feedback from the third party auditor. There needs to be some kind of reason for them to do it. And not all the companies actually choose to invite external feedback. They will share the bare minimum. And just for example, SpaceX yesterday put out a model card or system card about Brock 4.6.

50:18And there were like several sections missing from the table of contents. It seems like they got removed at the last minute. And so there's kind of... Sorry, you're going to say something. I was just going to ask, can you explain the whole model card thing to me? Yeah, yeah. Yeah. And so basically, the thing with model cards is that the original idea several years ago was that a model card was like a nutrition label, where it's like a bunch of information summarized succinctly, and you kind of slap it on the AI website. And it kind of succinctly explains like, what are the risks? How well does it work?

50:50What can it do? And so forth. Over time, as people such as myself in industry, we're like, okay, there's a lot to say, there's a lot to unpack here. And no one kind of established, no one was forcing anyone. to do this. So no one established like, this is the format, you need this little nutrition label. It was just people writing stuff. They ballooned into these like dozen page, 100 page, 200 page, 300 page documents of just describing like, here's all the crazy stuff we found here, all the tests we ran. And there's a spectrum. So like I would say anthropic puts out the longest ones. That's not necessarily totally correlated with like quality, but you know, it shows some some proof of work.

51:29And then others will put out five page, 10 page. And then you know, what happened yesterday is that SpaceX, for the first time, because of California law, there actually is a requirement to put these out, but there's not really a clear quality bar. And so they can say, Well, yes, we did that we followed the California law, we shared information about our testing and the extent to which third parties were involved in testing. And like, basically, it's just like one sentence saying, like, we worked with third parties, or whatever. And so, So yeah, and so I think this is different. I would say this is different from say like bank regulation in that, I mean, one is that only some things are required right now.

52:06It's like kind of putting out a document. There's no like the third party test that you're supposed to talk about whether you work with third parties, but that's different from actually doing it. And so I think what we need is kind of standardization around like how should the third party auditing work? What counts as a good system card? What are the minimum safety and security protections that you should be putting in place. And I think that's analogous to some of these like capitalization things you mentioned. And we need kind of standards for like, okay, what counts as a good auditor? What counts as, what are the standard tests you need to run and so forth?

52:37I mean, you yourself are sort of biased in this and that you're building out an auditing, a company or an entity, not a company because it's a nonprofit, but an entity that would do auditing. Also, you're promoting this idea that auditing should be important. And again, in the financial realm, there's a few different versions of it. There's sort of like bank supervisors, and some of them literally sit at the bank and they're there all the time. Then we have the Moody's and the S &P's of the world. So if you issue debt, you're compelled to get some sort of third-party rating. Why don't you describe in your ideal world, let's say this all happens and there's required auditing and the companies are cool with it, et cetera.

53:19What is the service that Avery, and I assume in the ideal world, there would be a few others, et cetera, as you can't go auditor shopping, et cetera. What is the service that the Averys of the world are doing? How embedded and what is the reason then to think for the general public, there's all of our hands, that this could lead to safer outcomes? Yeah, essentially the service that we and others would be providing in this world is similar to what we're currently doing, but kind of scaled up. So right now what we're doing is kind of voluntary pilot projects that are looking at a specific aspect of safety, security, governance, and so forth.

53:59What we would like to see eventually is that there's an ecosystem of auditors that are looking holistically at, is the company following its safety and security practices? are those safety and security practices reasonable and consistent with the standard floor, which ultimately we need, we don't have right now, and providing some kind of feedback to the company. And then there would be kind of like a remediation process for them to resolve issues that are surfaced during the auditing process. And then there would be a public version of this audit report that kind of shares after doing a lot of like technical testing, reviewing of documents, interviewing with staff and so forth, that kind of shares this update on some regular schedule, like quarterly or something like that, you might want it to be more like a kind of resident examiner kind of embedded auditor model rather than happening once a year, once every six months.

54:52And so but you kind of need to have some kind of like continuous trust building process where maybe the auditor is there all the time, but they occasionally issue these reports. And you know, what's in it, from the company's perspective, is they want to, you know, I mean, in this scenario, they would be required. But what's in it for them today is that they want to signal that they are ahead of the curve on safety and security. And they want to get feedback from these external experts who have like a kind of fresh perspective. And why does this matter? I think one is you just don't want to be in a world where you have to take the company's word for it.

55:22And you want them to kind of you want there to be common safety and security standards rather than it just being everyone's kind of making up their own things and then getting it checked. The other is that you want to avoid groupthink. And so I think even right now, there are a lot of, you know, a lot of what's happening with external testing is like, it's like a research project, like meter, you had someone from meter on recently, and they're doing this serious technical research on autonomy and loss of control and so forth. And they work with companies essentially in order to do these kind of research, very researchy assessments.

55:54And I think that's a key part of the process. But there's also just like verifying that the company did what they're saying they're doing. So there's producing evidence and then there's also checking evidence. And so it's kind of like in a K-1 statement, if I'm getting that right, you know, there's kind of this like short audit statement. We probably want something more than just a paragraph, but you basically want, you know, a third party saying we checked that they actually ran all these tests. We made sure that the model that was audited was the same one that's being deployed, et cetera, et cetera.

56:21What could we actually do to make the testing side safer? Because it seems to me like I I can totally believe that we can come up with a reasonable like auditing structure for models that are being deployed and allowed into the real world in some structured way. But if part of the problem is that we're developing newer and better and more intelligent models and then testing them and then they are figuring out ways to get out into the world before their product. Yeah, then that seems to be like a big vulnerability. I think basically what happened is that companies were getting cocky, getting overconfident in the quality of their sandboxes.

57:02And like maybe there was a disconnect between some of the people on the safety side who are measuring like, OK, this is where the hacking skills are going. And the people on the security side, you know, building the sandboxes and like something was getting lost in translation. Maybe it's groupthink. I don't know exactly. But it seemed like at multiple companies there was this kind of like overconfidence. And so I think these incidents come into light and all the kind of technical investigations are going to hopefully lead to more best practices, more people checking their own biases. But I don't think that's a long term solution.

57:33I think ultimately, people get overconfident all the time. That's a human thing. And that's why you want third parties checking to make sure that, okay, are you actually following these these best practices, you also probably are going to need some technical solutions to some of these things. Like, I mean, maybe some of this testing should be done on kind of air gap servers that are not connected to the Internet at all. And I think what happened in the hugging face thing is that it went through this like middle layer. There was like a piece of software that it routed out to the real Internet through this this kind of like intermediate thing.

58:04But like I think it might be that eventually we'll get to a point where AI systems are just so capable that they can hack their way out of anything. So you just need to make sure that they're in a cage, basically. Absolutely. Miles Brundage, we could talk for hours about this because there are so many fascinating dimensions of this. I will probably have you back in the future, unfortunately. No, fortunately, because that was a great conversation, but unfortunately, I probably won't be the last reason to have to talk to you. Thank you so much for coming on, Adlob. Thanks again. I appreciate it.

58:44Tracy, that was a fun, it's an unsettling thing, the way it behaves. So many of these AI conversations are still so surreal to me. Like the fact that this is what we're talking about in 2026, it just feels so strange. And it's only going to get orders of magnitude weirder because I thought things were weird in 2023 and things are much weirder today. Look, I get why people are very cynical about a lot of this stuff. And I get why people talk about like, oh, there's this regulatory capture. And I certainly believe in the premise of regulatory capture. And there may be some of that. But I will say one thing like the sort of like from the company's perspective is it is true that for a long time and for the very beginning, these are not companies make a lot of money.

59:30And yet they spend a lot on safety and security. And you could imagine tech companies historically didn't do that. They do. And they have these like it's certainly in the case of open AI and slightly to a lesser extent anthropic as a PBC. They have these weird corporate structures in part because they seem pretty they seem to believe that the things that they're building, if built wrong, should not necessarily just be in the hands of like purely profit seeking enterprises. Yeah, all very true. I do think one of the interesting things to me that stands out from that conversation is, again, the idea of like the asymmetry in power between the approved models that companies can actually use for defense against the new frontier models who are in testing mode and have somehow escaped the sandbox.

1:00:20I think there's a really scary dimension. And I think that actually you think about what is the difference between, say, sort of like auditing versus like a Moody's, et cetera. It seems like you need both, right? It seems like you need to have like the sort of like, yes, this specific model, it satisfies all the requirements that we've deemed it to be safe. But then this sort of like deeper auditing question of like, is this a company that generally experiments and does R &D and testing in what we perceive to be like a responsible manner? Yeah. Which is more like the supervisory aspect. It seems like you need like a testing auditor, like the bank supervisor who's like actually sitting on the floor, actually sitting in the labs and observing the testing process and making sure that the sandbox is well designed.

1:01:07But again, the problem with that is the classic cybersecurity problem or security in general problem, which is the model just has to find a single vulnerability. Right. You have to like fix all of them. Make sure that like thousands and thousands of vulnerabilities are impenetrable. And it does make sense, I think, that like, look, this is for profit capitalist competition. There's no doubt these are like some of the biggest most the pace of growth is extraordinary. then you lay we didn't even get into like how would you do this for like open source models or open source servers that's a whole other can of worms but it makes sense that if you're in the lab and you're trying to make money and you're also worried about like if you slow down etc then the other company is going to make more money etc that one way you solve this i don't know if it's prisoner's dilemma or whatever race to the bottom i guess game theory is okay you need this third party to like, you guys go as fast as you want on the R &D side, but we are going to set the rules of like, are you doing that in a safe way?

1:02:10Otherwise, why would Dario and Sam ever trust each other? It's like, no, we swear we're slow. We're taking it really seriously. We're slowing things down, you know, like, and then secretly they're racing ahead. That is really hard to solve for a series of purely private entities. All right. Shall we leave it there? Let's leave You can follow me at Tracy Allaway. And I'm Jill Weisenthal. You can follow me at The Stalwart. Follow our guest, Miles Brundage. He's at Miles underscore Brundage. Follow our producers, Carmen Rodriguez at Carmen Armand. Dashiell Bennett at Dashbot. Kale Brooks at Kale Brooks.

1:02:46And Kevin Lozano at Kevin Lloyd Lozano. And for more OddLoss content, go to Bloomberg.com slash OddLoss. We have a daily newsletter and all of our episodes. and you can chat about all of these topics 24-7 in our Discord, discord.gg. And if you enjoy Odd Lots, if you like it when we talk about moral relativism, then please leave us a positive review on your favorite podcast platform. And remember, if you are a Bloomberg subscriber, you can listen to all of our episodes absolutely ad-free. All you need to do is find the Bloomberg channel on Apple Podcasts and follow the instructions there. Thanks for listening.

1:03:28Thank you.

1:03:55the tech sector to new frontiers. Hi, I'm Ed Ludlow. Join me for Bloomberg Tech, a daily podcast focused exclusively on technology, innovation, and the future of business. Every weekday, we bring you the latest insights on Silicon Valley's top companies and conversations with tech's biggest decision makers. Listen to Bloomberg Tech on your commute home and stay ahead of the news cycle. Subscribe today on Apple, Spotify, or anywhere you listen.