OpenAI Slows Model Development After AI Hack, Anthropic Founders Grab Voting Power Ahead of IPO

0:13Welcome, everyone, to The Information's TI-TV. My name is Akash Pasricha. It is Wednesday, August 19th. Today on the show, OpenAI is pumping the brakes on developing one of its models. We'll talk about the rationale and the ramifications of that decision shortly. We also have a new story out about the limbo that AI companies have found themselves in as they await the government's guidance on model development. We're then going to bring on the reporter behind the scoop that Anthropix founders are taking moves to preserve their control over the company after it eventually goes public. And we'll close out the show with a conversation with the CEO of Whisper, which was just valued at$2 billion.

0:56It's going to be a great show. Let's get right on into it. OpenAI is pausing development on some of its models after it found that one of its upcoming models, Astra, reached a key cybersecurity threshold that the company has been monitoring for. I want to bring on our AI and robotics reporter, Rocket Drew, to walk us through this development. Rocket, welcome back to the show. It's great to have you here. Hey, Akash. Thanks for having me. Okay. So what's going on here? OpenAI says it's too good. It's scary. Yeah, that's right. OpenAI is taking some actions in response to some spooky cyber capabilities that they're seeing from its models, including the recent hugging face hack.

1:34And one of the actions that they're taking in response has been to pause training of some of its models. It paused a lot of training for a two-week period as it inspected and ensured that the training was safe enough to proceed. And then they have one massive reinforcement learning run that they say is still on pause as of yesterday. Did this decision surprise you? You know, I'm going to be honest, not really, because this is OpenAI. This is the company that sat on GPT-2 for like many months over safety concerns. Obviously, we're in a different kind of time period now than we were back when GPT-2 was released.

2:12But OpenAI has shown that it has some appetite to slow down development. There's been a lot of talk lately about pacing the frontier or potentially even pausing overall global AI progress to deal with some of these big safety issues that are coming up lately. So honestly, it feels a little on brand. And for OpenAI in particular, you know, they're facing a lot of heat still over this hugging face attack. to their credit they've been very transparent uh relatively transparent about how that attack happened but we're still awaiting more details so in the meantime it does really benefit them to say a little bit more about how they're responding to that incident internally right and i mean look let's talk about the the the sort of messaging of all this i mean we're in this moment where uh dario is the center of attention for better or for worse and you know anthropic has kind of taken the lead.

3:04And so they've become a little bit of the punching bag of the AI sector right now. So this, to me, from a PR perspective, I mean, it feels like OpenAI saying, oh, you know, we see an opportunity to maybe earn some brownie points with the sector and we'll take the good guy move here. I mean, that's how I'm reading it. Yeah, I see some of that as well. I mean, I think you can also read it as a more earnest move or something that's in good faith, but certainly they are facing public pressure right now over the cyber capabilities of their models. One thing I would add that maybe strengthens that case is like OpenAI was doing a lot of safety work before, right?

3:41This is not new. It's not like a whole cloth. They invented the idea that they're going to start doing safety work. So for example, one of the main steps that they're taking is monitoring their models. And in particular, they're now monitoring all uses of Astra, their unreleased model, internally. So when they do things, for example, like they showed Astra could solve some very hard math problems. Like when employees are using the model themselves, that usage will now be monitored. But OpenAI was doing a lot of monitoring before. In fact, OpenAI has championed a lot of the monitoring methods that are popular in the industry, namely reading the thoughts that the model has to try to catch if the model is doing something that it's not supposed to be doing.

4:24And they've been monitoring in the past, they've said over 99.9 % of all of their internal coding traffic in this way because they're concerned about issues like this. So they're scaling up that monitoring now, but they've been doing it before. It was costly before. It's sort of even more costly now that they're beefing up their monitoring. Well, and this is kind of interesting, you know, maybe if I just reflect a little bit on how I've thought about this issue. I mean, you know, Anthropic has always been perceived, I think, as the safe AI lab, or at least that's what they've championed a lot.

4:59And I guess from history that, I mean, you know this history better than I do, OpenAI's foundations seem to have been in safety the whole time. And so I'm trying to reflect on why it is that I have thought that OpenAI is perceived as less safe than Anthropic, just in the conversation. I'm not saying empirically speaking. I mean, maybe it was the, you know, the whole Department of Defense snafu and the fact that OpenAI was willing to do the deal with them. Maybe it was the, you know, the aggressiveness that they took towards a product. I welcome some, you know, thoughts. Why do you think that that perception existed for a while?

5:39Yeah, yeah. I think that's a really fair question. There's a lot of aspects to it. The DoD dispute is definitely an interesting incident to look to where Anthropic kind of like was seen as standing its ground and holding to its principles a little more, whereas OpenAI rushed into a deal that even cost it some employees. Like some employees were frustrated by that and left, including their head of robotics who wound up at Anthropic, which maybe is telling. Like, yeah, I think people both outside these two companies and certainly within OpenAI have been frustrated by this perception and the extent to which Anthropic has been able to run away with this reputation of being the safer lab when there are a lot of people within OpenAI that are focused on safety.

6:25One thing that contributes to this perception is that a lot of high up safety people at OpenAI have left over the years. Like their positions that are focused on safety and preparedness have been kind of defense against the dark arts positions where the people holding those jobs don't tend to last very long. I think you could respond and say, well, it's the AI world. No one lasts in any job very long. There's a lot of churn. A lot of people move in and out. OpenAI spins up new teams and retires teams all the time. Still, I think that has contributed to the perception. The other thing is that Anthropic really has set the bar with some of its policies and its safety disclosures.

7:04So Anthropic in some ways was first on the scene with having a safety and security policy, which now is an industry-wide norm and in fact has been codified in state-level laws in California, New York, and Illinois. But to your point, OpenAI also has their own version of these safety policies. They have a preparedness framework. And the preparedness framework is part of why they are now taking these steps that they've outlined, like pausing their training, because they've pre-committed that these are sometimes called if-then policies. If we see something spooky, then here's what we're going to do about it.

7:39And in this case, the if has been triggered. They're seeing cyber capabilities that are maybe strong enough to activate the mitigations that are required by their policies. So in particular, they say they can no longer rule out a critical cyber threshold according to their policy. And that means something very specific. It means that either one of their models can identify and exploit zero-day vulnerabilities in real-world critical software systems autonomously on their own. Or the model can do end-to-end novel cyber attacks against, again, hardened software targets given only a high-level instruction.

8:22So they're saying both of those are now within sight in addition to the hugging face hack, which actually happened. And both of those are contributing to the decision that they announced yesterday. Got it. So what's been the reaction broadly? People are supporting this move, I imagine? I think people are supporting it. I think it's mixed. I think for some people, it's a little bit of a wake-up call because they are recognizing how seriously OpenAI is taking this and the costly steps that OpenAI is willing to take to address it. On the other hand, I think you're also seeing the usual skepticism that there's probably some marketing angle here, that it's probably more talk than anything.

9:07I think the strongest evidence that it's more than just talk is that they're spending a lot of compute on this. So we don't know exactly how much, but it is kind of costly to beef up monitoring in the way that they're doing now. They say that about 20 % of however much compute is used to run the model will be spent on monitoring now, which is not a trivial amount. Right. Well, you know, we're going to talk to Leo, our AI and politics reporter, shortly about how this all connects to the regulations behind AI and safety and what the White House has in store. That, of course, is a bit of the background here to this decision that I'm sure had an impact.

9:48Rocket, I want to thank you for coming on and sharing with us your thoughts and explanations. That is Rocket Drew, our AI and robotics reporter here at The Information. Okay, despite OpenAI's move to pause development on Astra, my colleague Leo Schwartz has a new story out this morning about the big questions that many AI companies, including OpenAI, still have about the White House and how it is ultimately going to regulate the technology. I want to bring on Leo to walk us through his reporting. Leo, welcome back to the show. It's great to have you here. Thanks for having me. So just to remind folks of the timeline here, Back in June, the White House comes out with this executive order saying, hey, we're going to take some time.

10:32We're going to put some thought into a framework. It's going to be a voluntary framework. But AI companies, we hope, are going to follow it. And, you know, this is going to be our stab at regulation. It is now August 19th. The deadline was August 1st that it's set. Where are we now in this story? I mean, the hope with this framework is that as AI labs like OpenAionthropic create increasingly advanced frontier models, there would be some sort of testing program by the government, obviously on a voluntary basis to say, this is safe. This isn't going to crash the economy if it's out in the wild. And here's how the testing works.

11:12The deadline to do that was by August 1st. A few days later, the White House did host companies for a briefing that we reported on. And we're now about two weeks after that. And that framework has not been released to the public. It's also not been the actual written framework, even though it was handed out to companies during the briefing, as I report this morning, it hasn't been shared with those companies after the fact. And it hasn't been shared more widely with the rest of the AI industry. And a lot of critics look at that and say, this is a black box if you're not going to actually share details on what this testing program looks like, how the government is going to consider what a frontier model is, if it's going to also include open models.

11:53And at one point it will. There's tons of unanswered questions. And we're now a couple of weeks out from that deadline and people want answers. So, but there is a draft. There was a physical copy that was handed out at this briefing. And one of the details that I love in your story is this notion that they were not allowed to take it with them or take photos. they could just take notes on what was in the paper copy. Is that right? Yeah, so my understanding was there was, it was communicated that at a later date, this would be distributed to companies. Right, right. I just love the image of, you know, going to the government offices, you know, getting handed a document and then saying, you can't do anything with it, but here's what we're thinking.

12:43Let's talk about what's in the draft. I mean, this paper card, do we have any sense of, you know, which way the government is leaning towards things? Is it more strict than not? What do we know? Yeah, I mean, I think this is even more frustrating that there is a written version of it. The framework exists, but companies just aren't. I mean, they were able to see it. A few of them were able to see it then. But the industry obviously is a lot bigger than those companies that were at the briefing. In terms of the details of what's in it. I mean, the biggest thing to keep in mind is this is ostensibly a voluntary framework.

13:18So it can't really be strict or not strict because it's been called voluntary in the executive order itself. The White House takes pains to say over and over again, this is voluntary. It should not be construed to be mandatory. Of course, some companies may have a different interpretation of that, like Anthropic, which got slapped with export controls over its latest model a few weeks ago. I guess it was over a month ago at this. Yeah. But yeah, what the framework does is creates a classified benchmarking process basically for deciding when a model will be frontier or not. As I report in the story, there is explicit language around open models and ensuring that the framework should not be construed to be restricting the development or use of open models.

14:02Although, as I also report, it was verbally communicated that for now, the framework will only include closed models. It could include other models in the future. That still remains a confusing question for a lot of companies. There's been a lot of conflicting reports also from other media outlets after that, which I think stems from this confusion of nobody actually having the exact language shared to them by the White House after that briefing. And one of the interesting things you pointed out, again, is that the benchmarking process, from what you're hearing, it will remain classified. And that is kind of interesting to me because evaluating these models is not a black and white.

14:45It's a very subjective pursuit, as we've talked about on this show. And so the fact that that is remaining classified, I mean, that's even when this thing comes out, there's going to be a lot of unknowns, I guess, is what I'm taking away. That's definitely true. I mean, this was actually in the executive order itself from early June that the benchmarking process would be classified in part because it's going to be architected by the NSA. I think there are reasons for it to be classified. But at the same time, as I've reported before, one of the big questions for a lot of labs is how does it actually work?

15:21You know, it's a benchmarking process, but what is it benchmarking? What exact criteria is it looking at? And then, of course, the other big question for a lab that's not yet at the frontier level, but might be in the future, if they do reach that frontier, they won't even know if they have if the benchmarking process is classified. So there has to be some way for it to be communicated, okay, now we want you to come in and participate in this testing program. Right, right. And it's kind of like, it's like, you know, it's not like you get a notice in the mail because you had to voluntarily tell them, hey, here's what I'm working on.

15:56You know, what do you think about it? Let's talk about how this is affecting the companies themselves. So we talked about the big three labs that you reported were participating in these briefing meetings, OpenAI, Anthropic, and Google, if I'm not mistaken. As you said, I mean, there's tons of companies in this ecosystem who this will have an impact on. What is the state of limbo that they find themselves in? How are they making decisions? What are you hearing on that front? so there's there's no indication that the lack of details or the framework itself has slowed the deployment or training of new models what's interesting is that open ai which just released this blog yesterday i know rocket was talking about it has taken that decision by itself to actually pace the training of its new models this is theoretically something that could be obviously not ordered because it's voluntary or recommended by the white house as we reported a while back the White House did ask OpenAI in the past to stagger the release of its new models.

17:02But we know that Sam Altman was in town at the end of July before the deadline for the framework showing off this new model Astra. It sort of remains unclear if the framework can be considered live or not. I mean, certainly companies like OpenAI Anthropic are sharing their models with the government right now. It's going through some form of testing process. Whether that actually is the framework or not is a bit uncertain. But what is known is that for a company like OpenAI or Anthropic, they're taking it upon themselves to slow the release of their models for fears of cybersecurity risks or incidents like the hugging face hack.

17:42So, Leo, when this framework does come out, what will you be watching for? What are the questions you're hoping for answers for here? Well, we don't know if it's going to come out. That's still a question. The statement we got from the White House is that they're continuing to collaborate with companies on the implementation and the testing process. But if it does get released or this other detail I reported in the story, if this long-awaited event happens at the White House, a public event around the framework, certainly the biggest question is who does the framework actually apply to beyond OpenAI, Anthropic, Google, maybe some other labs that are on the cusp of Frontier?

18:23Will this at some point encompass all models, only the most advanced models? When will it be expanded to open models? These are all questions that remain unanswered. Well, sorry, and maybe this is a silly question, but how can they not release this voluntary framework now? I mean, if they just decide to not do that, what would be the rationale, like not picking sides here? Walk me through the calculus here. I wish I had a satisfying answer to that. Okay. Well, look, we got two weeks left in summer, so maybe September rolled around. Maybe they'll pick up the pace again. Leo, I want to thank you for coming on, as always.

19:05That is Leo Schwartz, our tech and politics reporter here at The Information. As Anthropic marches towards its IPO, the Information senior reporter Corey Weinberg and our deals reporter, Valida Pau, have some new exclusive reporting on the control that the company's founders would like to retain over the business. I want to bring on Corey to share more about what we know. Corey, welcome back to the show. It's great to have you here. Great to be here. Nice shirt. I like that shirt. Thank you. I feel like I'm trying to bring a little bit more of a colorful vibe to the show. No, it's good. Typically, the guidance is no pattern shirts but i'm gonna i'm gonna say that i like that shirt that message got lost in my slack uh i'm sorry to say well corey you've been doing this show for a year i mean and i love it every time i'm here for you and okay so um anthropic is gonna go public and you have some reporting on how much control the founders want what did you find out yeah um essentially what uh i find really interesting about this story is Anthropics co-founders are looking to cement their hard power over the company rather than just the current situation where they have a great deal of soft power.

20:29They are top of the business world. They have created a product that everyone's using. And they are essentially using this IPO or at least expecting to, planning to, things could change, but they are using this IPO as a chance to cement more hard power. And in the corporate world, that means voting power. That means ultimate sway over corporate decision making. Even if they have a relatively small economic ownership in the company, they'll have a very large sort of voting control of the company, at least how it stands today. And how does that compare to the actual ownership that the founders have in the business.

21:12Yeah. So we reported in our story that, for instance, Dario Amade only owns roughly 2 % of Anthropic. There could be some wiggle room there depending on how you calculate ownership before or after the IPO. But the upshot is that's a tiny, tiny percentage ownership for a founder, CEO. It's the lowest or among the lowest that I could find. What are these stakes typically? I mean, all these companies. I mean, they're totally dependent. The one variable that it hinges on is money raised. When you raise money, you are diluting yourself. You are adding more shares to the pool. And so money raised is a big variable.

22:03And Anthropic has raised a ton of money. You know, we're talking about some of the largest sums in the history of venture capital because the business is so capital consumptive. And so Dario owns a very small amount. However, the flip side here is that he has said publicly that he awarded his six other co-founders. Remember, Anthropic has seven co-founders, which is an anomaly also in the tech world. they all own a roughly equal share of the company, or at least were awarded roughly equal share packages. That's also very unusual. Usually the CEO has the biggest economic ownership just in terms of having the largest compensation package.

22:56And so you have a bunch of unusual variables here, but even if you add all that up, And let's say that all seven co-founders each have roughly 2%. That's still a pretty small economic ownership for a founder group in the grand scheme of things. And so they're going to need extra voting power for each of their shares if they want to even come close to being able to, as a block, control a lot of major corporate decisions over public shareholders. And so tactically, I guess this is where voting shares and different share classes come into play here. But tactically speaking, what do we know and what questions still remain about how the founders tactically are going to do this with these voting shares?

23:42A lot of specifics are still, from our understanding, TBD. I mean, all of this will be laid out in the S1. What we're trying to get ahead of is, hey, here's where things are headed. Here's where the voting structure is roughly likely to be from an outline perspective. And it's news because it's the first time that Anthropics founders would have had this. But we don't know how many votes per share each person will have. Is it 10 to 1, 20 to 1, 100 to 1? We've seen all sorts of schemes that tech companies have used over the years. We don't know exactly if it will all be the same voting structure for all seven co-founders.

24:22Will all seven get this or will just a few of them? So tactically, we don't know that. And then crucially, we don't yet know how it plays into the other powerful body within Anthropic, which is this. LGBT. LGBT. Yeah, the group called LTBT. I hate that acronym. Sounds like a boy band to me. I try very much avoid using it. I think it's a good name for fun. I mean, in a world where naming sucks. LTBT. I'm excited for LTBT Pride Month. But I think it's the long-term benefit trust is what it is. And essentially what it does is it is this non-shareholder group. It includes Ben Bernanke, the former Fed chair.

25:19Now, this is a group of three to five people that don't have any ownership in Anthropic, but have the power to elect the majority of the board of directors, which is a very powerful job. And they meet roughly every week. They go to most board meetings. They weigh in on anthropic safety considerations. And the through line through all of this is anthropic, it's trust, it's board of directors. They're all supposed to operate within the bounds of doing what's best for their mission, which is essentially creating AI that benefits humanity. And the reason we're writing about all of this and this government stuff is it does play into them as a public company.

26:10They have to navigate what is this going to mean for the world and for society, not just what it's going to mean for the next quarter and public shareholders, even though they're about to ask them for a ton of money at a high valuation when they're going public. Right. And, Corey, are there any analogs that you came across in your research at all for companies that maybe not have a long-term benefit trust, but any benefit corporations? there are there are certainly some um but they all are tiny uh so different than anthropic and so small compared to anthropic so so anthropic is two things it is a public benefit corporation um that is uh you know sort of a uh designation that forces it to serve its mission uh over shareholders and that is pretty rare among highly valued public companies.

27:20I feel like that would be a great trivia question for our audience. What is currently the most valuable U.S. publicly traded public benefit corporation? Pause if you want to take a stab at it. I'll tell you, it's Viva Systems, which is a biotech software company. They are valued at only 40 billion dollars. Anthropic is going to be valued at over a trillion. So that's incredibly unusual. And then the actual trust that Anthropic has, there's very few parallels. I mean, you could consider OpenAI with a nonprofit board somewhat of a parallel. So that'll be interesting to see how that works when it goes public.

27:58And then, I mean, Ben and Jerry's is a famous one that went awry, actually. And we can dig into that maybe in a future episode. But there are some parallels, but nothing nearly as big as Anthropic. Right. And last question, Corey, I mean, I just want to make clear for folks here. So the long-term benefit trust, their role, I think based on what I recall, based on what you said, they appoint the board of directors, right? They have to ultimately sign up on who the company board is, right? And that company board would then be the sort of the shareholder representative in a classic IPO public company sense, right?

28:42They elect, they have the power to elect the majority of the board seats. Anthropic is a seven-person board. The trust can elect four of them. And that is their main bit of hard power right now. We don't know how that could change in an IPO and how that will, but it's going to, The company is planning to keep the trust in place and largely its powers, is what we're reporting. But I don't know yet how it will play into the founders' voting power and how all that will balance or how it will change the balance. Great. Well, Corey, there is a lot of unanswered questions, but it is certainly novel to know that these are the steps that Anthropics founders are taking behind the scenes.

29:36And so I want to thank you for coming on and sharing that with us. That is Corey Weinberg, our senior reporter covering Anthropics, here at The Information. Voice dictation has skyrocketed in the AI era as voice agents have become the new great way to get work done. My colleague Stephanie Palazzolo wrote extensively about that trend in an AI Agenda column a few months ago. Whisper is a fast-growing startup at the center of that trend, and the company a few days ago announced it has raised$280 million at a$2 billion valuation. I want to bring on co-founder and CEO Taneh Kotari to share more about his company.