#157 Karen Hao: Inside OpenAI's Tumultuous Saga

0:00What does this actually achieve? I guess like the narrative that OpenAI tried to say was by doing this, we will be able to continue developing AI for the betterment of humanity. That's why it's called OpenAI. And I think that's the fiction. Microsoft has invested tens of billions. I mean, on paper, like what's been announced is they've invested over 12 billion in OpenAI. Microsoft's fate is tied to this company significantly. I suspect that this is not the end of the drama. I don't actually think that this resolution is going to be like, okay, great. Like everything is back to normal. This episode is sponsored by ISS, a leading global provider of video intelligence and data awareness solutions.

0:48Founded in 1996 and headquartered in Woodbridge, New Jersey, ISS offers a robust portfolio of AI-powered, high-trust video analytics for streamlining security, safety, and business operations within a wide range of vertical markets. So what do you want to know about your environment? To learn more about ISS's video intelligence solutions, visit issvs.com. That's issvs.com. They support us, so let's support them. Hi, I'm Craig Smith, and this is Eye on AI. In this episode, I talked to Karen Howe, a fellow journalist who is now a contributing writer at The Atlantic. She was previously writing for The Wall Street Journal in Hong Kong and before that for MIT Technology Review.

1:50She's the journalist with probably the best insight into OpenAI, and we talked about the events of the past week. Karen knows the players, knows OpenAI's history, and has some unique insights into what happened and what we're likely to see in the future. Beyond OpenAI, we talked about the likelihood of artificial general intelligence happening in the near future, as well as the existential risk that many AI researchers are concerned about. I hope you find the conversation as fascinating as I did. My name is Karen Howe. I am currently a contributing writer to The Atlantic and also writing a book about the AI industry through the lens of open AI's rise and impacts around the world.

2:41I became a tech journalist very much by accident. I had studied engineering in college and then I worked in Silicon Valley at a startup. And in the first year of working there, I sort of saw very rapidly kind of like the ills of Silicon Valley, if you will. The startup I was working for, the CEO was fired. So actually very relevant to this weekend's events. events. The CEO was fired by the board and I became very disenchanted with sort of the progression of events that followed around both leading up to and after the firing of the CEO. So I was looking for other opportunities and I wasn't really convinced that I would be able to find something different within the Valley.

3:34So I kind of, I had this vague inkling at the time that I would enjoy journalism, in part because I'd always enjoyed writing. And I was particularly interested at the time in climate change and how do you facilitate, how do you incentivize mass groups of people to change their minds and change their behaviors? And originally, I thought tech was the means to do that. Then I started thinking maybe actually journalism is the means to do that. Like you need to really build public opinion and public support for the science around something before people are going to act on it. So I started working as an environmental reporter for my first job.

4:17But when I was looking for, it was an internship. And when I was looking for full-time opportunities, it was very difficult for me to get hired as an environmental reporter because I didn't really have that kind of background. But consistently, I was asked, would you be on our tech desk instead because of the background that I had? And so I ended up becoming a tech reporter. And then I became an AI reporter also, not because I chose it, but because it was a job that was available. And then it became sort of the perfect match for me because I was really, really fascinated by AI technologies. And I actually had a lot of friends from college that had gone into AI research sort of on the other side.

5:08So I was like, able to kind of quickly embed myself in the community. And I realized that it was this microcosm of exploring all of the narratives that we have about technology, the promise of it, the power, the potential, the societal impact, the sort of moneyed interests that are involved, the egos that are involved. And so I ended up reporting on that for now more than five years. Yeah. And you were at MIT Tech Review, correct? I was at MIT Technology Review, and then I went to the Wall Street Journal. And then I joined Contributing Writer at The Atlantic. And while I was at MIT Tech Review, I guess this gets to your other question of when did I embed in OpenAI?

6:00While I was at MIT Tech Review, our focus was really on trying to cover like the bleeding edge of AI research. um so whereas you know the journal takes a very different stance it's like we the journal covered technologies that are starting to be commercialized that are starting to have um some kind of business potential MIT Tech Review was like if it had business potential it was already too late so it was always about how do we try to call the trends before they happen And because of that, we started covering OpenAI very quickly after they were founded. They were founded at the end of 2015. And we probably started covering their research in 2017 because that's when they started producing some stuff that was starting to push the boundaries.

6:55and then in 2019 I I can't really remember like how this came up but I basically had a discussion with my editor at the time where I was like I think opening eyes sort of just a really interesting lab and there hasn't really been that much coverage of it like we've covered its research but we haven't really covered its people and they were starting to become just prominent enough within the tech world that it felt like it was a worthwhile thing to do. And my editor at the time said, you should just profile them. And so I reached out to the company. They already knew me pretty well because I'd been covering their research.

7:35And I said, hey, you've never had a profile done before. I think I would be the best person to do it. I think MIT Tech Review would be the best publication. And let me come to the office for three days and sit in on some meetings, chat with researchers, chat with executives. So that's what I did. I ended up flying to San Francisco from Boston. And then I spent the three days there. And this was at the end of 2019. So this was a really, really interesting period of time in the company's history because I went there a month after Microsoft invested a billion dollars into the research lab. So in that year of 2019, the GPT-2 announcement happened, which GPT-2 was, as people may, listeners may remember, it was a few generations before ChatGPT.

8:29And initially, OpenAI took the stance of not releasing the model, but announcing to the world that they had developed it. Right. So that happened at the start of the year. And then it was a really big controversial decision because people thought, like, why would you announce it, but then not release it? That's really odd. And then the capped profit arm was created within this nonprofit entity. Sam Altman joined a CEO, and then the billion dollar investment happened. So it was a rapid succession of changes that made clear that the company, what was a nonprofit, was quickly evolving into sort of a company, and that it was sort of positioning itself to become bigger and bigger and more influential.

9:14Yeah. Yeah. And that GPT-2 debacle is how I regarded it, was sort of the first glimpse of what was to come because they said they developed it, but it was too dangerous to release. And it was, so everyone, of course, that got everybody excited. Like, what the hell is this thing? And then there were limited, you know, people were invited to review it. The creation of, I mean, in your article, in the Atlantic article, and it's really what I wanted to talk to you about. you have a line and I think it's what a lot of people are thinking and concerned about that and I'm going to try and find it here if I can find where I opened it.

10:15I don't have it open but you have a line about how you know this most important and powerful technology mankind has ever developed is controlled by half a dozen people who are fighting among themselves. And that's kind of frightening. And the whole sort of fiction of the nonprofit with a profit, even if it's a cap profit subsidiary. I wanted to ask if, is that a fiction as well? I mean, it's all the same people. It's like you and I have a nonprofit. Oh, and you and I also have a profit arm. You know, it's not like, you know, we put on one hat and we're a nonprofit, we take it off and we're a profitable, a profit-seeking company.

11:12So is there any real, is that just a fig leaf from your point of view? And what do you think about this kind of technology being in the hands of such a few people and people who evidently can't necessarily agree? so i think the non-profit for-profit arm i mean it's interesting the people that designed that structure were sam altman greg brockman and ilia setzkever who ended up becoming the main characters of the weekend um i can't personally speak for what sam believed when he designed this because I never spoke to him about it. But I spoke with Greg extensively about it during the time that I was embedded within the company.

12:06And he genuinely thought that this was the solution to kind of raise, they were trying to solve a problem. They realized that AI development, the type of AI development that they wanted to pursue would be very, very expensive. And they needed to raise more money than a nonprofit could help them raise. And they tried to, I remember Greg said this to me during that time that I was in bed. He was like, we did actually try because like this notion of like having this nonprofit was very, very like near and dear to us. So we didn't want to just immediately go, let's scrap it and move for a for-profit.

12:49So this, like at least for Brockman, he genuinely thought that this was like a really clever solution that they'd come up with to solve this problem of needing the money, but also staying a nonprofit. But the thing is that it's like, what is, what does this actually achieve? I guess like the, maybe the fiction is that this, the narrative that opening, I tried to say was by doing this, we will be able to continue developing AI for the betterment of humanity and with the participation of humanity. Like this was a really big part of their early days messaging as well was like, they were going to be open.

13:33They were going to be transparent. That's why it's called open AI. And the, I think that's the fiction, like the nonprofit for profit solving the specific problem that they wanted to. I think that is, you know, like they, they were genuinely trying to solve this like very particular problem, but does it actually get us more open, more transparent, participatory AI development? No, not at all. Like what it actually does is just entrenches the power of the people that designed this thing. And ironically, I mean, what we saw this weekend was that the nonprofit for-profit did end up working as designed in that the board did in fact do what their job was to vote out uh sam for like not aligning with the mission supposedly but then when the like the reactions that we see from sam from greg um and ultimately from ilia when when ilia flipped suggests that they're not actually here for this mechanism to be used against them, right?

14:47But if the mechanism were designed with true sincerity of maybe one day, I'd actually ask Greg, I'd ask, would you ever consider firing yourself if you felt that you were no longer up for the job? Which I could have even phrased it as, would you be open to the board firing you or the board firing the CEO? if they evaluate. And at the time he said, like, I would be open to it, but clearly like they weren't actually open to it. Right. So that's the fiction that I think kind of was, became very plainly displayed. Yeah. And, and the board is the board of the nonprofit. Is that right? The board? Yes.

15:32The board is part of the nonprofit and the nonprofit governs the capped profit. And that's why the board was able to exercise the power that had been bestowed upon them with this legal structure to vote out Sam. Yeah, it'll be interesting to see if that nonprofit profit structure survives because it doesn't seem to make a lot of sense. I mean, going back to GPT-2, that was the thing that upset a lot of people. Open AI was supposed to be kind of an answer to big tech, to Google specifically, and that they are going to be open source. They're going to share all their research. It's not going to be controlled by a for-profit entity.

16:26and maybe that was just naive that when anyone develops anything that has such profit potential, whether or not the logic is that they need to raise funds and investors need to have some profit participation, otherwise they won't invest. I mean, ultimately, this kind of technology is not going to survive under a nonprofit umbrella, I think. But more specifically, you know, I talk, and I know you do too, to Yen LeCun, who I have enormous respect for. I mean, obviously, he's a genius. but I mean in terms of his opinions on things like open source versus proprietary research. And do you think that this kind of tech should be open source, regardless of the dangers of open sourcing incredibly powerful technology, But simply to avoid this sort of thing, then you have the broader research community working on it, refining it, and then some sort of a license structure that allows people to use it, whether it's for research or commercial use.

18:06Yeah, I think it's a great question. To be honest, I haven't fully made up my mind about whether to fully open source technologies like these, but certainly we need more transparency than we need now. I think that is very, very clear. And also what's interesting, I will say that Meta, I mean, Jan has been a big advocate of open source or of transparency, but Meta's Llama 2 model does not actually technically fit the definition of open source. They open sourced the model weights, but by definition, they would also need to open source the data in order for people to audit it, to understand how it works.

18:45And Meta has refused to release any information about the data that it was trained on. And this is something that I think could easily become like a very low stakes accountability measure is releasing the data. Just saying like what's in the data already is a huge step forward and you haven't trained the model like that. The data is not the model. So if you're worried, if we were to buy into the idea that open sourcing the model could have dangerous potential, Open sourcing the data would not, you know. But the fact that we don't have any understanding whatsoever of what is being used to train these models, I think, is a very telling sign of why actually these companies that say that they can't open source the technologies, what is the true motivation behind their arguments?

19:42And do you think that's because they're afraid of liability? Absolutely. issue. Yeah. Absolutely. I think they're afraid of liability of reputation damage because a lot of the content that is put into these systems is not actually vetted very well. And that's precisely why open sourcing would create safer systems. Because if you force companies to open source, they would have to significantly do more work to clean up the data sets, which would actually result in better products. And if you have many more scientists within the community, many more other people within the community going through like more eyes on these data sets, they will naturally just become better.

20:33And I think it would also be a forcing function to then get to a place where, like some of the companies are now doing this, where they're striking data deals, where they actually purchase the data from a media company or from Shutterstock or whatever it is. That came very late in the stage of the AI development that we're in. like all of the original models did were not developed with these data deals. Right. And now like the companies kind of can continue to profit off of the data that wasn't paid for and that wasn't they weren't being transparent about. But if we had more of this transparency, it would be a forcing function to accelerate this trend, which I think is a really good one.

21:17Like there should be payments for data and potentially even dividend payments to the data providers, data creators over time. Yeah, yeah. That's an interesting idea that, you know, I know you know the same people I know, that Don Song at Berkeley has been working on this idea of, you know, using the blockchain to secure your data and then be able to sell it and have kind of this lifelong income stream coming from it, which sounds great to me now that I'm close to retirement. It'd be nice to have an income stream off the data that every company has ever used of mine. Yeah, so where do you think, I mean, Sam and Greg are back at OpenAI.

22:17uh ilia is uh i feel bad for him you know i've interviewed him he's such a a deep soul you know uh he clearly didn't mean to cause this this uh global ruckus uh helen toner i feel bad for her she's been raked over the coals people have made fun of her research um where do you think this is going to go i i i can't imagine that i think there they're what three people on the on the new board uh that again with this technology as as critical as it is i would assume uh at the very least microsoft will have a seat on the board uh where do you think it's it's going to go in in open ai's case and then we can talk about uh sort of government regulation i would think that uh that uh regulators would be looking at this and saying you know we can't have a bunch of you know 30 somethings uh in silicon valley like yeah yielding the future of the world so yeah Yeah.

23:33I think for OpenAI's case, I suspect that this is not the end of the drama. I don't actually think that this resolution is going to be like, okay, great. Like everything is back to normal. Sam's installed. All's happy and peaceful. and all, you know, like the piece that I wrote in The Atlantic talks about how there's all these different factions with the company, different ideologies, they all are kind of in this power struggle. And I really do think that the more powerful a technology is, the more you end up with a Game of Thrones style power struggle, because people think very, believe very strongly in their ideology for how AI should be developed.

24:16And it's both like this belief that's like a true, genuine belief and also of course like there is elements of desire for power desire for control and we've seen opening i go through different waves of drama before so this is just the third way if you i guess you could call it like the elon musk leaving opening i was the first wave and then the anthropic opening i split was the second wave and now this is the third wave there's definitely something else that's gonna come um but in terms of like what this means i guess for the course of AI development, I suspect that, I mean, Sam is certainly going to be a lot wiser about selecting, carefully choosing his board members and trying to make sure that he entrenches his power again.

25:01So if that is the case, then his specific ethos and his sort of habit around rapid commercialization, rapid growth is going to now be like the main driving seat of the organization. And that is going to continue. We're going to see like way more proliferation of products, way more downstream companies like building on top of it. And unfortunately, I think we will see way more ripple effects, negative ripple effects as well as like speed overtakes, you know, certain types of like trust and safety concerns, for example. um and so i i think that uh that's probably the most likely scenario but also it's really hard to know it's really hard to know because um i don't know if you saw there was like that that that letter that was circulating open letter to the board from former employees um so it's like is that going to be yet another episode in this particular weekend saga um or Or is it buried for now and we don't see something else for another one to two years?

26:22Yeah. Yeah. And then you've got Microsoft and Satya is publicly all smiles and everything's fine. But you can imagine. I mean, they must have been sweating like crazy. Yeah. Microsoft has invested tens of billions. I mean, on paper, like what's been announced is they've invested like something like over 12 billion in OpenAI. But it's way more than that in the sense that when you look at their investor, like their latest investor statement, they say that they're planning on laying down more than 50 billion dollars in new data centers for next year. And not all of that is for OpenAI, but they're laying that down because they're selling to Azure, their cloud compute customers, this idea of the Microsoft OpenAI partnership.

27:20And this is why Microsoft stock has been doing so well is it banks on this partnership. so like you know when the when the news came and like microsoft stock immediately started dropping and now that like things are back to normal microsoft stock is increasing microsoft's fate is tied to this company significantly significantly um so yeah it did

27:51yeah although a brilliant move because at one point it looked and this is well all you know all over Twitter and commentators who say, but that he in effect had acquired OpenAI or was on the cusp of acquiring OpenAI without having any regulatory interference or even having to pay a premium, actually paying a discount. So yeah, just on the data centers, this is something I've been talking to people about. this technology is is so much promise for enterprise but because of the constraint in in available compute which which goes all the way back to you know silicone starts at the foundry but then through to NVIDIA and their limited supply.

28:54You can't actually build and deploy a heavy use enterprise application using GPT-4 through an API. Just the pipe through which you're sending your tokens is too narrow. Is this investment by Microsoft intended to ease that? What do you think about that constraint? How long that'll last? I definitely think that, yeah. I do think that Microsoft's investments are meant to try and facilitate kind of all of their customer base to transition to an AI-forward business, I suppose. because, I mean, every company that I've been talking to these days, regardless of what industry they're in, is suddenly alert to the idea that they need some kind of AI strategy.

29:52And all of the tech giants, Microsoft, Google, AWS, Amazon, are all trying to capture that new market. And they're trying to build out their infrastructure to also facilitate that integration with these business customers. And I mean, I personally, I think there's sort of two interesting things that I'm personally watching for. One is how much of this talk is going to actually convert into implementation, because a lot of the companies I talk to, they say they need the AI strategy, but they're actually not sure what that means and whether or not it would ultimately be valuable for their business.

30:37You know, it is valuable right now for their business to talk about it, but will it actually be valuable later to implement it? So that's one thing to look out for. And then I guess the second thing is whether or not these companies, these cloud providers that are kind of jockeying for market share are even able to acquire the resources necessary to continue laying down the data centers to keep up with this kind of demand. I think those are two things that could potentially end up limiting AI adoption or bottlenecking AI adoption, but it's sort of difficult to tell right now what that will actually look like in five years' time, maybe.

31:28Yeah. Have you heard anything that you haven't published or that you have published about this idea of Sam Altman starting a chip company or OpenAI starting a chip company to compete with NVIDIA? Only what's reported. I mean, yeah, my understanding is that this was this is not a new idea for him, that it was something that he had always been kind of interested in, but had never maybe maybe not taken seriously before. I'm not sure. But that, you know, then it became much more real and viable potentially and potentially a smart business decision to be able to actually have that. But the thing is, I don't know that people fully understand sometimes that it doesn't matter how many chip companies we have.

32:25We only have one real chip manufacturer, which is ESMC. And you're not going to get, I mean, Samsung as well, and a couple other companies that are able to sort of produce these chips. but TSMC is sort of like the most consistent provider and everyone wants to use them. And no matter how many chip companies you have, there's still that bottleneck. So I'm not really sure what Sam's plan was with that. Like whether he was trying to create his own chip company to get around the waiting list for NVIDIA or whether he was doing it for something else, like maybe for optimizing, like trying to get to the next stage of AI development, maybe, by trying to optimize how model training works by going to the hardware level.

33:18I'm not really sure. Yeah, yeah. And a lot of people don't understand that when people talk about chip companies, they're not actually manufacturing the silicone chips. They're designing chips. um the uh you know i love the the the bit in the article about uh ilia chanting feel the agi uh i'm a big agi skeptic uh i agree with yen lacune again i mean i i a lot of his ideas really resonate uh with me uh that uh that language models are not the way to agi i mean they'll certainly advance to something um do you do you do you know as a journalist a very well-informed journalist how what's your feeling about that do you think uh And I get comments all the time on various things that I post that, you know, AGI, we're going to reach super intelligent sometime next year.

34:31It's like, really? So yeah, what's your sense of that? My feeling is that we don't have any agreed definition of AGI. So AGI could be here if you define it based on what we have. Or it could be like 100 years away if you define it totally differently. So for the people who are saying super intelligence might be here soon, I mean, we don't like even scientifically, we don't have a agreed upon definition of intelligence. but I'm not I'm not talking about AI I'm talking about like from biology and psychology and neuroscience there's no agreed upon definition of intelligence so yeah I mean like I'm sure that these people that are saying these things uh totally agree with them you know it's sort of it's sort of just like you get to define yourself what the goal is and where to go and I think this is the fundamental problem of of AI the AI industry as a whole and um as illustrated this weekend of of open AI is that by setting a goal towards something that is completely undefined, you kind of just get to do whatever you want.

35:40And say that it's under the banner of a thing that sounds really nice and magical even. And, and, and like de facto good, but, but that, yeah, but ultimately like that's actually the AGI is just like, it's actually just sort of like a rhetorical tool to continue advancing towards whatever you want to you want to advance towards yeah although i think we all have an idea of of i mean people who are paying attention to the space of what it would look and and uh feel like um i mean but you know I really like Jan LeCun's world model research because it's grounded in, you know, in that language you layer on top of that.