A Conversation With Prime Minister Justin Trudeau of Canada + An OpenAI Whistle-Blower Speaks Out

27:42We're not dictating to them. But we are saying, yeah, you have to think about protection of kids. What I want is not for government to legislate what platforms should do or not do because that's a recipe for disaster. We all know how slow governments end up working. We're always playing catch up. And, you know, the political debates are not necessarily these days as deep and thoughtful as we would sometimes like them to be. But can we put the onus of leadership and responsibility that goes with it increasingly on platforms around journalism, around protection of free speech, but also protection against hate speech?

28:22Can we find those balances? Can we make sure that there is a common grounding in facts and not just deep fakes, but the level of misinformation and disinformation that we've seen over the while? These are the things that I think needs to be shifted, and I think platforms need to be a positive part of that conversation. One more social media question while we're on the topic. Should Canada ban TikTok?

28:48About many months ago, the Canadian government made the decision that all government phones could not have the TikTok app because it was too much of a security risk in terms of that. as a byproduct of that, the prime minister's family is on government phones. So my teenagers don't have TikTok anymore, which is lovely for me. They must love that. They are grumbling about my job on this and on a few other things. But I think there's two questions around TikTok. One is the data security issue is real. And the second more nebulous one is, is this good for kids? Is this good for people to be on social media?

29:26And I think those two issues are being conflated a little bit. People are sort of leaping on TikTok as, oh my God, it's connected to China. So, you know, we should ban it and then we should get onto banning, you know, Instagram reels and YouTube shorts and all those things that are so, you know, popular for very powerful reasons right now. So I think on the data security side, I mean, the head of the spy agency just came out and said, yeah, it's a real risk that China is accessing so much of Canadians' data through TikTok. And We are looking carefully at how to counter for that. But to use those tools to sort of say, well, TikTok's also bad for bad for people.

30:07So let's ban that. I think that as satisfying as it might be for for some parents and some some people and some teachers, quite frankly, that's not necessarily the path we want to take. You mentioned deep fakes as something that exists. I understand you're going to be up for election here in a bit. Have you been deep faked already? And are any of them good? I know I have been deep faked many times. One of the ones that made me laugh the hardest was someone did a series of pictures of all the 23 Canadian prime ministers going back 150 years as classic rock 80s glam rock characters. And I found that very, very funny.

30:49Which one were you? Oh, I know I was, I was, I was, it was, I had very, very big hair and bulging packs and stuff. It was, it was very nice to see, but it was, it was sort of silly. I have a level of, of protection because I'm prime minister around that of people sort of expecting, although I did have a yoga instructor friend of mine who admitted that, hey, I just spent 300 bucks on that, that Bitcoin you were promoting on, on, online. I'm like, I don't promote Bitcoin online. You got scammed. How could you fall for that? It was, you know, so that happens. But I think much more about the fact that deep fakes are now impacting, you know, high school students, are impacting people in their works.

31:32The accessibility of that technology is something that I just don't worry about in terms of a political context. It's much larger than that. Yeah. I mean, we're always so sad on the show when anyone buys a Bitcoin. But to know that your own likeness had been used to promote it must be especially upsetting. Devastating. Can I ask one follow-up? Yes. Yeah. So I had a follow-up on deepfakes, which is that, you know, so far, you know, there's been some deepfakes of you. There's been some funny ones. You're not particularly, you know, as it pertains to, you know, maybe this election. And, of course, you're, you know, you're the most famous politician in Canada.

32:05So people are sort of already broadly aware of you. One concern that you hear sometimes about deepfakes, though, is that not all politicians are as widely known. and that in maybe a local race, a provincial race, one of these things might get a little bit further. So I just wonder, have you seen anything like that? And do you worry about, you know, what is sometimes called synthetic media, sort of just upsetting politics in general in Canada? Well, I mean, let's be very clear. Even before the arrival of deep fakes and increasingly sophisticated synthetic media, we have seen misinformation, disinformation, polarization, foreign interference by countries like Russia and China and India that are trying to destabilize our democracy on various ways, be very, very impactful, like some of the fake tweets or the bots that are out there to drum up support or criticize, like have become more and more sophisticated.

33:02So this is sort of a next level tool. And ultimately, the answer to that is sort of similar to the answer to everything, is to empower citizens to be more discerning, to be more thoughtful, to make sure that local journalists are able to actually cover those races that aren't with national figures, to make sure that there are consequences and frameworks and independent bodies not associated with government that are weighing in on fact-checking and saying that this does matter. And all those sorts of things, I think, will be more and more urgent to deal with deepfakes, but that might also give an impetus for more people to realize, okay, just because I saw this local counselor explaining that they would really want to do something horrible, that should clue in that we can't be just taking everything at face value.

33:58I'm curious if you turn to AI at all in your own life. Is it part of your work life? Do you use it for fun when you're not working? How have you intersected with these tools? Do my streaming algorithms count? You know, obviously, sometimes those algorithms work really well for good recommendations. Myself, I tend not to use it, even though my kids use it. I had a friend of mine who's big into AI. I said, well, listen, AI could write you a platform on how Canada should handle AI. I'm like, okay, why don't you do that? Do that and then show it to me. And they showed me two pages of work that they generated in about 10 seconds.

34:42And it looked reasonable. I could see how if you put that in the window, someone would say, oh, there's an AI strategy for Canada. But as soon as you scratch the surface a little bit, okay, well, why this and what does that work? You could see it was just, it was a facsimile of a platform, wasn't a real platform. So that reassured me a little bit that there's still a lot of hard thinking that needs to go into solving these problems that are not just cosmetic. Yeah. I know we only have a minute left. One question on national security and AI. Obviously, we could spend an entire podcast just talking about this, but I'm curious.

35:19Do you think governments need to be building their own AI systems in this day and age, or is it enough to just have the private sector building this stuff and governments regulating it? No, I think governments need to be using and aware of the various most advanced technologies out there, particularly because we know other people or adversaries or countries that don't wish the best for us. We'll be developing those as well. And there are going to be, I know how scary it sounds, oh, government's developing AI. And that's not what I'm talking about. But I am saying that if we're going to make sure we're keeping Canadians safe, like I said earlier, AI for good done by the good people with careful, rigorous oversight and transparency is going to be a very powerful tool to counter the AI for bad that we know is going to be out there.

36:22Yeah. Well, thank you so much for your time, Prime Minister. It's nice to talk to a world leader who is not guilty of 34 felonies. And I'm not saying anything. And if you want, as a parting shot here, I want to offer you a free slogan that you can use for your national AI efforts if you want. This is just an idea. Canada, we put the A. in AI. Oh my God. What are you doing? What do you think? I think that's a real winner. You have a future in politics. I hope you run against me because it'll be really hard. All right. Well, thank you so much, Prime Minister. Great to talk to you. Thank you, guys.

36:59Great to see you.

37:04When we come back, Kevin gets an open AI whistleblower to go on the record about what he saw inside that company that pushed him into leaving it.

37:40So a big week for you. I heard you were up all night on Sunday. I was, yeah. Well, not all night. Literally, that would be a little extreme. You stayed up till 930? Yes, way past my bedtime because I was working on this story. Do you ever think about just finishing it during the working day or do you always leave it to the last second? Oh my God. You know, some of us have deadlines and editors who want things by certain times. We can't just laze about and then fire off a quick email to our subscribers. Well, that was your first mistake was taking that job for yourself. But so let's talk about this story, Kevin.

38:13Because it was a very good story. I learned a lot, and it's something that I've wanted to know for a long time, which is, what do the people inside OpenAI actually think about what's going on there? Yeah, I mean, this has been a big question that I've been pursuing along with a bunch of other reporters ever since the board coup last year at OpenAI. There seems to be a sort of a steady drumbeat of intrigue and infighting at OpenAI that we've learned about since the board coup last year. And obviously, we've talked about some of the more recent developments, the departure of Ilya Sutskover and Jan Leakey, two of the most senior safety people at OpenAI who were concerned about the direction this technology was going.

38:52And then I got wind of this group of current and former OpenAI employees who were preparing to make some kind of a statement. So this was a story that came about because of a guy named Daniel Cocotelo, who's a former open AI researcher who left the company earlier this year. And he has basically organized this group. You know, you could call these people whistleblowers. You could call them, you know, concerned insiders, but either way, this was - Rabble rousers, traders. But, you know, whatever you want to call them, this is a group that was inside the company and became concerned about the direction that the company was moving and decided to speak out.

39:35And some of those people are still there, right? Yeah, so this group, I call them whistleblowers in my story, whatever you want to call them. It is roughly a dozen or so people who have now signed this public statement calling for changes at the big AI companies. And among those people are some former employees and a few more people who are signing this letter anonymously because they still work at OpenAI. So tell me a little bit about who Daniel is and how he got to this point. So Daniel is a 31-year-old former researcher at OpenAI. He was part of their governance division. And he's a person who's just spent a long time thinking about risk in AI.

40:16So he came up through the sort of effective altruist movement. He was part of this AI research nonprofit called AI Impacts. And then in 2022, he went to work at OpenAI, and his specialty is forecasting. He is a person who has strong views about when various sort of AI events will unfold, and he has really tried to be rigorous in studying those. And when he got to OpenAI, that was sort of the thing that he was focused on, but then he sort of gradually became worried that the company was not doing enough to promote safety and that it was just sort of racing to build shiny products and get them out into the world.

40:56And we'll ask him all about that. But in the meantime, this group that he helped bring together to raise their concerns, what do they think and what are they asking for? So the open letter is primarily concerned with what they call the right to warn. Basically, they want OpenAI and other leading AI companies to commit to not only creating sort of internal ways for employees to raise concerns about safety, but they also want stronger protections for whistleblowers or other people who sort of voice those concerns to maybe outside groups or regulators. Right. So OpenAI has this whistleblower hotline that you could call, but it just goes to someone at OpenAI.

41:40My understanding is they want this hotline to actually go to a regulator or some external authority who can help. Yes, and they are not just focused on open AI. They want these policies to be implemented all across the AI industry. And is this the sort of thing where these whistleblowers have given us a really detailed list of their complaints and helped us understand exactly what it is they're seeing inside these labs that is making them so nervous? Yes and no. So in my conversations with Daniel and other members of this group, they have been pretty light on specifics when it comes to incidents that made them concerned about safety at OpenAI.

42:14And in part, that's because they believe they are still bound by these confidentiality agreements that they have signed with OpenAI. So they're not getting into the kind of nitty gritty inside information. But they did provide some specifics and they provided a lot more sort of generalities about kinds of changes at OpenAI and messages from leadership that they believed, signaled that this company was not taking safety seriously enough. So what did OpenAI say, Kevin, when you asked them about all this? So a spokeswoman for OpenAI sent me a statement that said, quote, we're proud of our track record providing the most capable and safest AI systems and believe in our scientific approach to addressing risk.

42:59We agree that rigorous debate is crucial given the significance of this technology and will continue to engage with governments, civil society, and other communities around the world. They also pointed out that OpenAI does have an employee hotline for integrity issues, and they also said that they have started this new safety committee that will be reviewing feedback from these former employees as part of their sort of assessment of safety culture at OpenAI. Well, I'm excited to talk to Daniel about all of this, Kevin, because it plays into something I've been thinking a lot about this week, which is power and who wields power in OpenAI.

43:39So I imagine you saw this story in the Wall Street Journal about Sam Altman's investments this week. So the journal had this great piece. It revealed that Altman has more than 400 investments. Increasingly, it seems like companies in his portfolio are now signing deals with OpenAI for various things. And Sam has equity in those companies, which means that as those companies grow, he will get rich. But as I've Been thinking about this and talking about it with people, Kevin. I don't actually think that money is probably what this is about for Sam Altman, right? Because this is a person who is already rich.

44:11But if he has a piece of all the biggest and most important companies in AI, then if you fast forward to the time period that Daniel's worried about when AGI is arriving, one person is going to have a staggering amount of power in that universe, right? He will be the CEO of OpenAI, and he will be a major shareholder of many of the most important companies in the space. Why do I bring this up? In that kind of world, you want to know who is this person and what sort of governance structures are around him to rein him in? Because right now it does seem like we're on trajectory for Sam Altman to be one of the most powerful people in the world.

44:46Totally. And I think that raises one of the questions that I had when I first heard about Daniel and this group of OpenAI insiders, which is why are these people coming forward now? And I think this group really just feels like we are getting to a point where this technology is legitimately going to be dangerous. Now, there are people who disagree with that, who think we're nowhere near that. But these people are convinced that we are getting to an important inflection point, not just in the history of AI, but sort of in the history of humanity as we approach these generally intelligent systems.

45:19So for that reason, it's becoming more important. You know, five years ago, 10 years ago, it may not have mattered that Sam Altman had, you know, a hand in many of the hottest startups in Silicon Valley. It may not have mattered that, you know, AI was being sort of recklessly developed. But these people feel like as we get closer to a moment where the entire world is going to start caring deeply about the progress of AI, it actually becomes much more important to trust the people and companies that are building it. Yeah. Now, I do feel like we should say there are still a lot of people out here who think that the folks involved here are being ridiculous.

45:59They think that they are afraid of a ghost. They think there is no way that these systems, as they are currently designed, will ever amount to anything half as powerful as what is being discussed. And I go back and forth on this all the time, sort of depending on who is the last person I have talked to, right? But, Kevin, this week is a week where I am taking these people seriously, okay? because when you do what Daniel and others invite us to do, which is to just look at the trend lines of how this stuff is developing, and you look at how much more powerful it gets every time you add a little bit more compute, you can actually just extrapolate forward in a pretty linear way and say, you know what, the future might indeed look like the past five years where this stuff just keeps getting exponentially better.

46:41And I think if you believe that there is even a chance that that is true, then you do want to pay at least some attention to what folks like Daniel are saying. Absolutely. Yeah, and I think, you know, we've seen this kind of story play out before, like with social media, right? There were these companies, they had these grandiose missions of connecting the whole world. They hired all these people to make those visions a reality, including some pretty idealistic people who wanted the companies to live up to their stated values. And when those companies started, you know, making decisions, cutting corners, putting maybe growth and profits ahead of things like trust and safety, we saw people start to speak out about that.

47:21And so in some ways, this is a very familiar cycle that we've just seen play out with social media that I think we're just starting to see play out with AI. Yeah. Well, I think it's going to be a great discussion and we should bring Daniel in here. When we come back, we're going to talk with Daniel Cocatello, formerly of OpenAI.

47:58Daniel Cocotello, welcome to Hard Fork. Thanks for having me. Excited to be here. So let's start with your experience at OpenAI. You joined the company in 2022 as a researcher in the governance division. You were working on things like predicting the path of AI progress. At what point did you start to worry that safety was being deprioritized? It sort of crept up over time. I think the core thing here is that it's more important to prioritize good governance and safety the closer you get to incredibly powerful and dangerous AI systems. And when I joined early on, I thought like, we're talking a good talk.

48:42There's a bunch of people here who really care about this stuff. And we're sort of leading the industry, in fact, in terms of our like willingness to talk about these things. And I basically gradually lost hope that we were really actually going to pivot more to investing in these things to get ready, at least not to nearly the extent that we need to. You told me when we spoke earlier about some advice that you got after you joined OpenAI, basically from someone telling you sort of don't rock the boat, like don't come here and expect to sort of change the place. Tell me about that advice and what that meant to you at the time.

49:25I mean, to be honest, it was a relief to hear that advice because I I don't want to be doing that sort of stuff. Like I'm a researcher. I really like trying to predict the future of AI. By contrast, engaging in internal company politics and trying to like argue with people to change their policies is super stressful and not my, you know, cup of tea. So actually when I heard that, I was like, great. Like I didn't want to do that anyway. You know, I'm going to keep my head down and do my research. Yeah. Do you want to walk us through incidents that you observe that you found the most disturbing?

50:05Yeah, so this is tricky because I'm still bound by this confidentiality stuff. So there's a lot of stuff I would like to talk about, but I can't talk about it. I mean, I can talk about some of the things that are already public, such as the DSP thing. Yeah, give us that story. So this was, I think, in late 2022. and we had this internal self-governance structure set up called the Deployment Safety Board. And this board had some people from Microsoft on it and some people from OpenAI on it. And it was supposed to deliberate and then eventually approve major deployments on the GPT-4 scale or bigger.

50:42So it had been set up prior to reaching the GPT-4 scale, but it was set up with the expectation that once we hit that level and thereon, it would be something that would gate deployments, you'd have to get approval from the DSP. While the DSP was still deliberating about this, me and various other people started hearing rumors that Microsoft had just gone and deployed it anyway, specifically in India. So we had some conversations about this. We made inquiries to managers and so forth. I don't want to get into too much detail about who said what to whom and so forth, but I came away disappointed because it seemed like we had this self-governance structure in place and there was a violation of the structure.

51:30And rather than holding Microsoft accountable for doing this, we were afraid to damage our relationship with Microsoft, given the sensitivities involved. They have all this compute. We're trying to be partners with them and so forth. And what happened in India wasn't that big of a deal. GPT-4 isn't actually that dangerous or scary, but I think this is an ominous sign about the ability of tech companies to self-regulate like this. Companies can make all these commitments and have all these structures, but then they can be violated and the rest of the world can just not know about it. And Kevin, you actually verified with Microsoft that this happened, is that right?

52:06Yes, it was an interesting chain of events. So when I was getting ready to publish this story, I went to Microsoft, I said, hey, there's this claim being made that you all were testing this GPT-4-based model in India without approval of this safety board. And their first response was they denied it. They said, this never happened. We never used GPT-4 in India in these tests. And then after the story was published, I got a note saying, actually, that was wrong. A version of the model that eventually became GPT-4 was included in a small test in India, and it hadn't been reviewed by the DSB, although they say it was reviewed and approved later.

52:46So essentially saying our denial was wrong, and these allegations were correct. And so, Daniel, I think you're making a really interesting point here, because what you just said, you know, I would agree with, which is GPT-4 is not that scary. But for that very reason, it seems like it should not be a big deal to take it before this board that has been set up for this exact purpose and get approval to launch something. And I guess the question is, well, if you're not going to even test for GPT-4, which we can all agree isn't that scary, when are you actually going to start doing these tests? I mean, was that what you were feeling?

53:17Well, I mean, in OpenAI's defense, they are doing more stuff now. Like the DSP did, in fact, eventually approve GPT-4 employment. So I do think there's some progress in that direction. The point that I'm trying to make is that we shouldn't have to trust the company to keep their own commitments. We shouldn't have to trust any of these companies to keep their own commitments. There should be some way to tell whether the commitments are being followed. Yeah. I know you say you're bound by these confidentiality agreements, but I really want to press for just as many specifics as you can offer, because I think that will help people understand how you reached the conclusion that the company was being reckless, that they were not taking safety seriously enough.

53:56So anything more you can share, any incidents or conversations or things that maybe convinced you that there was a problem here? I know you want me to give like more concrete details, but I just can't. I'm sorry. I will say I can talk about some things that have already been discussed. But I'm going to try to avoid violating confidentiality. Sure. Well, so what else is out there that is sort of publicly known that maybe contributed to your sense that the company was acting recklessly? Yeah. So there's been this story about accelerating ship production globally. This flies in the face of some of the arguments that some of us at least have been making for why what we were doing was good.

54:39This is the story, just so I'm clear, about Sam Altman trying to go out and raise a bunch of money, perhaps as much as$7 trillion to build a whole bunch of the chips that you need to train powerful AI models. Is that the chip scaling story? That's exactly right. And again, I don't know all the details and the details I do know I'm not supposed to talk about. But, you know, that was disappointing to me because one of the arguments that had circulated at OpenAI prior to that was this hardware overhang argument. The idea was, yeah, sure, maybe the world is not ready for AGI yet. We have to improve our governance and regulation of this technology.

55:17But AGI isn't a single binary thing. It's not like you go from zero to one, nothing to AGI. It's rather going to be this intense period of rapid gains across all sorts of capability benchmarks. We want that intense period to be less intense. We want it to be smeared out more and for the rapid capability gains to be spread out over months or years rather than over weeks or months, right? And so the argument was, if we don't build HGI now, if we try to hold off or delay or pause, all of this compute is going to build up in the world. And so that when someone finally violates the agreement to pause, they're going to just have all this massive amount of compute and they can just scale up by orders of magnitude really quickly, blow through all those capability milestones really quickly.

56:03So this is the hardware overhang argument for why actually slowing down would be a bad idea. And we need to, you know, build AGI as fast as possible so that the transition from pre-AGI to post-AGI is slower. And then now, you know, it seems that Sam Altman is going to like just make there be more compute in the world, which directly undercuts that argument, at least. I'm very curious, Daniel, what the board drama at OpenAI last year when Sam Altman was fired and then brought back and the board was sort of reconstituted what that felt like to you from the inside as someone who presumably was already slightly worried about how the company was treating safety and then to have this whole big corporate governance blow up.

56:50I'm just curious what that felt like from your perspective. I mean, the short answer is it was devastating for me. Prior to the board crisis, I still had hope that we were going to get all that stuff in place. We would have, you know, pivoted to investing a lot more in alignment research and you know engage in some sort of internal governance or self-governance stuff that made us more democratically accountable and transparent but you know after Ilya Suskever and Helen Turner and after that whole fight it was like oh man things have sort of polarized like I don't think that we're going to like talk it out and and and pivot and then also just sort of on a personal level I think there was a lot of anger directed at safety people such as myself, because the perception was that the old board was being dishonest and that they didn't really fire Sam because he was insufficiently candid towards them, but rather they fired him because they wanted to slow down and he didn't, and they needed an excuse.

57:52So I think that as a result, there was this sort of like, you know, fuck you, safety people. That's not a fair move, you know? So, and I think that's false, by the way. Like, I don't think, I think that it really was just that they didn't trust Sam. And I think they probably had good reasons, although I wish they would say more about that. Did you take a side during this? Like, who did you sort of find yourself aligning with when that crisis happened?

58:18So, this is something that's been maybe a sort of formative experience for me. Like, when the news hit on Friday that this was happening, I got scared. And I basically went and hid for the weekend. So I just went and spent time with my family. And then towards the end, there was like the double crisis where the board appointed Emmett Shear and Ilya went into the office to talk about it. And people flipped out on Slack. And, you know, there's just this huge uproar.

58:49And then I did do something. I basically just piped up a little bit and said like, I tentatively support Ilya. I trust his judgment. I want to hear what he has to say. It was just a very stressful experience, basically. It really sounds like after this board crisis and, you know, coupled with a couple of those events that you just mentioned, you had the sense of this company is not for me anymore. It is not the company that I thought I was signing up to work for. Does that seem fair? Yeah. Yeah, I think that's fair. I mean, but it's really not about me. Like, this wouldn't be such a big deal if AI capabilities were going to be fixed at roughly GPT-4 level for the next five years or for the next 10 years.

59:30And then on your way out, something unusual happens. And this is how I actually saw your name pop up for the first time in all of this, which is that OpenAI has this standard sort of offboarding paperwork that when you leave the company, you're asked to sign. and there's a non-disclosure part of it, there's a non-disparagement part of it, or at least there was. And you did something unusual, which is that you refused to sign that paperwork, even though per the agreements you signed when you joined OpenAI, doing so might cause you to lose out on all of your vested equity, all of your sort of wealth that you had accumulated.

1:00:13And my understanding is that was a significant amount of money, about$1.7 million. So you were basically refusing to sign this paperwork that it would have, you know, guaranteed you a large sum of money in exchange for the right to speak out about the company. So walk me through that decision. Like the way it works is around the time you join the company, you sign something that says your vested equity goes away if you don't sign a general release when you leave the company within 60 days. And then when you leave the company, they give you the general release and it has all this stuff in it, including non-disparagement and including the fact that you can't talk about this release.

1:00:48At some point, I think I deliberated enough. I discussed this with my wife. She's been really helpful throughout this whole thing. I think this is a sort of decision that we would have to make as a family because obviously it affects both of us so much. And we were just like, no, we're not going to sign this. I don't think it's ethical. So I wrote this email to OpenAI saying, I don't think this is fair. I want to be able to criticize the company in the future. And so I cannot sign this. And then they were like, okay, we wish you the best on your future endeavors. And I was like, okay, well then, that's it, I guess.

1:01:19After 60 days, the equity is going to be gone. And so I started trying to move on with my life and get back to my research and, you know, try to figure out what my next career step was going to be. Right. And just to clarify, you discussed this with your wife because this was not a small amount of money for you guys, right? She doesn't work at OpenAI. She's not. No. Yeah. So first of all, it's hard to actually say how much this is worth. What I did is I just asked them, like, how much would this sell at the last tender? And they said a certain amount. And then I just multiplied that by the amount that I had and got$1.725 million.

1:01:54And that's like 85 % of our net worth at the time. Wow. And then after you refused to sign this paperwork, Vox reported on the existence of this sort of general agreement that all departing OpenAI employees were being forced to sign, which included this non-disparagement agreement and the sort of threat of maybe clawing back some of this vested equity. And the company actually responded to this. They basically said, we've never enforced this on former employees. We've never clawed back vested equity. We don't plan to. And Sam Altman said that he was genuinely embarrassed that he didn't know about the existence of this agreement and the terms of it.

1:02:34I guess I'm just curious if you believe that. I mean, I do find it hard to believe, but I will say for my part, I'm pleased that this happened. Like, I think if you had asked me like a month ago or whatever this was before the box story, what I thought would happen, I would have said OpenAI will be like, we're a corporation. This is just like our policies. What's the big deal? But actually like a lot of people inside OpenAI, as well as outside, were pretty upset about this, which was gratifying. And then also, you know, Sam apologized and it's going to change. And that's actually what leads to this to today with the proposal.

1:03:11Yeah, let's talk about that. So you leave OpenAI earlier this year, and then you sort of start putting together this group of current and former employees who share some of these same concerns. How did this whistleblower group, if you want to call it that, come together? Yeah. So after the Vox thing happened and this all started blowing up, a ton of people started reaching out to me, offering thanks and support and so forth. So this lawyer, Larry Lessig, offered pro bono services. And so I engaged with him. Various people were reaching out to me. And I was saying, if you feel similarly to me, like, here's a good lawyer, you should get in touch, you know?

1:03:54And so this group sort of congealed. and this core group of us met up and discussed what to do. And then we were like, how about this? How about we meet them with a positive proposal for what good policies would look like? And the core idea in this letter is, hey, let us talk about the risks that we believe that the companies that we worked at are creating related to artificial intelligence. Yeah, that's right. So it's actually kind of complicated. There's four points that we're proposing. So principle one is basically, don't do the sort of thing that OpenAngers did with the silencing of criticism and the equity clawbacks and stuff like that.

1:04:32Principle two is this, what would the ideal system look like? The ideal system would be an anonymous reporting hotline so that employees at these labs that are building this incredibly powerful technology can communicate with the people who need to know. People like regulators and like independent watchdog groups with relevant expertise. Principle three and four are sort of saying you should promote a culture of, you know, open criticism. Three is basically saying employees who are raising serious concerns about various risks should be able to talk about confidential information in the course of doing so to the public only insofar as it supports a risk-related concern.

1:05:15And then principle four says we can actually walk that back if principle two is working really well. If we have our good system in place where all the people who need to know can be informed, then we can walk back the thing about releasing confidential information, and we can just be like, nope, go back to silence. So that's the overall proposal. So that seems like a pretty reasonable set of requests to me. But there is this thing that I'm struggling with, Daniel, which is that you were mentioning earlier that the board didn't share a lot of details back when it fired Sam and this wind up sort of coming back to bite people who are really concerned about safety issues because it sort of felt like there was no supporting evidence, right, to back up some of the claims that were being made.

1:06:00And, you know, as I'm hearing your story, the only thing that I have to go on is that Microsoft tested a thing, you know, without running it by a board. So I just, my question is essentially like, why should I essentially believe that things are as dangerous as you're leading me to believe when no one involved is willing to really tell me anything of much substance? Yeah, good question. I think in my case, I'm not trying to call out OpenAI in particular. I mean, I do have my disagreements with the company, my criticisms. But honestly, I think that probably if I went to these other companies, I would feel pretty similarly.

1:06:42So I don't want to be like specifically saying OpenAI is really bad, and that's why we need these policies. Instead, I want to say, this policy would be good, and I can justify this without giving specific details, without violating confidentiality. Even the stuff that's already public, I think, is enough evidence to justify this policy. Yeah. Don't you feel like you'd build more momentum, though, if at least someone somewhere was willing to say, hey, you know, it turns out that OpenAI has GPT-6, like, you know, sitting over there in a vat or, you know, whatever it is that's going to raise our hackles.

1:07:11Like, I'm just curious how you think you can advance this movement when it seems like we're just sort of like stuck at this very high level of abstraction. I mean, OpenAI doesn't have GPG-6 in a vat. Okay, sweet. We're breaking news. I revealed some information there. I mean, this is an alternative strategy we could have taken. We could have been like, now is not the time. wait until it's like right on the brink of catastrophe with terrible things right about to happen and then you know blow the whistle and stuff like that and i think that's just cutting it too close for one thing agi by definition can automate a lot of work so that means that the the size of the silo of humans that you need to be in this inner circle who knows about it can be pretty small so you could end up in a situation where there's like this secret project within the company that has access to the latest model and is using it to make even more powerful models and is automating a ton of stuff.

1:08:11And they can be actually doing quite a lot of research really fast because most of the work is being done by the AIs. And there's only a small group of humans who even know about this. And that's one of the reasons why I think it's dangerous to wait until then and plan to whistleblower right when you're right on the brink. I'm empathetic in the sense that journalists wind up being in this position a lot. We constantly walk around ringing alarm Bell saying, hey, this bad thing could happen. And if we do our jobs right, someone intervenes to stop the bad thing from happening. And then that winds up often making us look foolish because the bad thing never happened, right?

1:08:43So we're sort of the boys who cried wolf. At the same time, I do think that this movement, like this contingent of safety folks, needs to find a more persuasive story to tell. Because I think that right now, it's not connecting with people because they use ChatGPT and they have trouble understanding how this is going to end the world. Oh yeah, that reminds me of the other thing I was going to say, which is that this whole issue about like, is AGI coming soon? Is AGI risk a real thing? Is like 90 % not about what's happening in the labs in secret, you know? I think that publicly available information about the capabilities of this models and the progress that's been happening over the last four years is already enough to make extrapolations and say, holy cow, it seems like we could get to AGI, you know, in 2027 or so or sooner, you don't need any secrets to be able to make that inference.

1:09:33And then for the second thing about like, is this a big deal? Is this really dangerous? Again, you don't need any specific secret. I did hear from some, you know, a lot of people when this article came out about you and this whistleblower group thought, you know, these people are taking a brave stand. I did also hear from people who had more critical views of this. And I just want to let you respond to some of these. One thing that really stood out to a lot of people who read this article is about you and your personal predictions about the trajectory of AI. So one of the things that you told me that was in the article is that you believe that AGI, which you define basically as an AI system that can sort of do any kind of job, like anything that would be economically valuable, that that could arrive as soon as 2027, so three years from now, which is a very short timeline.

1:10:21You also have a PDOOM, which is the probability that AI will catastrophically harm humanity, of 70%, which is among the highest I've ever heard from someone who has directly worked at one of the big AI companies. So people are sort of citing these figures and saying, well, Daniel, he's not like a sort of normal run-of-the-mill AI researcher. He is a doomer. He is someone who believes that this stuff is going to kill us all, and we shouldn't take him seriously as a result of that. So I'm curious what you make of that. Yeah. So, I mean, first of all, I will totally admit that my PDoom is higher than most people's.

1:10:59I want to inject a note of humility here. Like the PDoom thing is more of a vibe than a serious number because it's an inherently unpredictable sort of situation. I think that I feel much more confident about my timelines. I think this is something that I have expertise in. Like I was hired at OpenAI to try to forecast the future of AI and I spent years trying to do that. And I think it's, you know, there's now lots of evidence that you can draw from scaling laws, trends on benchmarks and so forth. So I do feel somewhat confident in my claim that HGI could happen in the next few years. But I also could say that, like, there's a bunch of other people who came together to make this proposal.

1:11:40and I bet they would all have much more like normal opinions about PDoom and they still are concerned and think this letter is a great idea and are advocating for it. So I think that this is something that hopefully a lot of people can get behind. I'm curious if you've heard from anyone at OpenAI since this letter was published this week. Yeah. Yeah. A few people. What did they say? some people reached out being like you go thank you for doing this it's very brave some people reached out being like i feel a little bit betrayed like i thought we were going to work together to get better policies but now you've been critical of open ai and it seemed like it's all our fault but like other companies are probably just as bad well the second group just sounds like they're whining and they need to get over it i mean i'm happy to say other companies are probably just like I think this is a systemic problem, you know.

1:12:36But I actually think it's a revealing point because I think that there are probably people inside OpenEye who think, well, because there's another worse company somewhere, that justifies some amount of the actions that we're taking or some amount of the flaws that we have. And, you know, I can understand why you would have that point of view, but that's really not what you're saying. You're saying a very dangerous thing is somewhat imminent and people working on it need to be able to discuss that openly with the public. Well, I'm definitely not going to tell you who it was who was saying these things.

1:13:03But it was more reasonable than you make it sound, I think. Got it. So what comes next for you? I want to get out of the media spotlight. I think this has all been very stressful for me. Yeah, so like technically I'm supposed to be on vacation right now. This is my parents' house. So mostly we just need to get Kevin to leave you alone, it sounds like. Yeah. Well, I hope you do actually get some vacation on top of your big week. I also am just really grateful for the time and the opportunity to speak with you. And I hope this continues a conversation that I think is fairly overdue. Thank you. And likewise.

1:13:41Thanks, Daniel. Thanks, Daniel.

1:14:11Hard Fork is produced by Whitney Jones and Rachel Cohn. We're edited by Jen Poyant. Today's show was engineered by Alyssa Moxley. Original music by Marian Lozano, Diane Wong, Pat McCusker, and Dan Powell. Our audience editor is Nolga Logli. video production by Ryan Manning and Dylan Bergeson. Go check us out on YouTube. We're at youtube.com slash hard fork. Special thanks to Paula Schumann, Pui Wing Tam, Kate Lopresti, Jeffrey Miranda, and thanks to Alex Goldberg for all the Canadian gear that he lent us and that we're not giving back. You can email us at hardfork at nytimes.com. Maybe it's time to blow the whistle at your company.

1:14:49Think about it. Come on. What do you got to lose? What do you got to lose? What do you got to lose?

1:14:58Thank you.