What the Ex-OpenAI Safety Employees Are Worried About — With William Saunders and Lawrence Lessig

0:00An ex-OpenAI Superalignment team member joins us to share his concerns about the company's trajectory along with his lawyer, the Harvard Law Professor Lawrence Lessig, who will shed light on the lack of protections for those who speak out. All that and more is coming up right after this. Welcome to Big Technology Podcast, a show for cool-headed, nuanced conversation of the tech world and beyond. We have a great show for you today. We're finally going to speak with some of the people behind some of the concerns you've been hearing about the trajectory of OpenAI, especially with regard to the alignment work within the company or really the super alignment work.

0:36So we're joined today by a former member of that super alignment team. William Saunders is here. Welcome, William. Thanks for having me on. Thanks for being here. And it's my great pleasure to welcome Larry Lessig back to the show. He's a professor of law and leadership at Harvard Law School, and he's also representing William Pro Bono here as he goes, and I guess many of his colleagues as well, as he goes and speaks out about these issues. Welcome, Larry. Great to be back. Let's begin just talking a little bit about the vibe within OpenAI. William, you left a few months ago. Take us a little bit inside the company so we can understand the environment from which you're coming out of.

1:12During my three years at OpenAI, I would sometimes ask myself a question. Was the path that OpenAI was on more like the Apollo program or more like the Titanic? And the Apollo program was about carefully predicting and assessing risks in doing groundbreaking science, building in enough safeguards to be able to successfully bring astronauts to the moon. And then even when big problems happened, like Apollo 13, they had enough redundancy and were able to adapt to the situation in order to bring everyone back safely. Whereas the Titanic came out of this competitive race between companies to keep building bigger and bigger ships, ships that were bigger than the regulations had been designed for.

2:04Lots of work went into making the ship safe and building watertight compartments so that they could say that it was unsinkable. But at the same time, there weren't enough lifeboats for everyone. And so when disaster struck, a lot of people died. And OpenAI claimed that their mission was to build safe and beneficial AGI. And I thought that this would mean that they would prioritize putting safety first. But over time, it started to really feel like the decisions being made by leadership were more like the White Star Line building the Titanic, prioritizing getting out newer, shinier products than really feeling like NASA during the days of the Apollo program.

2:47and I really didn't want to end up working on the Titanic of AI. And so that's why I resigned. It's kind of interesting that you use those examples and not the Manhattan Project, which is kind of the one that's been people have brought up, like the power and the destruction potential of nuclear energy has been something that's been talked about. And I don't know if Sam has compared himself to Oppenheimer, but I've definitely heard some people make those comparisons. Why did you shy away from that? As long as we're going through the analogy lens, why did you shy away from that one? I think that is another valid analogy.

3:26I think this example of the Titanic makes it sort of clear that there were, again, safety decisions that could have been made better or something. I think the Manhattan Project is more the analogy for the scope of impact that this technology could have, that the companies are claiming this technology will have and are raising billions and billions of dollars based on this premise of the scope of impact. I think it's also a tale of scientists who set out building a technology wanting to do something good in the world. The reason the Manhattan Project got started is because scientists looked at what was coming, what was possible, and were terrified that Adolf Hitler would get the bomb.

4:18And that this would be absolutely terrible for the world. And that's why they went to the Americans. But somewhere along the way, you know, at some point, like Hitler, you know, was dead. Germany had surrendered. And yet the project went on. And again, that's another situation that like, I would really not like to find myself in. Right. And I was going to ask you whether you think OpenAI is a product or a research company. And obviously it's both. But the question is, what leads? And reading between the lines or maybe just hearing you explicitly, your belief is it's a product company. Is that right?

4:54It's a bit different from just trying to make the products that are most useful today. It's coupled with a vision of the research for how to build towards something called AGI or artificial general intelligence, which is building systems that are as smart as most humans and can do most economically valuable work that humans can do, a.k.a. can do most jobs of people. And so it's like the combination of these visions or something is something that I'm more concerned about. It's where that they're like building on a trajectory where they are going to do like AGI, as stated, will be a tremendous change to the world.

5:43And so it's they're on this trajectory to like change the world. And yet when, you know, they release things, their priorities are more like a product company. And I think that is like what is most unsettling. You came out with this letter recently talking about how there needs to be a right for people within these companies to warn the public about some of the concerns they have. Here's a quote from the letter. There is no effective government oversight of these corporations. Current and former employees are among the few people who can hold them accountable to the public. Yet broad confidentiality agreements block us from voicing our concerns.

6:22So we're going to get into those agreements in a bit with Larry. But you obviously signed the agreement. You're here talking today. The main question I think the public has from you and the people who have signed this letter is, did you see something? Like, did you see something concerning enough inside the company to merit speaking out here? And if so, what was it? To set some things clear, if there was a group of people that I knew were being seriously harmed by this technology, first, I still really hope that OpenAI would do the right thing and address this if this was a very clear-cut case.

6:59I also personally would ignore any sort of considerations of how I might be retaliated against, and I would talk about that plainly. So that's not what I was seeing. It's also, I don't think that I was working on the Titanic. I don't think that GPT-4 was the Titanic. I more am afraid that like GPT-5 or GPT-6 or GPT-7 might be the Titanic in this analogy. And so I can talk about maybe like a couple of areas here. So one is a former colleague on the Superalignment team, Leopold Aschenbrenner, has talked on a podcast about how he was asking some questions about how the internal security was working at the company.

7:56And then wrote a document containing some concerns. He shared this around. He then got reprimanded because some of the concerns offended some people. And, you know, personally, I would have like written it in a different way. And I think but the the disturbing part of this was that, you know, the only response was reprimanding the person who raised these concerns. Right. It might be reasonable to reprimand the person and then be like, OK, but these parts of these concerns, we're taking these seriously and we're going to address them. Right. That was not the response. And then he later talked about like this was one of the reasons that was offered for him being fired.

8:39So I guess what I'm narrowing, I'll let you do the other example in a second. But like what I'm narrowing down on is the people that have raised concerns within the super alignment team. It's not like they've all seen some powerful, dangerous technology that they don't believe open AI, like in the immediate term, they don't believe open AI is going to handle appropriately. It's more of like, what is the path in the future? And that's where this right to warn thing comes down. I'll say at the beginning, we're going to get into right to warn. I'm fully in favor of this right to warn. And I'm really glad that you brought it up.

9:09But I think that it's important for the public just to establish that like, inside open AI today, there's this group that's just left. It's not like there was this question, like, did Ilya see something? Right? Did you guys see something? It's not that there's something immediate and harmful that you've seen. It's more of like, you're concerned about the path that this company can go down. Is that right? Yeah. And I think this right to warn, this is a right to warn responsibly. This is not a right to cause unnecessary panic or something. And most of my research was driven by, again, concerns along this trajectory that the companies are going along, have demonstrated progress on, and are raising billions of dollars to keep going along this trajectory.

9:50But I do think there could be things happening today that we don't know about. And so the sort of scenario that I'm worried about happening today is suppose that there's some group of people that wants to spread a lot of disinformation on social media. Let's say they want to manipulate an election or they want to incite violence against a minority ethnic group. and let's say that they're in a non-English speaking country. And so, you know, again, most of the people at OpenAI speak English. Most of the alignment work is done in English. And so, like, it would be, you know, somewhat harder for the company to notice this.

10:33Now, like, the models are, like, safety trained to, like, refuse requests to do things that are inappropriate, you know, or, like, that seem like they might be harmful. Now, you know, OpenAI has like caught actors like generating disinformation. And so clearly they were able to either like, you know, the safety training didn't apply or they were able to bypass it. Right. There's this technique of jailbreaking where you like change how you frame the request in order to like bypass the safety limits. Maybe you just ask it in a different language or you tell some story around it. So now you've got a group of people, they have the ability to get the model, to go along generating lots of disinformation.

11:16And then the other line of defense that you might have here would be monitoring the company looking at it. And I have concerns that there might be a lot of ways that monitoring might miss things. so for example like some systems that the company has talked about involve using a very like small and dumb language model to like monitor what a larger language model is doing and this will like clearly miss a bunch of things you know or there are like there might be ways to like send requests in that like go through some pathway that is just not subject to monitoring or like you know the people can't look at what the actual requests and the completions are just because the company just doesn't store that.

12:04And so now you might have a group of people generating massive amounts of disinformation using open AI products and the company wouldn't know about it. In this case, if this happens in an English-speaking country, somebody might notice and eventually tweet about it and the company would find out through that pathway. But if it's in a non-English-speaking country, I don't know how big it could get before people would notice. And I really would want a company taking this kind of step to have... This is a scary story. Tell me why this can't happen. And have somebody who is outside of the company who can have an independent assessment that can say, yes, this can't happen.

12:50And then I will be able to rest easy. But I can't. Yeah. And I want to get to Larry here because we should talk about the broader context here, which is that, first of all, believe it or not, William is the first OpenAI employee that we've had on the show in four years. He's the first one expressing criticism of OpenAI from within or like from previously within for four years. And recently we kind of had a, you know, at least from the former employee standpoint, we figured out why. And that is because there are these broad nondisclosure agreements that opening eye employees have to sign before they leave.

13:27So I think we're going to talk about those first. And then we'll talk a little bit about this, I guess, new regulation or law that you're both advocating for, which is basically a law that's going to allow employees to, or I don't know, a rule that will allow employees to whistleblow even if there's nothing imminently illegal that's happening within AI. But let's talk about the NDAs first. So when people leave OpenAI, what are they forced to sign? And why has that made it so difficult for people like William to speak out? Okay, but I want to actually first tag on to something William just said, which I guess I think is really important.

14:05I first got into this space of whistleblowing protection, helping Francis Haugen, who was the Facebook whistleblower. And what William just described, of course, is what happened inside of Facebook with Myanmar-Rawangi genocide, which was basically the technology wasn't able to monitor the hate that was being spread by the government in that country, which led to tens of thousands of people being murdered. And while this is happening, people in the company are trying to raise the alarms and the company is not willing to devote the resources necessary to address the harm that they are able to demonstrate their company is committing.

14:50And the reason why that experience is relevant is it shows exactly why you can't rely on the company alone. When you've got a company in a deeply competitive market that's focused on a, you know, Facebook's had a single dimension, which is like user engagement. Like, are we continuing to meet our target? And an employee, you know, there are a lot of great engineers in that company who raised concerns that were valid and serious. But if they're inconsistent with the objective of the company, it's not going to do anything about it. And so that's the structure that's real in Silicon Valley that you've got to build around.

15:28And that's why what we're talking about, which we'll get to in a second, as you've said, guarantees that any concerns that are raised are not just raised to the company, they're raised to people outside the company who can do something about it. Now, as to what you're bound by, I got connected to this incredible group of OpenAI employees, ex-employees, when I read about the struggle that Daniel had gone through. Another ex-OpenAI employee who didn't sign the agreement. Yeah, who believed as he was leaving that by not signing an agreement, he was giving up, as New York Times reported, something like$1.7 million in equity.

16:10And when I read that, I was like, wow, I mean, I don't know many people who would give up$1.7 million just for the freedom to speak. That's interesting. Like, what is it that you think you need to say? And when he raised that concern and we began to talk to people in the circle of the company, very quickly the company realized that the agreements they were forcing people to sign were technically just not legal agreements in the state of California. equity is wage in the state of California. If you earn equity, you're vested equity, it's like your wages. And when you leave, they can't say, oh, here's a bunch of other additional terms you must agree to in order to take what you've already earned.

16:57So the non-disparagement part, any other additional obligations that were demanded were not actually obligations that could be enforced. And right now, the company's in the process of revising and putting together exit packages that are consistent with the law. And I'm optimistic, we're not, nothing's settled yet, but I'm optimistic we can get to a place that the company's rules are exactly right, that they, you know, they say, you're leaving, remember, you've got secrets you can't share and don't. But of course, you're allowed to share secrets with government investigators or people who are doing work with the government for safety purposes.

17:39They're not trying to block any of that. And to the extent they do that, what they're doing is going to be consistent with the law. But we're in a transition right now, and it's not yet fully resolved exactly how much they've accomplished and how much this still needs to be done. It's interesting. So those non-disparagement that employees had to sign or else they could face their equity being clawed back. Seems like they're both like non-enforceable and potentially being revised, which is very interesting, good news. I think OpenAI also says that they've never clawed back any equity, nor did they ever intend to, but it's making people sign the agreement is strong enough.

18:16Yeah. I mean, but that's right because, I mean, I've spoken to ex-employees who've said, look, I've not done X, Y, and Z because I feared the club acts. So they can say they never enforced it. They didn't need to enforce it to have the effect, which it had for a significant number of people, especially when you've got people who are being very conscientious about the kind of obligation they're going to accept for themselves or not. And so if they sign something like that, they're going to live up to it. Totally. So it has an effect whether they enforced it or not. And that's the problem with the agreement.

18:49Right. And then so that's once people leave. So the deeper question is what happens if people are inside the company and they see something they don't like? And are they able to speak out? Because let me see if I get this right. In a normal whistleblower situation, let's say you're an Enron and you see the company committing tax fraud. That's obviously illegal. You'd be protected on whistleblower statutes. But if you're within, let's say, an open AI, and you find that the development of the technology is moving towards artificial general intelligence or super intelligence in a way that you find dangerous, you're not allowed to say anything because we don't have any laws against developing super intelligence.

19:33Yeah. So there's actually so there's two things that go together that's very important in this context. So one thing you're right, there's not a lot of regulation. So there's not an FAA or an FDA sitting on top of the company that has imposed regulations that the company is either living up to or not living up to. But, you know, some agencies like the SEC takes the view that most anything could potentially be the sort of thing you'd have privilege to complain to the SEC about because it could potentially affect the value of the company. And to the extent it's potentially affecting the value of the company, it raises SEC concerns.

20:13So if you say you're following the following safety regime as the company in order to make sure AGI is safe, and then you don't follow that regime, the SEC's view is you can come out and tell the SEC. You can whistleblow to the SEC, and the SEC would consider whether that's something to act on. The problem is, this is the second part, engineers inside of companies like this or policy people inside of companies like this need to have confidence that the people they're talking to know what the hell they're talking about. So it's one thing to imagine an AI safety institute where you can imagine going to that and talking to people like you, people who have a really good sense of what the risks are, what the technology is, and explaining here's why you think there's a concern.

20:59It's another thing to imagine like calling the SEC and telling the SEC, here are the seven safety related concerns that I have, because you're very anxious that they understand it and are able to act on it in the appropriate way. And so that's why this is a kind of unique situation. It's both that there's not adequate regulation. So there's no regulator on the scene. And that it's a technical field that doesn't easily open up to like non-technical lawyer types, the sorts that are going to be working at the SEC. And that's why, you know, when I spoke to the employees that I was representing, it became clear that they wanted to kind of craft something that was different and new.

21:41And that's what the structure of the right to warn is trying to produce. So are you advocating for both a new regulatory agency and a rule to protect AI whistleblowers? What are you going to try to get at here? Well, my own view, and I won't speak for my clients here. My own view is, yeah, absolutely. There needs to be a regulatory agency that is overseeing this. I'm not sure what the structure of it is. It's kind of academic to talk about it, given the dysfunction of the federal government right now. But yes, other countries are building things like this, and we ought to be doing the same. And if there were such an agency, it itself would have lots of whistleblower protections built in, and that would maybe obviate a significant chunk of the need for the rule.

22:24But the rule that we're talking about is a rule that initially we're trying to get companies to embrace. I think the most interesting part of the right to warn is the third point where it talks about creating a culture of criticism where the company says, look, we want you to criticize us. We want you to tell us what's going wrong. We want to encourage that. We're not going to punish that because that's the way we become the safest kind of company that we can be. And so that's really about the company itself creating that. And then the other part that I think is really critical is that the company says, we agree, if we create, we'll create this structure that says you can complain to us and to a regulator and to an independent AI, like a safety institute.

23:17You can do all three of those things, confidentially and anonymously. And if we do that, we expect you will use that channel. And if we don't do that, we acknowledge you can use whatever channels necessary to make sure that these safety concerns are out there. But that's obviously designed to create a strong incentive for them to build a channel for warning. But OpenAI would say they already have that channel for warning. So this is what they've said in the press reports. No, they don't. They say they have avenues for employees to express their concern, including an anonymous integrity hotline and a deployment safety board that they run products through.

23:54Right. But that's the company alone. Okay. So what I said is it has to be all three of those things together. Right. So it's the company and the regulator and the AI Safety Institute. So that, again, like we saw with Facebook, a lot of complaints were made to Facebook about the safety or the lack of safety of their product, and the company didn't do anything about it. And so the concern here is you need to have external review as well. And that's why the channel has got to be a channel that goes to three of these entities so that we have some confidence that somebody is going to do something if there's something that has to be done.

24:28Well, I'm just curious from your perspective, do you think going to the SEC, like Larry described, is something that you or your colleagues would consider given, you know, if there were things that you saw that didn't sort of hold to the safety protocols that OpenAI had lined out? Or is that a non-starter? Yeah. So what I would really want if I went to Whistleblow is to have somebody on the other end of the phone or the other end of the message line who I know really understands the technology. And I don't know who at the SEC would be the person who would really understand the technology. I think that a model that I personally think would work better would be the model more proposed in California Senate Bill 1047, where the law would create the office of the California Attorney General as a place where you could submit whistleblower complaints to.

25:30and you could have like you know if you had employees who understood the technology there and you could talk to them you know and ideally this doesn't need to be like ideally this is not a high stakes conversation ideally you can just like call up somebody at the government and say like hey I think this might be going like a little bit wrong what do you think about it and like talk to them and they can gather the information and then hopefully they say like okay, this isn't actually that bad, you know, and then you can like get on with your day. You know, I think the thing to fight for here is like being able to really like, you know, talk about things before they become big problems.

26:09Yeah. And in that circumstance, the SEC is insufficient. Going to the SEC sounds like very intimidating. and you know it sounds like the sort of thing one would only do you know like you know it would be again it would be better to be like you know again like be able to to talk to somebody in some agency who understands the technology and understands you know what the the safety like system should look like great well i have a few more questions that build off what some of the former colleagues of William have said within OpenAI, and then more about the nature of the company and where we might be heading.

26:52So let's do that right after this. Did you know your credit card points and miles can lose value to inflation? Credit card companies often reduce the redemption value of your points and miles. Now, imagine a credit card with rewards that can grow in value. With the Gemini credit card, you can earn Bitcoin or one of over 50 other cryptos instantly with no annual fee. Every swipe at the store or gas pump earns you instant rewards deposited straight to your account. Plus, sign up now for a$200 Bitcoin bonus to kickstart your rewards. Visit Gemini.com slash card today. Check out the link in the description for more information on rates.

27:30Again, if you're looking to invest in Bitcoin but don't know where to start, the Gemini credit card makes it easy. The Gemini credit card is issued by WebBank. In order to qualify for the$200 crypto intro bonus, you must spend$3 ,000 in your first 90 days. Some exclusions apply to instant rewards in which rewards are deposited when the transaction posts. This content is not investment advice and trading crypto involves risk. The Gemini credit card cannot be used to make gambling related purchases. What the hell is going on right now? And why is it happening like this? At Wired, we're obsessed with getting to the bottom of those questions on a daily basis.