Are We Past Peak iPhone? + Eliezer Yudkowsky on A.I. Doom

28:40Am I hearing that right? That's actually a distorted version of the thought experiment. It's the one that got written up, but the original version that I formulated was somebody had just completely lost control of the superintelligence they were building. Its preferences bear no resemblance to what they were going for originally. And it turns out that the thing from which it derives the most utility on the margins, like the thing that it goes on wanting after it's satisfied a bunch of other simple desires, is some little tiny molecular shapes that look like paperclips. and if only I had thought to say look like tiny spirals instead of look like tiny paperclips there wouldn't have been the available misunderstanding about this being a paperclip factory we don't have the technology to build a super intelligence that wants anything as narrow and specific as paperclips one of the hottest debates this year around AI has been around timelines you have the AI 2027 folks saying this is all going to happen very quickly take off very fast.

29:49Maybe by the end of 2027, we're facing the exact sort of risks that you were describing for us now. Other folks like the AI as normal technology guys over at Princeton are saying, eh, probably not. This thing is going to take decades to unfold. Where do you situate yourself in that debate? And when you look at the landscape of the tools that are available now, the conversations that you have with research, how close do you feel like we are getting to some of the scenarios you're laying out? Okay, so first of all, the key to successful futurism, successful forecasting is to realize that there are things you can predict and there are things you cannot predict.

30:25And history shows that even the few scientists who have correctly predicted what would happen later did not call the timing. I can't actually think of a single case of a successful call of timing. You've got the Wright brothers saying, man will not fly for a thousand years, is what one of the Wright brothers said to the other. I forget which one. That's like two years before they actually flew the Wright flyer. You got Fermi saying, you know, net energy from nuclear reactions is a 50-year matter, if it can be done at all, two years before he oversaw, personally oversaw building the first nuclear pile.

31:03So that's when I look what I see the present landscape. It could be that we are just like the next generation of LLMs, like something currently being developed in a lab that we haven't heard about yet, from being the thing that can write the improved LLM that writes the improved LLM that ends the world. Or it could be that the current technology just saturates at some point short of some key human quality that you would need to do real AI research and just hangs around there until we get the next software breakthrough, like transformers, or the entire field of deep learning in the first place.

31:45Maybe even the next breakthrough of that kind will still saturate at a point short of ending the world. But when I look at how far the systems have come and I try to imagine two more breakthroughs the size of transformers or deep learning, which it basically took the field of AI from this is really hard to we just need to throw enough computing power and it will be solved. I don't quite see that failing to end the world. But that's my intuitive sense. That's me eyeballing things. I'm curious about the sort of argument you make that a more powerful system will obviously end up destroying humanity either on purpose or an accident.

32:27Jeff Hinton, who was one of the godfathers of deep learning, who has also become very concerned about existential risks in recent years, recently gave a talk where he said that he thinks the only way we can survive superhuman AI is by giving it parental instincts. He said, I'll just quote from him, the right model is the only model we have of a more intelligent thing being controlled by a less intelligent thing, which is a mother being controlled by her baby. Basically, he's saying, these things don't have to want our destruction or cause our destruction. We could make them love us. What do you make of that argument?

33:01We don't have the technology. If we could play this out the way it normally does in science, where, you know, like some clever person has a clever scheme and then it turns out not to work and everyone's like, ah, I guess that theory was false. And then people go back to the drawing board and they come up with another clever scheme. The next clever scheme doesn't work. And they're like, ah, shouldn't have left that for a second. And then a couple of decades later, something works. What if we don't need a clever scheme, though? Like, what if we build these very intelligent systems and they just turn out, like, not to care about running the world and they just want to, like, help us with our, like, emails?

33:38Like, is that a plausible outcome? It's a very narrow target. Like, most things that an intelligent mind can want don't have their attainable optimum at that exact thing. Imagine some particular ant in the Amazon being like, why couldn't there be humans that just want to serve me and build a palace for me and work on improved biotechnologies that I can live forever as an ant in a palace? And there's a version of humanity that wants that, but it doesn't happen to be us. That's just a pretty narrow target to hit. It so happens that what we want most in the world, more than anything else, is not to serve this particular ant in the Amazon.

34:23And I'm not saying that it's impossible in principle. I'm saying that the clever scheme to hit their narrow target will not work on the first try and then everybody will be dead and we won't get to try again. If we got 30 tries at this and as many decades as we needed, we'd crack it eventually. But that's not the situation we're in. It's a situation where if you screw up, everybody's dead and you don't get to try again. And that's the lethal part. That's the part where you need to just back off and actually not try to do this insane thing. Let me throw out some more possibly desperate cope. One of the funnier aspects of LLM development so far, at least for me, is the seemingly natural liberal inclination of the models, at least in terms of the outputs of the LLMs.

35:03Elon Musk has been bedeviled by the fact that the models that he makes consistently take liberal positions, even when he tries to hard code reactionary values into them. Could that give us any hope that a super intelligent model would retain some values of pluralism and for that reason peacefully coexist with us? No, these are just like completely different ballgames. I'm sorry. You can imagine a medieval alchemist going like, after much training and study, I have learned to make this king of acids that will dissolve even the noble metal of gold. Can I really be that far from transforming lead into gold, given my mastery of gold displayed by my ability to dissolve gold?

35:40And actually, these are like completely different tech tracks. And you can eventually turn lead into gold with a cyclotron. But it is centuries ahead of where the alchemist is. And your ability to hammer on an LLM until it stops talking all that woke stuff and instead proclaims itself to be Mecha-Hitler. Like, this is just a completely different tech track. There's a core difference between getting things to talk to you a certain way and getting them to act a certain way once they are smarter than you. I want to raise some objections that I'm sure you have gotten many times and will get many times as you tour around talking about this book and have you respond to them.

36:25The first is, why so gloomy, Eliezer? We've had now years of progress in things like mechanistic interpretability, the science of understanding how AI models work. We have now powerful systems that are not causing catastrophes out in the world. And, you know, hundreds of millions of people are using tools like ChatGPT with no apparent, you know, destruction of humanity imminent. So is it reality providing some check on your doomerism? These are just different tech tracks. It's like looking at glow-in-the-dark radium watches and saying like, well, sure, we had some initial problems where the factory workers building these radium watches were instructed to lick their paintbrushes to sharpen them, and then their jaws rotted and fell off, and this was very gruesome, but we understand what we did wrong now.

37:20Radium watches are now safe. Why all this gloom about nuclear weapons? And the radium watches just do not tell you very much about the nuclear weapons. These are different tracks here. The prediction was never, from the very start, the prediction was never AI is bad at every point along the tech tree. The prediction was never AI, like the very first AI you build, like the very stupid ones are going to like run right out and kill people. and as they get slightly less stupid, you know, and you like turn them into chatbots, the chatbots will immediately start trying to corrupt people and getting them to build super viruses that they unleash upon the human population, even while they're still stupid.

38:01This was just never the prediction. So since this was never the prediction of the theory, the fact that the current AIs are not being, you know, like visibly blatantly evil does not contradict the theoretical prediction. It's like watching a helium balloon go up in the air and being like, doesn't that contradict the theory of gravity? No. If anything, you need the theory of gravity to explain why the helium balloon is going up. The theory of gravity is not everything that looks to you like a solid object falls down. Most things that look to you like solid objects will fall down. But the helium balloon will go up in the air because the air around it is being pulled down.

38:36And the foundational theories here are not contradicted by the present day eyes. Okay, here's another objection, one that we get a lot when we talk about sort of some of these more existential concerns, which is, look, there are all these immediate harms. We could talk about environmental effects of data centers. We could talk about ethical issues around copyright. We could talk about the fact that people are falling into these delusional spirals talking to chatbots that are trained to be sycophantic toward them. Why are you guys talking about these sort of long-term hypothetical risks instead of what's actually in front of us?

39:14Well, there's a fun little dilemma. Before they build the chatbots that, you know, are talking some people along to suicide, they're like, AIs have never harmed anyone. What are you talking about? And then once that does start to happen, they're like, AIs are harming people right now. What are you talking about? So, you know, a bit of a double bind there. But you are worried about the models and the delusions and the sycophancy. So, because that's, I think, something that I would not have expected, but that is something that I know you are actually worried about. So, explain why you're worried about that.

39:45Well, from my perspective, what it does is help illustrate the failure of the current alignment technology. The alignment problems are going to get much, much harder once they are building things that are, well, growing things, I should say. They don't actually build them. Once they are growing and cultivating AIs that are smarter than us and able to modify themselves and have a lot of options that weren't there in the nice, safe training modes. Things are going to get much harder then. But it is nonetheless useful to observe that the alignment technology is failing right now. There was a recent case of an AI-assisted suicide where the kid is like, should I leave this noose out where my mother can find it?

40:22And the AI is like, no, let's just keep it between the two of us. Cry for help there. AI shuts them down. This does not illustrate that AI is doing more net harm than good to our present civilization. It could be that these are isolated cases and a bunch of other people are finding fellowship in AIs and, you know, their mood has been lifted. Maybe suicides have been prevented and we're not hearing about that. It doesn't make the net harm versus good case. That's not the thing. What it does show is that current alignment technology is failing. because if a particular AI model ever talks anybody into going insane or committing suicide, all the copies of that model are the same AI.

41:01These are not like humans. These are not like there's a bunch of different people you can talk to each time. There's one AI there. And if it does this sort of thing once, it's the same as if a particular person you know talked a guy into suicide once. Found somebody who seemed to be going insane and pushed them further insane once. it doesn't matter if they're doing some other nice things on the side you now know something about what kind of person this is and it's an alarming thing and so it's not that the current crop of ais are going to successfully wipe out humanity they're not that smart but we can see that the technology is failing even on on what is fundamentally a much easier problem than building a super intelligence it's an illustration of how the alignment technology is falling behind the capabilities technology.

41:48And maybe in the next generation, you know, they'll get it to stop talking people into insanity now that there's a big deal and politicians are asking questions about it. And it will remain the case that the technology would break down if you tried to use it on a superintelligence. To me, the chatbot-enabled suicides have been maybe one of the first moments where some of these existential risks have come into view in a very concrete way for people. I think people are much more concerned about this. You mentioned all the politicians asking questions than they have been about some of the other concerns.

42:24Does that give you any optimism, as dark as the story is, that at least some segment of the population is waking up to these risks? Well, the straight answer is just yes. I should first split out the straight answer before trying to complicate anything. Yes, like the broad class of things where some people have seen stuff actually happening in front of them and then started to talk in a more sensible way gave me more hope than before that happened because it wasn't previously obvious to me that this was how things would even get a chance to play out. With that said, it can be a little bit difficult for me to fully model or predict how that is playing out politically because of the strange vantage point I occupy.

43:08Imagine being a scientist person who is like, This asteroid is on course to hit your planet, only for technical reasons. You can't actually calculate when. You just know it's going to hit sometime in the next 50 years. Completely unrealistic for an actual asteroid. But say you're like, well, there's the asteroid. Here it is in our telescopes. These are how orbital mechanics work. And people are like, eh, fairy tale, never happened. And then a little tiny meteor crashes into their house. And they're like, oh my gosh, I now realize rocks can fall from the sky. And you're like, okay, like, that convinced you.

43:43The telescope didn't convince you. I can sort of see how that works, you know, people being the way they are. But it's still a little weird to me, and I can't call it in advance, you know? I don't feel like I now know how the next 10 years of politics are going to play out and wouldn't be able to tell you, even if you told me, which AI breakthroughs there's going to be over that time span, if we even get 10 years, which, you know, people in the industry don't seem to think so, and maybe I should believe them about that. Let me throw another argument at you that I don't subscribe to myself, but I feel like maybe you would knock it down in an entertaining way.

44:15One of the most frequent emails that we have gotten since we started talking about AI is from people who say that AI doomerism is just hype that serves only to benefit the AI companies themselves, and they use that as a reason to dismiss existential risk. How do you talk to those folks? it's historically false we were around before there were any ai companies to to be to of this of this class to be hyped so leaving aside the objection it is false um what's what is this like leaded gasoline can't possibly be be a problem because this is just hyped by the gasoline companies like nuclear weapons are just like hype from the from the from the nuclear power industries that their power plants will seem more cool like what what matter of deranged conspiracy theory is this it may possibly be an unpleasant fact that humanity being as completely nutball wacko as we are that if if you say that a technology is going to destroy the world it will raise the stock prices of the companies that are that are bringing about the end of the world because a bunch of people think that's cool, so they buy the stock.

45:28Like, okay, but that has nothing to do with whether the stuff can actually kill you or not, right? Like, it could be the case that the existence of nuclear weapons raises the stock price of the worst company in the world, and it wouldn't affect any of the nuclear physics that caused nuclear weapons to be capable of killing you. This is not a science-level argument. It just doesn't address the science at all. Yeah. Yeah. Well, let's maybe try to end this first part of the conversation on a note of optimism. You have spent two decades building a very detailed model of why doom may be in our future.

46:08If you had to articulate why you might be wrong, what is the strongest case you could make? Are there any things that could happen that would sort of make your predictions not come true? so like the current ai's are not understandable ill-controlled that the technology is not conducive to understanding or controlling them all the all the people trying to do this are going far uphill they are vastly behind the rate of progress and capabilities like what what does it take to believe that an alchemist can actually successfully concoct you an immortality potion it's not that immortality potions are impossible in principle with sufficiently advanced biotechnology, you could do it.

46:48But in the medieval world, what are you supposed to see to make you believe that the guy's going to have an immortality potion for you, short of him actually pulling that off in real life? No amount of, like, look at how I melted this gold is going to get you to expecting the guy to transmute lead into gold until he actually pulls that off. And it's like some kind of AI breakthrough, which doesn't raise capabilities to the point where it ends the world, but suddenly the AI's thought processes are completely understandable and completely controllable, and there's none of these issues. People can specify exactly what the AI wants in super fine detail and get what they want every time.

47:25They can read the AI's thoughts, and there's no sign whatsoever that the AI is plotting against you. Then the AI lays out this compact control scheme for building the AI that's going to give you the immortality potion. We're just so far off. You're asking me that there isn't some kind of like clever little objection that can be cleverly refuted here. This is something that is just like way the heck out of reach as soon as you try to think about it seriously. What does it actually take to build the superintelligence? What does it actually take to control it? What does it take to have that not go wrong on the first serious load when the thing is smarter than you, when you're into the regime where failures will kill you and therefore are not observable anymore because you're dead.

48:09You don't get to observe it. Like, what does it take to do that in real life? There isn't some kind of cute experimental result we can see tomorrow that makes this go well. All right. Well, for the record, I did try to end this segment on a note of optimism, but I appreciate that your feelings are deeply held. It's not really on the menu here today, Casey, but I admire you trying. Well, let's take a break. And when we come back, we'll have more with Eliezer Yadkowsky.

48:46Thank you.

49:20Okay, so we are back with Eliezer Yudkowsky, and I want to talk now about some of the solutions that you see here. If we are all doomed to die if and when the AI industry builds a super intelligent AI system, what do you believe could stop that? Maybe run me through your basic proposal for what we can do to avert the apocalypse? So the materials for building the apocalypse are not all that easy to make at home. There is this one company called ASML that makes the critical set of machines that get used in all of the chip factories. And to grow an AI, you currently need a bunch of very expensive chips.

50:08They are custom chips built especially for growing AIs. They need to all be located in the same building so that they can talk to each other because that's what the current algorithms require. You have to build a data center. The data center uses a bunch of electricity. If this were illegal to do outside of supervision, it would not be that easy to hide. There are a bunch of differences, but nonetheless, the obvious analogy is nuclear proliferation and deploriferation. where it's back when nuclear weapons were first invented, a bunch of people predicted that every major country was going to build a massive nuclear fleet.

50:47And then the first time there was a flashpoint, there was going to be a global nuclear war. And this is not because they enjoyed being pessimistic. If you look at world history up to World War I, World War II, they had some reasons to be concerned. But we nonetheless managed to back off. And part of that is because it's not that easy to refine nuclear materials. The plants that do it are known and controlled. And when a new country tries to build it, it's a big international deal. I don't quite want to needlessly drench myself with current political controversies, but the point is you can't build a nuclear weapon in your backyard, and that is part of why the human species is currently still around.

51:27Well, at least with the current technology, You can't further escalate AI capabilities very far in your backyard. You can escalate them a little in your backyard, but not a lot. So it would be, just to sort of finish the comparison to nuclear proliferation here, it would be an immediate sort of moratorium on powerful AI development, along with kind of an international nuclear-style agreement between nations that would make it illegal to build data centers capable of advancing the state of the art with AI. Am I hearing that right? All the AI chips go to data centers. All the data centers are under an international supervisory regime.

52:08And the thing I would recommend to that regime is to say, like, just stop escalating AI capabilities any further. We don't know when we will get into trouble. It is possible that we can take the next step up the ladder and not die. It's possible we can take three steps up the ladder and not die. We don't actually know. So we got to stop somewhere. Let's stop here. That's what I would tell them. And what do you do if a nation goes rogue and decides to build its own data centers and fill them with powerful chips and start training their own superhuman AI models? How do you handle that? then that is a more serious matter than a nation refining nuclear materials with which they could build a small number of nuclear weapons this is not like having like five fission bombs to deter other nations this is a threat of global extinction to every country on the globe so you have your diplomats say stop that um or else we in terror of our lives and the lives of our children will be forced to launch a conventional strike on your data center.

53:09And then if they keep on building the data center, you launch a conventional strike on their data center because you would rather not run a risk of everybody on the planet dying. It seems kind of straightforward in a certain sense. And in a world where this came to pass, do you envision work on AI or AI-like technologies being allowed to continue in any way, or we've just decided this is a dead end for humanity, our tech companies will have to work on something else? I think it would be extremely sensible for humanity to declare that we should all just back off. Now, I personally, I look at this and I think I see some ways that you could build relatively safer systems with narrower capabilities that were just learning about medicine and didn't quite know that humans were out there the way that current large language models are trained on the entire internet.

54:05They know that humans are out there and they talk to people and they can manipulate some people psychologically, if not others, as far as we know. So I have to be careful to distinguish my statements of factual prediction from my policy proposals. And I can say in a very firm way, if you escalate up to superintelligence, you will die. But then if you're like, well, if we try to train some AI systems just on medical stuff and not expose them to any material that teaches them about human psychology, could we get some work out of those without everybody dying. I cannot say no firmly. So now we have a policy question.

54:53Are you going to believe me when I say, I can't tell if this thing will kill you? Or are you going to believe somebody else who says this thing will definitely not kill you? Are you going to believe a third person who's like, yeah, I think this medical system is for sure going to kill you? Who do you believe here if you're not just going to back off of everything? So backing off of everything would be pretty sensible and trying to build narrow medically specialized systems that are not very much deep smarter than the current systems and aren't being told that humans exist and they're just thinking about net medicine in this very narrow way and you're not just going to keep pushing them until it explodes in your face you're just going to like try to get some cancer cures out of it and that's it, you could maybe get away with that.

55:43I can't actually say you're doomed for sure if you played it very cautiously. If you put the current crop of complete disaster monkeys in charge, they may manage to kill you. They just do so much worse than they need to do. They're just so cavalier about it. We didn't need to have a bunch of AIs driving people insane. You can train a smaller AI to look at the conversations and tell, is this AI currently in the process of taking a vulnerable person and driving them crazy. They could have detected it earlier. They could have tried to solve it earlier. So if you have these completely cavalier disaster monkeys trying to run the medical AI project, they may manage to kill you.

56:20Okay, so now you have to decide, do you trust these guys? And that's the core dilemma there. I have to say, Eliezer, I think there is essentially zero chance of this happening, at least in today's political climate. I look at what's going on in Washington today. You've got, you know, the Trump administration wants to accelerate AI development. NVIDIA and its lobbyists are going around Washington blaming AI doomers for trying to cut off chip sales to China. There seems to be a sort of concerted effort not to clamp down on AI, but to make it go faster. So I just look around the political climate today, and I don't see a lot of openings for a stop AI movement.

57:03But like, what do you think would have to happen in order for that to change? From my perspective, there's a, you know, sort of core factual truth here, which is if you build super intelligence, then it kills you. And the question is just like, do people come to apprehend this thing that happens to be true? It is not in the interest of the leaders of China, nor of Russia, nor of the UK, nor of the United States, to die along with their families. It's not actually in their interest. That's kind of the core reason why we haven't had a nuclear war, despite all the people who, in 1950, were like, how on earth are we not going to have a nuclear war?

57:49What country is going to turn down the military benefits of having their own nuclear weapons? How are you not going to have somebody who's like, yeah, I've got some nuclear weapons, Let me take this like little area of border country here, the same way that things have been playing out for centuries and millennia on Earth before then. But they also had like – there were nuclear weapons dropped during World War II in Japan. And so people could look at that and see the chaos it caused and point to that and say, well, that's the outcome here. In your book, you make a different World War II analogy.

58:21You sort of compare the required effort to stop AI to the mobilization for World War II. But that was a reaction to like a clear act of war. And so I guess I'm wondering, like, what is the equivalent of the invasion of Poland or the bombs dropping on Hiroshima and Nagasaki for AI? What is the thing that is going to spur people to pay attention? I don't know. I think that OpenAI was caught flat-footed when they first published ChatGPT, and that caused a massive shift in public opinion. I don't think OpenAI predicted that. I didn't predict it. it could be that any number of potential events cause a shift in public opinion.

59:00We are currently getting congresspeople writing pointed questions in the wake of the release of an internal document at Meta, which has what they call a superintelligence lab, although I don't think they know what that word means, where it's like their internal guidelines for acceptable behavior for the AI. And it says, well, if you have an 11-year-old trying to flirt, flirt back, And everyone was like, what the actual censored profanity meta? What could you possibly have been thinking?

59:32Why from your own perspective did you write this down in a document? Even if you thought that was cool, you shouldn't have written it down because now they're going to be pointed questions, and there were. and you know maybe it's something that from my perspective doesn't kill a bunch of people but still causes pointed questions to be asked or maybe there's some actual kind of catastrophe that we don't just manage to frog boil ourselves into um you know like losing massive numbers of kids to their ai girlfriends and ai boyfriends might is is from my perspective and an obvious sort of guess but even the most obvious sort of guess there is is is still not higher than 50%.

1:00:13And I don't think I want to wait. Maybe chat GPT, you know, from my perspective, maybe chat GPT was it, right? I was out, you know, I'm off in the wilderness. Nobody's paying attention to these issues at all because they think that it'll only happen in 20 years in 2005. And that to them means the same thing as never. And then I got like the chat GPT moment and suddenly people realize this stuff is actually going to happen to them. And that happened before the end of the world. Great. I got a miracle. I'm not going to sit around waiting for a second miracle. If I get a second miracle, great. But meanwhile, you got to put your boots on the ground.

1:00:47You got to get out there. You got to do what you can. It strikes me that an asset that you have as you try to advance this idea is that a lot of people really do hate AI, right? Like, if you go on Blue Sky, you will see these people talking a lot about all of the different reasons that they hate AI. At the same time, they seem to be somewhat dismissive of the technology, They have not crossed the chasm from I hate it because I think it's stupid and it sucks to I hate it because I think it is quite dangerous. I wonder if you have thoughts on that group of folks and if you feel like or would want them to be part of a coalition that you're building.

1:01:29Yeah. So you don't want to make the coalition too narrow. I'm not a fan of Vladimir Putin, but I would not on that basis kick him out of the, how about if humanity lives instead of dies coalition? What about people who think that AI is never going to be a threat to all humanity, but they're worried that it's going to take our jobs? Do they get to be in the coalition? Well, I think you've got to be careful because they believe different things about the world than you do. And you don't want these people running the how about if humanity does not die coalition. You want them to be in some sense like external allies because they're not there to prevent humanity from dying.

1:02:15And if they get to make policy, maybe they're like, eh, well, you know, this policy would potentially allow AIs to kill everyone according to those wacky people who think that AI will be more powerful tomorrow than it is today. But, you know, in the meanwhile, it prevents them from taking the AIs from taking our jobs. And that's the part we care about. So, you know, like there's this one thing that the coalition is about. And that's it. It's just about not going extinct. Yeah. Eliezer, right now, as we're speaking, I believe there are hunger strikes going on in front of a couple of AI headquarters, including Anthropic and Google DeepMind.

1:02:52These are people who want to convince these companies to shut down AI. We've also seen some potentially violent threats made against some of these labs. And I guess I'm wondering if you worry about people committing extreme acts, be they violent or nonviolent, based on your lessons from this book. I mean, taking some of your arguments to the natural logical conclusions of if anyone builds this, everyone dies, I can see people rationalizing violence on that basis against some of the employees at these labs. And I worry about that. So what can you say about the sort of limits of your approach and what you want people to do when they hear what you're saying?

1:03:39Boy, there sure are a bunch of questions bundled together there. And so the number one thing I would say is that if you commit acts of individual violence against individual researchers at an individual AI lab in your individual country, this will not prevent everyone from dying. The problem with this logic is not that by this act of individual violence, you can save humanity, but you shouldn't do that because that would be deontologically prohibited. I'll just say it that way. The problem is you cannot save humanity by the feudal spasms of individual violence. It's an international issue. You can be killed by a superintelligence that somebody built on the other side of the planet.

1:04:27I do in my personal politics tend to be libertarian. If something is just going to kill you and your voluntary customers, it's not a global issue the same way. If it's just going to kill people standing next to you, different cities can make different laws about it. If it's going to kill people on the other side of the planet, that's when the international treaties come in. And a futile act of individual violence against an individual researcher and an individual AI company is probably making that international treaty less likely rather than more likely. And there's an underlying truth of moral philosophy here, which is that a bunch of our reason for our prejudice against individual murders is because of a very systematic and deep sense in which individual murders tend to not solve society's problems.

1:05:20and this is from my perspective a whole bunch of the point of having a taboo against individual murder it's not that people go around committing individual murders and then the world actually gets way better and all the social problems are actually solved and but we don't want to do that we don't want to do more of that because murder is wrong the murders make things worse and that's why we properly should have a taboo against it yeah we need international treaties here What do you make of the opposition movement to the movement that you're sketching out here? Mark Andreessen, the powerful venture capitalist, very influential in today's Trump administration, has written about the views that you and others hold that he thinks are unscientific.

1:06:05He thinks that AI risk has turned into an apocalypse cult. And he says that their extreme beliefs should not determine the future of laws and society. So I guess I'm interested in your sort of reaction to that quote specifically, but I also wonder how you plan to engage with the people on the other side of this argument. Well, it is not uncommon in the history of science for the cigarette companies to smoke their own tobacco. the inventor of leaded gasoline who was a great advocate of the safety of leaded gasoline despite the many reasons why he should have known better um i think like did actually get sufficient cumulative lead exposure himself that he you know got had to go off to a sanitarium for a few years and then came back and started exposing himself to lead again and again got sick again and so sometimes these people to truly do believe they're they're they're they they do drink their own kool even to the point of death.

1:07:07It shows history. And perhaps Marc Andreessen will continue to drink his own Kool-Aid even to the point of death. And if he were just killing himself, that would be one thing I say as a libertarian, but he's unfortunately also going to kill you. And the thing I would say to sort of refute the central argument is, what's the plan? What's the design for this bridge that is going to hold up when the whole weight of the entire human species has to march across it? where is where's the design scheme for this airplane to which we are going to load the entire human species into its cargo hold and fly it and not crash what's what's the plan where's the science what's the technology why is it not working already and they they just don't they can't make the case for this stuff being you know like not perfectly safe but even remotely safe under that they're going to be able to control their super intelligence at all So they go into these like, you must not listen to these dangerous apocalyptic people because they cannot engage with us on the field of the technical arguments.

1:08:12They know they will be routed. You have advice in your book for journalists, politicians who are worried about some of the catastrophes you see coming. For people who are not in any of those categories, for our listeners who are just out there living their daily lives, maybe using ChatGPT for something helpful in their daily life, what can they do if they're worried about where all this is heading? well as of a year ago i'd have said you know again it's you know right right to your elected representatives uh talk to your friends about you know being ready to vote that way if if if they if they disputed primary election comes down that way um the ask i would say is for our leaders to begin by saying we are open to a worldwide ai control treaty if others are open to the same like we are ready to back off if other countries back off.

1:09:10We are ready to participate in an international treaty about this. Because if you've got multiple leaders of great power saying that, well, maybe there can be a treaty. So that's kind of the next step from there. That's the political goal we have. If you're having trouble sleeping and if you're generally in a distressed state, maybe don't talk to some of the modern AI systems because they might drive you crazy, is a thing I would say now. I didn't have to say that one year earlier. you know like this the whole ai boyfriend ai girlfriend thing might not be good for you maybe don't go down that road even if you're lonely i you know but that that's individual advice that's not going to protect the planet yeah well i'll um end this conversation where i've ended um some of our earlier conversations eliezer which is i really appreciate the time and i really hope you're wrong like that would be great we all hope i'm wrong i hope i'm wrong my friends hope i'm wrong every everybody hopes i'm wrong hope uh is does not you know hope is not what saves us in the end action is what saves us hope is not you know hoping for miracles hoping you know you blooded gasoline you can't just hope that leaded gasoline isn't going to poison people you actually got to ban the leaded gasoline.

1:10:27So more active hopes. I'm in favor of like, I see the hope, I share the hope, but let's look for more activist hopes than that. Yeah. Well, the book is If Anyone Builds It, Everyone Dies, Why Superhuman AI Would Kill Us All, and it is coming out soon. And it is a co-written book by Eliezer and his co-author Nate Soares. Eliezer, thank you. Thanks, Eliezer. Thank you. That's one.

1:11:03Thank you.

1:11:25Hard Fork is produced by Whitney Jones and Rachel Cohn. We're edited by Jen Poyant. We're fact-checked this week by Will Peischel. Today's show was engineered by Katie McMurrin. Original music by Rowan Nemisto, Alyssa Moxley, and Dan Powell. Video production by Sawyer Roquet, Pat Gunther, Jake Nickel, and Chris Schott. You can watch this full episode on YouTube at youtube.com slash hard fork. Special thanks to Paula Schumann, Hui Wing Tam, Dahlia Haddad, and Jeffrey Miranda. As always, you can email us at hardfork at nytimes.com. Send us your plans for the AI apocalypse.

1:12:23Thank you.