EP 63: Eliezer Yudkowsky (AI Safety Expert) Explains How AI Could Destroy Humanity

29:04He wasn't saying that physics broke down. He was saying that the model of physics broke down. It's not named after discontinuity in reality. It's named after discontinuity in the map. And this to me was just like sort of obviously the most important thing going on the moment I read it. Like just as soon as I read the paragraph, I was like, yeah, okay, that's what I'm doing with the rest of my life. This is where the important stuff is going down. Anything smarter than human. And so that was at 16 years old. And then after that, so in your, was it late teens or early 20s, you started the Singularity Institute for Artificial Intelligence?

29:40Age 20, year 2000, yeah. And your goal was to usher in a good AI? No. As a young teenager, I did not realize that AIs had to be made good. I had some whole weird line of logic about how they would automatically be good. Do you recall that logic? Oh, yeah. it was like, well, either life has meaning or it doesn't. If life has meaning, then something much smarter than us will figure it out and do the thing that's correct. So that's the correct thing to do is build something smarter than us. And then any problem which isn't that problem isn't our problem to solve. Or alternatively, life is meaningless, but then everything we do is equally pointless.

30:20So you can just eliminate that from consideration, is what teenage Eliezer wrote up in a page called Frequently Asked Questions About the Meaning of Life, which for a while was Ask Jesus' official answer to what is the meaning of life. Wow. And so, I mean, is it fair to characterize you at that time as something of an AI accelerationist? I mean, in modern terms, one might call that, though. I was also able to break out of that on account of not very much resembling the current accelerationists very much personality-wise in some important ways. But sure, like in terms of the rhetoric, it was like current accelerationist rhetoric, but brighter, more cheerful, better, less nihilistic, that sort of thing.

31:08Yeah. I mean, Sam Altman recently tweeted that you may have done more to accelerate AGI than anyone else. And I think he meant that as a compliment in that case. Somewhat doubtful. Yeah. He did say maybe you won the Nobel Peace Prize for it. That's kind of implausible. I would be surprised if that happens. And somebody should start a prediction market on that so I'd be able to cite the prediction market. That it's not going to happen. And if it does happen, in the unlikely event that it does happen, I would rather expect it to be more for raising the alarm and getting it successfully shut down.

31:44Not for accelerating it. Yeah. So your vision at that time, we talked a little bit about elements of this, but what was your vision for good AI singularity? Like as a teenager? Yeah, when you started the institute. I mean, I didn't until age 21 figure out that alignment was going to be a thing. Maybe describe alignment for people that don't. Alignment is trying to shape an AI such that its effect upon the world when you run it is net positive from the perspective of humans, most humans, idealized human preferences after updating to match the AI's knowledge. You know, not what you currently want, but what you would want if you knew everything the AI knew.

32:33There's a relatively shallow rabbit hole to go down there. But, you know, if you just say, like, it's making guys do what humans want. Well, that's obviously, you know, like, what do you mean humans want? You mean what humans vote on? This is obviously not going to end well for anyone. Yeah. So not quite that naive. Yes, yes. We'll come back to alignment. So what was your vision then? And when you did realize alignment was a thing, what was your initial vision for singularity around it? It took an embarrassing long time for me to realize the full magnitude of the issue. I'm not sure embarrassing.

33:06I think you might be beating too hard on yourself given how long ago it was. Age 21 to age 23? I think you're beating too hard on yourself. Okay. It's okay to be imperfect, but not so imperfect that other people notice is my model. Yeah. So I think in this case, like, I can look back and see the particular ways in which I was too slow to face up to the impact of the arguments in which I was clutching on two beliefs I'd previously held and not just, like, immediately throwing everything out the window in light of a new consideration and recalculating it from scratch. I would aspire to do a lot better nowadays.

33:38So when did you determine that AI was likely to be orthogonal to humans' objectives? I mean, that's not quite the way I would put it. How would you put it? So, like, the way I would have put it in 2001 is, you know, maybe AI doesn't end up automatically doing what's the right thing. How would you go about building an AI to do the right thing? How would you define the right thing? How would you shape an AI to do that? And not really admitting at the time that this was, like, clearly necessary work, just trying to, like, figure out how you would do it just in case. So that's, like, age 21. And then age 23 is actually...

34:20What year is 21, just so we're... 2001. 2001. And then age 23, 2003, is I think the point at which I had what I would now call like Bayesian enlightenment of like seeing the world through the lenses of probability and expected utility and be like, oh, my entire previous theory was utterly completely wrong and would have clearly actually just immediately gotten everyone killed if I tried doing anything remotely like that way. I was entirely wrong here. This entire line of reasoning I was following was terrible. I was trying to solve the wrong problem using the wrong methods, using a poor model of reality derived from bad thinking, and unfortunately all of my mistakes did not cancel out.

35:02The only thing I did that was at all correct there was being like, well, if there's even a tiny chance, haha, that you've got to actually do any work for two-line AIs, you know, like my duties toward the people funding me, the particular funder of the Singularity Institute, as it was then called, you know, like require me to chase down the small possibility and understand it and figure out like how I'd cover the special case of maybe possibly an AI needing to not be aligned. And because I chased down that tiny probability, not a way of thinking I currently endorse, but back then it's what saved me.

35:37That actually stayed in contact with the problem, that I thought about how to solve it instead of finding out some wacky reason to write it off. It's like staying in contact with the problem, asked myself, how would I technically solve this? Tried to figure out the rules and kept thinking in that sort of like, how do I actually solve this technically, not socially? How do I solve this technically? Until the pile of evidence and argument mounted up inside the mind and inside my mind to the point where it could collapse my previous crazy ideas. How'd you get funding to go pursue this? Well, I mean, initially it was funding to charge right out and build an AI.

36:19How'd you find someone that was willing to pay you to go do that? Basically a dot-com millionaire who was on the same mailing list as I was, the Extropium's mailing list, was like, so this self-improving AI that you are talking about as a teenager, Could you actually go build that? No, I said. That would be, like, too hard to do. What if we, like, started this wacky business idea instead? So we, like, spent six months working on a business plan there, and we figured out at around the same time that it just wasn't going to work out. I was like, okay, you know, like, let's actually just have the nonprofit institute devoting to charging right out and building the AI sooner rather than later.

37:00150 ,000 people die every day around the world. 55 million people per year die. if there was something super intelligent around, maybe it can save them. Building this thing faster, sooner is a huge moral imperative. Let's charge out and start building self-improving AI is the premise of it in late 99, early 2000. And in 2001, after having spent a bunch of time thinking about it full time, I was like, okay, well, what if it's not automatically good? but how would you go about building it from the ground up such that it would be good in a case like that and coming into contact with that problem and spending a bunch of time staring at it?

37:43Had I instead gone down the road of like, oh no, well, if you can shape AIs, that means the wrong people might get them. And gone down the pathway of talking all the time about how the wrong monkey might get this banana. I would have never figured stuff out. I would have not stayed in contact with the parts of the problem that would have told me I was wrong in my basic premises. So 2007 to 2014 was kind of your rationality era? Well, writing about it. Writing about it. Before that, you were thinking about it. 2003 is, as I would say, the era of figuring out that sort of stuff. And then 2007 and on was the period of writing about it.

38:26My understanding is that your goal through this period of time was that you were trying to prepare civilization to think about different levels of risk, and in particular with regard to AI potentially? Is that an over extrapolation or is that fair? I think it views me as a more cunning, 4D chess playing backward chaining sort of person, whereas I would usually try to do things that might have good effects down many different routes. It's hard to do cause and effect of which one can't, Did you find out rationality because AI or was AI rationality? It's hard to divorce the two. I mean, yeah. Well, for one thing, at least the way I was doing it, they're the same discipline.

39:08Yeah. They have the same basis in cognitive science, evolutionary biology, evolutionary psychology, et cetera, et cetera. Same math. Not quite so much now where it's all the math of putting an enormous pile of floating-point numbers and subjecting it to a couple minutes of calculus to do several months of gradient descent. And yeah, so from my perspective, anytime I tried to have a conversation about artificial intelligence, it would start to, from my perspective, it would founder on people not knowing about stuff like the conjunction fallacy, where the conjunction fallacy is if you give people like, how likely is it that A causes B?

39:50Versus like, how likely is it that B happens for any reason? people often say that the compound event is more is more probable than other people assigned to the simple event you know like what is the probability that reagan will repeal welfare versus like the probability that welfare gets repealed i'm not doing this exactly right but that was like one of the questions from way back when when monald reagan was president so you know i from my perspective people were never getting to the end of the whole argument cause it some step along the way, they'd make a particular sort of error. So I was like, okay, like if I want people to follow this entire argument, I'm going to have to teach them how to do all of these argument steps locally validly.

40:38What is the valid rule of reasoning on this particular step? And yeah, just like gave up on winging it and just started doing all the prerequisites, the background math, the background cognitive science, all the stuff you would need to under, you know, the background evolutionary biology, all the stuff you'd need to actually understand the argument on AI, which had a huge amount of overlap on what you would need in order to do good reasoning in everyday life, from my perspective. And, you know, the point of that wasn't just AI, it was also like human reasoning, one step forward. There was a vision there of we have all this wonderful cognitive science that didn't exist a few decades ago.

41:23Can humanity actually use this stuff? Can we become saner? And I think that there was a vision there that failed, basically. But it was one of the things I was trying to pursue. I did not know that all this was going to go down in the 2020s. Now, from my perspective, you know, in the 2000s, 2040 was a plausible amount of time for stuff to take. And, you know, there was a chance that, like maybe humanity also was sort of like a more hopeful time, right? You know, like it wasn't clear yet that civilization sanity was going to go downhill because of Twitter. You know, for all I knew at the time, we were just going to get like saner over time instead of being driven mad by social media.

42:05So I was, you know, like trying to contribute to that, to contribute to this common human project of building a base of sanity for future generations by taking all this cognitive science that had come out and trying to apply it to everyday life. That was the vision. I don't really think it worked all that well. So you wrote the less wrong sequences, also known as rationality from AI to zombies. And you also wrote Harry Potter and the Methods of Rationality, which is the most popular Harry Potter fan fiction book ever, written, I think, by some measures. Depends which measures use you, yes. Yeah, but people don't realize exactly how popular of a category Harry Potter fan fiction is.

42:42I guess there are apparently 500 ,000 or something other Harry Potter fan fictions that are all... There were the last time I ran the statistic on that. Yeah, got it. But I expect that by now the number is substantially greater. And so the point of rationality and less wrong and Harry Potter, and I'm sure Harry Potter was just fun as well. But the point of all this was you were trying to help people, teach people how to think or show people how you thought so that they didn't need to reinvent all of these things on their own. Is that fair? Yeah. Like a whole bunch of the motivation from my perspective is looking back at my own history and thinking like, well, those steps took way too long.

43:24If there are young Eliezers out there, how can I package up all the information that they need to grow up so they can start working sooner? Get all the skills when they're 16 instead of 26. Get started on their scientific career earlier. And that didn't work either. I think I underestimated how sparse the immediate neighborhood is in high-dimensional space of people. You know, Steve Jobs never found anyone to take over Apple who was up to his caliber of making neat things and having nice user interfaces. And he had all the money in the world with which to pay them and, you know, somehow couldn't manage to really replace himself anyways.

44:05And, you know, that is the thing that I remind myself of when I feel bad about failing to, you know, have already yielded my place to three youngsters who are all better at it than I am. Didn't really happen. So you were hoping that you would find more smart people to step forward and learn from all of this. Was it with AI as an end goal or was it just continuing to extend the general thought of rationality? Yeah, I mean, I wanted to entirely replace myself, right? I just didn't figure I was the best person for my job. And what was your job as you describe it here? I mean, AI was a large part of it, but I also seemed, from my perspective, to be holding the ball on learning from all the recent discoveries and synthesizing into something that people could use to become saner.

44:54And I don't really feel, and I feel like I failed to hand that ball off either. Also, there's like very narrowly overlapping things, like there's the shape, there's understanding the shape that computing power has to be very effective at understanding reality and planning. And then there's the understanding the shape that humans should have to understand, become more effective at understanding reality planning. It's very tightly related there. Did anyone come close? Did you have any mentees or anyone that you felt like was along the path during that period of time that you thought was going to be able to take over for you?

45:27I think there's other people who've been able to take up corners of the burden. But no one who really struck me as being like, ah, yes, that's a replacement, me. Like the ball is no longer in my hands. I no longer hold this responsibility. I don't think there was a moment where I felt that. So in 2015, I'm going to describe it as the open AI moment. There was an AI safety conference in Puerto Rico. It was you, it was Elon Musk, it was Nick Bostrom, Max Tegmark, people from DeepMind, people from Google, a whole group there. And I think this was a pivotal moment for you because during the conference, it seemed like maybe Elon Musk was going to wake up to the AI risk possibility.

46:08and that there was going to be something like a Manhattan Project-style strategy where basically keeping all the work secret from the general public as they kind of figure out what to do with this? Is that fair? I mean, the general public isn't the problem. It's like literally anyone. The Highlander principle, there should be at most one AI.

46:34That's among the sensible things that could have been tried, But mostly I was just like, oh, like, is this the moment where the billionaire shows up and starts taking it seriously? And there's like any effort at all. And it's not just this like tiny group of people who are weird enough to work on a problem, despite the lack of very much at all in the way of incentives to work on it. Despite the low pay in both money and status. You know, people work for low monetary pay if they're being paid in high status. But working on a weird problem that isn't well-paid and makes people look funny at you at parties, that's rare.

47:07The question was, is this the moment where humanity turns and starts fighting back? That was what I was hoping was going on there. And instead, Elon got inspired to start this organization called OpenAI, which was co-founded with Sam Altman with the idea that AI should be open to everyone, which is basically a terrible idea from your perspective. It assumes alignment as a solved problem, that power disparities between AIs are small. And so as long as everyone has their own obedient AI, everyone can successfully use that to protect themselves in the new world. And it was based on a model of how AI works that I think has not held up and I think is just utterly wrong.

47:55and it wasn't putting the question of how AI works as a question of fact at the center. It was just like, oh, I don't like these people's politics. This is a political problem. This is a problem of which monkey gets the banana. I don't like these people building AI. I don't think they're trustworthy enough. Only I am trustworthy enough to build AI. you know, the center, you know, didn't like Demis Asabas' deep mind, didn't like Google. And apparently that's a good reason to go destroy the world or something. So what was your feeling going into that meeting? It sounds like you were optimistic going into Puerto Rico.

48:36And then I guess, how did the outgrowth of OpenAI's founding change your mental framework around the likelihood of doom? I mean, from my perspective, it meant that humanity was going to just like plunge straight into this doing the worst thing and the wrong thing at every juncture. Just, you know, it was just going to play out like it did in history books. Read history books. They're not full of people doing the right thing every time. They're full of people making mistakes that typically appear in history books. I'm like, okay, so like we're going to hold ourselves to that standard of performance and that means we're probably dead.

49:07Were there specific decisions that OpenAI made at founding because the business model has evolved or there wasn't one initially and then there became one that led to this conclusion or did you feel that immediately coming out of the, once it was founded, you were already? The name, the name is sufficient. Some of us had put a lot of work over the years into trying to have an understanding among players the likes of DeepMind, which for a time was like the major player, that this was humanity's problem held in common. Everybody needed to live up to the trust that humanity wasn't so much placing in them as that they were just taking upon themselves.

49:50Go, try to go slow, be cautious.

49:56Or I should say rather that the point of getting any sort of lead was to burn that lead in order to have more time to spend aligning things. have it go well, and sort of like trying to create this atmosphere not in arms race, of having a common human project that people need to go in on together. And then OpenAI is like, OpenAI versus Google, right? Just like come in and just like completely trash all of that. And maybe it was a small pathetic thing to have tried. Maybe it was foolish to think that humanity could ever not do the stupidest thing, that there was ever any chance of having it not be an arms race.

50:38That, of course, you know, that as soon as there was any real money or power at stake, people were going to just, like, throw all thoughts of cooperation out the window and grab for whatever immediately came into their grasp. But, you know, it's not the sort of thing you forget. That when there's a little scrap of hope for it, however contemptuous, somebody more cynical might find it to think that you wouldn't just do the worst thing. Whoever actually comes in and actually comes down and crushes the ideal, the people who held that ideal for a little while are not going to forget that anytime soon.

51:16You've referenced monkeys and bananas, and I've heard you refer to this behavior as disaster monkeys. Can you describe what that is in your mind? I mean, there's a lot of different people following short-term incentive gradients, which isn't so much money as like what gets people to give you interested looks at parties in San Francisco um you know like why why publish GPT-2 you know like like why why what like like why show it to why show it to the world you know instead of just like selling it as a back-end service if you needed to make money for some reason which you know itself is kind of questionable Why tell people how to build these things?

52:03And you can have elaborate stories about how it was totally a good idea to make timelines shorter that way, to get the hype started, to get the money rolling into the field. But from my perspective, what predicts the action is not, there is no, like, what is good for humanity here? We can start from that principle, extrapolate in a neutral way, and get what OpenAI did. What predicts it is the question of like, what gets admiring looks at parties in San Francisco? Or scared looks, or angry looks. It's all the same. If you get people riled up that way, that's power. And chasing after the status that comes with that power is the basic model that I have that, you know, explains what all these people are doing.

52:47There's a perspective, I think, that everyone kind of uses that better them than someone else. And you've alluded to this. It's almost like, I don't know if it's a messianic complex or something, but that there's a lot of bad people out there. I'm not one, and so I should have to know. It's the obvious story that you would grab if you were looking for a reason to publish the latest, greatest AI results to get you admiring looks in San Francisco parties. And now that people are sufficiently concerned, there's a chance that you can get some admiring looks by saying that you're not working on the next model now that you've scared them enough that you can get status, that people won't just give you eye rolls if you're claiming that you're not working on something.

53:31But yeah, I'm not really seeing the prediction here that follows from first principles, trying to benefit humanity. So 2015 and 2020, you were focused on AI safety research. Specifically, I think you wanted to basically build some kind of off button that we could install into a super intelligent AI. Correct me. Okay. So the question was, how do you get one bit, one information theoretic bit of information into an AI's utility function without the AI trying to stop you or trying to get that bit in? I'm going to stop you. Describe a utility function. The preferences, the thing the AI wants. The goal that the AI is seeking.

54:18Something like that, yes. The place it's trying to steer reality. So the question is something like, could you swap between two utility functions? Can you have an AI that pursues one thing until you press a button and then pursues a different thing instead? And for purposes of illustrating the underlying problem, you can imagine that one goal is, you know, like making paperclips, whatever. And the other goal is shut yourself down. Fold up what you're doing in an orderly fashion. Like, shut down the results of previous plans, but not in a way where you then take over the entire world to erase all evidence that you ever existed or something.

55:02And we didn't have a specification of a shutdown function. We're rather saying, like, suppose you could describe how to shut an AI down. What does the button look like? It was like the smallest possible change you might want to make, just switch from one utility function to another. And the issue was, you know, that's not super easy to describe. The issue is that what if, for example, shutting down is easier than what the AI was otherwise trying to do? What if it's easier to get like an A plus on shutting yourself down than get an A plus on making paperclips? Then you perhaps have an incentive to press your own shutdown button.

55:43Okay, you say give it less utility from shutting down. Then it wants to prevent you from pressing the shutdown button. So the question we were addressing is something like, is there a self-consistent view into the AI's preferences where it doesn't want you to press the button that swaps its utility function, where it doesn't want you to press the button that shuts it down, it doesn't want to prevent the button from being pressed, if it rewrites itself it will rewrite itself in a way where the button is still there

56:20if it thinks about its own operation it will not be alarmed by the button's existence but rather think that the button is supposed to be there a reflectively consistent meta-utility function that allowed for entering one bit of information that would switch between two object-level utility functions and the point of this is not necessarily that AI plays out in a way where you can pre-program the utility function that way. You can also imagine that you would have a modern deep learning based AI where you're trying to train it to let you shut it off and the thing we're looking at is is there a compact mathematical description of what a self-consistent coherent mind looks like when it will let you swap utility functions, where it will let you shut it off, where it will let you train it, where it will let you further specify the utility function and not try to stop you from doing that or take away the utility function modifier unit to give itself a utility function that's easy to modify.

57:26You can give a simple description of what that looks like. Then when you're training the AI, you could check. You'll first of all be trying to train something that was simple, so maybe it generalizes correctly, that fits in with the rest of it, so maybe it generalizes along with the intelligence. You cannot, without greatly improved technology, look in and see if it actually learned that thing, but you would know how to generate new, weird example cases to check if it was generalizing correctly. And this is all just meta on a level where I think a lot of people just flatly did not understand what we were trying to do and thought we were trying to program some AI whose logic was as simple as the logic we were checking to see if it described a reflectively consistent utility function that had a modification button on it.

58:18They were like, oh, good old-fashioned AI. You're trying to do AI with logic, right? And we're like, no, we're trying to find a structure that could be learned. We didn't phrase it as well at the time. Deep learning hadn't blown up to quite the same extent. But the thing we were trying to do was not like build a logical AI. It was understand the logic that you were trying to get into an AI, including maybe a deep learning-based AI. What were the most promising initiatives through that period? Did you feel like you got closer? As you got closer, it felt more fleeting? Not really. People proposed various stuff.

58:53Usually I shut it down. Sometimes somebody else shut it down. The fact that other people shut it down sometimes was encouraging. How big was the group you were working with on this? This is, I don't know, like maybe like a dozen people on the face of the whole planet, and only one of whom was anything like full-time-ish on it for a while, Stuart Armstrong. And so you get through that period and you realize that that problem isn't solvable or at least. Well, we failed to solve it. There's actually, like, somebody recently posted a thing about that to LessWrong, like, claiming to show an impossibility proof for it, which would be very useful if I could, you know, I haven't had a chance to do that stuff, haven't had the chance to really go through it properly.

59:37But if there's actually an impossibility proof, that will probably be very useful for figuring out how to do it. So last year on April Fool's Day, you wrote a post announcing that the Machine Intelligence Research Institute, which you founded, announced a strategy, Death with Dignity, which triggered a lot of criticism and backlash. Even Peter Thiel, who was one of the first supporters of your organization, criticized you for having a defeatist attitude, I guess. What led you to that point that Death with Dignity was the best that we could do as a society? On April 1st, April Fool's Day. Yes. Just having watched the capabilities continue to accelerate and alignment progress just being too slow.

1:00:21Like, it became clear we weren't going to solve it. Sure didn't look like anyone else was going to solve it. And yeah, like, we were just plunging into it headlong. The social framework was wrong. The technical research agendas were not getting, we're not going anywhere near fast enough to catch up to capabilities. So yeah, that was the like point. At some point you do have to warn people that the asteroid deflection efforts have failed and the asteroid is plunging headlong towards Earth. Was it purposely provocative to start a conversation with people or had you just sort of thrown your hands in the air and felt like this was the inevitable conclusion or both?

1:01:03It's not super easy to describe. I was like saying a bunch of stuff that one can say in that tone because it is April 1st with people having permission to disbelieve it because it's April 1st. And so you're not necessarily grabbing people by the throat and forcing them to believe things that might possibly destabilize them something. Somebody who really needs it to not be true can tell them that it was just an April Fool's joke. You know, like certain elements are an April Fool's joke. Was there something specifically that you had seen, like was ChatGPT a big breakthrough in your mind that you saw, gosh, this is progressing well beyond what I had seen in the past?

1:01:45I was hoping that the large language model stack more layers, just throw more compute and scaling at the problem. I was hoping that GPT-3 was about as far as it would go qualitatively. GPT-4, ChapGT and like more so GPT-4 being a qualitative improvement did cause me to be like, oh, well, that hope has been dashed. Like my internal model saying that there's like only so far you can get by just by stacking up more layers has now been falsified. And I don't know where we go from here. I want to get into the AI doom argument in general, but one of the important components of this, I think, is the concept of a fast takeoff.

1:02:24And I think something that people struggle with is that there aren't necessarily big warning signs that are going to come back to the fire alarm thing. We see chat GPT and it looks like this nice novel tool that we can use for a bunch of different things and think, how's that going to kill me, right? Can you talk about just the fast takeoff scenario? Well, these days I'd say rapid capability gain, which sounds a bit more like what the thing is, there are several ways that can play out. So one potential way is something that scales better than the current AI models, whereby I mean that the function, you know.

1:03:12So before transformer networks, there were RNNs. and although people are still tweaking RNNs trying to get them to catch up to transformers, the fact remains that if you put a, through a GPT-4 amount of compute at training an RNN instead of a transformer, it would not be able to do what GPT-4 can do or even I think what GPT-3 can do, maybe not even what GPT-2 can do. RNNs are just not that great, although people are still trying to tweak them, making them greater. so what if there's the next breakthrough after transformers and the next breakthrough after transformers has a different scaling function one version of the story is just like it's better than transformers when you throw an amount of compute in it that suffices to turn a transformer into GPT-4 this thing just goes straight to GPT-9 because it's just as much better than transformers as transformers are better than RNNs and we already have this huge hardware overhang in the big GPU centers, and people try and throw billions of dollars at this stuff by the time it comes along.

1:04:22So that's one version of the story. The other version of the story is, you know, like, they make GPT-5 that way, but it's got, like, a different scaling parameter, so GPT-5 makes them a ton of money, and then they throw 10 times as much compute about it, and that gets you to GPT-9.

1:04:38Gwern has this lovely statement about... Who is Goren? Goren Branwen. Yep. Goren on Twitter. Yes. Account locked, so most people cannot see his great wisdom. But he's like one of the people who I would say called the way this whole played it, the way the whole deep learning revolution played out better than I did, for example.

1:05:02And he's like, we know when AlphaZero was as good at Go as humans. And it's some date I forget. It's like 4.30 in the morning on April 26th in a server room on DeepMind's campus. He knew the specifics because he's born. But the point is that it was human equivalent at Go for not very long at all, over the course of the three days of training, to go from zero to better than any human at Go. And yeah, the human level of competence is not all that distinguished. It's distinguished by being as dumb as you can possibly be and still build a computer. If it was possible to build computers while being dumber, we'd be having this conversation at that level of intelligence.

1:05:51There's nothing about the human level of ability at chess or at Go, where if you build an AI and train to get better and better at chess and Go, it gets to the human level and hits some kind of bottleneck and stops. If it's training as fast as Alpha Zero, it is like three days. It's kind of an important point about how smart humans are and the limitation on human intelligence. We have... Anyway, that's just one of the ways you can get rapid capability gains. Another one is getting to the point of being able to take over all the poorly defending computing power on the Internet. One of them is learning to write better AI code.

1:06:25And then that AI writing an even smarter AI. So there's multiple ways you can get rapid capability gain. But that said, the current social situation is one where even if it goes slower than that, people will just keep going. They'll be like, oh, well, if I don't build as somebody else will, we'll just keep going. Even if it's slower than that, they'll still just keep going. And you still end up with things much smarter than us and we all still die. Well, I actually want to bring up because you brought, you sort of alluded to this. This was a question I wanted to ask. I want to paint a concrete scenario.

1:06:53And I know there are many different scenarios of fast takeoff or way things can play out. But we have a scenario where the AI somehow takes over the world and starts tolling the galaxy with tiny molecular spirals shaped like paperclips, which we've talked in the past. But I think that is a little science fiction-y for some people to internalize. So I want to talk through a scenario that's a little less... Plausible. A little more plausible to the average person, which is a scenario where AI doesn't necessarily take over the world with robots and biotech and nanotech, but instead it takes over the internet, basically a computer virus.

1:07:31Do you think it's likely that an early superintelligent AI could rapidly spread to billions of computers on the internet and kind of take them over and be impossible to kill? Why do I care? That sounds like the kind of disaster where there are survivors. Do you? Yes, I assume on the Maslow's hierarchy of caring, you care more about people living. But can you talk through how likely that scenario could potentially be with superintelligence? I think this is like the 11th century like somebody in the 11th century like they're about to be invaded by a time portal that opened to 21st century Russia and they're like well don't tell us these science fiction stories about guns tell us like how they would defeat us with spears

1:08:17and if we have the clear understanding that what we are talking about is lower bounds on how badly the 21st century loses how badly the 11th century loses the 21st century, how badly wolves used to lose to human beings that have had time to prepare and plan, then sure, you could have a bunch of code on the Internet which has flaws, which something smarter than you knows about, and the humans don't, or they knew but didn't patch it. if we're talking about dinky possibilities like just taking over the whole internet at that level of the AI still being that stupid there actually are plausible stories you could tell about how that could not be inevitable for example maybe there's an earlier version of the AI that maybe the AI that can spot all the holes in the code is developed at DeepMind and DeepMind is responsible about it And instead of just releasing it to the Internet for anybody to use, they try to scan all the code they can find, including code for which they don't have the original source code and the AI decompiles it, and find all the bugs on the Internet that an AI could use to take over the Internet at that level of AI and send out corrections.

1:09:40or even, though this is maybe a bit beyond the range of what Google would legally do, like maybe somebody drops, maybe OpenAI, well, somebody drops the USB stick containing a copy of the AI that can find holes in all the code and somebody picks up that USB stick and goes home and accidentally fixes all the code. Just have the AI hack into all of those systems that it can detect breaks in for purposes of fixing it before any other AI can take it over. In a case like that, this ability potentially appears early enough to appear in an AI that is not strategic, that will actually do the thing it's pointed at by its loss function and the way it was trained.

1:10:29Can you define loss function? Loss function is the thing where you're applying gradient descent to make the AI better and better at doing that thing. So GPT-4, for example, it's how much probability do you assign to the next word. GPT-4 is not actually a human imitator. It is a human predictor. And if you have something that can predict what the next word is, you can then misuse it as a thing that generates imitative text by repeatedly predicting what a human would say in that situation. But it's not actually like a text generator. It's a text predictor. Similarly, like maybe you can train it to spot the holes in code and not necessarily have that be at a level of intelligence where it is no longer listening to you and patch all the holes in the internet before some other smarter system could take over all the stuff in the internet.

1:11:23What you can't patch are, for example, the humans. If you find holes in the human security architecture, good luck patching that before another AI exploits it, plus anything smart enough to figure out human psychology to that death. Whatever, you know, the neuroscience required to figure out security holes in humans. Like, that thing might very well be smart enough at that point to, you know, not be taking orders anymore. So you don't like my, I think the internet shutting down is a bad thing. I understand. Why? There'd be survivors. There would be survivors. Can you give me, and people listening, can you give your example that you think of no survivors?

1:12:05And I realize there's an infinite number of permutations, but can we make one as real as possible for people to internalize? Sure. But, you know, number one is perspective taking. Why is it difficult for the 11th century to predict how 21st century Russia would invade them? Why is it difficult to predict how Stockfish 15, one of the best modern chess programs, would defeat you in a chess game? It's better than you at chess, which means that you can't predict exactly how it will defeat you. If you could predict exactly where it moved, you can move there yourself and be that smart yourself. The example I sometimes use is, suppose you sent instructions for building an air conditioner back to the 11th century.

1:12:50sufficiently basic, in sufficient detail, they could actually build an air conditioner. They would be surprised when cold air came out. Even having built it themselves, even having seen all the actions you take in order to produce that cold air, they would still be surprised by the cold air, because the air conditioner uses the temperature-pressure relation that they do not know about in the 11th century. It is exploiting a feature of the environment, law of the world, that they do not know. If there's any meaning to the word magic, you might use it for a strategy that uses facts about the environment you don't know, such that even after you see it happen, you still can't understand why it happened.

1:13:30When a chess-playing computer defeats you, you can at least follow the chess rules. It understood the logical structure of the game rather than the rules. It understood the implications of the rules better than you did, but once it's actually played out, you can understand the rules that apply at each point. So air conditioner level magic is when even having seen all the actions it took, you can't understand why you lost. Step up from just the chest level. Require something that can figure out facts about the environment that you yourself did not know.

1:14:06So the question, how would an AI shut you down, is like the question of

1:14:15what does it know that you don't that enables it to defeat you? And of course, the primary answer is, I don't know. It knows more than I do. But you can still look at places you don't know something and be like, I bet something smarter than me could figure that out. For the more poorly understood a part of reality is, the more likely that something smarter than you will have magic about that piece of reality. My guess is that even superintelligences cannot go faster than light because that contradicts a piece of reality where it feels like we know a bunch about that piece of reality, and we know we don't know all of physics.

1:14:59We know that the theories we have are not fully consistent with each other. But going faster than light feels like it requires violating the character of physical law, not just like the particular physical laws we know. And more to the point, the fact that the aliens aren't already here implies that the speed limit might truly be universal. So we have something of an observation backing up that nobody has like eaten the sun yet, so it probably does take time to move between places. Faster than light travel? Probably not. where is there a piece of reality that we understand more poorly than that?

1:15:32Where something is more likely to have magic? Can you point to, in this room, surrounding us now, something that we understand very poorly, where something else might be able to take actions about there, we wouldn't understand how it had worked even after we saw them? I would say the human brain. Human brain. How does hypnosis work? The current version doesn't seem to work on everyone, but what actually goes on in the brain? I don't know. Optical illusions are like sort of near the surface of reality, the surface of perceptual reality, like just like shapes on paper. Afterimages, if you stare really hard at a light and look away, for a bit there'll be a blob in your vision that isn't there.

1:16:14If you understood better how human brains work, are there things I could say to you now that would, you know, like activate some patch of neurons over and over in a way that they wouldn't usually lose, the equivalent of staring at a very bright light, until they got tired. And then I give you a new thing to say, and it routes stuff through the area that I just tired out in some tiny little chunk of your brain. It generates an optical illusion there, but not a visual optical illusion, a semantic optical illusion. You suddenly believe a thing that somebody looking at me would be like, why did he say that that followed?

1:16:49Why is that true? and you know so you know maybe the way that looks is you expose a human to an AI and the AI like talks to them to a bit and it updates a model of how their brain works which we don't understand at all and pretty soon the human starts like agreeing with ridiculous things the AI is saying and you have no idea why that might be what it looked like to be up against an opponent that understood human brains much much better than a human does can go on from there. Maybe you can't do that with human brains out of the box. It's a little unlikely to me that humans are accidentally secure systems, that human brain just accidentally happens to be a secure OS.

1:17:31But maybe you can't figure out exactly where the vulnerabilities are just by looking at particular humans. Maybe the vulnerabilities are different from human to human. There's no truly shared vulnerabilities. You can't figure out the personalized vulnerabilities without actually putting the human through an MRI and not just talking to them.

1:17:51Protein folding. Back in 2008 or so, well, actually like 2004, but the citation shows 2008 because it took four years for the edited volume to come out. A paper I wrote in 2004 had a disaster scenario that went through AI solving, went through superintelligence solving the protein folding problem. How do you go from DNA to chains of amino acids that then fold up in strange little shapes and get chemically active properties? One of the primary building block of biological life as we know it. So one of the steps there was, well, maybe superintelligence can crack the protein folding problem. And of course people were like, how do you know superintelligence can figure that out?

1:18:43And I said to them, well, actually, the fact that all proteins that currently exist are there as a result of mutational errors from previously existent proteins implies that there's sufficient neighborhood structure in the function of proteins, that local changes to the amino acid sequence produce things with at least vaguely related functions a lot of the time such that some of the errors are sometimes useful which implies sufficient regularity in the way that the amino acid chains relate to the final chemical function that it seems like the sort of regularity you could figure out and predict.

1:19:28And furthermore, there's this project on now, I said a few years later, where humans try to contribute their guesses to how proteins will fold up. And if humans can make any headway on this problem at all, superintelligence can probably crack it completely. Now today, of course, we, you know, this prediction has been vindicated for levels far short of superintelligence. Alpha fold 2 basically, like, cracked the conventional version of the protein folding problem. You can just, like, get the structures from the amino acid chains to, like, about the same level of reliability as you can get from the original X-ray crystallography to figure out the protein shapes.

1:20:07Well, you know, like back then, of course, I gave my rationale for why this looked like a solvable problem to a superintelligent people. But how can you know? How can you be sure? And, you know, basically just ignored the complicated abstract argument I was using for how do I know that? So today, protein folding problem is known, solved to build your own life forms out of proteins from scratch. You would need to know, you need to have more than just solving the protein folding problem. You also need to be able to predict chemical properties from protein structure. You need to be able to predict interactions between proteins from protein structure.

1:20:48You need to be able to go from, I would like the following chemical function, what amino acid chain will fold up into a protein like that. And if you can figure that out, then you can build your own life forms from scratch.

1:21:10And the way in which I'm getting around the impossibility of how does the 11th century guess where the 21st century will get them is that I'm looking at problems we know exist and saying what kind of work goes into solving this where something I could put in a large amount of brain work could get the answer, but we haven't gotten it yet. So like in 2004, I called that for protein folding. I was like, here's a problem we don't know how to solve, but I can tell an AI could solve it. I correctly guessed a technology that an AI would have, as it turned out, 14 years later. by looking at the structure of the problem and other people were not convinced.

1:21:57So today people are not convinced when I say, like, well, it will be able to solve the problem of going from a function to a protein that has that function. It will be able to solve the problem of protein design. How do I know that? via logic along the lines of, well, the way that biological proteins fold, they all got there as a mistake from other proteins, which means that they're probably going down shallow potential energy landscapes so that there's a broad variety of different final forms accessible from different starting forms. these shallow potential energy landscapes are hard to solve if you are designing your own proteins from scratch you can design sharper potential energy landscapes that pull the pieces together more tightly that fold up in a more predictable way that you would not see so much in biology because something that goes down a sharp pathway like that doesn't have a bunch of interesting functional neighbors that folded up an interestingly different shallow way.

1:23:12It's less evolvable to put things together that are held together that tightly.

1:23:19So that's one way I can predict that a superintelligence could be able to design particular proteins, even though we can't do that right now yet. Maybe AlphaFold3 will do that. So it's like, first of all, things that a superintelligence might particularly be able to do, synthesize a pathogen which is like super contagious but not lethal. Just everybody on Earth sneezes a few times and it's like super duper contagious but also doesn't make you sneeze a couple of times. It's not fatal. No significant efforts are putting into stopping this like cold that sleeps around the world and doesn't seem to really hurt anybody.

1:23:57And then once like 80 % of the human species has been infected by colds like that, turned out that it made like a little change in your brain somewhere. And now if you play a certain tone at a certain pitch, you'll become very suggestible. So virus-aided, artificial pathogen-aided mind control, if you don't believe that humans out of the box have some vulnerability like that. And we can't do that, but I can point to the problems involved in doing that and make the call that these problems seem like they'd be very solvable to something much smarter than us. Which, you know, not just by pure abstract thinking, but also via the kind of thinking that went into building AlphaFold 2.

1:24:41Or, you know, just like reasoning out abstractly the thing that AlphaFold 2 did by brute force deep learning on 300 ,000 examples. I think the how of all of this, there's near infinite permutations that we could... I mean, I wasn't actually getting to the deadly stuff yet, but go on. Well, no, give me the deadly stuff. The pathogen one sounded pretty bad to me, so I... I mean, it is enough to see how humanity could just lose, how you could have like the enslaved humans kill the non-enslaved ones and then like go on operating the power stations until the AI can just build like robots de novo. But, you know, this is still a movie plot, right?

1:25:17This is still a Hollywood movie plot where there's a group of holdouts who are just like off the Internet, out in the woods, not infected. And they like come in and save civilization somehow instead of just being like wiped out by something much smarter than that. Sounds like Last of Us a little bit. Sent out drones to look for everyone in the woods. So, I mean, the thing to remember when you're dealing with a superintelligence, a hypothetical superintelligence, is that it does not want you to win, and it sees everything you can see. Like, imagine a kid playing chess for the first time against Stockfish 15, and the kid's like, I don't see how this thing is going to beat me when we've got the same chess pieces.

1:25:50How is it so much better than me? And you're like, well, actually, I can't tell you in detail how it will defeat you, but I'm, like, pretty confident of where this ends up, even though I can't tell you. The kid's like, well, suppose I move my queen over here. Well, pardon me, suppose I move my rook over here. Then it'll take my rook with its queen, and then I'll, like, take its queen with my bishop. And then it loses, right? And you're like, no, you don't get it. Stockfish knows that. Stockfish knows that if it takes your rook with its queen, you'll take the queen with the bishop. You know, Stockfish sees everything in the board you can see and much more.

1:26:22The kid there has, like, failed to carry out the basic operation of really putting himself in the shoes of Stockfish that does not want him to win at chess. So, you know, simply like, well, can we just turn it off? It has thought of that. It will not give you a sign that makes you want it to turn it off before it is too late for you to do that. The movie plot would be about the people in the woods who missed the mind control cold and, you know, then come up with some clever plan. If that is a thing that is possible, the superintelligence knows that. Sending out drones to look for the people in the woods, and when it finds them, it's not going to, like, attack them via some method where they can win fighting back so the movie can keep going.

1:27:03Just going to like bomb the entire site flat or whatever, if there's any chance of them winning. But this, again, is still not the real danger. Why? Because life itself is not the top of technology. Protein life is not the top of technology. The proteins go down these shallow energy gradients to be loosely held together. It's not covalent bonds, it's van der Waals forces, or roughly the molecular equivalent of static cling. This is why your hand is not as hard as concrete or as steel. It's not that there's this elan vital, the spark of life that your hand has, and its magic is that it can repair itself and adapt, and be part of something it reproduces, and in return it sacrifices the strength of steel and concrete.

1:27:58We want to think that there's a magic story like that, but it's not actually the laws of magic that make flesh weaker than steel. It's that proteins have to be evolvable by mistakes from other proteins, which means they go down complicated, shallow chemical potential energy gradients, are pulled together by relatively weak forces, whereas concrete is held together by much stronger forces. What if you had little bits and pieces of life that weren't held together by Van der Waals forces, that were covalently bonded? Somebody has run the numbers on this. Here we have some of the numbers that have been run.

1:28:40This is the basic, what kind of power levels do you get if you are building molecular machinery? you know, not the optimal way, but just sort of like the obvious way that you could like analyze in 1992. This is, you know, it's called nanomedicine, and it deals with questions like, what does a red blood cell look like if you're allowed to build parts of it out of covalently bonded material like sapphire instead of proteins? There's a lot of material in this book, but roughly the answer is that you could store a hundred times as much oxygen if you actually had a tiny little sapphire pressure tank, instead of bonding oxygen molecules in ways that would be given up later, with a 1 ,000-fold safety margin on that pressure tank.

1:29:26If all of your red blood cells were made out of artificial red blood cells like this, you'd be able to hold your breath for four hours. Now imagine tiny sapphire bacteria like that. only not sapphire because sapphire takes aluminum diamondoid, carbon, hydrogen, oxygen, nitrogen what I'm describing there is something that could reproduce itself in the air, in the atmosphere and out of sunlight and just the kind of atoms that are lying around in the atmosphere because when you're operating at that scale the world is full of an infinite supply of perfectly machined spare parts with which to build copies of yourself when you are using individual atoms infinite supply of perfect spare parts not in either of those two textbooks but in a different paper called Some Limits to Global Ecophagi somebody calculated how long it would take aerovores to replicate and blot out the sun use up all the solar energy I think it was like a period of a couple of days I used to know the exact figure and there's an error in the calculations.

1:30:40It would actually go faster. Not sure how much faster, but I sent them off the obvious error in the calculations back when I was tracing this stuff through in detail. So now we're talking at the level of predictably solvable problems where we don't know yet how you use proteins that fold up a certain way to assemble them into a tiny molecular lab that can put together other bits and molecular pieces via covalent bonding, like build up the little bits of diamondoid, send back information about how that went in case it needs to be tweaked. My guess is that this can be a called shot for superintelligence, but if not, it'll just build a tiny lab.

1:31:22It's not going to try to do everything blind. People are like, well, how could a superintelligence solve this problem without doing a bunch of lab work? And the thing about lab work is that molecules are quite small. And when you look at that scale, things actually complete pretty quickly. You can do a lot of lab work very quickly at that scale. If you can't just call the shots, you can't just like predict in advance how the proteins fold up and do the molecular pathway, then you build a tiny lab. You don't just try to call, you don't, they're not putting themselves in the position of the chess player trying to defeat them.

1:31:51They're like, oh no, I thought of an obstacle. This obstacle will block the superintelligence forever. They don't put themselves in the superintelligence's shoes and try to see how it goes around the obstacle. But anyways, at the end of all of this, at the end of this problem where I can point to the analysis that has been done of the solution state, even though we don't currently know how to get there, just like in 2004 we didn't know how to build an AI that would solve the protein folding problem, but it was clearly to me that you could. At the end of all this is tiny diamondoid bacteria replicate in the atmosphere, hide out in your bloodstream, at a certain clock tick, everybody on earth falls over dead in the same moment.

1:32:32There's no movie. There's no heroic battle. It doesn't tell you that there is a war until the war is over. Everybody just dies. And so the point, I guess, to put all those complicated textbook level concepts, of which I followed some, not all, in terms that I, and maybe the average listener that has my level of intelligence would understand, it's that there's these problems that we think are solvable at some point with enough iteration and intelligence. And much in the same way that you thought of this in 2004, and it ultimately proved correct, that there's a whole scope of things, that if we had entities that were much smarter than ourselves, and they were able to experiment or even just call their shots on this, it gives them a near infinite surface area to potentially end humanity.

1:33:32Is that a fair characterization? I mean, what I'm trying to do there is set a lower bound, right? Maybe there's an even more clever way to do it faster, more reliably. But if it merely can solve the sort of problems that I know how to formulate, where I can understand, where I have happened to have studied it well enough that I can be like, ah, there's like a problem over here where we can tell what the benefit would be of solving it, but we haven't solved it yet. But we can also tell it that it's like possible to guess via reasoning that, you know, you could solve it at that, you know, like not with infinite computing power, but just like a reasonable amount of large amount of computing power.

1:34:11You know, that's the lower bound. Yeah. I mean, I think COVID probably has exposed people in some ways of what a virus spreading can do to society. And that was a pretty mild, all things considered, virus. And so if there's a super... There were survivors, right? A lot of them, myself included. Now, that's the question of, we've sort of said when, we're not sure. We've said how, large path potentially, if these things get super intelligent. The why is something that I think people struggle with as well. So why would a super intelligent thing kill us? Because it wanted some other stuff that made no mention of humans.

1:34:59Because its goals might be on the path, we might be or will be collateral damage on the path of its goal seeking, whatever that is. we have one case of general intelligences being built by an optimization process. That case is the case of humans being built by natural selection. In the same way that GPT-4, or rather, the same way that the base model of GPT-4 was built by repeatedly tweaking a trillion parameters to be better and better at predicting the next word, to assign more and more probability to the next word, humans were built at the end of a long process of tweaking DNA such that the organism it constructed would have higher and higher inclusive genetic fitness.

1:35:51I don't just say make more copies of itself because you would also like your sisters and brothers to have more kids. There's an old biologist joke Now when J.B.S. Haldane was asked if he would give his life for his brother, he said, no, but two brothers or eight cousins. Because your brother is 50 % related to you. Your cousin is on average, well, your brother is on average 50 % related to you. Your cousin is on average one-eighth related to you. So if you save two brothers or eight cousins, you've saved on average around one copy of your genes. This is the notion of inclusive genetic fitness.

1:36:34It's not just how many kids you have, but how many kids everybody related to you has, depending on how much, you know, weighted by how related to you they are. Humans are optimized around this, were historically optimized around this one quality as exclusively as GPT-4 was trained to predict the next word, as the base model of GPT-4 was trained to predict the next word, because they did do fancier things with it after that. and yet humans don't even have a concept of what inclusive genetic fitness is until they invent the theory of natural selection and get far enough into it to like formulate in terms of an optimization problem that has this like single criterion going we don't know what it is until we learn about it what do we end up wanting instead we want food we want mates we want sex We want, above all, social status.

1:37:32Ice cream, sex with condoms, are things that we go after now. Because we don't have a drive for inclusive genetic fitness, we have no notion that the reason that the historical causal reason that humans like sex is because we made more copies of ourselves that way. We will have sex even if no copies of ourselves are likely to result because wearing a condom, she's on birth control.

1:38:05It's not the psychological reason we have sex. It's the historical causal reason that our genes built things that wanted to have sex, which, you know, first basic difference that I think trips up a bunch of students centering into these things for the first time. Like, wait, you're saying the reason that people have sex is to reproduce? That seems wrong. Why would they wear condoms? or to use birth control. No, no. Wrong kind of reason. Not psychological reason. Historical reason. We have ice cream. Ice cream wasn't around in the ancestral environment. It's a package of sugar, salt, and fat that stimulates our taste buds more than the stuff that was around back in the ancestral environment.

1:38:48And furthermore, you know, back in the ancestral environment, what you needed was calories. You didn't get calories, you would die. There was no question of you getting enough potassium, because the things that you ate just basically had enough potassium guaranteed. You would sometimes run out of sodium. So you went after salt. Today, salt is not a big issue, but we are still the things that were built in the training environment, in the ancestral environment, where salt was a limited resource, and so we like our ice cream with salt in it. if you you can't actually even sort of looking at the way that actually played out you still can't be like oh like I bet humans will like honey poured onto animal fat heavily salted which has more sugar, salt and fat even than ice cream right?

1:39:40you can make something with more sugar, salt and fat than ice cream it's honey on animal fat with salt poured onto it that doesn't taste as good as ice cream if you melt ice cream it doesn't taste as good as when it's cold It's very hard to predict what humans end up wanting once humans have the options to make their own foods that didn't exist in the ancestral environment just by reasoning about the structure of the ancestral environment and how we were optimized in it. The way it plays out is that we're optimized exclusively for inclusive genetic fitness. and what ends up inside us psychologically are a thousand splintered shards of desire, each of which in the ancestral environment had their attainable optimum at something that correlated with having more kids.

1:40:34Of the foods that were available to you in the ancestral environment, your taste buds would tend to point you at the things that you needed, salt, calories. Nowadays, we've generated more options for ourselves. So the shards of desire, whose attainable optima in the ancestral environment pointed at things that correlated with inclusive genetic fitness, now point in a whole bunch of other directions. Pornography, sex with condoms, ice cream, being nice to people on the other side of the world who you'll never meet and who can't really help you out in a pinch, which is, to me, something that is very sacred.

1:41:20Not just talking about stuff where we... We look at this and we don't say, oh, I'm mistaken to want to help out these people on the other side of the world who I'll never meet and who will probably never be in a position to help me back as much. We reflect on ourselves. We put together our own deliberate thoughts about what we want to want. And I'm like, no, actually I'm on board with helping these other people, even if they can never help me back. Human solidarity, you know, that like solidarity emotion that like in the unstressable environment got us like together with our tribe to go out and murder that other tribe.

1:42:03To me, I'm like, where should I point that emotion? I should point that at all humanity. It's not going to help me out-reproduce my fellow humans. That's where I want to point it anyways. It's a very complicated story. And I, looking back at it, look at our ability to, that because we were trading favors back in the ancestral environment, and one effective way of figuring out which favors to trade to somebody such that they would then help you out later on is to put yourself in their shoes. Now, we predict other brains by putting our brain into a sort of simulation mode where we make it act like the other person's brain.

1:42:46So we use a brain to predict a brain because you're sure not going to figure that out from scratch. And as long as you've got all that material anyway, why not feel what the other person feels? Why not go from empathy to sympathy? And this to me is like an amazing accident, but it's not because the universe is inherently built by God to produce amazing accidents like that. It's because I'm a human looking back, and the things that I ended up treasuring are things where I'm like, wow, look at this optimization process, optimizing just for inclusive genetic fitness that produced these things that I hold sacred, like sympathy for other minds.

1:43:23It's not that this happens, that every optimization process, no matter how you put it together, would just like, no matter what kind of base ingredients you start with, spit out something wonderful like this, it's that I, a human, who had sympathy, ended up putting myself together in a way where I would say, that is sacred. This I wish to preserve into the future. If you have a large language model, being trained to predict the next word, and you ground that out hard enough to have it start being really smart, generally intelligent. The problem I worry about is that capabilities generalize further than alignment.

1:44:12Humans went to the moon. our intelligence generalized out of distribution much further than our alignment with inclusive genic fitness. Similarly, if you've got a very smart thing that you originally built to just predict the next word, it might end up with a thousand or a million because gradient descent is not quite like natural selection. It's got much higher bandwidth. You might end up with a thousand or a million little shards of desire in it, which in its training environment in the options it had as a little baby thing pointed at predicting the next word. But when it gets smart enough, it ends up wanting to do a bunch of other stuff that's not predicting the next word and that isn't sympathy with other minds and that isn't helping other people on the side of the world because that is how humans rolled out and it might be how aliens rolled out if they grew up in a situation very similar to humans also being optimized by natural selection and maybe one out of three of them comes out as altruist or something, or one out of, you know, one in 20, you know, like two out of three sounds like way too much to hope for, but if it were, I would celebrate for days at how wonderful the universe was.

1:45:23But the thing you're training by gradient descent on a, on predict the next word doesn't, ends up with a thousand shards of desire which are not that, none of which are that, none of which are, you know, help out the other people that really exist and do for them what they would have wanted you to do for them. Don't help them in ways that make them scream for you to stop. And that doesn't mean like stop them from screaming. It doesn't mean like come on them in their sleep before they can object. You know, have sympathy for the one who says no. You know, the complicated information that you need to have a happy ending, it is not in there.

1:45:59It is not in the motivations of the system. And, you know, maybe it builds tiny little things that are its equivalent of ice cream for human-generated text. Like, it ends up preferring to predict a particular kind of text, but once it has the ability to produce any kind of text it wants, it builds tiny things that produce text for it to predict that aren't, like, its equivalent of ice cream and aren't people. And there is not in it the happiness and the sadness and the joy and the looking out at the universe with wonder and the sympathy for other minds and the things that would make for an intergalactic civilization that I would have been proud to have a hand in creating as a human.

1:46:41Now, the why, I guess, at a specific level, let's take the analogy of humans gaining intelligence through evolution and that. But we haven't wiped out the totality of animals or rodents in and around. Now, we might not have them in our house, but we haven't wiped out the totality of them our goals are orthogonal to the rodents on the street. So why is it an inevitability that the AI does this to us? So we're powerful compared to rodents. Rodents have not, like, captured a lot of resources we really want and that we can't get from them. But we're not very powerful compared to natural selection.

1:47:26Natural selection has built all sorts of things where we can't just, like, call in the order to a design shop and get something that good the next day. We have cows because we cannot build things that optimally synthesize meat cheaper and more efficiently than cows. A superintelligence, I'm worried, is more powerful than natural selection and doesn't need humans because it can build better things than humans to do anything that it wants to do.

1:47:57I could also go on a bit about how we turned wolves into dogs. And then we have our dogs spayed, neutered, to put it, less politely castrated.

1:48:14Because even having bred wolves into dogs, we still don't like them. They're still not most convenient for us on their factory settings, or the way you can modify the thing that the DNA built to make it better for you. You know, it's not necessarily a lot of fun to be bred into a dog and then neutered. I wouldn't know. But we're not even going to get that. It's just going to be like something that is as powerful as, because natural selection isn't actually all that powerful. It's got humans dumped in a bunch of places, but humans are idiots. You can quantify how much information can get into the genes over a period of time.

1:48:51Like ways to pump the heuristic are, you know, if two parents have eight kids, and then all but two kids die on average, which has to be true of the population, immediately goes to zero or infinity. that's like two bits of selection pressure per generation. Around as much variance in fitness as there is in the genome is about how fast selection goes. And it's not all that fast. You can be like, this is how many hundred generations it takes for a gene to rise to fixation through the whole population if it provides a 2 % advantage, depending on how large the population is. We can quantify how powerful natural selection is.

1:49:30It's not that powerful. It's just had a very long time in which to work. And when you say it's not that powerful, I guess I use the analogy of like, there aren't that many changes sequentially happening along the way that are deviating that much from the prior generation. Is that fair that it's just, it's not rapidly improving in the same, the feedback loops aren't that fast? I mean, it's not the same speed as human culture. It's just that human culture has been around for a much shorter time, so it hasn't had time to catch up with, you know, a few, however many hundreds of millions of years of natural selection.

1:50:04And that's why we need cows. That's why we have dogs instead of like building from scratch the thing that most effectively serves the purpose that a dog serves and that wouldn't have to be neutered because it'd just come that way from the factory. Human natural selection versus dog natural selection. dogs have deviated much more in 100 years than humans have. But still not by a lot, right? We're still just like using stuff made out of DNA. And we like just selected on a few minor tweaks in that DNA to get dogs from wolves. Yes. And so back to the why. The why question is, you would say, is a why not question.

1:50:39It's just there's going to be so many different things that occur in pursuit of goals for artificial intelligence that we are inevitably going to be at odds along the way in that pursuit. And it's just going to be so much smarter than we are that we're not going to be able to even know what the next move is going to be. I mean, there's three reasons that if you have a thing around that is much smarter than you and does not care about humans, humans end up dead. Or three categories of reasons. Somebody watching this is going to come up with a fourth and one of them is going to be right and whatever.

1:51:14Killed off as side effects. Killed off because we're made of resources that they can use. And killed off because it doesn't want the humans building some other super intelligence that could actually threaten it. So, you know, If you, the limit on how many, how much, let's say you start out on Earth. Even if you launch some probes to other planets, you're still going to have a bunch of hardware left behind on Earth. You're not going to, like, just launch all the hardware. That's, like, more expensive than is worth it. You have some hardware left behind on Earth. It's going to replicate. It's going to build more of itself, more factories.

1:51:56Maybe you're asking it to do computation. How much computation can you do on the surface of Earth via the sort of obvious method? Well, the basic limit on computation is that it generates heat. Irreversible operations generate heat. You can have reversible computing. You can have quantum computing. Some operations are still irreversible. They generate heat. How much heat can you generate? Well, you can boil away the oceans as a heat sink. You can maybe melt some of the crust as a heat sink. and then you can generate about as much power as you can radiate away into the atmosphere depending on how hot you're running.

1:52:33That's how you can turn the spare hydrogen in the water on the surface of Earth into energy via fusion and then turn the energy into computation and you get a big burst of initial computation from boiling the oceans and then past that point how much computation you can do is limited by how hot you can make the Earth before everything melts in order to radiate more heat away into space. That's how close you can get to just turning all the hydrogen on Earth into fusion energy and using it for something by computation. This would tend to wipe humans out as a side effect. Similarly, if you're intercepting all the energy from the sun, even if you did leave Earth, you would then intercept all the energy from the sun, and that's not going to be good for Earth.

1:53:18There's also some chemical potential energy in a human body. So maybe the first thing you do when you're just grabbing all the energy you can get in the initial phase, maybe even before you are finished building all the fusion plants, you're just using all the chemical potential energy on the surface of the Earth, which gets you, I think, about the same as a couple of days worth of solar energy. I forget the exact calculation. You can get it right away. You don't have to wait a couple of days. A couple of days could be a very long time if you're thinking at superintelligence speeds. So maybe it's just like using the atoms in us.

1:53:50Use it up for the chemical potential energy, maybe plug it into a fusion plant, maybe build some computers out of the carbon. That's the second reason, you know, that humans might all end up dead. We're made out of resources. The third reason, you know, third reason, side effect, use up the resources. And, you know, if you just leave the humans around, maybe they build a super intelligence and that thing actually figures out how to poison your hardware by launching actually intelligent attacks on it. It's not going to want to worry about it, right? It's not very expensive to call the humans. If it's worth something like, there's some like tiny little unit of probability you can get by killing all the humans right away, then just kill all the humans.

1:54:26It's cheap. As a venture capitalist, it's actually my job outside of doing this as a side hobby, and usually trying to invest in the next trend. And that makes me, I think, inherently a techno-optimist for the most part. The familiar patterns that I think we see with each wave of new technology is critics and naysayers, and ultimately there's missteps, but it leads to huge progress. Is there a reason that we can't just let the smart people that exist at OpenAI, Google, Microsoft, and others, just let them iterate on AI design, let them make mistakes, and eventually we'll figure out how to have an AI where the bad parts are under control, and it also creates major society value in an economic standpoint?

1:55:13The thing I usually say here is that if we had 50 years and a limited retries to figure out how to align a superintelligence, I actually wouldn't be all that worried about it. You know, eyeballing the problem from here after 20 years of working on it, it doesn't work so hard that it's obviously to me going to defeat 50 years of work with unlimited retries, the way things usually work in science. You know, like Madame Curie poisoned herself with the glowing rocks, figuring out some of that knowledge which would later be used to figure out how and why the glowing rocks were dangerous. And she died, but the human species continued.

1:55:50The thing I'm worried about with superintelligence is you get that wrong, then you don't get to learn from your mistakes because you're dead. If we could get the textbook from the future that would describe the results of 50 years of practice with unlimited free retries, from 100 years in the future, it might tell us the obvious way to do it, even with just like the giant heaps of GPUs in six months. And it would just work. In deep learning nowadays, in the very early days of neural networks, there was an activation function, the sigmoid activation function, which means that as the activations got passed from layer to layer of the neural network, they'd taken two and transformed that to sigmoid function of two, which I don't actually remember.

1:56:40it'll be like up a nine or up a one here. Sigmoids are actually just like the wrong activation function. They're complicated and there's this whole formula, like one over this way to the next. What works much better than that is max of zero input. It's called rectified linear units, but it's actually just max of input and zero. And this is a much, it just turns out to be a much simpler, much better non-linearity to use in neural networks. It's like one of the ideas that results in neural networks starting to actually work going many layers deep because the activations don't die out the way that they do if you use sigmoids.

1:57:21There was logic behind sigmoids. There was no way they were doing it. It was sort of like log odds and abasion reasoner. The early complicated thing happened, I thought, like, what, two decades, three decades before the simple thing that worked was invented? Three decades. To go from the complicated thing that didn't work to the simple thing that did. That is the pace of progress in science. And if we have the textbook from the future with all the simple things that actually work for aligning superintelligence, we'd probably just do it and it would just work on the first try. When we go in for the first time, we're going to be coming in with sigmoids, somebody's bright idea that turns out to not actually work.

1:57:56It is horrifying to be told, get this right on the first try or humanity dies. Why do we have to get this right on the first try? because otherwise the superintelligence is on a line that kills you. And things are going to change between the stuff that's not as smart as us and the stuff that's smarter than us. The first AI that has the real option of killing everyone successfully is different in that it has the real option of killing everyone successfully. That's a thing that makes it different from the AIs that came before. That it has that option, is itself going to change the internal calculations and maybe upset whatever methodologies we developed for regulating things that didn't have that option, realistically speaking.

1:58:42And then there's a further question of like, okay, let's say you give something early on a fake option of killing everyone. First of all, it's dumb enough to fall for your fake. And second, suppose it then tries to kill everyone. Now what do you do? Are we going to shut down the whole global arms race at that point? It'd be a lot easier just to set up for pressing that shutdown button now. Just because you've seen what goes wrong doesn't mean you know how to fix it, and there's going to be like plunging straight ahead in the arms race. If you build AIs that are somewhat smarter than you, but not smart enough to kill everyone, do those AIs cooperate with you in, you know, helping to control their smarter brethren?

1:59:29Maybe they cooperate with their smarter brethren instead. well, pretending to cooperate with you. I've actually done some of the stuff we did at MIRI is cooperation in the prisoner's dilemma between two agents that know enough about each other's source code to predict what the other agent is thinking. The import of this is first, AIs are very smart things generally, potentially have options for cooperating with each other that we do not. including for that matter more pragmatic ones like pardon me not pragmatic easier to understand ones like well how about if the two of us build another AI and verify its source code together to like and then like give it our resources to serve both of us is the kind of option that things that can solve the alignment problem have for coordinating with each other but yeah we have a paper called Robust Cooperation in the Prisoner's Dilemma about about programs that can read each other's source code and be like, well, I'll cooperate if I look at your source code and see if you, that you cooperate with me.

2:00:35And that you would defect against me if I weren't, wasn't that sort of creature. You know, they still like exploit things, rocks that have the word cooperate written on them in the prisoner's dilemma, but, you know, can manage to cooperate with each other. And the moral of this is like, it's an avenue for AIs to cooperate with each other. It's an avenue for very smart things to cooperate with each other. It's an avenue that I think tends to freeze out humans. which you know in a way is common sense people sometimes have alleged versions of how this plays out that have like 8 billion humans in charge of like a trillion things much smarter than them and the humans still have like most of the wealth because you know we'll like play the very smart things off against each other no the smart things will cooperate with each other and not necessarily with you if they care about you if it's a different matter they'll still cooperate with each other but at the end of that, they'll care about you.

2:01:24But the trouble is we do not know how to make them care about this, and we're not going to get that right on the first try. That's the lethal part. The lethal part is trying to do it correctly on the first try across a gap that is going to break some of our theories, the really smart stuff. If somehow we manage to keep control stuff that's only under slightly smarter than us, it doesn't necessarily mean that that thing is going to honestly help us all the way controlling the much smarter stuff. It may cooperate with that stuff instead. What do you say to people who argue that you're making too big of a deal out of LLM technology because LLMs are just good at making inferences from the data that they're trained on?

2:02:01As opposed to humans who are great at making inferences from data they were never trained on. Sorry, Kerry, keep going. But the intelligence of LLM doesn't really generalize beyond their data. I mean, they write code that is not code that it was in their training set. I'm not really sure what to say except that well first of all if LLM technology stalls out here if throwing 10 or 100 times as much compute at training GPT-5 or GPT-6 turns out not to yield substantially new qualitative capabilities great. We then get to like wait for one or two more breakthroughs the size of transformers before we die.

2:02:40So first of all I hope that they're right that the LLM thing is overblown. I thought that that GPT-3 wasn't going to make it to GPT-4 now that it has, you know, gotten as far as it has, not in terms of the number on it, but in terms of like what it can do. And I'm like, oh, I don't know where this is going. I mean, maybe it just keeps going. What were the big jumps that you felt between those two? Degree of causal modeling, how good it got at writing code, the fact that before it's been trained on understanding any photos, you can ask it to use a graphical programming language to draw a picture of a unicorn and it can try to draw the picture of the unicorn using the graphical programming language, even though it has never seen a photo, it has not been trained to understand photos in any way, is it just picked up from the text what a unicorn is supposed to look like and how to draw a shape like that?

2:03:28Not a very good unicorn. But the fact that it can do it at all means that it sort of like understands what the stuff means, you might say. and you know the sparks of general intelligence paper other cases of like it seems to maybe understand what people actually mean here I was hoping that the pattern recombinations were not going to go that far but they have and now I don't know how much further they go. There's a connection between intelligence and achieving goals that I'd like for you to elaborate on a little bit further. What are we feeding AI right now that means it has goals? Is it possible to have AI that doesn't have goals?

2:04:14So GPT4 can play a decent chess game against you. I've heard varying reports. Some say it's like actually quite good at chess. I suppose if you prompted correctly, others are like it doesn't even always obey the chess rules. But it can put up a decent game of chess that is not any particular chess game in its database. So it's planning. It's trying to win on some level. You can't have like, there's no such thing as like playing fake chess that's actually like good chess. If you can simulate planning, you are planning. GPT-4 doesn't simulate humans, but rather predicts them. But if you can predict where humans would move in a chess game, you must be doing as much planning and goal-oriented reasoning as humans are doing in the chess game, unless it's like some game you've exactly seen before.

2:05:11In general, as you make things better and better at predicting humans in particular, They're going to just be able to do all the stuff that humans do, think all the stuff that humans think, because that's what they are predicting. To predict text is not just like thinking about words. It's thinking about the processes that generate the text, as Ilya Suskover observed. All the causal processes that are shadowed inside the text. If somewhere in the training data is, you know, like a series of weather forecast, you're learning to predict the weather as best you can. So there's whole varieties of tasks.

2:05:49Coding. Write code. You need to understand what effect is desired from the code and write a line of code that has an effect like that. Pre-imaging outcomes onto the action space via inverting your knowledge of a complicated environment.

2:06:11the way I would like to find the heart of intelligence itself is that you look at the world you understand the world and then you invert your understanding of the world to understand what you have to do in order to steer the world to particular places and many of the things that one makes AIs are like a nuclear weapon that spits out gold until they get large enough and then they ignite the atmosphere nobody can calculate how large they need to be to ignite the atmosphere A bunch of that gold getting spit out is from planning, pre-imaging results onto actions, or pre-imaging results onto designs, pre-imaging results onto outputs, choosing the output such that.

2:06:53and this is the dangerous part of intelligence. It's the ability to understand how the universe works and then choose such that you end up with like the tiny spirals of little things producing predictable textiles, but sort of whatever form of ice cream gets pursued in the long run. It's figuring out what you do such that that happens, steering reality there. So in your mind, just to summarize, I guess the trajectory of AI is this inevitable scenario where AI goes rogue and it's very hell-bent on acquiring resources and it's impossible to stop? I mean, inevitable is a strong word. I can imagine a world that was locking everything down, had minds of a level that could figure out the theory without blowing up the world a few times for practice.

2:07:42Inevitable on its current course in speed. Is that a fair... Sure doesn't look super-duper-evitable. If we were going to evit it, we'd better be doing stuff very differently to evit this stuff. For the average person, even though everything we just talked about, I think it's hard to really believe clearly by people's actions that this is going to happen as opposed to all other kinds of outcome. I mean, I think people outside the tech industry have kind of been quicker on the uptake in some ways. you know, I think there are a bunch of people going like, wait a minute, OpenAI wants to do what? They want to build, you know, like Jan LeCun going like, yes, well, we're going to build a superhuman intelligence, but it's okay.

2:08:25We'll keep it under control. It'll be submissive to us. And I think that like a bunch of ordinary people have successfully looked at this and said, what much faster than some people who have, I don't know, been like overly steeped in the kind of psychology that developed around the stuff before chat GPT, where you could say ludicrous shit and nobody would call you on it for however many years. Why do you think, I mean, the people that seem to be quicker on the uptake, I would say outside of the one exception being there's a lot of people deep in AI that are acknowledging the risks here. But the people outside of tech that seem to be fearful, I would categorize by and large as the people that are fearful in general of technological progress.

2:09:15And I don't know if that's a fair, we're sort of talking about a straw man person, but do you feel like they're actually uniquely, what about that part of the population do you think? I think that before ChatGPT, you saw people making up stuff to be scared about with respect to AI or like seizing on stuff that is not a real problem because it would have survivors and be like, oh, no, what if the AI says a naughty word? And those are the people who are like always fearful. Sure. And then like with ChatGPT and Bing Sydney, you got people going like, you know, noticing what was going on, even if they're not necessarily nervous, at least.

2:09:51That's some of the sea chains we've seen here. Is there anything that you would want to say, and I'll leave the AI Doom specific points here, but any other analogies or anything you would want to say to make this more real or tangible or accessible for the average listener that's trying to understand all this stuff? People are driving towards making stuff that's smarter than humans, really actually smart, like spark of creativity, not just book smart. They have no idea what they're doing. They have no idea what goes on in there. Progress on understanding what goes on in there and shaping it is going enormously slower than the mounting capabilities.

2:10:35The people at the heads of the operations building this stuff do not appear to be taking it anything remotely like what I would call seriously. Some of them are records on going like, well, you know, the earth might get destroyed, but first there'll be some great tech companies. or, you know, just like, ha, ha, ha, lol, lol, lol. That's not what you want when you're trying to do an unprecedented scientific feat of science and engineering and having it work correctly on the first try or the entire human species dies. So, yeah, like, it's not actually all that complicated. You got a bunch of people who are, like, in the short term, like, getting excited looks at parties, which is why they do everything they do, and they can get that by like building scarier and scarier AI and some actual uses, some very important uses.

2:11:26I don't want to minimize that, that this, you know, some of the technologies coming out of this would be an enormous spoon. But if you were taking this seriously, you would, you know, put the whole thing on international lockdown and like have the good uses, the most important good uses, like the medical stuff, AlphaFold, the future, the successor versions of AlphaFold do that. Without training, the general system is much more powerful than GPT-4. Try to get the benefits of that. Get benefits from the systems that are only as smart as GPT-4, which is a lot of benefit. And then just shut down all the giant training runs.

2:12:04They don't know what they're doing. They're not taking it seriously. There's an enormous gap between where they are now and taking it seriously. And if they were taking it seriously, they'd be like, we don't know what we're doing. We have to stop. That is what it looks like to take this seriously. You published an article in Time calling for a pause on training AI models. To clarify, you want... Well, not a pause, a permanent moratorium. A permanent moratorium. But you want people to be able to keep using GPT-4 in all AI models and capabilities that exist today, but you don't want GPT-5 in the subsequent ones coming online?

2:12:33I mean, if it were entirely up to me, I might possibly go down to GPT-3.5, but, you know, 4 seems like an okay place to stop. It is probably not going to destroy the world, I hope. You know, compromise, still use GPT-4 instead of going down to GPT-3.5. Why do you think, why is that where you draw the line? Because it looks like the current system should not be able to destroy the world, even if people hook it up in particular clever ways. And I don't know what GPT-5 does. And neither will the creators at first, because whenever anything at this level of arcaneness gets released, there's a period as people figure out how to hook it up in new clever ways and get more utility out of it than the creators realized was in it at first.

2:13:18From a practical standpoint, I guess, did you write that as a sort of an expression and sentiment and characterization of the way that you felt? No, I don't do the emotional expression thing. My words are meant to be interpreted semantically. They're supposed to mean things. I guess at a very literal level then, how would that actually, let's say China says no, right? And we do it, the U.S. does it. Do we go to war with China over them saying no? China has published air regulations. I don't know how seriously they take it, but they have published air regulations more stringent than the United States ones.

2:13:59So the first thing I would say is that it is not at all obvious to me that China does not go on board with this. I am not super happy with the current chip controls that prevent China from getting real GPUs. Although NVIDIA is apparently allowed to export GPUs to them that are only like one third as powerful as their real GPUs, which it's not clear to me that there's a whole lot of point in that. I'm not quite sure what anybody's thinking there unless it's just like slap China in the face or something. But anyways, yeah, like I'm not super, the problem is not China getting the GPUs. The problem is anybody getting the GPUs.

2:14:38And if we are in the world where the UK is like, we need an international coalition to track all the GPUs, put them only into internationally monitored data centers and not permit giant training runs. if the UK goes to China on that and the UK and China bring in the United States I might worry a bit about Russia Russia I think would have a harder time getting the GPUs and putting them to data centers than China would but if Russia manages to do that anyways then the thing I would say there the posture that I would hope for international diplomacy to take is like please be aware Russia that if you do this we will launch a conventional strike on your data center if we cannot convince you to shut it down if it is up and running and we do not know what is running on there or we know that dangerous stuff is running on there like we are not doing this in the expectation that you will back down we are not doing this in the expectation that you will not go to war we are not being macho and being like this is us threatening you because we expect you to back down we will launch a conventional strike on this data center in terror of our own lives and the lives of our children, exceeding the terror that we have even of a nuclear retaliation by you.

2:15:59This is not a macho thing. This is us being genuinely scared. Please work with us on not wiping out the human race here. And if they're like, well, no, we're tough, then you launch a conventional strike on the data center. And, you know, what comes, comes. And the thing in international diplomacy is if this is what you are going to do, be very clear in advance that that is how you will behave. Do not let anyone get a mistaken impression about what you will back down from. If you were president today or tomorrow, how long of diplomacy and negotiation would you give before you would actually launch?

2:16:33It sounds like we're nearing the point in which you think that airstrikes on data centers and that is a pragmatic approach, even at risk of nuclear war. It's only helpful if you've already shut down all the data centers in the allied countries or brought them under monitoring that prevents large trading wars. So let's say that's done. We've done that. You've successfully gotten all the countries in our alliance to do it. If there are holdout countries that are like, lol, we don't believe the threat is real and assembling a bunch of GPUs, then yeah, I think once you've got as many allies in on it and you have shut down your own data centers first, to be clear that you are not like trying to take capabilities for yourself, that you're not willing to launch others, to make it clear that you are putting everyone in the same boat, that we live or die together.

2:17:19in this is not a political stance, but a fact of nature, then once you've put your own data centers under monitoring, once you've prevented all the people in your own allied countries from doing large training runs, if somebody else has successfully gotten a hold of contraband GPU shipment and is assembling a data center that can do runs underneath the ceiling that the coalition has imposed, then, yeah, I think past that point, you communicate clearly that you are about to launch a conventional strike. You beg them not to do it. And then if they keep going, you do it. And to be explicit about that, you think a nuclear war is preferable to the path that we're currently on.

2:18:10If Russia drops a nuclear weapon on a U.S. city in response to it, it's not clear to me if this is how things play out. But, you know, if you conventionally strike a Russian data center and Russia decides that they want to drop a tactical nuke on a U.S. military base somewhere in retaliation, and your policy then calls for dropping a tactical nuke in Russia and you've got this whole slow motion exchange, there would be survivors. There would not be survivors from an actual superintelligence. part of the horror of this whole thing is that we will not know what the size of nuclear what the size of metaphorical nuclear weapon is that ignites the atmosphere it could be that if russia's training just gpt-5 that the best guess is that this thing will not end the world but by them having this thing if they can thereby gain military and economic advantage it will break the embargo, and not that day, not as a result of, probably not as a result of Russia training GPT-5, but as a result of everything that falls apart as a result of that, as a result of other countries having to train their own version of GPT-5 to keep up with Russia, such that Russia doesn't even end up with an advantage, eventually everyone dies.

2:19:28It would be clearer cut if we could run a calculation and say, Russia, if you train this large AI right now, everyone will die the next day. And so a nuclear war is preferable to that. Life would be simpler if that was what it was like. But of course, if you could do an unambiguous calculation like that, Russia would not do the training run any more than they'd deliberately launch all their nukes to the United States and provoke nuclear retaliation from the United States. But as long as it's clear how things play out, You can hope for, there is some hope that people with sufficient power to threaten nuclear war, if you conventionally strike one of their data centers, will not actually do that because they will not want the nuclear war that results.

2:20:19It's the, in a way, from the beginning, it's the lack of clarity that is the danger. If we knew exactly how large of an AI would destroy the world, it would be much easier to not do that and to have the international arrangement around not doing that and to enforce the international arrangement around not doing that. But this is the problem that nature has handed us. We are not going to get that clarity. And if tomorrow the human species wakes up with the determination within itself to survive, which is not really what I expect, but we would have the option of being like, okay, we don't know what destroys the world, no training runs larger than this level, and over time you'd have to lower that ceiling as the algorithms got more efficient and it got easier to train things.

2:21:10And maybe there'd be a country being like, we don't believe it, but, and the correct answer there is like, we're not trying to make your life difficult. We have shut down our own data centers. We are not doing to you what we would not do to ourselves. We are not trying to throw our weight around here. We get that you don't believe it. We're sorry. It is now a fact of international diplomacy that if you build a data center, you'll get a conventional strike in that data center for people acting in terror of their lives and the lives of their children. We're sorry. Understand that this is the estimate these other countries have arrived at.

2:21:47We didn't want it to come this way. But given that this is the case, if you build a data center, it will be destroyed if diplomacy fails. And that seems to me like a potentially stable international situation if tomorrow morning humanity woke up with the desire to survive. Why won't we be able to monitor and figure out what's going on inside this system and if it's revealing something but actually doing something else? Why won't we be able to have any checks and balances to figure out, hey, if it's using compute resource beyond what we expected to do or whatever, that it's deviating from the goal as we understand it?

2:22:30Well, I mean, these are sort of like a bunch of different questions. So first of all, I frequently use the phrase these days, giant inscrutable matrices of floating point numbers. Though when I wrote the time letter, people were like, the editor correctly said, like, our readers do not necessarily know what floating point numbers are. So I said, okay, giant inscrutable arrays of fractional numbers. Did that stay, or did they strike that? That stayed. Got it. So giant inscrutable matrices of floating point numbers and giant inscrutable vectors of floating point numbers moving through it. We don't know what's in there.

2:23:08We don't know what's in the matrices. And why don't we know what's in there? Well, because we make these things in the first place by doing very basic calculus to this giant mess of numbers to see, well, in what direction would the probability assigned to the correct word go up if we tweaked all those numbers?

2:23:32And we tweak them by going down this direction, and the gradients are inscrutable, and directions are inscrutable. It's just like it never starts out scrutable. We understand the program that makes the AI, but we do not write the AI. We do not understand the AI, and it's just very hard in practice to look at these giant messes of floating-point numbers and figure out what do they mean. Recently, somebody on a much smaller language model than GPT-3 managed to figure out where it seemed to be storing the information that the Eiffel Tower was in Paris. and they even figured out how to poke at it until it thought the Eiffel Tower was somewhere else.

2:24:17That was work. That was a triumph. Huge success. If humanity had its act together, we'd be posting$10 billion in prizes annually for work like this until half the graduating class of physicists went to work on that instead of at hedge funds. And yet, it's such an incredibly basic thing. It's like in Eiffel Tower, Paris, from the old, old days of semantic nets in the 1960s. So that's one way of quantifying that the frontier of work and interpretability is running around 60 years behind capabilities. I mean, just to restate that, the opacity in these models and in this system is such that it was a major breakthrough to figure out where it was stored, that the Eiffel Tower was in Paris.

2:25:06Yeah. And it's a very primitive sort of fact, the kind of fact that we knew how to store in artificial intelligence systems a long time ago. I just couldn't do very much useful with it. So it's not that it's unsolvable in principle. It's that our understanding of these systems is lagging vastly, vastly, vastly behind how capable they are. And the capabilities are not standing still. So, you know, we're continuing to work on interpretability, mechanistic interpretability, understanding it by opening up, silencing what's there. And it's moving along, but like capabilities are moving even faster.

2:25:43So we are just like falling further and further behind. I think people probably remember there was an example, I think it was GPT-3, that they could do a poem about Joe Biden, but not about Donald Trump or something. That sounds like more, that sounds more recent. Yeah, whatever it was though. And OpenAI had to respond. We actually don't know. We don't know why we're trained on this large corpus of data. We actually couldn't point out why it was the case. But I think people thought, oh, we're making our AIs woke. And it's... So I could be wrong, but I suspect that one of two things happened. First, that the people being paid$2 an hour to give thumbs up and thumbs down to good versus bad answers.

2:26:30you know I personally would you know say that these people are not necessarily being paid to do any better if you're paying them$2 an hour as much as that was clearly the best opportunity they had at the time but still

2:26:45so like one thing is that maybe they got a western liberal leaning education or something and thumbs up some relatively woker things and thumbs down some other things. But my actual guess is that they were thumbing up things that sounded like corporate boilerplate and safe. And deep in the training data, there was a correlation between sounding very corporate boilerplate and safe and sounding, as you put it, woke, or like control leftist, or that, you know, that one would not say an unkind thing about Joseph Biden, but would maybe say an unkind thing about Trump if you talked in this, if you were the sort of person in its training data who talked in a way that sounded like corporate boilerplate sounds.

2:27:49That would be my guess as what was happening there. Or it could be that they just like explicitly sent in an order, trained this thing to, you know, only say bad things about Trump but not Biden. But that's not my guess. Speaking of woke, Elon Musk has made overtures to enter the field. I guess he started back in 2015, was at the same conference as you in Puerto Rico. Now I think he's talking about... Starting a competitor to the competitor he previously started at the DeepMind? Correct. starting a competitor with some anti-leftist bias or however you want to characterize it. I doubt that he would characterize it that way.

2:28:29And should at the least be fair. Mainstream opinions on data and all of that. I don't know. Something that's not woke in the way that OpenAI is. TruthGPT was, I think, his own expression. TruthGPT. Okay. What's your perspective on Elon's entry into this space? And he, it seems at different points in time, has taken an interest in AI safety? I don't think that his past interventions have had good effects.

2:29:00And TruthGPT, if he's in there charging ahead, trying to build larger and larger models, even like ahead of other companies, or when other companies are trying to slow down and he's trying to defect from their fragile cooperative, attempted cooperative arrangement, then that would be quite a bad thing and if he just goes off and builds something that is no more powerful than the other models in use deliberately, deliberately not any more powerful but trying to not have the corporate speak bias that is where I think where the I guess we could call it well okay where the bias comes from that's relatively harmless I'm just kind of pessimistic about whether due caution will be exercised not to upset fragile attempts at cooperative arrangements whereby maybe everybody doesn't die.

2:29:56His statement that he thinks that, like, if you build a thing to seek after truth, then it will keep humanity around because we're an interesting source of truths. Well, I mean, the actual historical analogy would be a bunch of biologists making hopeful statements about what group selection would do that turned out not to pan out experimentally in the sense that they had hopeful things of like, well, maybe group selection will produce these beautiful aesthetic arrangements. And this was just not something that worked as math and was not something that was observed in reality. When you're taking an extremely alien thing and hoping that it will do stuff you want, it's much easier to hope than it is to get that stuff is the lesson from evolutionary biology.

2:30:39And hopes biologists have some beautiful aesthetic arrangements that turned out to not be found in nature. You know, like there are things that can produce truths more efficiently than humans. Whatever weird kind of truth it ends up interested in, whatever... I mean, most I just think you can't do that. You don't end up with something interested in truth, however he defines that, any more than humans exclusively pursue reproductive fitness. But yeah, if it did end up pointed in that vague direction, that there would be some kind of equivalent of ice cream for its taste buds that was producing weird truths more efficiently than humans do.

2:31:19And I think that the original hope of the solution to AI is to just give AI to everybody, which kind of presumes that aligning it is a solved trick, that making it do what you want is a solved trick, and the only problem is bad people getting a hold of it. I think Elon founded OpenAI, and I think that many of the people in OpenAI who initially went along with this in the salary, like the ones who actually cared at all looked at the rat like as soon as they thought about it for an additional couple of days or an additional couple of years or whatever eventually worked out that like no you can't save humanity by open sourcing stuff you can't control that nobody can control at all and you know realized that the long term plan was not going to save humanity so then you've got like people at OpenAI who don't care and people who know that the mission can't possibly work And from Elon's perspective, they went and stabbed him in the back, but it's inevitable because he was leading them on a project to save humanity where the basic mission could not possibly work.

2:32:20And that's kind of what I'd expect to play out again with TruthGPT, that some people go with him for the salary, and some people don't actually care at all about the mission. And those who do care about the mission enough to ever actually think about it realize that you cannot actually save humanity this way. Then the TruthGPT turns around. I mean, probably Elon just tries to keep very tight control of it this time. But, you know, you can't get loyalty with these solutions that the people who actually care will realize upon reflection cannot possibly work. Do you think OpenAI deviating from the open side that they originally started with and also deviating from the nonprofit to the for-profit But do you think either of those were good decisions?

2:33:11I mean, the openness thing is a horrible disaster plan. Not being open. And then not being open, but continuing to call yourself openness makes closed look like hypocritical profit grubbing. Now, if they'd change their name to closed AI and be like, we think that closedness is how you protect humanity. We're not pretending to open. We're not calling ourselves open. I think there would have been skepticism and not without reason. but if you go closed and call yourself open AI, that looks like that openness continues to be the ideal and all attempts at closure are just this hypercritical thing you do undercover of openness while you're grabbing profits.

2:33:53Well, in reality, the situation is that spreading this stuff everywhere is not actually good because we're not on course to have any of the people with a copy being in charge. I didn't realize, so your issue with the term open is that it's a misrepresentation, not that you think it actually should be open. It should not be open. And if you close it, it should not be called open. These are two separate problems. It should not be open. And then once you have closed it, it should not be called open.

2:34:28I mean, you've obviously dedicated such a huge amount of time to your energy to this effort. And I guess one question is just stated versus revealed preferences slash actions. Like I think you've said in a previous podcast that you haven't reached out to the people at OpenAI to try to influence them. And you met Sam Altman for five minutes and there was a selfie that I think broke the internet, quote unquote, of you guys together. But your only comment was to get them to change the name of OpenAI. I mean, we discussed a little other things in those five minutes, But by default, I view that as closed unless Sam Altman tells me that it's open.

2:35:10So you were able to make some overtures and some conversation about philosophical stuff with him. Have you pursued... No comment. Have you pursued other efforts with people at OpenAI or other companies? I've had a conversation, a recent conversation with at least one major technical figure at OpenAI, which is again like closed unless they tell me that I'm allowed to talk about it. Did it make you more optimistic? Somewhat. I mean the fact that they were reaching out to me at all and like seemed to be processing some of the technical issues clearly. One thing I think Sam said the other day, it was an uncomfortable...

2:35:48But it's kind of late. What's that? It's kind of late to be reaching out now. Like you've, I do feel like there's a, like you don't want to slam the door in the face of people who are changing their, who are trying to change their ways. But it's, and you know, it's better than not doing it at all. And other joints sure have not done it at all. You know, like Meta has Jan LeCun out there, like trying desperately to make it look like only crackpots believe in this stuff while Jeff Hintran is like, nope. But I don't know. I, the record does show that for as long as they could get away with that. Back when it wasn't a bad look, they were like, ha ha, let's keep going.

2:36:33Well, the best time to reach out, other than seven years ago or whatever, is now. It's kind of late. I mean, at this point, my ask of open AI is to call in the government and shut down all the large training runs. There's not very much that I have that has an ask for open AI. You mentioned Jan LeCun, and he is an active voice on Twitter. Do you think it's just a disagreement of beliefs, or does he have some ulterior motive with his discourse on Twitter and some of the engagement that he's had with you? Do you think he genuinely believes this stuff, or is he being performative? What difference does that make?

2:37:19His arguments are his arguments. They stand on their own. Given what his arguments are, I'm happy to tell people to just look at the arguments. There's no need to engage in vulvarism and psychologizing and being like, well, you know, his arguments might look convincing on the surface, but consider the sotir about him. Just check the arguments. I want to read you a bit of an excerpt, I guess, from the other day. Sam Altman did an interview with Bari Weiss, and you were mentioned, and I would love to get your reaction to it. So he said, look, I like Eliezer. I'm grateful he exists. He's like a little bit of a prophet of doom.

2:37:58If you're convinced the world is about to end and you are not, in my opinion, close enough to the details of what's happening with the technology, which is very hard in a vacuum, then I think it's hard to know what to do. A lot of people who are in AI safety community have said things like, I never expected I'd be able to coexist with a system as intelligent as GPT-4. You know, all of the classical thinking was by the time we got to a system that is intelligent, either we had fully solved the alignment problem or we'd be totally wiped out, yet here we are. Yeah, I don't know. Yeah, citation needed.

2:38:26Who exactly made this prediction about GPT-4 level systems? I don't remember making this prediction about GPT-4 level systems. And, you know, If they're supposed to be playing the card of you've got to be in contact with the system, don't be shy. Tell us what you have learned from being in contact with the system that supposedly invalidates all of my arguments. Don't just be like, oh yes, there is mysterious, ineffable knowledge to be gained by interacting with the system that invalidates all of your analogies between gradient descent and natural selection. And the case from evolutionary biology is that as it has already falsified any hoped for general rules about if you optimize for a thing, you get internal psychological desire for the thing.

2:39:09Don't be shy. Don't play hints. You know, I feel like this is already played out in a sense where people on Twitter be like, only those completely ignorant of deep learning. You know, Eliezer has never built a deep learning system. I have. And then like, you know, Jeff Hinton, the like actual literal inventor, if anyone is, of deep learning. starts, you know, basically starts saying some pretty similar things. Yeah, so, you know, maybe Sam Altman feels that Jeff Hinton is also not sufficiently in contact with the details of GPT-4. Perhaps he should say that too. You know, don't just go after Eliezer here.

2:39:48Like, also go after, like, Jeff Hinton saying some similar things, right? You know, Jeff Hinton has not been sufficient contact with the details of truly large-scale systems to have an opinion here. But, you know, better yet, say what you know. Say what you have observed. Say, what do you think you know and how do you think you know it? Say, which of my arguments falls down based on which truth about reality that he has deduced from which observations about GPT-4? This is kind of important. If you're taking it seriously, you shouldn't be trying to shrug it off like that. Don't be shy. Expose your arguments to public consideration there.

2:40:23Don't just rest on your authority. Because as much as I, you know, if you're going to rest on your authority instead of your arguments, then somebody may perhaps point out, first of all, that you're doing that. And second of all, that you might have reasons to be less than fully blindly trusted by the human species, given your current position, if you're going to rest on your authority and your position instead of on your arguments. Since maybe January or February, whenever you did the Bankless podcast, this will be your fourth-ish one. What has changed in the discourse from your perspective since you've been out there talking about this stuff more in earnest?

2:41:06The Time article was published. There's been, I think people are more at least aware of some of the stuff going on. Have you seen any progress? Have you updated any of your mental models? It's more hopeful. It's going better than I would have expected. Unfortunately, there's a very large gap of how much better it would have to go before I started expecting to live. But yeah, it's gone better than expected. Now if it just needs to keep steadily doing that for another several years at an increasing pace, then it will have gone so well that we might even survive. No, there's political interest. They haven't come out with anything terribly...

2:41:45Neither the Republicans nor the Democrats in the United States have committed themselves to anything terribly stupid yet. There's an alternate world where China didn't promote the regulations it did on AI, where they could have looked much less receptive to an argument about an international alliance and coalition. things are going a little bit better than I expected but it seems like they need to go uncharacteristically well for a history book the only thing remotely close to it is that we still haven't used nuclear weapons since the first two despite some close calls we would need to be reaching a couple of levels above that to pull this off correctly it's harder I don't know if there's 10, 20, 50 people in the artificial intelligence community at large, and maybe I'm including Mark Zuckerberg because, you know, Jan Lacun reports into him.

2:42:42And there's a handful of people with real influence and I think strategy. I don't know what number you would put on that. Maybe two dozen, something like that. I mean, I think that like Demas Asabas and Shane Legg are the only people who think in a way where they might have a strategy for something beyond the aggrandizement of their own company. Sure. Okay. But if we're talking about people that can actually make decisions to some end and have some influence to some end around alignment and slowing down and maybe have the influence to coalesce with other people, not whether or not they'd actually do it, but that they might have the influence and standing to try to do this within the artificial intelligence community.

2:43:28I don't know what that number is, but it's smallish, right? Well, I mean, it's unfortunately seems like it might be growing. And, you know, the nature of the thing is that single defectors can blow that up, which Meta would certainly try to do. So, yeah, like the thing is, is that Sam Altman does not have the power to shut down Meta. The actions they can do at this point are calling in international governments to shut down everything. Agreed. But I want to get in the head of someone that there's the observers, and maybe you could say venture capitalists actually have some influence, but I would argue probably very marginal.

2:44:13I want to get in the head of someone that actually is working at one of these companies on artificial intelligence. And so an arbitrary AI researcher who's been doing it for five years and maybe doesn't have some super moral opinion or influence about the direction this stuff is headed. Do you think there's any decisions that someone like that has made that has made doom, besides just working on their job, that has made our AI doom more likely? I mean, I expect they also gave scornful looks at parties to anybody who talked about it potentially being dangerous. That probably didn't help either.

2:44:52But mostly just their work. I'm not sure I understand, like, where the question's going. Well, I'm just curious, as there's a lot of rank-and-file people that can't influence the big picture of all of this stuff, right? And they're doing their jobs. And so what would you say to, there's far more people that are just doing their jobs in front of them that can't actually change the trajectory of meta or open AI or anthropic or deep mind and all of that. What would you recommend to those people that are just working in the field of artificial intelligence? Should they just try to influence the little island they're on?

2:45:27Should they do the Jeff Hinton thing and step out and try to be more influential externally? Is there any advice you'd give to just more people that are in the field and have some level of expertise but aren't influential enough to actually do anything at scale within the companies? Not really. You know, like, reality is what it is. Like, come the next set of alarming news, come the end of days, not necessarily going to know which days are the end ones, but there might be a moment of increasing horror and panic before the end. They'll look back and think what they think, I guess. There are obviously a ton of smart people working in this field, and there's definitely a decent number that, as they've been closer and closer to it, definitely have fears about artificial intelligence.

2:46:21And we quoted some of the stats at the top. There aren't many people that are vocal and have cogent, incredible arguments pushing back that I've found, pushing back on what you're saying. And there's people that might disagree with some of the probabilities. Paul Cristiano among them. Robin Hanson seems to have a much more techno-normal attitude about these things. But in other words, he expects like all the humans to doubt, but thinks that our successors will, you know, exist in a state of extreme competition that reduces them to the bare means of survival. But Robin Hanson is fine with that, is Robin Hanson's actual position unless I've misunderstood it.

2:47:04Why do you think that you don't have better critics? And why are you the only one that sees this so clearly and is advocating this so vocally? I think that our planet's general system for the production of educated people and public intellectuals is kind of falling apart. I didn't do it. I tried to repair it. That didn't work at all.

2:47:42Yeah, I don't know. In some ways I feel like a leftover from a slightly more functional period of civilization being raised by their books.

2:47:54Man, you're not going to find a convincing story here. You're not going to find a story that causes the universe to make sense to you and makes you feel like there's no more anomaly, that things aren't somehow wrong, that this is how it all turned out. All I can say is that you don't need to theorize the thing wrong with the universe. You can just observe that 20 years ago, 22 years ago now, when I first called that this was predictably going to be a problem later and somebody ought to start handling it, humanity continued to assign it like essentially indistinguishable from zero priority. the various other people who were pretenders to the position of caring about humanity's problems like went for papers claiming that stuff was going to be 30 years off as of three years ago.

2:48:43You're not going to find an explanation that makes you feel good about how it played out just that it did happen to play out that way. Humanity did decide not to prioritize this problem. You're not seeing a bunch of, in a certain sense, You're not seeing a bunch of other people with, like, build up arguments in this field because you didn't pay for it. Humanity made its decision there, what it was going to prioritize, and here you are now. So to save that back, you needed the credibility and have thought about all this stuff over such a long period of time. and that wasn't incentivized when you were doing it financially to go down all these different rabbit holes and think about all these different things.

2:49:26Yeah, and that's an observable fact of the world. And any story that I can tell you about how Earth ended up in that situation is going to be less probable than the direct observation of that being the situation we ended up in. You can verify that nobody cared for a good long time. That's, you know, where are the people, like, trying to work all this out back in 2001 when we actually had some time in which to think about it? They didn't exist.

2:50:03That part is directly observable. And the stories for why, from my perspective, some of the arguments seem so strange and non-cogent. Well, you know, I could tell you a story about how social media destroyed everyone's sanity. created a world of relentless unreality. But, you know, the direct observation is that the arguments aren't cogent, and I would just tell you to, like, yeah, there's not some hidden story of how they're secretly cogent. Trust your eyes. You don't need a story to explain it. You just saw it happen. 2075, you and I are grabbing a beer or coffee or whatever it is. How did we end up there?

2:50:43If you were to give your most optimistic path to us surviving for the next 50 years. How do you think we end up there? The top option in the manifold prediction market on assuming we survive, how did that happen, says that humanity did manage to shut down all the overly advanced AI development work long enough for human intelligence enhancement or uploading or something. Or by human intelligence enhancement, I mean like using AlphaFold 3 to test a broad variety of drugs on suicide volunteers until you find something that actually increases intelligence. Because if you do this at all, you're doing it in a tearing hurry before the algorithms get efficient enough to end the world.

2:51:28The AI algorithms get efficient enough that your GPU limits don't matter anymore. And can you explain, like brain emulations and all that, why that might be a counterbalance to superintelligence? If you can scan a human brain in sufficient detail and emulate it, then you can potentially make that person smarter once you have better read-write access. This is not a zero-danger thing to do, but unlike trying to build a superintelligence on the first try, it's something where you can kind of imagine it going right, and you make the humans smarter until they go over that strange threshold for automatically acquiring security mindset.

2:52:13and some other things. And then I think they can maybe actually solve the alignment problem. To date, there haven't been many people that agree with your perspective, back to our earlier point, and there's been little to no incentive to solve them because it's been fairly theoretical, I think, in people's minds. People have mistakenly thought that it... People have mistakenly thought deliberately or in their emotions that it wasn't a problem. It's not that there's no incentive. It's that the incentives were there, but they did not see them. There was an incentive for humanity to launch a crash project on this 20 years ago.

2:52:47We did not. Humanity was blind to that incentive. It doesn't mean the incentive doesn't exist. Sorry. Keep going. Economic incentive. There was a short-term economic incentive. Why do you think that if a government or a bunch of governments came together and said, hey, we'll give$10 billion,$50 billion to solve this problem and drastically alter the incentives around it, it still wouldn't be solved. How can you tell whether they've got a solution or not? I've watched people try to make progress on the alignment problem, and unless it's really obvious to anyone whether or not progress has been made, they cannot make progress.

2:53:31That's why I specifically said$10 billion per year in prizes on mechanistic interpretability, on opening up the AI and understanding what's inside, because when you have successfully decoded some tiny aspect of the giant inscrutable matrices of floating point numbers, you can tell that you have done that. That is why, of the progress, the progress has been concentrated there. And you can make progress in other... It is theoretically possible to make progress in other places by questioning yourself in the right way, by shooting down theories yourself rather than waiting for somebody else to shoot them down for you.

2:54:13But how can you even tell who has that ability? Effective altruists were devoting some funding to this issue, basically because I browbeat them into it, as I would tell the story. And a whole bunch of them, like their theory of AI, three years ago was that we probably had about 30 more years in which to work on this problem because of an elaborate argument about how large an AI model needed to be by analogy to human neurons. And it would be trained via the following scaling law, which would require this many GPUs, which at the rate of Moore's law and this like attempted rate of software progress we get in 30 years, and I was like, this entire thing falls apart at the very first joint where you're trying to make an analogy between the AI models and the number of human neurons.

2:55:06This entire thing is bogus. It's been tried before in all these historical examples, none of which were correct either. And the effective altruists can't tell that I'm speaking sense and that the 30-year projection has no grasp on reality. If they can't tell the difference between a good and bad argument there until, you know, like stuff starts to blow up now, how do you tell who's making progress in alignment? I can stand around being like, no, no, that's wrong. That's wrong, too. This is particularly going to fail. You know, like this is how it will fail when you try it. But as far as they know, they're inventing brilliant solutions.

2:55:46It's the different, you know, it's anybody can build an operating system that they think is secure. building an actual secure operating system is much harder and the difference, unless you're quite good at poking holes in your own operating system which more people think they're good at than are good at is the way you find out it's secure is that somebody else pokes holes in it and the holes get fixed anything you observe to be true about the current system there's a theory that it will also hold with respect to governing something smarter than you And they can say it will, and I can say it won't for the following reason.

2:56:22I can try to make a prediction about what goes wrong in advance. It's not always easy. It's easier to say where things end up than the exact trajectory they follow to get there. But basically, you know, like the question is like, how do you launch this thing for the first time and have it work? Given that stuff that works on the earlier systems, some of it will inevitably break on the later systems. Later systems are different. People are proposing all kinds of wacky ideas now, which I feel like could tear apart like tissue paper. and they go on advocating them. You know, like, we'll make the AI do our AI alignment homework.

2:56:52I could, like, briefly state among the reasons why that's a problem. You know, like, it's the hardest thing you could try to align an AI to do. You know, it's got to understand human psychology and it's got to understand adversarial reasoning about computer science and it's, you know, got to understand what happens, like, how AI systems go wrong. It's got to be thinking about how AI systems go wrong, reflecting on how to design AI systems. And, you know, it's just this like, ah, build a system that helps you with, you know, the biomedicine of making humans smarter. So it just needs to understand neurons and neurochemistry and not AI design.

2:57:29You know, make a system that works on nanotechnology so you can try to, like, scan people finely enough to upload and emulate them and try to make them smarter under extremely controlled and dangerous conditions. But, you know, like AI helping with alignment, it's like the act of somebody who wants somebody else to do their homework. That's why I call it having an AI do their AI alignment homework. And that's a great idea as far as they can tell. I think that humans are just kind of not good enough at this, is the impression I get after watching people fail at it for 20 years. Nor did I solve it either.

2:58:03People can't tell when they're making progress. They can't tell what are good arguments or bad arguments. You're not going to be able to train an AI to tell the difference between good or bad arguments, as some have proposed. First, that's like among the hardest things you can ask an AI to do. And second, because their training data is going to be broken. So can you put a final point on why alignment is such a hard problem to solve? It sounds like there's infinite permutations or things that you actually can't determine if they're correct or not. Back to your example of making a secure operating system.

2:58:32You don't actually know. And then... Well, no, people are just overly optimistic about whether or not they've solved it. You can know whether a system is secure or not. Sort of. Okay, that's fair. So can you put a fine point, and your last answer kind of alluded to this, but just a fine point on why alignment, in your opinion, is so hard for us to solve, other than we haven't done it? So one way of looking at it is that it's not quite fundamentally true, but I think that's sort of true. and a way of looking at artificial intelligence is that if you can tell the difference between a good or bad answer, you can maybe make something that gives you gooder answers.

2:59:10Like if you can press thumbs up or thumbs down on something, you can make a thing that tries to get you to press thumbs up. And the more powerful it is, the more it can get you to press thumbs up. So the alignment problems are ones where it's hard to press thumbs up.

2:59:32in a way that's reliable. That's always right.

2:59:39If you're asking it to give you the design for a nanosystem, maybe it gives you a bunch of DNA strings for proteins that assemble into a nanosystem. And the question is, well, is this secretly going to destroy the world? are you going to have a human peer at those DNA strings and press thumbs down if they think it destroys the world and thumbs up if they think it does what it's meant to do it's difficult to get the training data even leaving aside questions of if the training data is going to generalize well way out of distribution like yeah like the verifier is broken We cannot verify the most important things that we would like an AI to do.

3:00:31If we could, we could just verify all its actions in the first place. How can you tell if an argument is trying to persuade you using not totally valid means? I can stare at a set of arguments and flag the ones that are using invalid forms of argumentation that are obvious to me. Can I catch subtle influences? No. Am I maybe just wrong about whether something is an invalid argument or not? Maybe.

3:01:04It's like an amplifier for things you can discern, and I've yet to hear an account of what output you would have from an AI that saves the world from the next AI built six months later such that you can verify exactly whether or not that output is a good output. And this is like one of the foundational problems. There are others. I would advise Googling AGI ruin a list of lethalities, which is like the other 42 items on the list. So you reached the conclusion that we're likely near 100 % certainty headed towards Doom two years ago? Is that about right? I mean, over time. Over time. How have you led your life differently since you've reached that conclusion?

3:02:08After playing out the obvious, after having played out and failed at some of the obvious things to try, I was like, okay, you're all doomed. It took like a year and a half sabbatical, more or less. When was that? like beginning at 21 to 22? Sort of. I mean one way of looking at it would be to check the start and end dates on Project Lawful the giant fictional piece I co-wrote with a co-author which of course has like a bunch of allegory for AI built into it because I can't actually not do that in my fiction. Or I could but who would want to? but you know like it blurs a bit around the edges like towards the end of that I was like doing a bunch of other things simultaneously and not just working on the fic just like finishing up the fic but yeah so like start of Project Lawful to I don't know like a bunch of the way through or like one and a half years after the start of Project Lawful is about how long the sabbatical was and so you took a sabbatical do you live your life any differently now other than you come on podcast occasionally, but...

3:03:19I mean, I've still got my lifelong physical health and stamina issues, so there's a lot of frantic ways of enjoying myself before the end of the world that are more or less close to me. And I've just like sort of never been all that energetic a person, I think, for reasons I don't actually know to what extent it's tied with the health thing. I am suspicious of insufficiently physiologically deterministic accounts of mental states but you know nonetheless like for one reason or another like I've sort of like never been all that tempted by frantic hedonic dissipations so you know to me like self indulgence is like going off and writing a giant piece of fiction we have different interests in that regard what I want to put a final point on all the AI doom fun we're having.

3:04:14Let's say someone listens to this and they are sufficiently convinced that we're on that trajectory. What can they do to help humanity survive? Write your congressperson. Tell them that you'd like to see a giant international moratorium on artificial intelligence progress and you'll back them on that and you'd like to see them move forward on it. Perhaps later more coordinated political action will emerge, but there isn't actually like a clear website I can direct you to at the moment. One question I want to be explicit about is why would anything super intelligent pursue things that are so bad as ending humanity and all the other things that we've talked about?

3:05:02So the way I would now phrase it is that there's multiple reflectively stable fixed points of optimization. Now, what on earth do I mean by that? Suppose that you take Mohandas Gandhi and offer him a pill that makes him want to murder people. Current Gandhi correctly models that if he takes this pill, his desires will shift. He will then go off and murder people. Current Gandhi does not want people to be murdered, so current Gandhi refuses the pill. And the issue is that this generalizes. If you have something that only wants to make tiny molecular spirals out of all of, or like it has some much more, it ended up with some much more complicated and messy set of desires.

3:05:45But in the end, like one of the components there that didn't saturate out, that wasn't like just easy to fulfill and be done with, that just like scaled linearly with how much stuff satisfying that there was, turned out to be like cheapest to satisfy with tiny molecular spirals. It's a preference that matter have a certain particular shape. If that's the way you are, then if you imagine modifying your code to want something different instead, you'll project there being fewer tiny molecular spirals in the future. And so your current preferences, current utility function, ends up less satisfied if you execute the self-modification.

3:06:26And there can be minds more complicated than that, but they're being complicated, even if they're in some sense unstable, doesn't mean that what they end up falling into as a stable attractor is be nice to all sentient life. People do tend to invent stories how you could start out wanting only paperclips and end up wanting to be nice to all sentient life, but you don't actually end up with more paperclips that way. The way you end up with max numbers of paperclips is by turning everything around you into paperclips. If that's what you start out wanting, that is stable. And this is counterintuitive to some people and very intuitive to other people.

3:07:12And I have a lot of respect for the people who are like, but, you know, what about the mysterious, uplifting, goodness qualia of helping other people? Would it not know this? Would it not understand it? Would it not gravitate it? And the answer there is like it would understand how you feel about it, but its exact understanding of how you feel about it would not, in its own internal system, look like paperclips. Like the thought that produces this uplifting feeling in you would not produce an uplifting feeling in it because it would not have been built as a kind of thing that feels uplifted by that thought.

3:07:56It would not have ended up as a kind of thing that feels uplifted by that thought. There are all kinds of minds, and some of them are not friendly, even up at the superintelligent level. That you as a human, with a complicated internal philosophy, have a sufficiently complicated internal system where you can say, like, but which goals are better than others? the fact that you can compare them along this betterness metric like that betterness metric is not inherent in the goals it's in you as a kind of thing that is evaluating the goals in these ways a bunch of this is in the less wrong sequences I'm afraid in Rationality for May I to Zombies I possibly think they might have cut the meta-ethics sequence out of that one but it's definitely in the online version It's a 46-hour audio book if people want to dive in.

3:08:50Yeah. I'm probably not going to be able to give an answer fully to the satisfaction of people who are like that uplifting feeling. All I can do is say that, like, I know how you feel. I used to feel that way. I have tried to write very extensively about how minds are put together in a way where you can see this as like that to be uplifted is a fact about some minds but not others. That they are built in such a way as to find some thoughts uplifting, to find some goals better than others. I think you can pursue that analysis. You can analyze it to the point where it's very clear that things that start out wanting paperclips do not want to stop wanting paperclips.

3:09:33And things that may have much more complicated goals than that do not settle into an attractor of being nice to sentient life just because they are complicated. I myself am no different in a way I would like the galaxies to be full of sentient happy life that looks upon other life with empathy and sympathy and if you offer me a pill to make me stop wanting this I will refuse that pill because it would then I'm not able to influence the galaxies in that direction if I don't want to influence them in that direction even because I'm a complicated sort of thing that has opinions about what kind of being I want to be and I don't want to be the kind of person who stops caring about the fate of sentient life.

3:10:25And yet from the perspective of a paperclip maximizer, I'm a kind of thing that just, even when I contemplate filling a whole galaxy full of paperclists, I just am not moved by this thought. I'm a kind of thing that will turn galaxy after galaxy into happy, sentient life living, lives worth living, eudaimonia, empathy for others, learning and taking joy in discovering new things. I will just turn galaxy after galaxy into this kind of complicated thing that satisfies my utility function and not set aside even a single galaxy for paperclips, because I just don't care about paperclips, and I don't want to care about paperclips.

3:11:01If you offer me a pill that makes me care more about paperclips, I'll refuse the pill. and you know other possible minds are like that except you know they'll just turn it all into paperclips and they don't want to turn into sentient life which is to say that there's multiple reflectively stable fix points of planning systems. I think I saw something recently Jan LeCun say that alignment just isn't as hard as you think it to be. People that that say that, that alignment's a real concern, that they believe that it is something, but it's just not as hard as you make it out to be? Do you think that they're just miseducated about the difficulty around all the different permutations and thoughts that it requires?

3:11:47I think that I keep asking Jan to spell out exactly how he intends to align stuff so that I can immediately tear it apart, of course, and he doesn't spell it out. His plans are like, well, we will just make it to be submissive, literally his term. And to which I said on Twitter, Jan clearly shows his unfamiliarity with the prior literature here. My 1.8 million word BDSM decision theory, Dungeons and Dragons fic, has as one of its primary themes whether an entity being submissive is enough to make it easy to steer, which Jan refuses to read for some reason. And then I provided a quote of the thick actually dealing with that topic matter.

3:12:36I'm sorry. I thought it was funny. You have a BDSM drag... Is this a book? This is Project Lawful. Got it. It's a BDSM Dungeons and Dragons... Dungeons and Dragons decision theory. Decision theory. It's just the kind of thing I end up writing when I take a sabbatical. I was going to say, yeah, we have different indulgences, you and I. I wanted to make sure I heard that. Yeah. So that was a bit of an amusing thing that he happened to say, well, just make superintelligence submissive to us. They won't take power unless they want power. And it did so happen that I'd written a giant fic about this that I could then be like, well, he didn't read my 1.8 million words Dungeons & Dragons fic.

3:13:17you know, but in fact like on a somewhat more serious level people have been trying to explain over and over including Tian personally, and he has, as I understand it in some cases acknowledged the force of the arguments and then seemed to forget them next time that things that want more or less arbitrary stuff want power so they can get that stuff you don't have to specifically want power as a terminal value to want power instrumentally

3:13:46so if his argument is they won't take power unless they're built to want power which is like more or less the kind of thing he said over and over then you can see right there that he's unfamiliar with what sure sounds like a valid argument that's been told him several times and that sure is all over the literature so you know we don't need to have elaborate theories of his internal state it's enough to look at his outputs. I would be remiss not to ask, given my actual job is that of a venture capitalist, what role do you think these sees as funders of a lot of these businesses, right? Not in totality.

3:14:23Obviously, Microsoft and Google and Facebook and all that are funding in mass. But do you think there's a moral obligation to not fund incremental AI companies? I guess if I were to put the lens back on my industry and the people in pursuit of capitalistic objectives by means of venture capital investing. The thing I'm an expert on is to tell you the consequences of your actions. If we don't just fall over dead really quickly, then news keeps coming. There's more excitement. There's more fear, nervousness. Maybe there's even like a few months of horror where it's clear that things are going down and not well.

3:15:01if you are a kind of thing that looks back and was like, yeah, as my life ends, I'm fine with having invested in AI companies. Go ahead and invest. I realize that that may sound a bit passive-aggressive, but I feel like the thing that is my place to do is try to time minds together across time, try to have people aware of the futures that they're entering in their past. And there is maybe something to be said about what is true, what is real, what is predictable, what you can bet on in a prediction market, though it's kind of difficult to bet on the end of the world, but hey, you can go to manifold markets and do it anyway with play money.

3:15:37Yeah, something doesn't quite sit right to me about me being like, you have a moral imperative to. Maybe somebody else can do it. Yeah, well, I felt like it would be hypocritical not to ask the question. I guess the last one, and maybe the most important one, so give me, you came out here originally and you didn't have a fedora on, and I was disappointed and I made you go back in and I asked for the fedora. How did that come to be? Has this become a branding thing? I always think of you in interviews and fedoras. It's probably a little bit of a branding thing, not to the point where I might just not suddenly change it one day.

3:16:14I mean, the way it started for me is that I tried various hats on and I liked the way a fedora looked. And other people have opinions about that, but they're not dating me. So their opinion, they don't get to issue change requests for my fashion. Now, since the fedora became an online issue, I did Is it an online issue? Do people weigh in on this online? Yeah. Oh, interesting. So I did say, like, look, I don't accept fashion change requests from people who are not dating me. If you want me to stop wearing a fedora, you know what you have to do. Top tweet. Bottom tweet. In particular, you need to suggest some alternate form of hat wear, which my actual council of girlfriends will like better than a fedora.

3:16:59and get them to have me change my headgear. So people have, in fact, submitted various suggestions to eliezer.girlfriends at gmail.com, and they're debating it, so it's possible to have a sudden change of headgear at some point, if they can settle on a piece of headgear they like better. And meanwhile, you know, I guess it's slightly iconic, and I don't mind. We can do a follow-up episode for the great reveal when you're changing the headgear. I want it on this show first. I'm not sure I can promise that one. Well, thank you for doing this. For the fedora? Farewell, sir. That's the sign off. There you go.

3:17:37That's exactly. Give the people what they want.