#1011 - Eliezer Yudkowsky - Why Superhuman AI Would Kill Us All

39:13Actually, their AI is just going to be written by GPT-6 by hand because GPT-6 can do that. And it's really all those GPUs are doing the GPT-6 tests of training GPT-6.1. So augmenting its own intelligence, making itself smarter, getting itself up to level where it can do the same sort of work that's done by current AIs like AlphaFold and AlphaProteo with respect to thinking about biology. Now, the current AIs that are top at biology tend to be special-purpose systems. They're not general-purpose AIs like Chachupiti. But they can do things like you feed in the genomes of a bunch of bacteriophages into the AI, and the AI spits out its own new bacteriophage, and you build a hundred of those, and a couple of them actually work.

40:11A couple of them actually work better than the existing bacteriophages. A bacteriophage is a virus that infects a bacteria. It's the sort of thing that you would research for the sensible-sounding reason of, well, sometimes bacteria attack humans, so if we have a virus that attacks the bacteria, maybe that works as a kind of antibiotic. So the current AIs are already at the stage of designing from scratch their own viruses that can infect bacteria, which are, of course, simpler targets than infecting a whole human. They can predict from a DNA sequence the protein that will get built, how that protein will fold up, and they are starting to predict how those proteins interact with each other and with other chemicals.

41:03That's today's AIP. so if you want the equivalent of a tree that grows computer chips not not quite your not not quite our kind of computer chips the kind of chips you could grow out of a tree

41:25the protein folding protein interaction protein design route is where gpt 6.1 would go down to, is one of the obvious places GPT 6.1 could go down in order to get its own infrastructure independent of humanity. It doesn't take over the factories. It takes over the trees. It builds its own biology because biology self-replicates from simpler raw materials much faster than our current factory system self-replicates. Oh, that is fucking scary. That is some terrifying shit.

42:09And then, as I spin the story, you know, the more you will let me pull out books like these. Okay. Nanosystems, Molecular Machinery Manufacturing and Computation by Eric Drexler. Yeah. Robert Freitas Jr., Nanomedicine, Volume 1, Basic Capabilities. Yeah. So I can try to describe capacities that sound more like you've seen from trees, grass, bamboo, algae. I will take a solar-powered self-replicating factory and miniaturize it down to the one micron scale. That's an algae cell. That's not the limit of what's possible. the algae cell is made out of folded proteins now there's two kinds I'm going to be immensely oversimplifying a bunch of stuff when a protein folds up the backbone of the protein is held together by covalent bonds but the folded protein itself is more something like static cling why is your flesh weaker than diamond diamonds are just made of carbon your flesh has a bunch of carbon in it you've you're made of the raw materials for diamond why is your flesh weaker than diamond and a bunch of the answer there is that when proteins fold up they're being held together by van der waals forces which is the thing i was glossing a static cling their their their backbone, like it's a string that folds up into a tangle, and the backbone of the string is the kind of bond that appears in diamond.

44:01Not as many bonds as appear in diamond or as solidly arranged, but covalent bonds. But then it folds up into something with static cling, and that is why your flesh is weaker than diamond in a certain basic sense. Why does natural selection build this way? Well, some of the answer is that natural selection has figured out how to make your bones be a little tougher than just like your skin. It's not quite as tough as diamond, but the proteins build, instead of just your bones being made directly out of protein, they're made out of stuff that is built by protein, synthesized by protein, and put in place by proteins, and so your bones are a bit stronger.

44:46Not steel beams holding up skyscrapers. Not titanium holding together airplanes. Not diamond, but stronger than flesh. An algae cell doesn't contain bone. It's a self-replicating solar-powered micron diameter factory held together by static cling. the flesh-eating bacteria that will that you know you will potentially you know put you into a fairly gruesome fate the multi-antibiotic resistant uh strep that you know will kill people in hospitals that's does that's not that doesn't have bone running through it that's the static cling that that's the strength of static cling, the strength of protein.

45:40You can look at physics and biology and see how you could have things that are the size of bacteria, but more with the strength of bone, more with the strength of diamond. You could even do it with the strength of iron if you're figuring out how to do a whole new set of biology from scratch and just like putting together some iron molecules. Probably wouldn't. Diamond works well enough. but um this is why you know i talk about you know it's it's it's scary to to imagine trees that are making you know like enough computer chips to run gpt 6.1 and also spawning things the size of mosquitoes or even smaller than that dust mites you can see dust mites under a microscope good luck seeing them with the naked eye and so so but you know it's it's sort of easier to imagine if you imagine that the things here are visible and not often the mysterious fairyland of stuff that only the scientists can see so you know it's scary enough to imagine that that that the trees are making mosquitoes and mosquito lands on the back back of your neck and stings you with butylenum toxin which is fatal in nanogram quantities to humans and so you fall over dead that way but this is nowhere near to the world it's just that i have to start dragging out this kind of textbook if i want to say how we know that it gets worse oh my god how have you not gone insane uh i'm excited not to okay um well wonderful that's i suppose that answers that all right um a couple of questions that i've had llms how likely are they to be the architecture that bootloads super intelligent ai in your opinion as far as i'm aware total muggle in the room um there are some limitations to the level of creativity that llms have in terms of the way that they are uh able to um be creative to come up with genuinely novel new sorts of things have you got a real concern that llms are going to be the architecture that bootloads this is there something else that you're more concerned about which is currently in dark mode or whatever else so the thing is from my perspective i have been at this a couple of decades at this point or or three decades if you want to start to count my like crazy youthful self who just wants to charge up and build super intelligence as fast as possible because it would inevitably be nice um and llms have not always been the latest thing in AI.

48:22There have been many breakthroughs over the years. LLMs are powered by a particular innovation called transformers, which in some ways is crazy simple by the standards of people doing math things in computer science, but possibly not to the point where you want me to launch into an explanation of exactly how it works right here. There's better YouTube videos about that anyway um but the point is the the the underlying circuit that gets repeated to build an llm the circuit that gets repeated and then like mysteriously trained and tweaked until nobody knows what the actual contents are but but the the the form the structure the skeleton that was that was invented in 2018 and we've had some breakthroughs since then but nothing quite as logjam breaking as transformers which were the technology that made computers go from not talking to you to talking to you and you know so that's what seven years ago it's not the only breakthrough that's ever happened in AI

49:27there was a more recent breakthrough of latent diffusion which is when AI started drawing pictures that would you know be okay would be decent to look at there were ways of drawing pictures before then called generative adversarial networks or gans but uh the the latent diffusion algorithm was what broke the logjam on image image generation and made it really start working for the first time uh and when was that that i don't remember off the top of my head like i want to spitball 2021 or something but i'm pretty sure that's wrong um so that's like a weaker breakthrough and it's like, I don't know, four years ago or something.

50:10The entire field of AI started working because somebody got backprop to work on multi-layer neural networks. You know this as deep learning. It did not always exist. It's a batch of techniques that were developed at around the turn of the 21st century, like I could I could arbitrarily say 2006 but there was more than one innovation there it's it started with if I recall correctly with unrolling restricted Boltzmann machines it's now been a while I didn't do it Jeffrey Hinton did it and then from but once they once they sort of got that working on multi-layered neural networks at all there were more innovations since then um clever more clever ways of initializing them uh the atom optimizer sgd with momentum is like much older than that but you know still important that the point is this is what made sort of the entire modern family of ai systems start working at all.

51:22Before then, Netflix, when it was much smaller, ran the most famous, huge, expensive prize there had ever been in artificial intelligence, open to anyone for a better recommender algorithm for movies. There was a$1 million prize. It was so much money. everyone got interested in it one million dollars was a lot of money back at the turn of the the the 21st century which is around when netflix was running this i'd have to look up the exact year it might have been like 2001 2005 i don't remember um i'm not sure there was a single neural network in the on some in the ensemble of algorithms that won the netflix prize i'd have to look it up but you know you it wasn't uh they didn't wasn't just like a mighty training run with many gpus that was producing a very smart recommender algorithm because before deep learning you couldn't just throw more computing power at training a more powerful ai if you were to say when that happened that was about 20 years ago so how far are we from the end of the world it might be that you just throw a hundred times as much computing power at the current algorithms and they end the world or they get good enough at coding in ai research to end the world it could be that it takes one more brilliant algorithm on the level of latent diffusion i think if you throw something that breaks as much loose as as transformers did, my guess starts to be, yeah, that sure sounds to me like it ends the world, but maybe not immediately.

53:09Maybe you need like another two years of technology burning first. Or, and then if you talk about a breakthrough on the order of deep learning itself, that seems to me like that just sort of like ends the world in a snap. A quick aside, using the internet without a VPN today is like leaving your front door wide open and hoping that no one walks in. Websites, apps, and data brokers are constantly collecting your personal information, what you search, what you watch, what you buy, where you are, it all gets tracked. And Surfshark protects you from that. It encrypts your internet connection so your activity stays private, even on sketchy public Wi-Fi at airports, cafes, or hotels.

53:46And it lets you change your virtual location with a single click. Their clean web feature also blocks ads, trackers, and malware before they even load so you stay safer and your browsing is smoother. You can run Surfshark on every device that you own, unlimited installs on one account. And right now you can get four extra months of Surfshark for free by going to the link in the description below or heading to surfshark.com slash modernwisdom and using the code modernwisdom at checkout. That's surfshark.com slash modernwisdom and modernwisdom at checkout.

54:18Okay, so LLMs could be a really big deal. And there's also a ton of other stuff that we can't see that would be dangerous as well. i don't know if the lms go there some people are saying that they're that it seems to them like the lms are as smart as they get and other people are like well did you try gbt5 pro for 200 a month or whatever it is at that cost and other people are going like yes i did it's and like the 200 version of claude is no better than the 200 version of this and and the thing i would say about about this is that if you have some perspective, if you've been watching this for longer than three years, if you've been watching this from before ChatGPT, stuff saturates, and then other stuff comes along and breaks through.

55:08It doesn't matter if LLMs take you to the end of the world because people are not, because they're not going to stick to LLMs.

55:17Okay What are the range of timelines for this sort of transformative AI that you think are likely? I mean, again, everybody wants answers like these, just like they'd like to know next week's lottery numbers. But if you look over the history of science, I am hard-pressed to name a single case of successful prediction of timing of future technology. There are many cases of scientists correctly predicting what will be developed. You can look at the laws. You can look at the physical laws. You can look at the biology laws. You can say, and you can look at that like, hmm, yeah, this sure looks like it ought to be possible.

56:04And you can look at it and say, this sure looks like it ought to be possible, and I think I see the angle of attack there. Leo Sillard, in 1933, was crossing a particular street intersection, his name I forget, when he had the insight that we would now refer to as a chain reaction, nuclear chain reaction, a cascade of induced radioactivity. Even then it was known that you could put some materials next to a source of radioactivity and induce secondary radioactivity. And so Leo Stillard was like, hmm, we've got these naturally radioactive materials. What if we find something that's naturally radioactive and furthermore has the property that you can induce radioactivity in it?

57:00Geranium-235 was what was eventually settled on, but back then they didn't know that. And Leo Sillard saw way ahead in that moment. He saw through to nuclear weapons. he saw that this was not something he should publish in a journal for immediate fame and fortune he realized that hitler specifically was likely to be a problem he did not say this is going to take two billion dollars to turn into a weapon by 1945 there are as off the top of my head there are zero instances of a science of a scientist ever making a call like that it is the difference between predicting that an ice cube dropped into a glass of water is going to melt and predicting how long it takes to melt and where all the individual where like the individual molecules end up if you point out that on a quantum level the molecules are indistinguishable i claim that there's some deuterium in there so you can't predict i get it look look i imagine i imagine that uh that's probably got to be number one on the list of things people who work in ai safety are sick of being asked uh a lot of them a lot of them will run off an answer.

58:10A lot of them are not wise enough to realize that they can't answer it. Okay. I'm going to guess that your confidence interval that it happens before the end of the century is probably pretty high. Yeah. I mean, unless we deliberately shut it down, and even then, getting all the way out to the end of the century sounds hard. If you had an international treaty planning this stuff, I would say to go really hard on human intelligence augmentation, because eventually the international treaty will break down. All you can do with it is by time to have smarter people tackling this problem and tackling humanity's problems in general okay um but that's a bit of a topic change there the ai could the people at the ai companies themselves are sometimes naming two to three year timelines and there is a lesson of history which says that just because you can't predict some when something will happen does not mean that it is far away.

59:03Two years before Enrico Fermi personally oversaw the construction of the first self-sustaining nuclear reaction, the first nuclear pile that went critical, he said that that was 50 years off if it could ever be done at all. Fermi not being wise enough to realize that he couldn't do timing. A couple of years before the Wright brothers flew, one of the Wright brothers said to the other, I forget if it was Orville or Wilder, man will not fly for a thousand years. But they kept on trying anyway. So it was two years off, but their intuitive sense was it's 1 ,000 years off. And of course, AI itself, very famously, there were some people in 1955 who thought they could make progress on AI, learning to talk, be scientifically creative, and self-improve over the course of a summer with 10 researchers.

59:54This was not a completely unreasonable thing to think because nobody had ever tried it and maybe ai would turn out to be that easy but it wasn't actually that easy not in 1955 so the um it you know it could be two years away it could be 15 years away um the ai companies themselves say two to three years but it's questionable whether we should be taking their words at face value as meaning things as opposed to like hype but at the same yeah the llms if that architecture is not the one that is going to end up at a place that is super dangerous then what what did they know if they have got all of their chips on this one particular architecture they're all in on this we don't know that they could every time every time i think i've managed to get some sort of like reprieve they're like oh no what about the super secret open ai project that's actually using some other approach so the most recent you know reasonably large breakthrough and and large language models was successfully applying reinforcement learning to chain of thought and it was a very can you explain what so um if if you haven't learned anything about llm since they they uh they started getting heard about if you haven't so like you might have heard that that llms uh just imitate humans this is false they also you can also have an llm try to think about how to solve a problem and then of the like 20 tries it takes at solving the problem one of those tries works or works best and then you say think more like that try thinking about the problem that succeeded this is how llms go past imitating humans or it's one of the one of many ways that llms go past imitating humans so this is a very so this is a relatively very obvious thing to do with llms like paul christio and myself were you know talking about that 10 years ago um but uh before llms actually existed because it's that's how obvious it is um but getting it to work uh was the what was like last year or two maybe and open ai was had this like thing called strawberry and it was, you know, their like super secret special LLM sauce that they weren't going to tell anyone.

1:02:40It was actually just like reinforcement learning on chain of thought. But the point is that this is the level of innovation that AI labs have in the past proven to have and keep secret and that when we later found out what it was and, you know, they did get a fair amount of mileage out of that. Out of having AIs try different ways of thinking and reinforcing the one that worked to solve objectively verifiable problems like math or programming and so on. So this is, you know, like the AI companies could potentially have a replacement for LLMs that they've discovered and are keeping secret from us.

1:03:22More likely is that they would have something that was on the order of reinforcement learning on chain of thought, which is, you know, when AI started to get good at coding. or they might have nothing on that order up their sleeves at the moment. And that's why people are currently claiming that the latest wave of LLMs do not seem fundamentally smarter than the LLMs from three months ago or six months ago, which is what today's young whippersnappers think is an AI winter. Let's see your field stagnate for 10 years and eventually break through before you talk to me about winter, young kids.

1:04:05Okay, brilliant. Brilliant. I have no idea what I even want to ask you. I want to know why experts aren't worried, and I also want to know what you make about AI companies. Let's talk about the experts. Why?

1:04:24Obviously, some people's wages are dependent on this train staying on the tracks. That means it's very difficult to convince somebody, what's that quote? It's very difficult to convince somebody of something that their wage depends on them not being convinced of. What about the other thinkers, researchers in this space? What is it that you, that they are most commonly missing that you think? Where are they making their fundamental thinking errors when it comes to we will be fine with just continuing on ai growth so first of all um jeffrey hinton the noble the guy who won the nobel prize in physics for being among the people most directly pinpointable as having kicked off the entire revolution in getting back to work on multilayer neural networks, or as it's now currently known, deep learning, like the point where AI started working at all.

1:05:32Jeffrey Hinton, I think, is on record as recently saying he quit his job at Google and then could speak freely, saying something like, intuitively, it seems to him like his 50 % catastrophe probability, but based on other people seeming less concerned, he ingested down to 25%. I could be misquoting here. I'm trying to do this from memory.

1:05:58So many people would consider this to not be a lack of concern. Like somebody being like, well, it looks to me like a coin flip whether or not you destroy the world. This is not what you want to hear from your Nobel laureate scientist who helped invent the field and left Google to be able to speak freely about it so he no longer has a financial stake in making it bigger or smaller one way or the other. Many people would call this already a high degree of scientific alarm.

1:06:31Joshua Bengio was one of the co-founders of deep learning. He co-won the Computer Science Award with Jeffrey Hinton, the Turing Prize, for inventing deep learning. yashua benji is also i think on the on the concern list i don't off the top of my head have a direct quote from him about probabilities it is true that i am more concerned than they are i would and i realize that this may sound you know somewhat hubristic attribute this to them being relative newcomers to my field who may not have like gotten acquainted with the full list of reasons why it is hard to align ai that said coin flip odds of destroying the world is still not what you want to be hearing from your relatively more senior scientists who are relatively newer to the field relatively newer to my field they are vastly my seniors in artificial intelligence itself of course i i am like speaking tongue-in-cheek whenever i accuse people of being young whippersnappers like jeffrey hendon could say that with a stick with a straight face i i'm just like you know a bit of light self-mockery there about how i'm not jeffrey hinted um but but that said you know um if you are relatively newer to this you might think like well you know we maybe we've just got to use reinforcement learning to make the ais love us the way a child loves a parent or love us the way a parent loves a child and not quite have at your fingertips the top six reasons why the why that is hard and principled obstacles to that and what will go wrong there so that is what prevents the the the like the the the famous inventors of the field who only started speaking out about their concerns relatively recently after leaving their companies to you know and are now financially dependent of stakes on their opinion that's what that's what makes them be like 50 50 the world gets destroyed instead of my own thing where i'm like yeah it's predictable that the world gets destroyed if you keep doing this but if you ask like what's responsible for sam altman at open ai not uh you know possibly having less than 50 odds who knows what that guy's really thinking well you can like trace out his his long trail over time of um him initially saying like ai will end the world but in the meanwhile there there will be great companies um to him sort of like saying less and less alarm astounding things in front of Congress, where Congress asks him, well, you talk about the world ending.

1:09:04By that, do you mean mass unemployment? And Sam Altman hesitates for two seconds and replies, yes. Was the lovely congressional hearing thing that happened, I think, about a year back now. So what's going on with the AI companies? I'm not telepaths. I can't read their minds. I would point out that it is immensely well-precedented in scientific history, in the history of science and engineering, for companies that are making short-term profits to do really sad amounts of damage, vastly disproportionate to the profit that they are making, and to be in apparently sincere denial about the negative effects of what they are doing.

1:09:51Two cases that come to mind are leaded gasoline and cigarettes. I don't know if you would be familiar off the top of your head with the case of alleged gasoline, probably even the kids today have heard about cigarettes. The cigarette companies did way more damage to human life in cancer and other health effects than they made in profits. They did make a few billion dollars in profit selling cigarettes, but nothing remotely compared to the cost of human life. It's not that they were, you know, like, this was an immensely negative sum game. They were doing enormously more damage than the profits that they were making.

1:10:28And any particular advertising professional who got up in the morning and figured out how to market cigarettes to teenagers, any of the scientists that they paid to write stories about how you couldn't really tell whether or not cigarettes were causing lung cancer, would have made a tiny, tiny fraction of the total profit of the cigarette companies. Their CEO would not have made that larger fraction of the total profit of the cigarette company. So they went off and participated in this thing that caused lung cancer to I don't know how many millions of people. And for what? For this very small profit.

1:11:05How could a human being bring themselves to do that? Through a very simple alchemy. First, you convince yourself that what you're doing is not causing the harm, which is just a very easy thing for human beings to do all the time, all throughout the entire recorded history of humanity. And then once you've convinced yourself that you're not doing that much harm, well, what's the harm in taking money to not do any harm? Leaded gasoline caused brain damage to tens, maybe hundreds of millions of developing brains in the United States and elsewhere. It caused brain damage to children. For what? the gas companies making leaded gasoline could have made unleaded gasoline.

1:11:54It's not that they would have gone out of business if they'd somehow gotten together and decided to stop making leaded gasoline. If they hadn't opposed the regulations that were trying to bend leaded gasoline before it turned into a big deal back in the 1930s, there was an attempt to have regulations against leaded gasoline. Lead was known to be poisonous in large quantities. Why let people spray it all over the place, even in smaller quantities? But the gas companies got together. They managed to prevent that legislation from passing. They poisoned an entire generation. And for what? For gas that burned about 10 % more efficiently, I think was what leaded gasoline basically got you.

1:12:42for it being more convenient to add lead to the gas instead of adding ethanol to make it burn more smoothly inside of car engines. Trivial. Trivial, trivial compared to the, this is not a conspiracy theory. This is standard medical history I'm talking about here. Like I've seen estimates of five points off the tested IQs. and you can look at the chart of which states banned leaded gasoline when and watch the drops in the crime rate because it makes you, you know, it disposes you to be more violent, not just stupid, that tiny little bit that hit child after child after child. Why? Why would anyone cause that amount of damage?

1:13:38because you got your CEO's salary of a company that didn't have, then didn't need to go to the inconvenience of adding ethanol to gasoline instead? Because first you convince yourself it's safe. First you convince yourself you're doing no harm, which is just an easy thing for human brains to convince themselves of. And then why not oppose the legislation against leaded gasoline? It's not doing any harm, right? Ronald Fisher, one of the inventors of modern scientific statistics, testified against it being knowable that cigarettes cause lung cancer because, you see, no proper controlled experiment had been done on cigarettes causing lung cancer.

1:14:24And so how could you possibly know from mere observational studies showing 20 times the chance of cancer if you were a smoker? How could you possibly know from mere correlational studies? And Fisher himself was a heavy smoker. He actually drank his own Kool-Aid. The inventor of leaded gasoline, I think, had to go away to a sanitarium at one point because of how much he managed to poison himself with lead. He drank his own Kool-Aid. They really managed to convince themselves that they were doing no harm. And so they could do arbitrarily vast amounts of harm in exchange for these tiny, comparatively tiny, tiny profits.

1:15:07and to say this is not a substitute for actually tracking the object level arguments about whether or not ai will kill you and for what reason you cannot figure out what will happen as a matter of computer science if you build a super intelligence and switch it on by pointing out at who has what tainted motives to you know who has what incentives to say what but it but having tried in in my book to in my in my innate stories his book to to make the case for why on an object level this is what happens if you build a super intelligence and switch it on to ask why the people paying being paid literally hundreds of millions of dollars by meta to be ai researchers why people like sam altman who you know i mean he what you know doesn't quite get paid billions of dollars.

1:16:00He was supposed to be CEO of a non-profit. He actually stole billions of dollars. But, you know, why the guy stealing billions of dollars in equity from the public that was supposed to own it? Like, how does he manage to convince himself that what he's doing is okay? Well, maybe he's not even convinced. You know, we do have him on the record as saying a few years earlier, like, AI will end the world, but in the meantime, they'll be great companies. You know, maybe he's just like, yeah sure you know like the world's gonna end but i get to be important i get to be there you know sure who but i could be trusted with this power that that's you think you think that that's the position that a lot of the guys at the heads of these uh ai companies believe i'm not a telepath i can't tell you what these people are actually thinking you got to distinguish between stuff you can possibly know and stuff you can't um but their overt language and it has has often been like, well, building superintelligence is inevitable.

1:16:56Who could possibly stop that? An international treaty could possibly stop that. A coalition of major nuclear powers could stop that. But leaving that aside, they may have convinced themselves that's not going to happen. Who could possibly stop anyone from building superintelligence? So I need to build it. Only I can be trusted to build it, is what their overt rhetoric has sort of been. Okay. But the main thing I'm trying to point out is that having presented the object-level case that superintelligence will kill everyone, to ask the question of how could these companies possibly believe that the Singh bringing them immense short-term profits and letting them be the most important guy in the room is not going to end the world is something enormously well-precedented in the history of science.

1:17:42If I'm saying that to the extent you might think that's what happened, a very ordinary thing happened, not an extraordinary thing. A thing happened that has happened a dozen times before. If they managed to convince themselves that they were doing no harm. Okay. Or, you know, only an acceptable amount of harm. Only running the 25 % chance of destroying the world. Whatever it is they think is acceptable. I'm trying to work out what the solution is. Do you have any proposed solutions that makes this seem slightly less apocalyptic? my best i have to offer is is the same solution that humanity used on global thermonuclear war don't do it like don't instead of having the nuclear war the global thermonuclear war and trying to survive it which which which for nuclear war might have worked don't have the nuclear war we managed to do that it's the best sign of hope i can offer you it is it is slightly harder for ai in some ways, if not others.

1:18:45But, you know, people going into the 1950s, 1960s, they thought they were screwed. And that wasn't them indulging in some nice doom-scrolling pessimism, luxuriating in the pleasant feeling of being doomed. This was people who did not want to be doomed. But they looked at the course of human history over the last century. They looked at World War I. They looked at how in the aftermath of World War I, everyone had said, let's not do that again. And then there'd been World War II. They had some reason to be worried about nuclear war. They had some reason to expect that no country was going to turn down the prospect of making nuclear weapons.

1:19:30They had some reason to believe that, you know, once a great, a bunch of great powers had a bunch of nuclear weapons, why, of course, they would go to war anyway and use those nuclear weapons. It was apparently to them what had happened with World War II, all these people saying, we must not have another world war, and then the world war happening anyway. Why didn't we have a nuclear war? Well, on my account of it, it is because for the first time in all human history, all the great powers, all the leaders of the great powers, understood that they personally were going to have a bad day if they started a major war.

1:20:06And people had pretended before to proclaim that, you know, war is a very terrible thing that should never be done. It wasn't quite the same level of personal consequence. You know, maybe as a general secretary of the Soviet Union, you would think that if you started a nuclear war, you would personally survive. You'd end up in a bunker somewhere. They wouldn't be going to your favorite restaurants in Moscow ever again. and that was not the situation that obtained before the start of world war one the start of world war two people might make a bunch of you know like it only takes one side to think that they might have a bit of an advantage in the sport in war the sport of kings to you know to kick off that that fun adventure of trying to conquer another country which you know wasn't as fun as much fun for adolf hitler as he expected but you could see how adolf hitler might have thought that he was going to have a nice day as a result of invading poland and that's what changed that the general secretary of the soviet union and the president of the united states actually personally expected to have both sides expected to personally have bad days if they start a nuclear war they would not have any better of a bad any better of a good day if anyone anywhere on earth built a super intelligence.

1:21:24Yeah, it's this sort of lack of the tragedy. It's kind of like a tragedy of the commons. It's just tragedy that everybody's fucked, right? It's like everything, everything gets blown up no matter who it is that builds it. Well, okay. The commons is that the commons get overgrazed because the individual farmers benefit from setting their cows loose on it. And the thing with nuclear war is that you might get a bit of a benefit by dropping a tactical nuclear weapon on, you know, like, you know, like, the united states could get an immediate benefit by dropping tactical nuclear weapons on the russian troops in ukraine and russia could get an immediate benefit by dropping tactical nuclear weapons on ukraine but neither of them is going to risk the global thermonuclear war that might follow happening with a greater probability so it's it's not the so it's not a tragic like it's not a classic tragedy of the commons the the thing that stopped nuclear war is that although you could get a short-term advantage from dropping a tactical nuke or even like dropping a strategic nuke on one city, the leaders understood how this was a, you know, like increasing the probability of a global thermonuclear war.

1:22:37They managed to hold off from doing that for that reason. They understood the concept of how it escalated things. They saw the connection to not getting to go to their favorite restaurants again, even if they were surviving in a bunker somewhere. and with artificial intelligence what we've got is a ladder where every time you climb another step on the ladder you get five times as much money but one of those steps of the ladder destroys the world and nobody and maybe if this true fact can become something that is known and believed by the leaders of a handful of major nuclear powers, they can all be like, all right, we're not climbing any more rungs of this ladder.

1:23:23It is not in my interest that you start to climb this ladder. And it's not even my own interest to break apart the treaty by climbing another step of this ladder, because then we're all just going to keep climbing and then we're all going to die. That is the best ray of hope I can offer you, that we manage to not do the stupid thing, the same as we managed to not have a nuclear war, despite many people being concerned for excellent reasons that it was going to be an impossible slope not to fall down. Okay, so what do we actually do? Well, you know, voters do not necessarily have all that much power under the modern political process.

1:24:05But I think, but like the next step for the United States might be something like like the president saying, you know, like, we're of course not going to give up AI unilaterally, which wouldn't even solve anything in its own way, but we stand ready to, you know, join with an international treaty, international alliance, whose purpose is to prevent further escalation of AI intelligence, further escalation of the AI ladder. No, we're not going to do it unilaterally, but we're ready to get together and do it everywhere. And China has already sort of, like, hasn't quite said that, but they've sort of indicated openness to international arrangements meant to prevent human loss of control from AI.

1:24:51You'd want Britain to say the same thing. And if a bunch of leaders of major powers have said, like, yeah, we would join in an arrangement to prevent this from getting out of control and everybody on Earth ending up dead, then from there you can go on to the actual treaty. what can voters do well you can write your elected officials is among the things you can try to do there um there's a uh there if you go to if anyone builds it.com i can't believe that you got that url brilliant okay yeah yeah anyone builds it.com and you and you click on where it says act you'll see our guide to calling your representatives and if you click on march you'll see a place where you can sign up to march on washington dc if 100 000 other people other also pledge to march on it.

1:25:52And for this to just happen in the United States does not solve the problem, because this is not a regional problem where you ban superintelligence inside your own country, and then your own country is safe. But this sort of thing can exert some amount of influence on politicians, and more importantly, can make it clear to them that they're allowed to discuss it, that they're allowed to want to not die themselves. There are multiple Congress people who I'm not going to name, but whom we have talked to, who would prefer that America not die along with the rest of the world, but it doesn't quite seem like the sort of thing you're allowed to speak out in public about yet.

1:26:34Voters can make it clear to their politicians that the politicians are allowed to speak out. there's already there's already like 70 like if you actually survey american voters 70 of them say they do not want super intelligence but you know that's not enough for the politicians to feel licensed to act but maybe you know if you call them and if you march on washington you know that's that's what you can do as an individual voter well i applaud you for uh trying to get some grassroots stuff going congratulations um you've been frank throughout this conversation I think it's fair for me to be frank here it does feel a little bit like you're outgunned legislation tends to move more slowly than technology does by many many years sometimes decades it just feels bleak it feels if what you say is true it really is kind of fluke that gets us to a stage where this goes well because the likelihood of some moratorium being placed where all AI development is halted and all efforts are placed on this.

1:27:41You only need one bad actor to do it, because again, if anybody builds it. You don't want the international treaty to fall over if North Korea steals a bunch of GPUs. You do want the treaty to say, if North Korea steals a bunch of GPUs and builds an unlicensed data center, then we will clearly communicate diplomatically what is about to happen. and then if North Korea still proceeds, we will drop a bunker buster on their data center. That assumes that you know that you are somehow able to detect and that no one can do it surreptitiously. It is hard to surreptitious a data center. They consume a lot of electricity.

1:28:21Okay, so we can see most of the ones in Russia and China and North Korea?

1:28:29um like i'm not sure who is looking for them at the moment and if you can you know and to what extent these things show up on satellites and to what extent these things show up on on you know intelligence reports but there has previously been an issue of detecting covert nuclear refineries um in terms of of nuclear non-proliferation and and this was not an unsolvable problem. And data centers are, if anything, even higher profile than nuclear refineries. Right. So we are going to threaten some people with... I mean, I wouldn't use the word threaten. I would say that if North Korea is building an unsupervised data center, then you should actually be terrified for your lives and the lives of your children.

1:29:15And you tell North Korea this plainly and truthfully. And then if they don't drop, If they don't shut down their data center, you drop a bunker buster on it, then you do this even though North Korea has some nuclear weapons of its own.

1:29:31Okay, so pressure from people on their elected representatives through mail, marches, more awareness, to get the government officials to come up with an international treaty, to get countries to agree that what, specifically? We're not making AIs any smarter than they are already. We are putting the chips that can be used to build the more powerful AIs into locations where their uses are supervised. supervised i would say ideally you are putting the chips that run the ais into locations where they can be supervised as a minor side effect maybe you can stop the ais from driving people insane it seems like the sort of thing you could better do if this was all happening under supervision by international treaty as you know it's not vital to humanity's survival that ais be prevented from driving people insane but it serves as a kind of test case of can you you know stop the damage.

1:30:41Like, is humanity in control here? Can we stop AI from predating upon some of our human people? But, you know, that's not the main thing here. It's a thing that some people find attractive, but it's not the main thing. You're trying to, like, just get the whole AI thing under control, and then you're trying to stop the further escalation of AI capabilities up the ladder.

1:31:07It is scary. It is one of these things, and I imagine that it feels, it must feel a little bit like this to you, that everybody is sort of dancing their way through a daisy field. Oh, I've got this personal coach in my pocket, and it's so cool, and I get to talk to it about all of my psychological problems. God, I can bitch to it about my husband. It just listens. and at the end of this daisy field that everyone's having a load of fun in is just like a huge cliff that descends into eternity and there's like a balrog at the bottom or something. Is that what it feels like? Yeah, pretty much. But the future is hard to predict.

1:31:56It is genuinely hard to predict. I can tell you that if you build a superintelligence using anything remotely like current methods, everyone will die. That's a pretty firm prediction. The part where people maintain the daisy field attitude that they had a few years earlier toward AI, that has already shifted to some degree just because of the chat GPT moment, and nobody predicted that in advance. Nobody at OpenAI, as far as I can tell, had any idea that when they released chat GPT that were going to be causing a massive shift in public opinion about AI as people realized AI's were actually talking to them.

1:32:35Now it's not kind of intelligent about it. So maybe it also, maybe, I don't want to wait for anything else to happen. Maybe ChatGPT was the miracle we got. I wasn't expecting that much of a miracle. I did not call that in advance. But maybe we get another miracle. I don't want to sit around waiting for it because I can't tell you the miracle will occur on such and such a day, but maybe the AI has managed to do something more destructive than driving a few people insane, breaking up a few marriages, and causing whatever further decline in birth rates is going to be caused here. Maybe they do worse than that, and that shifts opinion.

1:33:20Maybe they just get more powerful and smarter and are clearly no longer choice, and that shifts opinion even without a giant catastrophe. It's not clear to me, you know, as much as people love to bitch about their elected leaders, it is not clear to me that we are looking at permanent obliviousness to the aliens getting smarter and smarter. Like, people are currently saying completely wacky and oblivious things because they think that's what's politically mandatory to say in the current political environment and that you have to talk about jobs rather than the other extinction of humanity but it's just not clear the future is very hard to predict in general it's not clear to me that the current state of obliviousness is something supreme unmovable and impossible for any event to change or to change or that it won't just disintegrate on on its own as more people talk about it there's a level in which you you kind of have to be pretty dumb to look at this smarter and smarter alien showing up on your planet and not have the thought cross your mind that maybe this won't end well can can even elected politicians be that dumb yes absolutely it is not known to me to be prohibited that this can be the case do they have to do the stupid thing it's not clear to me that it's mandatory.

1:34:49We did manage to not have a nuclear war, and people did not think they were going to get that much luck. Eliezer Yukowski, ladies and gentlemen. I was prepared coming in, but I'm not sure that the rest of the audience will be. So, dude, the best compliment I can pay you is i hope you're wrong uh but i fear you're not yeah being wrong it'd be great to be wrong i'd love to be wonderful you know i'd be wonderful i i want let me assure you everyone by making it by way of like making you know like destroying any shred of optimism you might previously have had i completely would have other career options lined up and other ways of supporting myself if I was completely wrong.

1:35:37Like, not just me, but some sensible people who donated me a bit of appreciated currency wanted to make sure that I could, if I changed my mind about this sort of thing, just retreat from my entire career path and not end up in financial trouble. And yet here I am. So I'd love to be wrong. We have tried to arrange it to be the case that I could at any moment say, yep, I was completely wrong about that and everybody could breathe a sigh of relief, and it wouldn't be the end of my ability to support myself, and I would have other things to do. We have made sure to leave a line of retreat there. Unfortunately, as far as I currently know, I continue to not think that it is time to declare myself to have been wrong about this.

1:36:22Heck yeah. All right, Elliot. Well, if the internet is still alive in a little bit of time in the future, we can check back in and see just how right you are. Well, every year that we're still alive is another chance for, you know, something else to happen. What a wonderful way to finish. Dude, I appreciate you. Thank you so much for your work. Thank you for having me over to deliver the bad news. Rinse takes your laundry and hand delivers it to your door, expertly cleaned and folded, so you could take the time once spent folding and sorting and waiting to finally pursue a whole new version of you.

1:37:03Like tea time you. Mmm. Or this tea time you. Or even this tea time you. So did you hear about Dave? Or even tea time, tea time, tea time you. Mmm. So update on Dave. It's up to you. We'll take the laundry. Rinse. It's time to be great.