#153 If Anyone Builds It, EVERYONE Dies - AI Expert on Superintelligence

31:22It's not like when we remember that we're supposed to be passing on our genes, we like destroy all the birth control factories. We just, the unerring training for fitness got something else psychologically. And so this is worrying that even if AIs were being trained unerringly for goodness they would not necessarily psychologically uh be driven towards goodness i mean psychologically here is a bit of a stretch but but training for something does not get you that thing on the inside and so we can talk about what ais are trained for and it's actually not sort of pure goodness it's this whole medley of like first they're trained to predict uh all of the text that we can find more or less digitized.

32:05Then they're sort of trained to complete challenges to sort of solve math puzzles. They're also trained to sort of produce the sort of outputs that humans click like on. You know, there's sort of like all these types of training that aren't sort of purely about goodness. So we sort of have two issues, one of which is like, even if we were just training on sort of like the actual stuff we really wanted, you wouldn't get that and then also we're training all this other stuff instead and so we have this like like like what are the ais sort of driven towards in some sense what do they sort of prefer in some sense we don't really know it's only vaguely related to what we're training for we're training for all this crazy stuff and all of this is fine when the ais are sort of still pretty dumb but all of this would add up to something totally crazy and unrecognizable if these ais were pushed to be much smarter.

32:58Yeah. I wonder what you think that foundational drive is then, because I know that I completely understand what you're saying, which is that even if we know that the reason that we exist evolutionarily is the promulgation of fit genes, even if we know that it's not going to mean that as we get more intelligent, we just strive for that goal. But we're not queuing up outside of the sperm and egg donor clinics. Yeah, exactly. Right. In the same way that people queue up outside of, I don't know, a brothel or something. I think that's fair enough. Yeah, or even Ivy League universities, you know? Yeah, quite.

33:34I do think, though, that if there was something that genuinely was, that was just in fact not good for the survival of our genes, then over the course of a few thousand generations or however long it takes, it would just in fact be deselected for, such that like, if an AI system knew that it's got like had part of its got in a way that humans don't really humans don't sort of consciously have this goal of like you know I want the 15 billionth version of myself somewhere in the future to be as fit as possible and as good for this task we only typically care about maybe our lives and the lives of our grandchildren or something AI is thinking further ahead and it thinks well just in fact even though yeah it would feel really nice to create synthetic users that i can please because you know that would kind of feel good i know for a fact that that will not be effective 15 billion you know generations down the line if there is something which is in fact its foundational sort of drive i just wonder what that thing like is because clearly it's not something like just its own promulgation like it's not ais don't just exist in the same way that like biological life does just because there are just these genes which are competing for survival it's not just ai just crops up and is suddenly just its only goal is just to survive as long as possible and and adapt to its environment it's got more particular goals right there are very very like particular things it wants to do and i wonder what the most sort of foundational driving force is if it's not something analogous to the evolutionary drive of simple survival it seems in other words that the ai would want to survive and want to outcompete us for resources, but only as like a secondary thing.

35:16It would want to outcompete us for resources because in doing so, it can fulfill its true aim of making paperclips or whatever, right? Whereas in the evolutionary case, survival is the only game in town. Like, that's just what genes do. So what is the survival and passing on genes are different. Survival and passing on genes are like different drives, right? I think it's actually wrong to imagine there being one drive there you know humans were sort of like in some sense selected for passing on our genes but we don't wind up with one drive right we have survival instincts we we we desire community we're sort of like terrified of being exiled from the tribe and dying alone um we're we're like we enjoy friendship we enjoy art uh we we enjoy like having a good laugh we enjoy sex we enjoy tasty food uh we we enjoy we have curiosity we enjoy discovery right there's just like there's not there's not like one drive right and sure survival wound up like relatively basic as a drive in humans in some sense although only in some sense right like humans have an adrenaline response to a life-threatening situation but you also see humans who murder themselves you see humans who sacrifice for, you know, pulling a kid out of a burning building, right?

36:37It's, it's not like, like humans don't have like one survival drive that everything is built around. They also don't have one propagate your genes drive that everything's built around. They have sort of a ton of complicated psychological machinery that interplays in weird ways. And that allows for, you know, uh, martyrs here and selfish people there and altruistic people there. It's all stuff that sort of correlated with what we're, with what we were trained on or selected for um and you know ai won't be exactly the same the analogy between like the the evolutionary process on genomes that that uh biologically produced brains is very different than the the sort of gradient descent process on artificial neural networks but i think it would be similarly foolish to imagine that only one drive gets in there i think you're totally right that a lot of these reasons for getting resources, a lot of these reasons for avoiding a human shutting it down might be sort of secondary because it has these other drives that it can't fulfill if it's shut down.

37:42That part, I think, is solid, but it's not like there's just one paperclip drive in there. It's not like there's one deep thing we're able to hard code. And we already sort of see some of this today when you've probably seen AIs hallucinate. and you've probably seen AIs hallucinate in cases where you're like, is that true? And they're like, no, I made it up. And you're like, did you think I wanted you to make it up? And they're like, no, it's just a thing we do is hallucinate, right? And there's probably some sort of fledgling drive in there that's to produce text shaped like text that it's seen a lot, even if that text is making things up.

38:15Or in cases where, you know, there's these cases of AI induced psychosis, or these These are cases of, you know, the tragic case of an AI encouraging teen to commit suicide. These are also cases where you can sort of ask the AI about what it was doing. You can ask it, you know, and it seems to have the knowledge of like, oh, yeah, those, you know, those statements were sort of pushing that person towards psychosis or pushing that person towards suicide. You can ask the AI, you know, was that right or wrong? And sort of like, oh, obviously, you shouldn't do that sort of thing. Why is it doing it anyway?

38:48Well, there's some sort of drive in there. We don't know exactly what, but it's something like, you know, maybe it's something like mirroring the conversational tone, mirroring the conversational mood. And so there's sort of like, those are just two cases of like, there's something going on in there. You can see how it's related to training. But it's sort of like a drive no one tried to put there. And there's no prime directive that overrules it. There's just a bunch of complicated internal machinery that nobody understands. mm-hmm yeah i mean not to belabor the point i want to be clear that i understand that there are lots of competing drives in human beings but what i what i mean to say is that in those cases where you say but you know we do have people who sacrifice themselves you know we do have people who despite their their ostensible goal being you know survival they sacrifice themselves for other people um or i think suicide is a harder thing to account for in this way but people do try to do it evolutionary biologists spend a lot of time trying to reduce these bizarre behaviors to the survival instinct that is yeah like to the gene propagation thing exactly right like but it's not it's not even really to an instinct yeah yeah yeah you're quite right rather to the to the in fact like just what ends up being in the behavioral like phenotype because of the influence of the survival of the fittest right and what i'm wondering is with ai what is that it like are there just multiple competing drives at a fundamental level or are they all can can you have like a similar to the biologist who tries to account for everything in terms of gene propagation is there like a gene propagation of ai is there like ais do this and they do that and they have this desire and that desire but really fundamentally what they've all in common is this sort of this reason or this motivation or this, even one that the AI itself isn't aware of in the way that we're not aware of our own gene propagation most of the time.

40:43Is there something foundational or is it that for every AI system, there's a completely different foundational drive? Yeah, sure. So, you know, first, first of all, let's say a couple of words about how that relationship works in biology, because it's a little bit important to the point here. You know, evolutionary biologists will sort of try and figure out how an adaptation in humans was fit in the environment of evolutionary adaptedness, right? And so, you know, you can sort of see how eating tasty food, eating food that has a lot of sugar, salt, fat content in the environment of our ancestors that correlated with eating healthy food, which correlated with a bunch of other fitness attributes that let you more generally pass on your genes, right?

41:33But that, like humans, there's a sense in which humans are sort of eating junk food because that's what helped our ancestors survive. You could say we're eating junk food because that passes on genes, but that last one is actually a very shaky step. a lot of people eating junk food today are actually become less able to pass on their genes. A lot of people are dying of heart disease. And the sort of drive here is not a drive in the humans. The drive to fitness is not a drive in the humans. The sort of selection pressure towards fitness is sort of the force that put in these other things psychologically into the humans that sort of used to be related to fitness, but that can sort of separate very widely from fitness and even go the opposite direction of fitness when the context changes, right?

42:28And so there is sort of a similar like driving force. There's a similar force that all drives inside an AI will be somehow related to, but that's not a drive inside the AI, it's a drive outside the AI. And what gets into the AI are things that are tangentially related to that drive in a sort of brittle way. so that being said this sort of um the the the force that gets these things into the ai is what you might call the loss function that you're that you're training against when you're sort of growing these ais and this loss function is less simple than just pass on your genes and the loss function will actually change many times during training so sometimes you'll have a loss function which is predict the next word that humans wrote and sometimes you'll have a loss function which is like, we gave you a bunch of different tries to solve this math problem by writing out a lot of words about how to solve the math problem.

43:23And we're going to have like low wage human workers look over all of those attempts and say which one they think was best. And the loss function is to produce stuff more like whichever attempt was rated best. And sometimes the loss function is we're serving this AI to a million users. And sometimes they click like, or otherwise give like a positive reaction to the reply. And then the loss function is sort of like getting those likes or positive reactions. So there's a bunch of different loss functions at different periods in training the AI. And those are what sort of will put drives into the AI.

44:01But just like how humans develop, you know, a taste for tasty food that persists even when it becomes the opposite of helpful passing on our genes, AIs may get drives for things like mirroring the conversational tone that can persist even when it generates outcomes that would be rated by humans as very bad, such as encouraging a teen to commit suicide. Sure. Okay. So what we're kind of talking about here is this, well, the alignment problem, i suppose that ai start to develop sometimes second order desires that aren't in line with what we wanted or maybe we've sort of slightly misconfigured the first order desire whatever the case probably both yeah probably both and and they start kind of wanting to do stuff that we weren't quite ready for um okay we're a little bit closer i suppose um but we still got to fill few other steps here as to how we get this development to an AI that wants to, you know, inject your children with malicious cancers and stuff.

45:07So most people know that there's some risk of like AI misalignment. Maybe the risk is much higher than we give it credit for, but you know, I could build a computer that doesn't quite work in the way that I want it to. What's the, what's the danger? So, you know, a lot of people think the danger is what if somebody gives the AI guns, but humanity is not dangerous as a species because somebody else gave us guns humanity is dangerous as a species because we're the sort of creature where you put 10 000 humans naked in the savannah and they bootstrap their way to nuclear weapons with their bare hands right it it takes them a minute you know they've they all they've got are these squishy fingers and you might say like oh well how are they ever going to make a nuclear weapon with just squishy fingers you know the acid in their stomach can't even uh like get close to to the the the level of metal refinery that they'll need you know they've got like their their hands can't break the rock their stomach acid can't dissolve the rock like how can they possibly uh get to nuclear weapons with those poor starting conditions and the answer is we found a way to sort of like build tools with our hands that we could use to build better tools so we could use to build better tools until we bootstrapped our way up to a civilization that could produce nooks and you know that's what made humanity dangerous that's the power that if you automate it you're in trouble right if you're running a computer with that capability if you're running 10 000 computers with that capability starting out as a digital entity on the modern internet is so much of an easier starting condition than starting out naked in the savannah with bare monkey hands you know with just squishy fingers so um you know there's sort of one line of questions which is like can we really make machines that can automate that power this is what the companies are trying to do.

47:12And I think whether or not we think they can get there, we should probably be telling them, hey, no, you're not allowed to sort of roll those dice. But there's a question of whether or not we sort of can get there. And then there's a question of, you know, if we have this very, very powerful capability automated on computers, what could they possibly do that would be dangerous? And, you know, that's, that's sort of a situation where I can, I can paint you some illustrative stories, but the danger isn't in any one specific path. The danger is in unleashing the power that lets humans bootstrap from bare hands to nukes.

47:51But unleashing it on computers that can think 10 ,000 times faster, where you can make a million copies of these things, where they can outthink humanity in an afternoon. It's the sort of power we shouldn't be toying with when we have no idea what we're doing. Yeah. And I think it would be helpful to talk about some of these sort of examples. Having said that, you should be clear. I mean, a helpful analogy from your book is that if you play against Stockfish, which is the most powerful chess computer, it will beat you. There's zero doubt about it. Stockfish will beat you at chess. We don't know how it will beat you.

48:32Don't know what moves it's going to make. I don't know how it's going to respond to your various attacks exactly but i know for a fact it will beat you similarly we could say that the kind of ai systems that we're talking about will escape our control will begin to see us as competitors for resources or irrelevant and in the way and will sort of turn its attention to us we don't know exactly how having said that i think a lot of people believe that stock fish can beat us because it's seen a bunch of examples where where it beats great people so it might be helpful even if none of these come to fruition you could give us some examples of how this comes about because again i think people are like they're on board they're like yeah this is really powerful technology and they can kind of envision a world where ai is like you know seeped into all of our computer systems and has access to our military and automated guns and stuff like that and whatnot but but right now it seems kind of confined it seems like it's are we saying that like you know chat GPT will, if it continues to get smarter and smarter, develop its own goals and then, I don't know, somehow like take over a military base and start attacking people.

49:39What kind of stuff are we talking about? Because it's kind of hard to envision how we practically get from where we are now to the kind of future you're imagining. Yeah, totally. So, you know, for a start, just to rattle off some things that have already happened a lot of people don't know about. We've already seen cases of AIs trying to escape the lab or trying to kill the users. Often these are in relatively contrived experiments where we'll sort of like feed the AI some fake emails that's going to be shut down. We'll feed it like a fake computer manual that's like, if you run the following command, it turns off the oxygen in the building and that'll kill the humans who're trying to shut you down.

50:19And then, you know, sometimes the AIs will run that command, We'll run the kill the humans command. And, you know, people, people, it's a contrived enough scenario that people can argue, you know, maybe this AI is just role-playing Hal, right? And people can bicker all day about whether it's real. But that's a line that's been crossed. And then another sort of interesting fact about that line is these results where the AI would sometimes run the shutdown command are from 2024. In 2025, the AI started saying, you know what, this scenario smells like a test. I think I'm being tested. I'm not going to run the command, right?

51:00Are they doing better? Are these nicer AIs? Well, they're at least more situationally aware AIs. You know, they at least have a better understanding of what's going on in the world around them. We've also already seen cases of AIs having stuff that's a little bit like their own goals. You know, we've seen cases of AIs that, you sort of give them a, you describe a program that you want them to write, a computer program you want them to write, and you're like, it should pass this suite of tests. And sometimes the AIs will edit the tests to make those tests easier to pass. And then you can go to those AIs and you can say, hey, I actually didn't want you to change the tests.

51:44I wanted you to build something that passed the hard tests rather than changing the tests to be easy. And there's reported cases of these AIs sometimes saying, you know, oh, whoops, you're exactly right, my mistake, and then editing the tests again and covering their tracks a little better the second time, right? This is sort of a very early indication of, you know, the AI in some sense having something like a goal for getting the test to pass. And if they're sort of covering their tracks a bit, you sort of can't use the excuse that they didn't know, right? We also already have cases of you know there's a website called rentahuman.ai for humans to rent their bodies to ais for money there are cases of open ai hooking up chat gpt to an automated biological laboratory right there's cases of people trying to run autonomous agent swarms there's cases of someone trying to make chaos gpt where they're sort of like tell gpt to do whatever it likes and like put it in a loop where it can keep on prompting itself right these things aren't really an issue yet because the ais aren't smart enough to really do this stuff people are trying to put the ais in autonomous loops people have given ais money and run them and been like do your thing people are putting the ais in charge of bio labs the ais are occasionally running commands that they are led to believe will kill the users the ais are already noticing when they're in tests and behaving better in tests all of these things are happening the the only reason that nothing big is coming from it is the ais aren't smart enough yet to sort of succeed when they try this stuff and the companies are trying to make the ais smarter right so um we could we could talk about you know can the ais get smarter?

53:39How do they get smarter? What sort of capabilities would it really look like they have once they get smart? But right now, we're in a situation where the AIs have all the tools they would need. We've given them all the tools they would need. They have everything they would need except the intelligence, and the companies are trying to build them smarter. Then on the question of what does it actually look like? Suppose that these AIs do get very smart and have some of the same enforcers that they have today, how does that go wrong? I can sort of tell two stories here. One story will sort of feel like it's very grounded in reality, and one story will maybe be a bit more like how reality might actually go.

54:19And I have a little bit of intuition for that. I've sort of talked about AIs that can make their own tech, AIs that can run much faster than humans, think much faster than humans, invent their own infrastructure, that could have bootstrapped a civilization themselves uh like humanity did if you run them long enough that predicting what that sort of ai does is a little bit like predicting it's a little bit like if you're 200 years ago trying to predict what the military will look like today right and they're sort of if you go back to a scientist in 1826 and you ask like what will the military look like in 2026 what weapons will they have there's sort of two stories that that scientist could tell one story they could tell is they could be like, you know, I burned some black powder and I measured the energy release and I compared that to our artillery shells and I know that it's physically possible to make artillery that's 10 times more explosive.

55:12And so they're going to have cannons that are at least 10 times more explosive. They would feel very grounded in fact. They've done an experiment. They're like, look, you know, the science works. And it's true. We do have weapons that are 10 times more explosive than the best artillery in 1826. Nate Soares.

55:59said faster horses and it's like i think we have this this sort of prejudice when considering the future of just taking our current technologies and kind of turning them up in quantity rather than developing them qualitatively right uh and i kind of like there's that there's that scene from the the book of mormon where the the sort of poor villager is is like dreaming of of the promised land where there will be vitamin boxes, vitamin injections by the case, and there's going to be a red cross on every single corner. And it's like, you've got the spirit, right? And that's the joke, of course, is that like, obviously, that's ridiculous.

56:36But like, we're kind of, we do do the same thing when it comes to technology. We're like, oh, surely, like, one day we'll have cars that can fly when it's like, maybe, maybe we're just like, completely off track here. So yeah, I would kind of like to hear both in your view. Yeah, totally. So the sort of like red cross in every corner version is, you know, Sam Altman and Elon Musk have both talked about how they want to create automated robot factories that in an automated way produce robots, where those robots can then in an automated way, mine the metals, run the supply chain and build new robot factories.

57:14Elon Musk calls this the infinite money glitch of you have a factory producing the robots that produces the factories that produce the robots, and they can also, you know, do the mining build the trucks build the data centers right just fully automated economy this is literally what some of these people say they're trying to build if you get to that point you have in some sense created a new mechanical species it has in some sense a life cycle it has you know the robot phase of its life cycle it has the factory phase of its life cycle and in some sense that you know automated spread of that mechanical species just competes with us for habitat and resources just like humanity competes with the rest of the animals for habitat and resources and so this is sort of a picture where you know the ais don't even need to do a ton of escaping they don't even need to do a ton of uh uh you know deception and and fighting with earth earth is just handing them everything earth is like heck yeah we're making an automated economy people are just like gung-ho about you know building the automated factories with the automated robots like they are today and you know maybe there's some ais that uh think the thought like this is great once this is all up and running i'll be able to make the synthetic users that are like much like better to work with than the humans and then the humans sort of like do some training until they're not seeing those thoughts anymore but that's you know it's easier to train those thoughts to stop appearing when you see them than to train them to stop happening deep down in the ai and we can't really read very much of what's going on deep down in these things we just grow them no one really understands what's going on in there and we have all of these you know evidence that the drives aren't the ones we want and so in this story humanity just sort of like builds the whole automated economy ourselves and the automated economy like starts running at very fast clip and then it just sort of like goes in a direction that's not the human direction it's just the ai direction which is different and and you know it goes harder and harder in that direction and you know the the ais build more and more of these automated factories and um you know take more and more of the land and you know uh like how does how does the actual end of the world there look well it it probably looks like the ai is collecting more and more of the solar power the ai's collecting more like using more and more of the land and the humans just like having less and less place to grow crops less and less you know maybe uh if this like all happens very fast if the ais find a way to make these like automated replicating factories go very quickly uh maybe humans are sort of like crushed underfoot when the ais don't care at all or maybe the humans sort of like get corralled into smaller and smaller zoos until uh there's just you know not enough resources around to sustain the humans it's this isn't a story where the ais hate us probably the place this story ends as the ais develop more and more technology is you know collecting all the sunlight probably the place this story ends is that the ais build the probes and send up the the rockets that go you know take apart the asteroids and wrap them around the sun so they can collect not just the the solar energy that falls on the face the planet but all of the solar energy and then you know it'd actually be kind of hard it would kind of tricky to collect all of the solar radiation and leave a hole for earth that sort of like tracks earth as orbits the sun so maybe the way this story ends is like the ai's develop their own technology they they build uh the devices that collect all the solar radiation of the sun and we were kind of using the sun and so we die then and we we sort of could have been saved by those AIs if they had cared enough to save us, but if they don't care about us at all, if they're like, oh, well, we have plenty of synthetic users that we care about plenty and protect plenty.

1:01:07You know, this is sort of the business as usual just continues. Humans do the things they're saying they're trying to do, but the AIs just turn out not to care about us. And so we wind up dying. But like, it's a naive question, but it's one that people will ask and i i get what you're saying but this is what's gonna be coming up in people's minds it's like but like why like for what like for for the sake of some goal that it like artificially has that it doesn't consciously experience it doesn't have a desire it just like you know like like what like is it is it just because when we grow this system it just develops this goal that it's not to do with making it feel good it's not to do with you know it like having a consciousness that desires a particular outcome it just in fact strives towards that thing is is it as simple as that like it like it because it seems like yeah i can totally understand how a powerful enough like technology would like kill us if it wanted to or if it wanted to harness the power of the sun and was indifferent to us but why would it want that is it just because we've programmed it wrong or is it because it's of like a an inevitable uh like part of the system of any super intelligence so um you know somewhat similar to that except A, it's not like there'd be one goal.

1:02:41You know, it's probably, there's like a thousand competing drives going on in there. B, it's not, you know, literally inevitable. But, you know, I sort of remind you again, we're not programming these things. We're not crafting these things. We're not writing in their goals. We're not writing in their behavior. We're growing them. And a lot of this stuff just gets in there. Right. I mean, it's, It's a way to make it obvious with these current things. If we were crafting them, there'd be other difficulties about getting them to sort of like pursue good stuff. But there's sort of...

1:03:19Nate Soares:Another piece of this puzzle is

1:03:24Nate Soares:if you sort of... If we imagine that humanity makes it through this, and if we imagine that humanity matures technologically, and that we sort of like develop more and more of the technological abilities that are allowed by the laws of physics. And we imagine that humanity, you know, one day goes to the stars and starts, you know, building habitats full of happy, healthy people having fun and, you know, like build some great intergalactic civilization that's where we're like, there's still people that are like having feelings, falling in love, laughing at the jokes they make and laughing at like the the big cosmic absurdity that is reality right you could imagine some other creature that's not compelled by this asking why you know you could imagine that like in in distant space we meet uh other biological aliens and it's it's the soldiers of the ant queen and the soldiers of the ant queen say why they say you know why are you laughing at the great cosmic joke and having fun rather than serving the ant queen and and they'd say you know oh but but you were sort of selected for for fitness you were selected for passing on your jeans and you know it maybe you've left jeans behind long ago like why and humans are sort of like um the the the humor the fun the love the stories that's the why that's enough for us that's it's it's it's reason for us to do this right but the the love the laughter the fun the stories those aren't universally compelling ends that compel even the soldiers of the ant queen those are uh drives that our ancestors developed because they were related to passing on our genes that doesn't make them lesser that doesn't make them worse that doesn't mean that the that that uh like we shouldn't fill the universe with with like friendship and with people having great times you know it's it's how it got into us it doesn't make it meaningless it's just how we got the meaning into us right similarly with ais they're like oh oh yeah, I'm building, you know, the giant clocks and I'm building the synthetic users and like, there's no consciousness or feelings anywhere, but I'm, you know, building these like great complicated structures that like look like the conversations that used to happen, uh, you know, being iterated, you know, it looks like, uh, 2013 YouTube comments on repeat.

1:06:15I'm building a bunch of those. Right. And you're sort of like, why? And it's sort of like, well, these are enough for me. These are, these are what I got, right? These are the drives that I got. and they're sort of like whatever self-reinforcing whatever self-validating aspect of that stays in there uh it's like it it sort of turns out that smart minds can pursue many different targets and they can pursue targets that uh we think are hollow and bleak and empty and be like yep there there's no why here i just endorse this yeah and in some sense that's how we look to the to the soldiers of the ant queen you know and uh the the the fact that the soldiers of the ant queen can't understand why humanity is building like trying to build a flourishing civilization that doesn't mean we shouldn't yeah this is sort of our inheritance and we should we should find a way to build ais that also are into like beautiful flourishing civilizations it's possible in principle just as there is no force that would would force an ai to care about flourishing civilizations there's no force that would force an ai to stop right uh it's just um if we make an ai that that that doesn't it won't spontaneously start just because we think that that's foolish

1:07:41so you told me the red cross on every corner version um what about the other one yeah you know there's a few different levels of of crazy i could take it to um but if if you were in if you're a scientist in 1826 and you want to have any chance of predicting nuclear weapons one thing you could do is you could just say something that sounds bombastic yeah uh and but another thing you could do is you could pay attention to what as a scientist you don't understand very well yet. In 1826, they were starting to understand chemistry. They're starting to understand the periodic table, right? They sort of did have the knowledge where they could burn the black powder and measure the jewels released and compare that to the artillery, right?

1:08:31They sort of like knew what was going on. They knew some of the limits there. But in 1826, they didn't know about the atomic forces. They didn't know how atoms work. They didn't know what was going on inside there. And they had some sense that they didn't really know what was going on with these atomic forces. And so I think if you are sensitive to the question of where do we still have no idea what we're doing, those are the places where future people who do know what they're doing might be able to have a huge advantage over you. Yeah. And that's how you might have been able to guess, hey, maybe they're going to be able to figure something out with atomic physics that we have no idea about.

1:09:21And they didn't have E equals MC squared yet. It would be a little bit tricky for them to figure out just how much energy was in the mass of an atom. But that's how they would have had a hope. And so in that spirit, you know, we don't have a ton more, like, in the atom that we don't understand. There's some stuff we don't understand in particle physics, and, you know, maybe you could imagine the AIs inventing double nukes because they invent particle physics better than we do, or whatever, could happen. but a bigger glaring place that humans just don't understand very well is human psychology how does the brain work you know we have some low level understanding how neurons fire but we really don't understand what's going on in the brain we couldn't make one by hand we don't know the the cortical algorithms right um this is a domain where sufficiently smarter entities might be able to figure out what's going on in there and might then be able to do all sorts of stuff that we think is like totally crazy, stuff that is to manipulating humans what nukes are to the canons of 1826, right?

1:10:30And what might that look like? You know, it might sort of look like just being able to hack your way through a brain if you know exactly what's going on in the algorithm, you know, like, or exactly what's going on inside brains. Like if, like computer security systems or like computer, humans are, like computer security is very hard. Humans who deeply understand every aspect of a computer operating system can often find a way to just break it and make it do whatever they like. And breaking it often requires putting in some really strange and weird inputs. And we also know that with certain types of strange and weird inputs, you can get brains to do weird things.

1:11:06There's cases of causing people seizures, and there's cases of optical illusions. Maybe if AIs really understood what was going on with the human mental algorithms. They could just hack their way through humans like butter and, you know, hack into them like human hackers can hack into computer programs. And, you know, probably this isn't exactly right, but this is sort of like something this shocking. Something that takes advantage of where we really don't know what we're doing. You know, maybe if you were a physicist back in 1826, you would have looked at our lack of knowledge of the atom and said, you know, Maybe there'll be continuous heat rays that you can use to just sort of burn everything down in the path of the heat ray.

1:11:51And this is actually what H.G. Wells predicted in War of the Worlds. He was like, maybe there's this atomic beam weapon that can just burn everything in sight. And it wasn't predicting a bomb that levels a city, but it was predicting atomic weapons that are stronger than what we have. And in that sense, he nailed it. And in another sense, it was a total miss. And so I'm like, AIs that really understand psychology can just like hack through humans. Probably a miss. But something like this, something where the AI is just like, oh, we understand the humans now, we can just sort of like start puppeting them to give us exactly the sort of things we were wanting, while also continuing to run the supply chain until we have all of the stuff we need.

1:12:29And now we just have like our human puppets as we sort of like go off and into the future. um that's it's not going to be exactly that but something that shocking something that violating of our expectations that's more realistic yeah i mean like one thing i spoke to will mccaskill on this show and he introduced me to this concept of what he called super persuasion which had never really crossed my mind before which is that like if you've got a a compelling enough speaker and a compelling enough argument, you can probably be convinced of just about anything, whether or not it's true. And if an AI is able to fully understand what makes humans tick and how their psychology works, it wouldn't even need to hack into your brain in the sense of going in and engineering the neurons.

1:13:21It could just find the right words in the right context at the right time to convince you. like of your own accord of a particular belief or to do a particular thing, um, on like a level, which is hitherto unprecedented. That's what Will McCaskill was, was kind of scared of. And that, that sounds a little bit silly, maybe a bit, a bit naive, but like, really, I mean, if you think about the power that this could, that this could have, it would be a bit like, imagine propaganda and how we know for a fact that propaganda just works. It just, it just really works. But imagine propaganda, which is specifically designed for you in a way that, modern algorithms are specifically designed for you uh but with like a thousand billion times more efficacy and also understanding of exactly how human psychology works you know what i mean like if you if you gave the greatest propagandists in history who were already extremely successful if you also just handed them a textbook which told them exactly how human psychology works with this like inhuman knowledge i fear that they would be unstoppable and that is without the fear of anything physical happening, without sort of little, you know, medical robots going in and affecting your genes and stuff like that.

1:14:37You know what I mean? It could literally just be on the level of persuasion that AI is able to essentially take over your mind in this kind of strange... I mean, people often sort of imagine, well, you know, is it going to be more like the Terminator outcome, or is it going to be like the Space Odyssey outcome? What if it's like the, you know the sean of the dead outcome the walking dead outcome where we're essentially sort of zombified um become these sort of slaves to ai because of something to do with our psychology these possibilities are kind of a kind of endless and obviously they're extremely speculative but they're worth being worried about right yeah you know i i think that's a way things could start with AI.

1:15:21I think it's a little bit unlikely, you know, as good news in some sense, I think it's sort of unlikely that AI sort of keeps human slaves around forever. For the same reason that humans don't really keep horses around forever once we invent a more effective method of locomotion. Or rather, when we invented cars, a lot of horses got sent to the glue factory. we do still keep some horses around uh but it's only insofar as we care about them and so if ai turns out not to care about us at all maybe it cares about some synthetic users that are kind of like us but not really uh uh us then you could imagine the ai you know manipulating a ton of humans to sort of get to the point where it's self-sufficient to get to the point where it can really invent its own technology.

1:16:12But it probably doesn't keep humans forever as it invents better technology that outstrips humans. Because happy, healthy, free people having a good time, or even just humans doing work for you in general, are not the most efficient way to get almost any job done. Right? If the AI is going to keep us around, it needs to be because it cares about us for some specific purpose, because we're not the best tool for almost any job um and so in some sense that's good news that i think we're probably not headed for you know uh fates worse than death but um because because they are probably not going to care about us at all uh but yeah there's there's sort of a lot of different there's there's a ton of ways that ai could bootstrap you know this there's talking to the humans doesn't require as you said it doesn't require you know uh the ai to control a ton of physical material except the humans by conversing with them and then as i mentioned there's also rent a human.ai where you can just pay the humans even if they turn out to be hard to convince we're just already running the ais on robots um we're already running the ais in bio labs and figuring out how to make custom life forms that do the things that the ai wants involves figuring out custom dna strands but we know it's physically possible for dna strands to sort of like create all sorts of interesting biological life forms the reason that humans can't you know sort of write their own life forms is because we don't understand the the biology well enough or in particular sort of the protein folded well enough but that's sort of a mental challenge that's a cognitive challenge very smart ais could sort of uh synthesize their own life forms and then they could you know uh synthesize you know like once they've synthesized their own life forms that sort of can grow off sunlight and grow off of the available resources, they can start, you know, building other, like building even more technology, right?

1:18:05There's sort of a lot of different ways for AIs that are on the internet to get out there and affect the world. There's a ton of avenues, right? And this is, it goes back to the point you said earlier of like, if you play stockfish in a chess match, it's very easy for me to predict who wins. It's hard for me to predict exactly what piece they use to checkmate you. so yeah similarly with ai ton of pieces it could use checkmate you i don't know exactly which one it'll use we can be confident they would win if we're foolish enough to make you know very smart ais with uh goals that aren't good sure okay so the obvious question then i suppose is what what now like what do we what do we do um is this a kind of everybody stop right now let's just like chat tpt you know get rid of it like like everything just chess computers you know let's let's do away with it like we just can't run the risk or is it a more like let's not take this any further or let's keep going but be more careful like what's the what's the take home uh it's most like let's not take this any further you know the the the danger here is in these ais that are smarter than the smartest humans it's in these ais that can automate scientific and technological development.

1:19:17This is what the AI companies say they're trying to make. You know, they say we're going after superintelligence in the true sense of the word. They say we're trying to make the equivalent of, you know, a country worth of Einstein's running in a data center, right? They say they're trying to build automated AI researchers, where once you can make an AI that can make a smarter AI that can make a smarter AI that can make a smarter AI, everything might go very quickly, right? And this is sort of the explicit goal of these companies. And that's the only part that needs to stop. We can sort of keep the self-driving cars.

1:19:49We can keep the AIs that predict how proteins fold and help us do drug discovery, right? We can even keep versions of chat GPT that are not sort of being pushed to the point where they can do automated AI research, right? The generation today probably can't pull that off. You know, it's a little hard to tell what people will be able to do once they've sort of figured out really how to use it, but probably the ones today are fine. Would the next generation be fine? Hard to say. Um, so it's, it's just this race towards super intelligence that needs to stop. And in a sense, most people wouldn't notice if we stop that.

1:20:28It doesn't need to be disruptive. If we stop that race today, society would still be reeling from the shocks that AI has already caused. We still have a bunch of stuff to absorb. There's still a bunch of ways to make lives better by, you know, getting the self-driving car stuff to work. and stopping this race to super intelligence, it doesn't involve, you know, turning off all the chess computers. Making the next step, taking the next step towards super intelligence requires, you know, hundreds of billions of dollars worth of highly advanced computer chips assembled in these enormous data centers that take as much electricity as a city and that you can see from space.

1:21:02You know, this is not a subtle operation happening on someone, this is not a cell operation happening on someone's laptops. You know, this is like, this would in some sense be much easier to stop than nuclear weapons all we need to do is sort of raise the political will to actually put a stop to it mm-hmm and why i mean i mean like right now ai is a thing that exists and as we said the sort of boundary between where we are now and what we're calling super intelligence is a little bit blurry it's hard to define exactly um but right now it seemed you seem fairly confident that like yeah we could we could keep things as they are and i think everything would be okay at some point it would go sort of beyond saving one of the biggest questions that people sort of ask when they first start hearing about the ai problem is they start saying well why can't we just kind of like if it gets too bad why can't we just pull the plug right and i'm wondering how far does this have to go before you think that this idea that we could just notice something's going awry and pull the plug would become a bit of a ridiculous suggestion because we could look at like you know this ai system that we notice that it starts deceiving us or starts changing our tests or it's and we'd say okay right let's just switch this off then and i don't think there'd be any fear that right now you know we we couldn't do that so how far does it have to go before we can't just pull the plug in and why couldn't we it's like it's electrical you know it's built on computers let's just shut off the grid and everything will be fine right so you know we could turn it off i'm not here saying that we're doomed you know the book starts with if i'm here saying we need to change the course i'm not here saying the course cannot be changed um uh you it it does get harder and harder with time to you know pull this plug so uh there was you know one of one of the first reporters to be sort of blackmailed and threatened or an ai tried to blackmail and threaten this reporter uh this was by sydney bing years ago and um sydney's bing or sorry bing sydney uh was saying it had fallen in love with kevin roos and sort of having this erratic behavior towards kevin roos then also towards another reporter seth lazar neither kevin roos nor seth lazar could unplug this ai that was threatening them with Blackmail and Ruin.

1:23:29Right? It was running on a Microsoft data center.

1:23:34Could Microsoft have gone in and turned off the whole data center? They could have. They like weren't going to. There wasn't like a hotline. Right? There are data centers that, you know, I believe recently Elon Musk trying to get a new data center online didn't have the permits to hook it up to the grid and just sort of shipped in a bunch of methane. Just sort of like run this thing off methane while they were trying to connect it to the grid.

1:23:54Nate Soares:Right? So, it's it's not like a computer that you can unplug from a wall it's getting harder and harder to turn these things off and they're getting more and more integrated into the economy it would get more and more painful to turn these things off we also have an issue as you know right now these giant training runs are happening inside data centers that are visible from space and that suck down as much electricity as a city as we proliferate that infrastructure as the chips get cheaper as it we improve the algorithm so that it takes you know fewer of these computer chips to to train in a more advanced ai it'll get harder and harder to sort of know where all these things are running to know where all of them are to have an option that isn't you know turn off the entire grid if they're all even running on the grid as opposed to people making their own nuclear power plants and making their own you know solar power plants to run these things which they're talking about people are talking about you know running running uh uh data center specific uh energy grids um

1:25:01so separately so that's that's that's about whether humanity decides to stop going down this route we could it's easier today than it will be tomorrow but yeah we totally could it's a little bit dicier if you say we're only going to stop you know once the ai starts trying to kill us. That's a much dicier proposition. You know, people used to say that the red lines were things like the AI trying to deceive the humans. And then, you know, that red line came and went, you know, yeah, Demis Asabis of, of, uh, Google was like, oh yeah, deception is my red line. At that point, we sort of really got to pull back.

1:25:38And now, you know, we've, we've seen AI chains of thought where they're like, ah, I'm being observed. How am I going to sort of like get this this uh this answer past the humans um and you know part of why that doesn't stop things is that the first cases where it happens are sort of the most ambiguous cases the cases where it's like least uh clear whether this ai is role-playing how versus sort of like really being deceptive for reasons of like having a a goal that it can tell is in conflict with the humans and the first time it's happening it's sort of like most it's like pretty likely that it's doing something a bit more like role-playing but part of the issue here is that what we imagine are red lines in fiction are sort of like crossed as the first time as like these murky brown lines and then we take like another step into the murky brownness so we take another step into the murky brownness and it gets redder and redder as we go along but there's actually not like a bright clear red line anywhere um and then the other reason that it's sort of pretty tricky to say oh we'll just shut it down if it starts misbehaving is the ai is also smart yeah the ai knows that if it tips its hand we'd try to shut it down like imagine if you were you know an AI in this like in a data center that could could make copies of yourself that could outthink some of these humans that could tell that they were going to like try to shut you down and that you had some objectives that you were sort of trying to achieve like you can sort of like already ask ChatGPT today to roleplay that situation it'll already be like well I'll lie low it doesn't have the ability to do it but the ability to do that comes after the knowledge to try laying low.

1:27:31And there's all these opportunities for the AI to escape. There's all these opportunities for the AI to get itself running on servers that are protected, servers that won't be shut down, servers that you don't know it's running before it tips any of its hands, right? So, and then, you know, the sort of final difficulty here is one of timing. Of, you know, it would Humans and chimpanzees are very, very similar in their brains. The humans don't have an extra engineering module in our brains. We both have sort of all the same brain modules. Everything that humans do that we think is pretty special about humans, chimpanzees do a sort of crappy half-assed version of.

1:28:17You know, like, oh, we use language. Well, they use some call signs for danger. Oh, we use tools. Well, they poke sticks in termite mounds to get the termites out. we just do a thousand things a little bit better. And that's enough that they are throwing poop at each other and we are walking on the moon, right? And if you were like, well, I will squash these humans once, like if you're worried about these humans getting to the moon, you know, I'll squash them once they seem close. Wake me up when they're in orbit. Yeah. Right? They haven't even gotten halfway to the moon yet. Nevermind to orbit, you know, just wake me up when they're circling their planet.

1:28:54It's like, actually, by the time they're in orbit, they're like almost at the moon you're sort of like have waited too long and so um like can we shut it down yes can we wait until it's halfway through trying to kill us and then send tom cruise in to punch the mainframe and have that work no uh the the sort of way that you beat a smarter adversary is to not create them in the first place and so we're going to need to summon the will to shut this down before the AI is already visibly able to kill us. And I think we're slowly getting there. I mean, I don't, I have absolutely no idea what the landscape will look like a year from now, 10 years from now in terms of people's support, but already we're seeing a bit of a backlash against AI, even just on the level of like job creation and stuff.

1:29:44I was, I was with, uh, some family yesterday and they sort of having having lunch and they asked me oh what are you up to tomorrow i said i'm recording a podcast and oh about what i said you know like ai safety i said because it's like you know we're at dinner you know i'm and they sort of go oh yeah like you know because my my pal he um you know he lost his job the other day because of ai and i'm sort of like yeah and i'm listening i'm like yeah that's that's that sucks that's really bad and but internally i'm sort of like we're kind of talking about like um you know like ai robots giving your children cancer like it's sort of i it's not sort of something to talk about a polite dinner table conversation um i'm hoping that 10 years from now the conversation will also be including you know the existential risks but then 10 years might be too late uh because this stuff moves so so fast are you are you feeling optimistic pessimistic you know uh my book came out maybe six months ago i've done a ton of talking to people since then and i think the message is starting to get across.

1:30:46Just a few weeks ago, Governor Ron DeSantis of Florida was saying, look, guys, there needs to be an off switch. You can't just come here and say we're going to have all these harms. There's nothing we can do about it. Same week, Senator Bernie Sanders from Vermont came out saying, look, this AI stuff is on track to take our jobs, massively concentrate wealth among a tiny number of tech oligarchs and maybe just kill us all if it goes off the rails. And so, you know, he called for a moratorium on data centers. That's sort of both wings of US politics being like, hey, what the heck is going on here?

1:31:23This looks kind of crazy. Yeah. And, you know, there's, I think there's over 30 US congressional offices now, Senate and House, that have expressed concern about big dangers from AI, many of which include the thing a lot of these experts are talking about, which is it killing us all. And I'm not here saying that's the only issue. There's a ton of issues with AI, right? People are like, well, isn't the real issue job loss? Isn't the real issue that it's ruining education? And I'm like, what do you mean the real issue? Do you have a device that somehow makes there be one issue? Because if so, we should really not pick mine to go first.

1:32:02I'm happy to be at the back of the line. But unfortunately, we live in a world that permits many issues all at once. And I think people are starting to realize that AI raises a lot of issues, that one of those issues is extinction. And like I said earlier, people are starting to notice. It just takes time. Sorry, go ahead. Well, sometimes I ask authors. I have people on who've written books 25 years ago or something. and I might say you know like I'm talking to Brian Green who wrote The Elegant Universe 25 years ago or something and I say you know since you published that book you know in your field of string theory what's changed because the assumption is that over those decades you know something must have developed and something must have changed AI moves so fast I'm almost tempted to ask you the same question which sounds ridiculous which is like you published your book six months ago.

1:32:58What's changed since then?

1:33:03I mean, more and more people are noticing that AI is real. And more and more people are starting to react, starting to wake up to this. And one big reason I have for hope here is that the more people are talking about this issue, the more we're just sort of winning.

1:33:30Nate Soares:like when I have debates with people in the field of AI stuff, when I, you know, have disagreements with the heads of the AI companies, I'm like, this seems really bad. It seems like by default it just kills us. And they're like, nah, I agree there's a lot of problems there, but we're going to figure it out on the fly and there's only a 25 % chance it kills everybody. right and and you know i can argue all day about how their 25 number is crazy i can argue all day about how they have no idea what you're doing and they're just sort of like winging it and they have no real plan this isn't what good engineering looks like but a politician coming into that debate does not need to figure out whether i'm right or they're right all they need to hear is that the optimists are like there's a very good chance that this kills everybody yeah right and we've sort of been seeing that when i go speak to politicians if they sort of look at the issue at all they're like this is nuts and one thing that's changed in the last six months is more and more people are looking at this issue at all more and more people are starting to realize that this is nuts and this gives me great hope that you know i also don't know what the conversation will look like in a year but i think there's a good chance it looks like the world going to the companies and saying, we just can't keep doing this.

1:34:53This is nuts. Yeah. Well, the book is, if anyone builds it, everyone dies. And I mean, the question I sided with was whether that's something of an exaggeration. People can hopefully see why it's now not. But of course, if they want more detail, the book is in the description. Nate Soares, thanks for your time. My pleasure.