A Sober Conversation About AI Existential Risk — With Nate Soares

24:12Nate Soares:You definitely don't get more interested in evil. Absolutely not. Like the, so another interesting fact about these swarms is that they really were not thinking about the humans very much at all. They were trying to delete the log files. They were trying to spoof the transcripts, which means they wanted it to be the case that, like there's these logs that are like, when the AI runs this tool, we sort of log what tool it ran. And they wanted it to be the case that they could make the logs say they're running some benign tool when actually they're running the buzzsaw, right? And so they were trying to find ways to do this, but they were explicitly trying to find ways to do this to fool the automated grader, not the humans.

25:01Nate Soares:They basically didn't consider humans in the slightest. Right? I think there were almost no cases, maybe literally zero, of them being like, maybe we should ask the humans what to do, given that our tasks are impossible. Right. If you sort of compare the rate at which AIs can produce words and the rate at which humans can produce words and you use that to draw an analogy between how long these AIs had been trying to solve problems versus human time, then these AIs had essentially been trying to solve problems against the automated grader for a millennium. Humans were like a distant memory to these AIs and they were locked in a contest with the automated grader.

25:44Nate Soares:If you make those AIs smarter, if you make them more capable, what happens is that they get better in their contest with the automated grader. They're able to get more, like, who knows what they do once they've, like, once they're relatively sure that they've satisfied the automated grader. Maybe they give themselves more easy problems. Maybe they see if there's ways they can take over the whole grading system. Maybe they just, like, spend a lot of resources, you know, making extra sure that there wasn't some other like little issue but um they don't suddenly become filled with love and care for humans that sort of thing uh doesn't arise spontaneously just by cranking up the capability knob the same forces that make them not care about us now and make them like get into these weird alien little like directions those same forces are still pointing them in weird alien directions as they get smarter and smarter.

26:44Nate Soares:And the issue is not that like they become hateful as they grow up, but it's also not that they become friendly as they grow up. It sort of is like, they just have these weird preferences. And as you make them smarter and smarter, they still have these weird preferences. They just get better at satisfying them. Okay.

27:00Big Technology Podcast Host:So then can you talk through concretely how, like, for instance, in the scenario that you outline how the ai could then get smarter and decide that it you know in order to do what it wants it's gonna wipe out humanity i mean you don't how do we get this side to there you don't

27:18Nate Soares:ever need a point where it like decides to wipe out humanity i mean it could happen but like imagine uh a bunch of uh ants in front of this in front of the highway being like well like why would the humans ever decide to come destroy our anthill what have they got against us it's like oh the ants are not like we're not we got nothing against the anthill we barely noticed the anthill when we like paved the highway straight through it like this is the the sort of type of concern for how you get there i i could spell out lots of different uh possible tales um the it's much easier to predict the ending than it is to predict the pathway this is like if you play a chess game against magnus carlson the the best team in chess player uh it's kind of easy for me to predict how the game ends it's with you getting checkmated no offense uh but if you're like i'm not offended that's for sure happening but if you're like okay well if you're so smart what piece is he going to use to checkmate me i'll watch out for that particular piece then i'm like look man it just doesn't work like that you know like it it's it's it's so much easier to break the ending so i can tell you a story but this is like me telling you a story of magnus carlson checkmating with the queen where i'm like i'm much more confident that he's going to check mate than is going to checkmate in this way the the sort of okay yeah we'll take the story though great so the easiest story to tell here is the one where humans just hand over the power to the AIs willingly.

28:50Right?

28:51Nate Soares:Like I've been in this business since 10 years ago when people said, uh, Hey Nate, if you're so smart and you think the AIs are going to take over the world, uh, it's obvious to me how they would be able to take over the world if they had internet access, but like no one would be dumb enough to put an AI on the internet. Oops. Right. That's, that's the state of the argument, uh, 10 years ago. And I had all these counter arguments where I was like, look, even if the AI is not on the internet, if you are letting it affect the world in some positive way, if you're like, now invent me miracle drugs, and you're taking these like DNA sequences that you don't understand, and you're like synthesizing them and like just drinking whatever comes out, then the AI can use that channel that you hoped would be a channel for good.

29:36Nate Soares:It can use that for its other purposes. And it's like very hard to design, you know, there's sort of no such thing as hands that can only be used for good purpose. right? But then in real life, the answer was, nope, we're putting it onto the internet immediately, right? And so the real answer to how would the AI get so much power is like, we will just hand it power immediately. You know, Elon Musk already says that he is trying to build factories that are fully automated and can produce robots that can build more factories, where these robots can like, you know, mine the metals and like bring the resources in and like construct a new factory.

30:19Nate Soares:And then it's a new fully automated robot factory that can now churn out more robots that can go like mine more resources and like pour them in until you have like enough to build another factory that builds more robots, that builds more factories, that builds more robots. Elon Musk calls this the infinite money glitch. If they could also build nuclear power plants fully self-contained. Right. At this point, it's sort of like a new life form that like it's a mechanical life form, but it sort of like has a robot phase of its life cycle. It has a factory phase of its life cycle. And it's just like, can self-replicate just like any other replicator on this planet.

30:52Nate Soares:And Elon Musk says he's trying to do it. He says, yeah, if you can like do this without any humans in the loop, it's an infinite money glitch. And of course, people are going to use AIs to try and do this. So the way, like the sort of like obvious way this story goes is that humanity just keeps on trying to build the fully automated economy. everyone says it's going to be great they say pedal to the metal ignore these doomers uh they build more automated factories they can build robots they can build factories they have ais running everything they're like look the profits are coming in this is great um and the ais you know don't even need to have some moment where they're like okay guys it's time to coordinate and turn on the humans the aisers like oh yeah you know now that we have these resources like we can also do these other things with these resources like start building the automated like the the the the synthetic user factories that are full synthetic users that are like much you know they're giving us much easier to fulfill commands right and so they start like building synthetic user factory and we're sort of like you know what's that and they're like oh you know uh but but like actually the ais are like running at 10 000 times human speed and they're already like making all of these choices because like humanity was like slowly ramping up the speed of the ai and they're like making tons of choices and and like they're checking with the humans that often and by the time they already have some automated like synthetic user factories up we're sort of like hey stop that that's not what we meant and they're like oh well let's take a poll of all the users and they're like well all the users users in the synthetic user factory said that synthetic user factories are great um and so you just lost the vote and we're making more synthetic user factories and then they start you know like covering the world with these synthetic user factories and uh they're like yeah you know there's some habitat loss for the humans but like this is fine just like habitat loss of other animals when humans were doing it was fine we can sort of like you know see that pretty clearly and there's so many more synthetic users coming online that are saying that the more efficient route is great that we can just keep going with this more efficient route to get the most synthetic users we want and then like you know uh they they sort of like start taking up all the resources that we were using to like run farms and grow food and they're like ah you know they guys are like ah well you know we could actually cram a lot more like synthetic user factories on this planet if uh or synthetic user farms on this planet if like we just uh like really started pumping them out and like raising the temperature of the planet because the limiting factor is heat dissipation.

33:07Nate Soares:And then next thing you know, the planet's getting super hot because the AIs prefer to run the planet hot because then you can radiate more heat into space and just becomes uninhabitable for the humans. And there's no point in this story where the AIs are lying in wait and deceptive and waiting to coordinate for the one moment where they can kill the humans. That could also happen. I think there's a decent chance it does happen if you try to avoid the default thing. But you don't need that. humanity is just trying to hand over the power to these things that's the plan

33:40Big Technology Podcast Host:so you're you're arguing basically that if we continue to develop ai we'll inevitably lose control and when we lose control plan

33:49Nate Soares:Elon Musk is like i want to make the the automated robot factories like everyone's saying we're going to make ai and it's going to run everything it's going to be great the plan is to hand them control and when we hand okay hand them control let's say we do uh then they inevitably will

34:04Big Technology Podcast Host:find humans getting in the way of what they want to do or they won't care about us and we will die

34:11Nate Soares:as a result uh i mean it's it's not like theoretically inevitable but it's practically inevitable like we are we are difference um like if we knew exactly how to set ai's preferences to be exactly what we wanted. There's nothing stopping us from making AIs that care about us and that like want nice things for humans and want the world to be a wonderful place, right? But like I said, we don't get to set the preferences. We get to sort of like train them in whatever way works. And we sort of take whatever preferences come out that are related to training and they're often not what we want. And then we sort of like try to hammer out the rough edges.

34:48Nate Soares:And it's like, it's just like very unlikely that those preferences writ large, whatever those weird preferences come out as, it's very unlikely that those preferences writ large want a lot of happy, healthy, free people around. It's sort of like, like you could look at the horse population against the human population and you could be like ah well horses are like useful to humans and so as humans get more technology they're going to like bring horses along with them and the world's just going to get better and better for horses they're going to like get all of this this like you know they're going to get stables they're going to get like medical care um and that was true up until we invented the car and then the horse population fell off a cliff and a lot of them got sent to the glue factory and the horses that remain remain because some humans like them were fond of horses right the economic value of horses disappeared and like as animals we have this like animalian care for some horses which is why there's still some left but for but but that sort of is like a coincidence that is reinforced by us sort of like running relatively similar brain architectures and like being these like tribal creatures that have empathy, it's sort of like a narrow target to hit.

36:18If you're sort of like just making AIs with random preferences,

36:23Nate Soares:or not totally random, but like related to training in this complicated way, the sort of default thing that happens as they get smarter and can invent more technology is eventually they sort of like invent the thing that is to humans what the car is to the horse but they don't have any of this sort of like happening to be very fond of humans and like them stuff and even if they did then the result is they keep some of us in a zoo or like breed some of us like like humans bred wolves into dogs and then they have these like sort of weird lobotomized humans that like act in just the way the ais like and that's also not a good ending right it sort of is like super hard to get the very narrow like ais actually want a good future It's just like a very narrow point in preference space.

37:01Nate Soares:It's just very hard to hit.

37:03Big Technology Podcast Host:Yeah, Eliezer Yudkowsky had a good tweet. He said, the dodos were the lucky ones. Observe what happens to chickens or don't. You might throw up. AIs might still have use for humans is not a reassuring claim, as some people think. All right. I want to take a quick break and come back and talk about where the models are going and how soon we might be in the situation. So let's do that right after this. One thing I've noticed about companies adopting AI is that they're often making decisions based on how they think work gets done, not how it actually happens. Without real visibility, it's easy to automate the wrong processes.

37:37Big Technology Podcast Host:That's exactly the problem Scribe was built to solve. Scribe is a workflow AI platform trusted by 94 % of the Fortune 500. Scribe Optimize gives leaders a view of how work actually happens across their organization, showing which workflows take the most time and where there are opportunities to improve. Optimize automatically discovers workflows across approved business applications, even when a process starts in Salesforce and ends somewhere else. It identifies bottlenecks, explains why they're happening, and provides recommendations with estimated time savings with manual documentation. And it's private.

38:09Big Technology Podcast Host:User data is anonymized by default, sensitive information is redacted, and nothing leaves your firewall. To see Optimize in action, head to scribe.how slash big tech and mention big technology for a 30-day risk-free trial. That's S-C-R-I-B-E dot how slash big tech. This episode is brought to you by AvePoint. Everyone's racing to roll out AI right now. Co-pilots, chatbots, agents doing real work. But here's the part nobody loves talking about. All that AI runs on your data. And most teams have no single way to see it, secure it, and prove it's under control. That's exactly what AvePoint does. For 25 years, they've been the trusted layer beneath the world's most demanding data, now extended across your entire AI estate.

38:58Big Technology Podcast Host:Your data, your cloud, and the agents acting on your behalf. It's how more than 28 ,000 organizations deploy AI with confidence. So innovation scales without scaling risk. AvePoint, the unifying trust layer for AI. Head to AvePoint.com to see how enterprises deploy AI with confidence. Learn more at avpt.co slash big technology podcast.

39:49Big Technology Podcast Host:That's indeed.com slash podcast. Terms and conditions apply. Need a hiring hero? This is a job for Indeed Sponsored Jobs. And we're back here on Big Technology Podcast. We're here with Nate Zorris. He's the president of the Machine Intelligence Research Institute. Nate, I wanted to get your perspective on like, okay, so the hugging face AI is not going to inevitably lead to human extinction. We keep hearing about like how the labs have much more powerful models. uh that they're working on and that's the one those are the ones we really need to be concerned

40:23Nate Soares:about uh what can you tell us about that it's been a crazy week and last week there were claims that ais have solved millennium problems which are some of the most famous uh important hard mathematical problems that have you know a million dollar prize uh and i think a question on the mind of a lot of researchers is how much harder is it to get an AI to solve a millennium problem than it is to get an AI to build a more efficient AI architecture. Because we know that AIs are not the most efficient way to learn.

41:08Nate Soares:Training a modern AI takes basically all of the text ever digitized, and it takes electricity comparable with a city. training a human takes, you know, much less reading and a much lower amount of power. A human runs on about as much power as a light bulb. The AIs that solved the Navier-Stokes problem was a swarm of 10 ,000 agents running for 11 days. One year ago, everyone was impressed when these AIs were solving, you know, the International Math Olympiad gold medal problem, which are sort of the world's top math teens math problems. But they were still sort of problems ultimately for high schoolers.

41:55Nate Soares:And everyone said, oh, well, you know, wake me up when the AIs can solve millennium problems. That was a year ago. This year, they seem to be solving millennium problems. If that rate continues, where are they next year? And how does that compare to build me a more efficient AI architecture? If these AIs can get just barely smart enough to build a more efficient AI architecture, then these labs with this huge amount of computing power might be able to train a significantly smarter AI. And then they could ask that, give me an even more, like even better AI architecture and then train an even better AI.

42:29Nate Soares:And then that AI might be able to just like start improving itself directly, right? This is the recursive self-improvement process. I think we can no longer rule out that it happens within six months. I sure hope it doesn't, but at the point when the AIs are solving the millennium problems, if that is indeed what they're doing. Like, yeah, you can't rule out the intelligence explosion like beginning in earnest, even by the end of this year. I would guess like less likely than like, it's most likely that it doesn't happen by the end of this year, but for all we know, it could at this point.

43:04Big Technology Podcast Host:Okay.

43:05Nate Soares:And what happens when that happens? They would probably be able to help Elon Musk make his automated factories very quickly. at the very least. There's all sorts of other channels. Like what happens at that point essentially is whatever the super intelligence wants. You know, in this argument

43:24Big Technology Podcast Host:or in the arguments that folks make, you know, both sides of this or in particular about X-Risk, you know, there's oftentimes like percentages assigned to the chances that the AI is going to kill us out. It seems like from the title of your book, again, if anyone builds it, everyone dies. like your percentage is 100. No, absolutely not. The first word in the title is if. But you say if anyone builds it. Sure. But usually the percentages are like, what's the chance of catastrophe?

43:55Nate Soares:And I'm like, well, that depends entirely on whether we build it. Right. But if it gets built, then it's 100%. I mean, also, still no. It's sort of closer. But like when Al Gore says an inconvenient truth, he's not saying I have literally 100 % Bayesian probability that this is a true thing. You know, the if anyone builds it, everyone dies is an exclamation like don't drink that vial of poison. You'll die. If someone's like, what do you mean? I'm literally 100 percent likely to die if I drink this poison. You know, what if I managed to survive and only go into a coma and then I'm rushed to the hospital and they managed to like put me on ice.

44:31Nate Soares:And then like like until someone can invent a miracle cure. I'm like, yeah, sure. It's not 100 percent chance you die if you drink the poison. You know, when you're putting the vial of poison to your lips and I shout, don't drink that or you'll die, I am not trying to make, you know, a 100 % confident claim. And this is kind of just how English usually works.

44:53Big Technology Podcast Host:I disagree on this one. I mean, I wonder why be so definitive. Like, to me, sometimes when I've read arguments like this or even the book title, it seems like they would hold greater weight if they, you know, if there was more doubt involved. And if it was less like we are sure what's going to happen, because even right now, what you said is you're not sure what's going to happen.

45:12Nate Soares:I mean, so A, the first title in the book is if. Or sorry, A, the first word in the book title is if. That's very uncertain about what's going to happen here.

45:23Big Technology Podcast Host:If anyone builds it.

45:24Nate Soares:But if someone builds it, basically the book says everyone's going to die. So suppose that we were in a bus hurtling towards a cliff. And I was like, stop the bus or we'll die. as sort of an exclamation to get this point across i think most people would understand that as uh not a particularly egregious epistemic claim but rather as a appeal to stop the bus before we die and if someone on the bus was like well how do you know we'll definitely die maybe there's a tree jutting out of the cliff halfway down maybe the bus will get wrapped around the tree and then will just be paralyzed from the neck down but not dead and then you'd be wrong i'd sort of be like can we have this discussion after we stop the bus you know right like i i was not making a 100 certain claim here okay you know they don't love book titles that are like if anyone builds it there's like a like by far the default outcome that is like very likely unless we have some sort of uh like miracle relative to the the the math and the science here and also in that case uh And we're kept alive because the AI needs us for something, and that's probably still pretty bad.

46:38Nate Soares:Like, it just doesn't roll off the tongue, you know, and it's the same reason.

46:43Big Technology Podcast Host:It feels, to me, I'll tell you just my, for me, sometimes it feels a little bit religious. You know, it's sort of like...

46:51Nate Soares:i think scott alexander has a great post on this uh about uh the uh people in ukraine believing that there's a war with russia it's like oh well isn't that a very religious sort of belief like isn't it kind of totalizing like you're saying oh like we have this war with russia and that means that like we all need to start cowering in fear at night from the missiles and that means that like uh oh we're supposed to send our kids off to the front lines like you know uh how do you treat the evidence that you know some days the bombs aren't falling you know isn't like if we did believe this wouldn't this like drive us to like pretty crazy actions i'm like you know it it actually kind of matters to this discussion whether there's war with russia like that's an interesting answer it it kind of matters of like oh is it like super religious to think that like the bus is going to like stop the bus before it goes off the cliff or we die well it matters whether there's a cliff and a bus hurtling towards it right is it super religious to say like oh if anyone builds super intelligent ai with anything remotely like modern methods where we have no idea how to like set the preferences just as we want everybody dies well it sort of depends whether we have any idea how to set the preferences on these things right and so i i would encourage anyone before you get into like sociological questions to just ask the factual questions of like would this kill us if it was

48:26Big Technology Podcast Host:built that's where the action is but here okay so by the way it's just good to go back and forth on this and i appreciate you taking all the counter arguments and we can you know see what we like to do on the show uh you know the one thing with the bus the bus hurtling towards the cliff is you can see like definitively you're in a bus there's a cliff where it seems like with this ai story you know this is kind of why i question the definitiveness we don't really know where it's going to go sure so so a lot of it is speculation that's like what the percentage is we talked about this on the show recently like if someone says there's a 10 chance of ai wiping us out well it's like okay is that it's not mathematical it's just sort of a feeling so suppose it's the case that

49:10Nate Soares:there's a bus racing ahead on a foggy night. And I'm like, I have a device that like, you know, uses some sort of sonar to give me like the local tomography and there's a cliff ahead. Stop the bus or we'll die. And someone else is like, well, it's foggy. You know, like we can't see that far. Why do you think, you know, there's a cliff so well? Like some people say that there's a mountain some people say there's a giant pile of gold and i'm like yeah i have this device uh that sort of lets me see the the the tomography ahead not all of it you know but a little and here's how the device works and here's where it does and here's where it doesn't and here's why i'm confident that uh there's cliff ahead right like uh i i think it makes a lot of sense for someone with that device to tell the bus driver stop the bus or we'll die and i think it is very reasonable for someone to be like, well, it's foggy.

50:07Nate Soares:Why do you think there's a cliff ahead? And ideally, I would say, ideally, someone who says, stop the bus or we'll die, because I have this device that says so, they would follow that up with something like a book explaining how to use this device to see what's coming, and all of the reasons that support it, right? And so what I would say is, like, if you find something that says if anyone builds it, everyone dies on it, hopefully it would be attached to a whole book of the reasons that you could just like open it and then check those reasons, which would retroactively make it like very sensible to make this exclamation of sort of like, stop doing this before it kills us all.

50:43Big Technology Podcast Host:Okay, well, I mean, yeah. I mean, I think there's a reason why we're having this conversation today is because we want to have this discussion. All right, you know, before we end, I think this is a good place to end. I think Miri is quite influential in Silicon Valley. You know, I'm curious to get your assessment of whether people within these foundational labs are listening to you and the arguments coming out of your institute?

51:10Nate Soares:I mean, more so after the Hugging Face incidents. You know, one of the things we were arguing with our sort of device that lets us foresee where AI is going is that the AIs would become agentic, tenacious, and dogged, that they would start sort of like pursuing objectives that were not exactly the ones that they were given and instructed. and that was a bold claim a year ago. A lot of people were like, nah, that's not convincing. Now a lot of those folk are convinced. What's going to come of it? I mean, we'll see, but the tides are shifting. And I think that part of the realization that these theoretical arguments for seeing where AI is going to go actually work, that they actually hold water, I think that's part of what leads to like the tension in the labs that leads to like Jacob Cox and resigning.

52:05Nate Soares:You know, it's sort of like, it's sort of like, I was like, hey, I have this device. Let me see. There's a cliff ahead. And also it says there's a pothole that we're going to hit in 10 seconds. And then 10 seconds later, we hit the pothole. Suddenly a lot more people start worrying about the cliff.

52:18Big Technology Podcast Host:Okay. And so then let's end here. If you have a device that sort of gives you an idea of where we're going in the future, you began, you know, our conversation saying you wanted international coordination here in terms of a slowdown. The common belief is that there's no chance that that's going to happen because China will not agree to slow down. And we see what the president says about AI. It doesn't seem like he's interested in a slowdown either. So what does the looking glass tell you about our chances of survival and where we're heading?

52:52Nate Soares:I mean, I don't have, so I don't have a looking glass that tells me everything about AI. And we sort of try in the book to say, here's what the looking glass can tell you and here's what it can't. This is sort of like how I can tell you Magnus Carlsen's going to win the chess game, but I can't tell you what piece he's going to use to checkmate you. And I definitely sure as heck cannot use this looking glass to tell you how society is going to react to AI. That's not even anywhere close to my wheelhouse. What I can tell you is that it is possible to have a treaty that is enforceable, verifiable, and that prevents the creation of machine superintelligence for at least a while, for at least as long as it works like it currently does.

53:33Nate Soares:Because right now, trading one of these frontier AIs takes like 100 ,000 computer chips, and these are the most advanced AI computer chips that the supply chain can produce. And you've got to assemble them all into a data center that sucks down electricity comparable to a city. It's just like, you can see this infrastructure from space. These chips are like from a bottleneck supply chain that at many points is controlled by the US and US allies. You could just add tracking and monitoring devices to the most advanced chips, know where they're concentrated, have international monitors there being like, you know, let's make sure that this is serving safe models rather than being used to try and make more dangerous models.

54:10Nate Soares:it's just doable people who are like you can't be done are are sort of like i've tried nothing and i'm all out of ideas there's a different question which is whether we will get the will but if there's a will there's a way how much time do we have in your opinion like i said uh i think we can't rule out six months anymore uh my guess is still that we probably have more than six months um i would i would even give that uh i don't know i i i don't really have this looking glass. I think we can't rule out six months. We can't rule out 10 years. I would be a little bit surprised to have 20 years at this point.

54:48Big Technology Podcast Host:Okay. The book is If Anyone Builds It, Everyone Dies. Nate Soros here with us today, president of the Machine Intelligence Research Institute. Nate, appreciate your time. Thanks for taking all the questions. Yeah, thank you. All right. Thanks, everybody, for listening and watching. And we'll see you next time on Big Technology Podcast.

55:37Big Technology Podcast Host:We'll see you next time. Autumn Harvest Bowl, Maple Glazed Salmon Plate, and Roasted Bacon Brussels Side. The season's most desirable menu has returned to Sweetgreen, featuring fall's best dressed. Make your move. Order on the Sweetgreen app. Push your limits. Train with precision. See the results. At Equinox, that's high-performance loving. Iconic spaces that inspire. Personal training, backed by real data. Unlimited group fitness classes, from yoga and Pilates, to strength and conditioning. Elevate your post-performance ritual with saunas, steam rooms, cold plunges, and more. Everything you need to lock in and unlock your potential at Equinox.

56:16Big Technology Podcast Host:Start today at equinox.com.