AI's Doomsday Philosopher Says Maybe It'll All Be Totally Fine — With Nick Bostrum

0:00The Oxford philosopher who warned the world about the dangers of AI superintelligence comes on to talk about how we might end up in utopia instead. and whether we should want that. That's coming up right after this.

0:14You're used to hearing my voice on the world bringing you interviews from around the globe. And you hear me reporting environment and climate news. I'm Carolyn Buehler. And I'm Marco Werman. We're now with you hosting The World Together. More global journalism with a fresh new sound. Listen to The World on your local public radio station and wherever you find your podcasts.

0:41Welcome to Big Technology Podcast, a show for cool-headed, nuanced conversation of the tech world and beyond. We're here today with Nick Bostrom. He's a philosopher and the best-selling author of Superintelligence and also the author of a new book, Deep Utopia, Life and Meaning in a Solved World. I have it here with me today. Nick, welcome to the show. Great to see you again. Good to see you. You know, I've really struggled to figure out, like, where exactly to start this interview. because I was going to ask you about like your past, talking about superintelligence and the dangers of that or the beginning of utopia.

1:12And then, you know what I said? I'm just going to go back to our last conversation that we had. I'm not sure if you recall it, but I was writing my book Always Day One and I had a Black Mirror chapter talking about what could go wrong with artificial intelligence technology. And I was like, all right, I'm going to call Nick Bostrom for the AI Black Mirror chapter because you became famous predicting or talking about the probability that we might end up with dangerous super intelligence, and we should be prepared for that. And this is what you told me. You said, I don't necessarily think of myself as being in the dark.

1:42Mirror people come to me for a quote on the negative side of AI, and then other people will read me saying something negative, and then more people will come to me to get the negative side. It kind of gets to be self-amplifying, and people then assume I only have negative things to say about AI. We spoke in 2019, and now you have this book coming out a little bit about how we might end up in utopia with AI. And of course, there are some concerns there. But did I kind of catch you in part of the journey between the dangers of superintelligence and maybe the utopic living that we might end up in?

2:15Not really. It's always been there, both these sides, in my thinking and my expectations. The previous book, Superintelligence, did focus most of the pages on what could go wrong. That is true. it seemed at the time more pressing to try to draw attention to the potential risks of developing artificial general intelligence and super intelligence so that we could hopefully develop ideas for how to avoid those pitfalls. But even back then, it always was clear to me as well that if things go right, they could go very right indeed. And that there was this enormous upside as well. And the more recent book tries to like actually analyze what that would look like if things, as it were, go maximally well.

3:04And we're going to get back to some parts of the interview that we did back in the day as we talk through like what this utopia can look like. In the meantime, I do want to talk to you about what it was like to be effectively this focal point for AI fear or negativity. Even if you had this more nuanced view, you certainly have lived sort of the perspective of being this vector for people with fears of AI, they would come to you. And this is from this New Yorker, a story written about your book in 2015. The title was The Doomsday Invention. And this is describing your arguments. True artificial intelligence, if realized, might pose a danger that exceeds every previous threat from technology, even nuclear weapons, and that if its development is not managed carefully, humanity risks engineering its own extinction.

3:51Central to this concern is the prospect of an intelligence explosion, a speculative event in which AI gains the ability to improve itself, and in short order exceeds the intellectual potential of the human brain by many orders of magnitude. So first of all, how do you feel looking back to that portrayal? Well, not wrong. I think still today that there are very substantial risks in developing greater than human machine intelligence, including existential risks. They have become a lot more widely recognized since the book Superintelligence came out. Back then, it came out in 2014, but it had been in the works for six years prior.

4:37At that time, this whole idea of, well, A, superintelligent AI in the first place, and B, that it could pose existential risks was very much neglected. Certainly academia completely ignored that until the book came out. And more broadly in the world, there were basically nobody, maybe a handful of people on the internet who were starting to think about the alignment problem. That has radically changed in intervening years. Now there are many research groups focusing specifically on trying to develop scalable methods for AI alignment, including all the frontier AI labs. Also, the governance challenges have started to receive a lot more attention, including from top-level policymakers in the last two years.

5:22So the landscape has shifted dramatically. And I guess it's in some sense validating to see that these concepts that used to be super fringe and totally outside the Overton window are now kind of very mainstream. Where do you think society is in terms of their concerns with AI now? Is there a proper amount of concern? Are they overly concerned? Because the truth is that the sort of doom contingent has really elevated dramatically and is very prominent, you know, even in the most serious research houses, open AI, Anthropic, etc. Yeah, there has been this surge of the doomers in the last, I mean, two years or something like that, where they really started to become vocal, like on Twitter and stuff.

6:06I think probably we are still below the optimal level of concern, but the kind of first derivative is that we're moving rapidly towards increasing levels of concern. And I've started worrying a little bit about us not, as it were, increasing the level of concern to the optimal level and then stopping there, but kind of overshooting. If this negativity about AI snowballs, like it might, in some scenarios, get out of control. And we might then lose out on the enormous upside that could be unlocked by developing the good kind of superintelligence. What does that overshooting look like? Is it just that these concerns grow, I mean, bigger and bigger?

6:52Well, there are different ways. I mean, one scenario class, which still looks unlikely to me, although less unlikely perhaps than two years ago, is that you could imagine so much stigma developing that we get some ban. Like maybe it starts with a pause or something, but then like after six months, nothing has really changed. So you extend the pause, then you maybe set up regulatory agencies to police it to then have and all they are doing is trying to prevent dangerous AI like and or just a kind of negativity might might develop into a new orthodoxy where like anybody who says anything positive about AI gets gets cancelled or shadow banned it just becomes like impossible to and so it still seems a bit unlikely but we can't really accurately predict the kind of social dynamics that will be playing out and now under different circumstances than historically because we have new technologies available now that might at some point make it possible to lock in an orthodoxy in ways that were never possible historically because you could have much more fine-grained control over what people are saying.

7:59So that's one class. Another might be that instead of AI being banned, it kind of gets escalated to the national security level, which may or may not be good, but you could certainly imagine scenarios in which instead of it being a kind of peaceful pursuit for civilian purposes by cooperating scientists generally motivated by some sort of, you know, humane cosmopolitan goals, like it becomes more like a kind of how can we get an edge over our military rivals? And it's not clear whether in that context, the outcome looks rosier than if it were more like a kind of freewheeling civilian enterprise.

8:36So that could be like a concern level with possible negative effects, even short of stopping AI development altogether. I mean, if you look at today's large language models, and I know we know this isn't the end of development, but they are very smart in some areas and really stupid in some areas, and they continually fail tests that are put in front of them to try to demonstrate real-world understanding. Where does this fear come from? And what could eventually happen as we continue this path of development that could lead to some of the negative outcomes that you and others anticipate as one possibility.

9:10Yeah, I mean, I don't know how much we should read into the fact that current like AIs still have limitations, that that will remain true until it isn't true any longer, right? But the idea has always been that we can anticipate that AI capabilities most likely will continue to increase and at some point AI will succeed. And then we can realize that at that point, like some really powerful dynamics will kick in. And we could try to use the intervening time to prepare ourselves in different ways, including ideally solving the alignment problem, you know, maybe making some progress on the governance challenges and on like the ethics of, there's a bunch of difficult ethical questions that will arise with like the moral status of digital minds and the distribution of benefits, like a lot of quite tricky questions that we could use these intervening years to try to, you know, think about.

10:03But unfortunately, we wasted most of the time we had. I mean, to me, at least, even back in the 90s when I started thinking about these types of things, it was clear that we had a good chance of eventually developing AI and that it was going to pose these risks. We could have used those decades to do our homework. And it's become increasingly clear. But really only in the last maybe five years or so has to have been like a significant effort to actually work on these issues. So we still have some time left on the clock. We don't know how much, but hopefully we can at least make a good use of the remaining months and years.

10:42Yeah. Why do you believe that it's a certainty that will sort of achieve artificial general intelligence? Certainty is too strong a word, I think. But you feel it's very likely. Yeah, at least conditional on science and technology continuing. And we could have some sort of civilizational collapse or we could eventually go extinct or destroy ourselves in some other way, not unrelated to AI. There are developments in synthetic biology and other new weapon systems and whatnot. But if kind of these development efforts in hardware and in algorithms continue, then it looks very likely that we will succeed in this.

11:25I mean, from first principles, we have the human brain as an existence proof that general intelligence is possible. There is no reason at all to suppose the human brain is in any sense optimal, like neither from a hardware point of view, nor presumably from the algorithmic point of view. So just as we have machines that are physically much stronger and faster than human bodies or any animal body is, then likewise we will have eventually cognitive systems that are much faster and clever. So that's like a very high level argument. But then you can also look just at the kind of advances we are seeing where more and more things that used to be impossible for AIs to do have become done.

12:08And there are just a lot fewer left of these kind of milestones. There is this phenomenon that like before it is done, it looks really hard. Once AIs have done it, then we kind of quickly forget just how impressive it was and just take it for granted. Like, yeah, of course, computers can play chess. Like, of course, what's the big deal? But like at the time, like it was a big, and then you can see like, oh, we have, they can play Go. They can see, they can imagine, they can write poetry, they can write computer programs. They can talk to us in ordinary language. They can pass undergraduate level exams in all these different subjects.

12:49This is a lot of stuff that to people 30 years ago would have been like, wow, you must be really close to AGI if you have done all of these things. Okay, so let's talk about Utopia a little bit. Just give us your perspective on what could go right in the best case. Well, why don't we do this way? What could go wrong in the worst case scenario of AI? What could go right in the best case scenario of AI? And how do humans have an influence in terms of which direction we go? So I think there is like the real existential risks that will arise as we develop. And possibly not even super intelligence, but you could imagine even something short of that, making it very easy to develop new weapons of mass destruction using synthetic biology.

13:35Are you more concerned about humans using AI to hurt each other or AI hurting us? Well, I think they are both worth worrying about. I think with the X risks, I mean, maybe slightly larger on the AI being the kind of agentic part there. But certainly, we really need to do a decent job on both sort of the alignment and the governance for us to have a good outcome. So, yeah. So now on the upside, like there is a huge unlock. And a lot of that is just removal of a bunch of negatives. If you look around at the world as it is now, it's not that rosy a picture. In many ways, it's quite a horror show with people dying from, you know, Alzheimer's or kids getting cancer and like starvation and people being bombed and like all kinds of, or just at the more mundane level, People spending most of their adult life, you know, working in a sort of boring occupation that gives them no fulfillment, but they just have to do it to, you know, to pay the rent.

14:43And like, you know, headaches and stomach aches and like all kinds of just the totality of all of this extreme misery and very common, more everyday misery. That's just within the human sphere. And then you add the animal kingdom with like ameliorating all that suffering. would already be, I think, a very strong argument that something at some point needs to be done here. Can't like go on like this. I mean, do we want like another, you know, 10 ,000 years, 100 ,000 years of just this? But I think on top of that, there is like the potential to also unlock like new levels of flourishing beyond those which are possible even under ideal conditions in the current world.

15:30That's a lot harder to paint a very concrete picture of because we are sort of limited in our ability to imagine and appreciate just as, you know, if you imagine like the great ape ancestors of Homo sapiens kind of thinking about what could be so good about being, you know, human. And so they might like realize a few things like, oh, we could have banana plantations and have like a lot of bananas and stuff. and that is true. We can have a lot of bananas now, but there's more to being human than just unlimited bananas, right? Like we have sort of, you know, music and poetry and film and humor and romantic love and like all kinds of stuff, science.

16:14So similarly, there is probably like if we unlock, as it were, a greater space of possible modes of being, there are some in there, I'm sure, that are extremely valuable, that I think AI would be the most plausible path towards realizing. If I then really dives in and tries to think more specifically about what would the best possible continuation of a life starting from our current human starting point look like, then there are some quite interesting philosophical questions that arise. And so this book, Deep Utopia, it's not really an attempt to sort of, well, before we looked at the downside, Now let's make the case of how wonderful the upside could be.

16:56I think the upside could be extremely wonderful, but that's not sort of the thrust of the book. It's more like, let's just look at this. What would happen if we actually did succeed in creating a solved world, as I call it, like where all the practical problems are already solved, or to the extent that there are problems that are not solved, they are in a way better dealt with by advanced AIs and robots than by us. And there is like some aspects of that condition that at least prima facie look quite unappealing to our current sensibilities. We often define our sense of self-worth on the idea of being a contributor.

17:40Like you're a breadwinner or you make like a positive difference in the lives of your friends or of society at large. You bring value to the world. so much of our existence is kind of constructed within the constraints of various instrumental necessities that have been with us since you know the dawn of the human species there have always been a lot of things that we need to do just to survive and if you remove all of those there is at least initially the sense of kind of disorientation or an undermining of like we feel like kind of what's the purpose? Like we would just be these blobs. But this is different from what we spoke about the last time you and I were on the phone.

18:28This was in 2019, I think. You said that we'd have to find some news source. This is from our conversation that I put in my book. We'd have to find some news sources of self-worth. But in Disneyland, the job of children there is to enjoy the whole thing. And Disneyland would be a rather sad place if it weren't for the kids. So you say we would all be like kids in this giant Disneyland. maybe one that would be maintained and improved by our AI machine tools. So effectively, that even if we didn't have to do any sort of sustainment work that gets turned over to AI, we could be actually quite fulfilled in life.

19:01Yeah. So how do you get from there to where you are today? Well, I think that's basically correct. But we can distinguish two different senses of fulfilled or of having purpose. So there is, first of all, what you might say, the subjective sense, like the feeling of fulfillment or the feeling of having purpose. Like it's like the emotion of being motivated and you're really excited about what you're doing, right? Like that kind of psychological state. That certainly you could have that in a solved world, in utopia. The utopians could have like extreme levels of motivation and immersion and subjective purpose.

19:41That's easy. That's like a checkmark. And more broadly, you can go through different plausible human values, and for some of them, you can just right off the bat say, well, yeah, sure, of course, that would be trivially easy to do in utopia. So in this case, through the psychological engineering techniques that they would have. I mean, already, you could have it like, imagine a drug without side effects and without addiction potential that just induced a state of fascination and motivation. We already have simple versions of that, but you could imagine far more sophisticated ways that would give the utopians like very fine-grained direct control over their mental states and their psychology.

20:21That would follow from technological maturity. Right. So that's easy. Some people, however, think that there is also a more objective concept of purpose, where it's not just that you feel motivated, but that what you are doing is actually objectively worth doing. and that's a little bit less obvious to what extent the utopians would have that in as much as at least at first sight it looks like anything they could do they wouldn't have to do because they could just press a button and a machine could do it instead except for those those few things that you bring bring up that actually we want humans to do like for instance like ordain a marriage right like that is something or read a poem people might want this to be humans Potentially, yeah.

21:08So you could automate everything, but except there might be certain jobs where the, as it were, consumer has a direct preference that the job be done by human. In which case, as it were, almost by definition, it's not automatable. Exactly. And this whole idea of a solved world is basically where AI can effectively take care of everything, all of our needs, all of the production. And we are in this utopia because the machines have done all the hard stuff that we don't want to do anymore. So you could imagine like a very dystopian scenario with advanced technology if you have like a sort of totalitarian despotic, right?

21:42But imagine also that to whatever extent governance problems can be solved, they have been solved. Maybe you can't solve governance, but to whatever extent they're like better and worse in terms of social political structures. Imagine like we get something at the good end of that combined with technological maturity. Then that's basically the definition of a solid world. Sorry, I know I took you on a bit of a tangent there. No, so yeah, so like that kind of layer. So you could say first, well, you could have like a simple utopia might be a kind of post scarcity utopia where we just have abundance of material goods.

22:20So we already, if you're fortunate enough to live in like a developed country, you know, with a decent education, et cetera, you're already pretty close to that. You might not be able to have like the ideal yacht of your dreams. Like there are some limitations. But if you sort of plot a line that has like the starting point hunter-gatherer and the end point is like complete post-scarcity, I think we are more than halfway there. Like it's a bigger difference to go from not having enough to eat to having enough to eat than, you know, to get like a slightly more advanced version of an iPhone or like a third house if you already have two houses.

22:58It's like diminishing returns. So that's like, you could first consider this concept post-scarcity utopia. Okay, so then what's a level of, as it were, more radical utopia than that? Well, you could have a post-work utopia where not just do we have plenty, but we don't have to work to produce that plenty. So it's not just that we work all day long and then we have a lot of money and we buy stuff. But imagine you had all this plenty and you didn't have to work. is slightly more radical conception, but not that radical. I mean, there are already people who, you know, are born with a trust fund or something, and they never have to work, and they have plenty.

23:36You know, again, there's limits to how big their palaces could be, but, like, at least some approximation. But I think we can then go further and consider even more radical conceptions. So I've already alluded to, there is the post-instrumental utopia that you could have where it's not just that we don't have to work like to make money but we also don't have to do any of the other things that we currently have to do for instrumental reasons so if you know if you are Bill Gates you still have to brush your teeth you still have to you know do a whole host of things just in your everyday life to get the outcomes you want.

24:18There's like a limit to how much you could ask your assistant to do or that you could, you know. But in this scenario, like a lot of those other instrumental reasons we have for doing things would also drop out of the picture with like super advanced automation technology. And I think there's like a step farther than that, which is a, I call it a plastic utopia, where we also have complete control over ourselves, over our own bodies, minds, and mental states using advanced biotechnologies or neurotechnologies. I think that's going to, we're going to achieve that. That's wild. Yeah, but I think if you consider what would be possible at technological maturity, which we can at least play some lower bounds on through a kind of theoretical analysis, So we can sort of estimate what kinds of computational systems could be built.

25:14We can see what kind of molecular manufacturing systems are possible to build in our universe, even though we can't currently put them together with the tools we have now. We can see that there is like a path there. Other things like cures for aging and stuff like that, like we don't have them yet, but there is no, you know, no laws of physics prevent people, you know, from living indefinitely if you had like repair technologies. et cetera, et cetera, perfect virtual realities, perfect like ways to manipulate the brain, like that they're better than drugs. Like there's like a whole, you could, there's this kind of, in fact, like a table in the book that outlines some of the affordances you would have at technological maturity.

25:55And maybe there will be additional things we haven't yet thought of, but at least these. And so they would, I think enable us to instantiate this condition of plasticity where human nature itself becomes malleable. It's crazy. So that means there are further questions about purpose. So right now, like if you didn't have to work for a living, like maybe some people would say, well, you know, maybe I would start, you know, going to the gym or to get fit, right? Like that's, you can't hire a robot to, you know, go run on the treadmill on your behalf. But with plasticity, there would be a shortcut to that.

26:33You could pop a pill that would induce exactly the same physiological and psychological. I think we're already on the way there with those MPIC. Yeah, exactly. That's like one more step in that direction. And so the thing is with superintelligence, you get the telescoping of the future. So all these sort of science fiction-like technologies that maybe we would develop if we had like, you know, 20 ,000 years for human scientists to work for it, we probably will have a cure for aging and perfect virtual reality and space colonies and all the rest of it, right? but all of that could happen very quickly if you have super intelligence doing the research and development.

27:11So you get this kind of telescoping of the long term. But yeah, so then there's like a further set of things that currently fill the lives of people that we wouldn't need to do, including things we do for fun. So maybe some person say, well, you know, if I didn't have to work and I had like, maybe I would play golf all day long Like, why would you play golf for a long while? Because it's fun, it gives me joy. Let's suppose somebody says that. Well then, in this condition of plasticity, there would be a different and easier way for them to get joy. They would just pop a pill and they could get exactly the same level of subjective well-being as like a beautifully manicured golf course could induce.