#20 - Nick Bostrom - Utopia, Superintelligence and Digital Minds

0:00The problem of deep utopia is what would we do in a solved world? For example, to know mathematics is to invest a bunch of time and effort into learning mathematics. But at technological maturity, if you could just sort of download the algebra 2 module at the press of a button, there would be no need to exert effort to learn mathematics. You can kind of start to cross out a lot of these activities. So then you enter this more problem of deep utopia, which is more radical, where you have to start to rethink, really, questions like the purpose and meaning and ultimate value. Hello, everyone, and welcome to the Win Win podcast.

0:39Today's episode is all about utopias, dystopias, and thought experiments, because I am speaking to Nick Bostrom. Nick is one of the world's greatest living polymaths. He's worked as a philosophy professor at the University of Oxford for the past 15 years. Plus, he's also got an extensive academic background in theoretical physics, computational neuroscience, logic, and loads more. He's probably best known though for his work around catastrophic risk and AI. He wrote the book Superintelligence around 10 years ago, a seminal work that has influenced many of the people you nowadays associate with AI.

1:14Today's episode mostly focuses on his brand new book, Deep Utopia, Life and Meaning in a Solved World, as big a topic as one could really try and address. So on that note, here is my and Igor's conversation with Nick Bostrom.

1:41So Nick, thank you so much for joining us. One of the reasons, one of the many reasons I wanted to speak to you today is because you have just written this book, Deep Utopia, that should be coming out just as we release this episode. Given that you've sort of historically been known more for your writing around global catastrophic risk, this is quite the pivot to now be talking about Utopia. So what inspired this change? This little ray of sun shining through the dark, gloomy clouds. No, I think both sides have always been there. And in fact, there have been little glimpses. I wrote something called A Letter from Utopia a good many years ago, which was an attempt to poetically evoke some of the positive things that could be attained.

2:30It's just that it seemed more pressing at various points in the past to try to do some analysis and draw attention to various things that could go wrong so that we could avoid them. And then if we avoid these pitfalls, then eventually we'll have plenty of time to think about what to do with this big future that we would have secured for ourselves. Yeah, I think it's a misconception that exists that many of the people that work or think about existential risk or global catastrophic risk do so because they like thinking about it. It's more that, no, I like thinking so much about the good things that could come provided we avoid the pitfalls on the way.

3:13That's why it's an instrumental thinking about the pitfalls rather for the ends of achieving the utopian potential. Yeah, yeah. So it's not like kind of some big fascination with film noir or sort of dark that makes most people, researchers, go into this field, I don't think. I guess to dig into the topic of utopia, it would help to try and define exactly what it means because so many people have different impressions of it. So could you take us through some of the various different definitions that you give in the book? There are a lot of concepts and ideas, but I guess one key concept is that of technological maturity, which I define as a condition in which all the technologies that are physically possible and for which there is some feasible pathway of development have been developed, or at least some reasonable approximation to that.

4:20I should say the book, it brackets a lot of things. In particular, it brackets the question of how to get from here to there. So it kind of sets aside all the practical difficulties that lie ahead of us in order to sort of get to the point where you can actually ask the questions of what then, And, you know, whereas superintelligence, the previous book kind of asked what could go wrong with AI. At least 95 % of the pages were on various risks. This asks kind of what if things go right. So then I think eventually we have like condition of technological maturity. So that's kind of one word that helps set some of the premises.

5:08then there is the concept of a solved world which you might think of as a world in which we have technological maturity but on top of that we have also solved all coordination problems or political problems or fairness problems so you have a world that is in some sense but maybe with important qualifications maximally malleable to human wishes or values The problem of deep utopia is what would we do in a solved world? If human instrumental effort becomes OTOs and we don't really need to do any, the robots can serve our food. And then what gives meaning and purpose to human existence in such a radically transformed condition?

5:59Yes, you also go over the different types of utopias that have been previously described or that one could describe as kind of like a governance and culture utopia, which I suppose is like the brave new world or Marxism tries to achieve as well. Very badly. Which is usually not a technologically enabled one, right? In difference to the other ones that you described like a post-scarcity, which is kind of like the lower level where we now have abundant resources, I suppose. a post-work utopia, post-instrumental, or even the plastic one. So the one you just talked about is closer to the plastic one, meaning high malleability.

6:42I found this delineation between the different types helpful in that one notices that the different utopian writings that have existed talk about actually different stages of it, where a post-scarcity one is still one where people might work, but a post-work one is where you have now full automation. And then post-instrumental would be where you can't even learn for the sake of knowing the thing later because you can just download something. A lot of efforts in this realm, utopian imagination, I think have kind of stopped early in a journey that can go on for a lot longer. So, and it's very natural.

7:27I mean, so you have like the kind of the most primitive form of utopia or there's like a whole, almost every culture had this. I mean, if you imagine you're some medieval peasant who works all day and like you eat some porridge and you're still hungry. Like the kind of obvious thing you dream of is a state of abundance. Just having like a lot of food and you can rest and, you know, money grows on trees. and maybe if there's intense social pressure and constraint, maybe you're also imagining a kind of feast time, a party time where social mores are loosened up. And that's already enough to be a fantastic fantasy in that condition.

8:10Then you have the class of utopias that are really more stand-ins for, like it's kind of symbolic political struggle. So you have some utopias that warn of various tendencies in contemporary society, which if allowed to continue would lead to some more overtly bad condition. And so you have like 1984 or Brave New World and a whole bunch of other sort of political writings in the guise of utopias. And then more recently, you have these attempts to think through what about automation? if AI is doing all this stuff, what will happen to the human labor market? And the more radical forms of that think, well, what if robots could do all economic human labor or all except with a few carve-outs?

9:00Then maybe you need some universal basic income and you could have some discussion about that and what would happen to the education system and culture in that kind of world. But I think there's like a step farther on this journey, at least one step farther, which is to realize that it's not just human economic effort and labor that would become unnecessary in a solid world, but all kinds of other human effort as well. So you mentioned learning, like that's one thing that you might spend effort doing now, because the only way, for example, to know mathematics is to invest a bunch of time and effort into learning mathematics.

9:42But at technological maturity, if you could just download the algebra 2 module at the press of a button, there would be no need to exert effort to learn mathematics. Right now, if you want to be fit, you have to go to the gym and work out. But if you could just pop a pill and your body does the same thing. And so you can kind of go through activity by activity, the things that would currently fill the days of somebody who had achieved, I don't know, leisure, entered retirement and have a lot of money. You can kind of start to cross out a lot of these activities, or at least raise a question mark above them.

10:25Because maybe we would choose to do some of these things anyway, but it seems that a lot of what gives them their allure today is that there is some outcome that you secure by doing them. and that would be a kind of sense of pointlessness. Yeah, you could study mathematics, but why if you could get equally good at mathematics by downloading it? Or maybe people like to go shopping as a fun pastime, but part of maybe what gives that activity its allure is that at the end of the day, you might end up with some item that you really like that kind of enhances your life or it looks good on you if it's a clothing.

11:07But if you had a kind of recommender system that could just pick out better stuff than you would pick yourself and then order it, then is it still as fun? If the upshot of that is that you actually get worse items than if you just let the AI do its thing. And so then you enter this more sort of, yeah, the problem of deep utopia, which is more radical, where you have to start to rethink really questions like the purpose and meaning and ultimate value from the ground up. Right, because what it sounds like you've just described is basically a world of instant gratification. Anything your heart's desire, you can just have immediately without any kind of striving.

11:51And yet, so much conventional wisdom. Certainly, my intuition is going off going, ah, this is bad. Like, you know, there is, we derive meaning from the struggle so often, or at least from striving in this sense of achievement. Do you think that is actually a real problem? Or is that something that a future society will be able to solve for as well? And if so, how? Well, first of all, it's not really so much a book of conclusions as a book of that, like, helps maybe the reader to ask questions and to think about them, to sort of bring them into conscious awareness. And second, it's not, it's an ambivalent book in many ways.

12:27I think ultimately hopeful, but it's not trying to lay out the case that the problems are not really real and that it's all hunky-dory. I think there are kind of, once you start to think through this, a lot of, I don't know, disillusionment or something. I think once you get through that, you can maybe construct something really good at the other side. but it will be probably in many respects quite non-human or feel a little alien, at least until you get used to it, the kind of existences that would make sense in a post-instrumental condition. These various instrumental constraints that we currently face, all of us, I mean, even people who don't have to work for a living have to do a whole lot of other things.

13:14If they want to have a decent life and they have to brush their teeth, they have to spend time doing this, that, and the other and put an effort. I think it forms just as insects have a kind of an exoskeleton that holds the gooey bits in place. I think these instrumental constraints that we face in our lives kind of hold the human soul together. And if you imagine removing that, then there's the question of what we could be other than a kind of blob, like a drugged out pleasure blob or something amorphous. And you also point out things that probably couldn't be done by machines, even in that utopia.

13:56And you're right in particular what I found was a cute example that sentimentality might be one of those areas. A child's work with crayons may be especially dear to its parents precisely because it was the child who made it. This little labor might be harder to automate than the work of a neurosurgeon or a derivatives straighter. I like the idea for current parents to assume, to keep that in mind when they see the child's drawing. What other things do you think might stay uniquely human, even in that plastic utopia? Well, I can look for places where it would be instrumentally necessary for humans to put out effort.

14:37We might put out efforts for other reasons as well, just because if we think putting out effort is intrinsically valuable, we might choose to do it for that reason. But one might think that there is some special value in putting out effort and where there is an instrumental reason for doing it, that something is achieved or done or accomplished that couldn't be accomplished without you actually making the effort. One might think that those kinds of setups have particular value. And there, I think, a lot of the potential for those have to do with various forms of social cultural entanglements. So basically, let's suppose you have one of your value preferences is that the other person's preferences be satisfied.

15:21Let's suppose so. Put it crudely, unpoetically in terms of preferences. Now, if the other person happens to, for whatever reason, want you to do something that requires effort and wants specifically you to do it, as opposed to having the thing done by some machine that might just be a brute fact of this other person that they happen to have that desire, then you, with your desire that their desire be satisfied, would have no other way of accomplishing that than by putting out the effort, because that's specifically what they want. So this seems kind of a little bit hokey in that simple setup, but I think more subtle versions of that could be quite pervasive.

16:01I mean, it doesn't seem that hokey, the example basically. I mean, I could imagine, this would be a simple form that applies today. Liv would currently want that I exercise, for example, and the only way for me to achieve that is by exercising. And she might not want that for my physical change or health improvements. She might want it for me to be the type of person who themselves does it, right? and then I have no other way to fulfill that than by actually doing that task. Do you think that's an example of the thing? It's a potential example.

16:38If she really wanted you to exercise in order to, I don't know, presumably, I don't know, Liv, you could say, but if she wants you to exercise, probably it's because she wants you to take care of your body. Yeah, that is the primary reason. It's not so much for tyrannical reasons. It's not for the reason that you want me to be the type of person that does the things that you want me to do? Definitely not. If anything, I want the inverse. I like the resistance, but I want you to, health is first, and then some resistance. Okay, fair. So maybe it's only a potential example because the specific goal that she would have in wanting me to do that would not exist in that world anymore.

17:18But if we assume that that kind of desire in her still existed, then it might be the case that only I can do it by being the person who does it. I think in reality what you have is more company. You shouldn't necessarily think of this atomistically as there being individual A and individual B with preference one, two, and three. These can be more diffuse culturally interconnected. So we might have some commitment to a certain cultural form or tradition that we value. So this is how our, from generation after generation, they celebrated this holiday or whatever and then we want to uphold that. That's beautiful.

17:58It's been going on for long enough. And now the only way that we can continue that tradition is by ourselves doing various things. It would not count as continuing the tradition if we launched some AIs to enact it. It has to be we who do it. So there is a certain class of things that people care about that seem to require our own participation. And so those would be carve-outs where there might be demand for instrumental effort from humans because of the need to achieve some other thing than the effort itself. In the concept of technological maturity, which limits do you place on the potential maturity?

18:39So I would imagine that if there are, a quadrillion large biological humans, for example, in that world, and even more digital minds. And each of them wishes for something to happen in the observable universe that requires, say, something as complex as creating a biological being on a molecular precision level, that this would have such a high energy and compute demand that maybe it's outside of the limits of the observable universe's ability? Certainly there can be a lot of values that are unrealizable in the real world or in any future we could attain because you could have kind of arbitrarily resource-hungry preferences.

19:30And certainly if you make more people, then eventually resources will run out per person. If you have a finite cosmic endowment, then for a sufficiently large population, you will have a Malthusian state. You make the disclaimer early in the book, which is an important one for the book, as well as for the conversation here, that it's not about how good a utopia looks from the outside, but rather how good it feels for the individual within it, for that specific individual from the inside. Not exactly necessarily feels, but how good it is to live in. It's like a kind of lens perspective in how one approaches these questions.

20:16That I think you get very different answers if you think what kind of future would actually want to live in versus what is the coolest kind of future to explore in a science fiction scenario. And obviously, when you present the questions like that, it's impossible to confuse them. But I think in reality, when people think about these futures, some people allow intuitions from the outside perspective to creep in to a larger extent than they should. Now, there is a general difficulty in making one's thinking amount to something more than just a kind of regurgitation of the more or less random influences one has had or the literature one has read.

21:03So there's kind of more formal work that people do in population ethics and stuff where you can do some analytic work and it's easy to see how you could make intellectual incremental progress by kind of debating that. But when it comes to sort of the values themselves, it's very hard to have opinions about that, that add signal, as it were, because these are highly subjective in many ways. And if you want your thinking on these things to reflect something more than your mood or your own idiosyncratic personality or the particular cultural milieu you happened to grow up in, it is hard to do that.

21:50I think in general, the likelihood of having signal is increased if one allows, and maybe to the extent that one allows, different alternatives to play with one's soul and to kind of enter into the frame of mind of a certain value and, as it were, run with it or let it fill you up for a period of time and then maybe another one. And if you're just operating at the level of words and some bit you're a member or an opinion survey or stuff like that, I think you're unlikely to kind of add more information to the extent that it is a question of information about these ultimate value questions. Again, it feels like if you're in a radically abundant world where you have super intelligences that can just solve everything for you and there is nothing you necessarily have to strive for, that there is no place for competition.

22:52Basically, we've eradicated competition from the universe and the way that humans interact. And it's obviously competition is a huge part of what drives change and evolution. so do you think that people will find ways to derive meaning by introducing artificial scarcity back in and essentially creating fake I guess like little pockets of zero sum games in order to derive meaning yeah I mean so competition we do a whole bunch of just deliberately athletic competitions and computer game competitions and there is certainly like artificial scarcity generated in order to enable the right kinds of competition.

23:36And that seems like an obvious thing for Utopians to include various forms of game playing, I guess you could call it. And these might be zero sum in the narrow sense, but positive sum in a broader sense. Like if two people are playing chess in a narrow sense, it's zero sum. Like the more likely one is to win, the more likely the other is to lose. But in a broader sense, they are both hopefully having fun and learning more tests and participating in a community, etc. There would be a lot of opportunity in a plastic world where so much is without constraints to deliberately construct constraints in order to enable particular types of activity.

24:22just as somebody might do mountain climbing or something. Even if there were a lift that went up to the peak of the mountain, they might choose deliberately not to use that. And they might go further. They might kind of put themselves in a position in which once they are in that position, in fact, there are no shortcuts. So if you are on the rock face with no cell phone or something, then it is actually life or death for you. Even in a plastic world, you could kind of create these pockets where your instrumental effort is needed for practical reasons like that. But then in addition to these kind of artificial purposes, as I call them, where you sort of deliberately engineer the condition just so that you can then engage in this activity with instrumental reasons.

25:20We might ask, to what extent do these various natural purposes that would persist suffice to give a lot of structure to life? And I think that a lot of the most obvious and stark purposes that permeate the current world would go away. You don't have to work to make a living. You don't have to exercise to keep healthy. You don't have to like a whole bunch of these.

25:52But I sort of suspect that there is many, many more subtle purposes or values that we are kind of more or less oblivious to at present because there is so much more pressing and urgent things to worry about. But that once those kind of urgent screams die down, we can begin to hear more of these subtler whispers. So, I mean, whilst there are like kind of kids starving, like it feels frivolous to be too concerned about various subtle aesthetic dimensions that one might otherwise think should have a say on how we act. But if there were no more starving kids and no body getting cancer and no political injustices and no risk of war, et cetera, et cetera, then it might make perfect sense to kind of allow these quieter values to play a larger role in shaping what we do.

26:52And it might be that there are enough of these natural purposes that we would then come into view to significantly constrain and give shape to human existence. You have a quote that made me worry slightly less about the purpose problem in the future, which is, more people jump out of their seats when their soccer team scores a goal than when an international agency publishes a report saying that 100 ,000 fewer children died from preventable diseases this year than last. if one was to take an outside view, then the preventable disease eradication seems like something that people in some form ought to jump out of their seats more for, but they don't.

27:33It made sense, just on the basis of that quote as well, to imagine, one, these contained universes where we create artificial scarcity, or other things might just come up that will loom very large. Yeah, I mean, certainly it would be trivial to ramp that up, if one wants, with neurotechnology, our drive. and motivation, or you could sort of tweak somebody to take an immense interest in this, that, or the other, some random game. So that certainly would be possible. Like if we are thinking of things that boxes we could check with Deep Utopia, like so certainly like, oh, if pleasure is the thing, like, well, that would be easy.

28:13That's just kind of manipulating the hedonic system in the brain. If the feeling of purpose, having motivation, like this kind of jumping out of bed in the morning, if that's what we want, well, that also is something that definitely we could do with some simple neuroengineering if we wanted that. So the question is not whether we could create a situation where we would have drives and motivations to do various things, but also whether those things would be ones that are in some sense worth doing. me. Which brings us back to another set of constraints. So we've alluded to there's some ultimately limited finite pool of resources.

28:53And so if there were sufficiently large population growth, eventually you would kind of run out of resources. That's one kind of constraint. There are obviously physical constraints like the speed of light, et cetera. There might also be moral constraints that limit what can morally permissibly be done, even if it is technologically possible to do it. And I think one, maybe the most obvious area would be if there are certain types of technological affordances that would require the instantiation of minds with moral status. So for example, if somebody had a preference that would require them to interact with another entity, but the only way to make that interaction fully realistic would be to say for that other entity to be conscious or to be a morally significant being, then it might be that the first person's preference is to have certain types of interactions would not be satisfiable, even in this kind of idealized situation.