#803 - Nick Bostrom - Are We Headed For AI Utopia Or Disaster?

30:31They also would be unnecessary in that there would be this shortcut to joy and pleasure and happiness, like the direct brain manipulation. So you have this quite radically different condition, where the world is solved in the sense of the practical problems have been taken care of, but also in the sense of maybe this solved in that a lot of the fixed points and hard constraints that shape our current lives are kind of solved in this solution of technological advancement. And then we kind of get really the heart of the problem that the book is trying to think about is like, in such a condition, what would a great human life look like?

31:22What could we actually achieve in terms of realizing human values if we had all of these affordances, all of these options? It's so strange that the thing that reading the book that stood out to me is how much of what we seem to value and take pride in are kind of like clever ways to deal with scarcity and the fact that much of what we do is instrumental to striving and achieving some future goal which requires effort. It seems like, in a sense, much of human philosophy and value is just negotiating with a world which is effortful and constrained, and we are trying to find ways cognitively to deal with this sort of pressure that we have to lean up against in order to control the world, to deliver the thing that we want.

32:14Yeah. These practical necessities have been with us since, I mean, through the entire history of the human species, and indeed beyond that, it's like the human nature has kind of evolved than been shaped in a condition where this is always present. There are all kinds of things we have to do and cope with and struggle against. So it's almost like if you think of like a little bug that has an exoskeleton, right? And then it holds this kushibits inside together. But if you might end up removing the exoskeleton, there's like just a blob there. And similarly the human soul might have as an exoskeleton all these instrumental necessities.

33:01And that, that, that, that, like evolution can just assume or present because I've always been. But if you were to remove those, then what becomes of the human soul and the human, human life does it become a kind of pleasure blob? Or is there something that could give structure to our existence even after these instrumental necessities are removed? So much of what we seem to take pleasure in as well is the absence and then satisfaction of some desire. There is a thing that we want, we don't currently have it and then we get it and then it gives us something. We work hard to achieve a body that we're satisfied with.

33:40We are thirsty for a while and then we get a drink. We want to have sex and then we do. We are looking forward to having a child and then it's born. We are all of these things are on the other side of something and yeah, if like you do X to get Y, but you can just always immediately get Y without having to do X, it does ask the question of where does the absence, there are no longer any absences. There's this quote, something to do with in a perfect world, the only desired, the only lack would be for the want of lack itself. which is this sort of, the absence actually makes the presence of something finally valuable.

34:24And if you don't have any more absence, then what does all of this presence kind of mean? Yeah. I think there might always be a whole bunch of absences in as much as human, if not human need, at least human desire, or at least some human desires are kind of unlimited. It's maybe most clearly seen if you have two people who want exclusive possession of the same thing, or two people each of whom wants to have more than the other. Like, if they think two billionaires who want to have the world's longest yacht, and so one has like 150 meter long yacht, and then the other has a, builds a slightly bigger wire that like is kind of intrinsically unlimited because there's no way that they could both have everything they want.

35:14And so there might be a bunch of desires like that that are quite common that could never be completely fulfilled or like just inviting somebody who is like utilitarian and who wants there to be as many happy people as possible in existence. But like however many they are, there could still be more and so they would always prefer to have more resources. But even if there are some such desires, it wouldn't give these future utopians necessarily any reason for laboring or exerting effort because there might just not be anything they could do themselves to increase the degree to which these desires are satisfied.

36:01I mean, the person has a trillion dollar, maybe they would want to have two trillion dollars, but they can't actually make more money by working because all the work is more efficiently done by machine. So yeah, even with unlimited desire, you might still have this condition that is post -work and post -instrumental. Do you think that humans would be running the risk of getting bored in a utopia? Not if they didn't want to, at least if you buy boredom, refer to a subjective state of, I don't know, like some kind of restless, discontented, uneasy feeling of having a difficulty keeping your focus.

36:44So that certainly would be amongst the things that could be trivially be dispelled through advanced neuro -technology. I mean, you already have drugs that could do it for a limited period of time now, with side effects and then they wreck your body. but it's easy to imagine how you could just have a better version of that. That would make it possible for you to always feel extremely interested and excited and motivated. And in fact, some people have, I mean, there's a lot of variation among students. I have a friend who tells me he's never bored and I believe him. I've never seen him bored. He's kind of interested in everything except sport.

37:28And like he writes papers on all kinds of different weird topics and it's like just constantly excited about learning new things and you can have a conversation with anybody about anything and he's like really, so it's an existence proof. It's possible to be that kind of being and in the future we could all become such being said we want to. So subjective boredom would be like trivially easy to dispel under this condition. Now, it is possible also to have a more objective notion about boredom or maybe we say boringness to refer to this objective notion, which is the idea that certain activities are intrinsically boring, like meaning maybe that it is appropriate to feel bored if you were spending too much time doing them.

38:28It's like kind of maybe an open question whether this notion of objective boringness makes sense, but you might think of like say counting blades of grass on the law. Like, suppose you had a being who found it extremely fascinating and like, and never ending source of joy to just count and recount the blades of grass on a college lawn somewhere. You might say that although, subjectively, he's not at all bored, that objectively what he's doing is boring. It's uninteresting. Like, no variation or significance or development and the appropriate attitude for somebody to have if they were spending their whole day doing that would be to be subjectively bored.

39:18If you have this notion of objective boringness, then it becomes a much less trivial question to ask in this hypothetical condition of a solved world, like would it be possible for us to avoid objective boringness? Like yeah, we could like engineer ourselves we always felt interested in what was going on, but would we be doing anything inappropriate? Like, would our circumstances be such that the appropriate attitude would be to be bored? And so there's a big discussion about this, but yeah, and I think it's, I think there are certain forms of interestingness that we might run out of. For example, you might think it's especially interesting to, I don't know, be the first to discover some important truths.

40:13Like Einstein discovering relativity theory might be like a kind of paradigm case of an extremely interesting discovery and experience. But it's plausible after some while that most fundamental important insights about reality that we could have we already have had and in any case the machines will be much better at doing the discovery than than we would be and so we would kind of run out of the opportunity to achieve that kind of Interestinglyness in our lives. Yeah, I mean so much of what Humanities done is been chasing down answering big questions. So where do we go when all of the big questions have been answered?

41:02Yeah, I mean, I think fortunately for the most part, it's not what we're actually doing in our lives. I mean, most people are not spending most of the time trying to chase down the answer to the big questions, right? Like most of the time we're just going about our daily business. And you couldn't make the case that already. I mean, if you really looked at it, I mean, from a kind of on biased outside, like if the alien super brains came to Earth and said, okay, so, look, these guys are worrying about allocing out on what's interesting in life. Well, let's look at their current life and see how interesting there is.

41:46How many times did this guy brush his teeth? Okay, well, 40 ,600, like, how interesting is if the brochure tees for the port is thousand seven hundred sixty's time. And then, um, all right, so he commuted into the office and then, you know, he ate a steak. Okay, I mean, how interesting is it to do that again and again and again? Yeah. And even the big highlights in our lives, like, I mean, maybe they are like really novel and exciting. If your scope of evaluation is a single life, like the first time you see your own newborn, he just happens once. So there are a few of those. But if you zoom out and look at humanity, it's kind of well, it's already been done, tens of billions of times.

42:35How different is this particular newborn from all the other newborns? So depending on how you look at it, you might either think that we are already at the very low rung of the ladder of objective interestingness. or if you shrink the focus of evaluation enough to a single life, or perhaps even just to a single moment in a single life, then there is more novelty, but also opportunities for the same kind of thing to happen in Utopia. If you're just looking at the most interesting possible moment, and you don't care about whether similar moments have existed before or after, then you might think the average human moment of awareness is very far from the maximum.

43:23Interestingness. What do you think would happen to religion? That's obviously a place that an awful lot of people take their meaning from currently. Is there a place for religion in a D .P. Utopia. Yeah, so this is one of the things that plausibly survives this transition to a sold world, like which could remain highly relevant, even if we had all these fancy technology. And it might constitute a bigger part of people's lives and attention than it does today, because there would be fewer other distractions if you want.

44:09What else? What are the other areas that are uniquely human or that would survive this transition? Well, yeah, so I mean, you can kind of build it up, starting with the most basic value, perhaps, which is just this sheer subjective well -being, pleasure, enjoyment, which obviously would be possible to achieve in utopia and not just achieve, but like you could have prodigious quantities of this bliss. And so that's intellectually, not maybe super exciting to discuss at Great Length, but I think actually super important. Like it's easy to dismiss, oh, these are like, sort of junkies just having their like harrowing drips.

44:59But the key question here is not like how exciting is this future or admirable is it from our point of view as if we were sitting in the audience like evaluating a stage play. Like that's one perspective and then we want the stage play with a lot of drama and suffering and tragedy and overcomings and heroism and right but the question here is which future would you actually want to live in and there like one of very great levels of subjective happiness and well -being might that might be the most important thing about the future in fact and you definitely have that in extreme degrees. So that's like, it's worth making a note of that.

45:45Like, let's put that in the bank. At least we could have that. And that's already possibly, like according to some people, it's the only thing that matters. Like if you're a hedonist, a philosophical hedonist. But for most other people, it's at least one of the things that is important, even if not the only thing that is of value. So that's the first thing. then you could add to that experience texture. So it's not the case that you could only have subjective well -being. You could attach that to some intricate, complex mental states that relates to some important, like so for example, you could experience the pleasure not just as a sort of unanchored sensation of well -being, but you could attach it to say the appreciation of aesthetic beauty, appreciation of great truths or great literature or contemplating the divine, and that that's what you derive the pleasure from.

46:46So your conscious state is one of insight, let us say, or understanding or appreciation of things that deserve to be appreciated and understood. So some people think that that is also a locus of value, not just the hedonic, like the scale of whether it's like plus 10 or minus 10, but having plus 10 whilst you're like understanding or seeing or appreciating something that is like actually lovely and worse understanding or profound makes that the more valuable condition. So you could have that. But then if we go to some of the other values that seem more at risk like purpose, you could have, you could certainly have artificial purpose.

47:36So you could set yourself goals in utopia in order to then enable the activity of trying to achieve them, like playing a sport against another person. Yeah, so games is a paradigm example of this in today's world. You set yourself some arbitrary goal, like maybe to get the golf ball into a sequence of 18 holes using only a club. And there's no other reason for why you'd need to achieve this goal, right, other than to enable the actual activity of golf playing. And that could become a much larger part of the utopian lives, various forms of game playing, like you could make all kinds of new, much more sophisticated and immersive games alone or with other people that involves setting yourself sort of arbitrary challenges, or at least semi -arbitrary challenges, simply in order to then create an opportunity for the activity of striving to achieve them.

48:41And yeah, so to enable play, you kind of like deliberately limit the means available to you to achieve this arbitrary goal. So just as a golf like because if you were in the post scarcity world, you press the button and there's never any doubt about whether or not hitting the ball goes into the hole, which makes the hitting of the ball completely arbitrary. Yeah, and uninteresting and maybe objectively boring, but you could just set the goal to achieve this sequence of outcomes, the ball falling into the different holes, whilst also not availing yourself of various shortcuts that could you could sort of make this more complex call of achieving X while only using means to know why there needs to be some constraint which then gives a degree of satisfaction when you achieve it.

49:33Yeah, yeah. And like, either to give you the satisfaction, you could achieve the satisfaction just directly through the newer technology. But if you're also in addition to the pleasure, wants to have the experience texture and you want to have the sort of effortful activity that's driving, then yeah, you could achieve that by having these artificial purposes. You kind of feel to me like you very quickly keep coming back to the same question of, is a quicker route to achieving the outcome that I'm trying to achieve here. I've had it in my head since you were talking about this, since I read the book about churning butter.

50:09So there's very few people that would look at the butter that they use now and think, I know that it's here and it's convenient and tasty and does what butter needs to do by being lovely on bread or whatever. But I feel like it would have been more meaningful to me if I'd gone out into the field and got the cow and and the thing into the bucket and then churned it and then got it and then put it in the fridge and all of those steps. So we can see, and there's kind of this inherent sense, there's sort of naturalistic fallacy, that this is taking us away from what it means to be human. The set point that we have grown up in, this is a misalignment evolutionarily, there's something sacred about the process of being human.

50:55It imbues you with meaning to go through the challenge and the struggle beforehand, but there's very few people that would make that argument about churning butter. And when you think, okay, so if you are happy with more convenient butter, I think that something right now, which is assumed by most people to be natural, but in future we'll be looked at as barbaric, we'll probably be driving our own cars. I think that in 50 years time, 100 years time, it'll be like if you looked at someone riding a horse down the street now. And you go, I mean, isn't that so quaint? And while that people used to do that, that was the way that they got around it.

51:31You know, they had to have these special people in New York that would sweep up all of the muck from the street. This entire industry builds around horses. So again, right now, something that we can almost begin to see the transition of. We're about to let go of this thing, which is a less efficient, less safe, more effortful way of getting us from A to B. And yet some people, I love driving. I take great pride in driving. Some people even compete in it. You know, F1 is an entire competition around people that are driving. So you can see different frontiers of human endeavour being eroded away by technology, whether it's from churning butter or driving a car, and you just continue to slice that ever more thinly all the way to why you hear what are the sort of relating to other people having to get yourself out of bed and move yourself down the stairs on the morning.

52:22Each one of these different things could begin to look okay, isn't it cute that people used to you know like pick themselves up out of bed and put their own clothes on and walk down stairs and brush their teeth and then it asks you ask yourself the question well if everything is open to you and you can manipulate your own internal state why not just spend the rest of your life counting table legs or blades of grass. Indeed, yeah. So you are forced to confront these fundamental questions of value in this condition. What things are you doing for the sake of something else versus what things are you doing truly for the sake of the activity itself?

53:03So even the guy now maybe likes to do the their own butter. There is a question of is it because they intrinsically the valid activity, or is it maybe because of the pleasure they get out of it? Or because of the way it teaches them about their own body and about the cow and the physical objects and puts them in touch with that. That's a kind of extrinsic element, but yeah, so these things that is currently we can conflate them because in reality the only way maybe to get various kinds of pleasure is to dive into activities and give it your best and then you get the satisfaction We can't separate these today, but in this hypothetical condition, they can be separated and then you do have to ask the question of what precisely is it that you actually value.

53:53So, this conception, I mean, so for me, it's like interesting because I think there's a real chance that, if things go well, we might actually end up in something like this condition with the whole machine intelligence revolution, etc. But even if you thought that was not going to happen, you could view it as a kind of philosophical thought experiment. Just like physicists build big particle accelerators at which they smash atoms together at extreme energies to sort of see what their constituents are. And then you can assume that if there are quarks, when you smash the particles together in CERN, then maybe there are quarks in other matter, and you can kind of learn from, like, you expose basic principles by looking at extreme conditions and extrapolate and think they might be there all the time, even though we can't see them.

54:49Similarly, with human values, if you kind of smash them into one another under this extreme condition of a soiled world, you can study their constituents. And then you might think that, well, in our ordinary lives, maybe those same constituents are there, they are just kind of invisible top, because they're hidden by all the kind of practical necessities. What are the implications if humans live for a very long time? Does anything change that? Yeah, so certain values are more jeopardized by extreme longevity. For example, interest in us, as we discussed earlier. If your notion of interest in us involves the idea that something has to be novel, to be interesting.

55:30Like it's uninteresting to just do the same thing over and over. And if the domain within which it has to be novel is your own life, as opposed to say the world as a whole or your species or the current moment. but if the relevant sort of locus of evaluation is human life, then the longer the human life goes on, the harder it is to do important things for the first time. I think we already see this in our current lifestyle. If you think about what happens in the first year or two, there's some pretty big epistemic Earthquery. Like you discover there is a world out there like that. It's covering right?

56:16Oh, it has objects like the objects remain there even when I'm not looking at them. Like there are other people there. Like my friend in the world, I can imagine just discovering for the first time that there are people my friends. I would have a body. Yeah. Wow. And you're separate from mom and dad and you can communicate. Yeah, My friend described a, his son was born during COVID. And I think the first maybe year of his life or something had only seen four people. He'd seen like mum, dad, grandma and nanny or something like that. And then apparently one day he saw a fifth person and it fucking blew his mind.

57:00He was like, what, there's more than four? Right. And then, yeah, you discover that you can do things, move your body, and then later in life, it's like, oh, what happened this year? Well, we got the puppy, we bought the caravan truck. It's not really at the same order of magnitude in terms of how much it reshapes your view about reality. So I think there is already within the human lifespan, a kind of rapid diminishing. If you measure interestingness in a certain way, where it's kind of the delta between your previous conception of the world and what you could do and what you're able to do after the event that is then interesting.

57:46There is another conception of interestingness where it's less the rate of change and more sort of the complexity of what you're engaging with at the particular moment in which case maybe the level of interestingness of a day of the typical adult by that metric might be higher than that of an infant because of all kinds of complicated things going on at work and relationships. Right. So rather than having big questions, being answered, you have increasingly dexterous small questions with more magnification and you can sort of see them with more complexity and you derive some pleasure from that.

58:24One of the things that I've got in my head, if humans do live for a much longer time, you're going to be able to continue producing humans. That will have to be in relation to the increase in computing power. So there'll be this kind of mouth -usian tug between how much computing power have we got to be able to support how many humans and which can move more quickly. Have you considered this sort of tension between the two things? Yeah, so in the long run, I think economic growth becomes really a consequence of growth through space, the acquisition of more land as it were by space settlement. Like once you have achieved technological maturity, you can't have economic growth by like inventing better methods for producing stuff.

59:14And you also probably can't have more growth by just accumulating capital assets and machines because you already like have built all the machines that results in optimal productivity for the volume. So ultimately the limiting constraint become what economists call land, but it basically means those resources that you can't make more of. And so in the long term, you could imagine human civilization kind of expanding through space, but there is a limit to that, which is the speed of light. So if you have a sphere, you know, with Earth at the center, maybe, and then growing at maximal speed in all direction, at some fraction of the speed of light, that the volume of that would grow polynomially.

1:00:00But population could go exponentially. It could like double every generation, or, you know, so in the long run, an exponential overtakes a polynomial. So at some point, you would need to moderate the rate at which new beings are brought into existence if you want to maintain a sort of above subsistence. That's a really interesting point that I hadn't considered. So you can have a solved world in which almost all problems have been defeated, but there are still some constraints. Speed of light is one of them. What are some of the other constraints that a utopian world would encounter? Yes, there's a bunch of basic physical constraints, like the speed of information processing, the amount of memory you can store, like the size of a mind that is integrated.

1:00:51Like if you make a mind much bigger than a planet, then you get conduction delays, like it just takes time for one, like what happens in one part of the mind to kind of communicate to what happens in a different part of the mind. So either you have to run the mind much slower, or you have to keep the mind relatively small. That might be, I mean, we are hoping not, but you could imagine if there are other alien civilizations out there, there's like the potential for all kinds of competition and conflict. So yeah, so there's like a bunch of, of those external physical constraints that I think, define the ultimate envelope of what could be done, but it looks like the space of possibilities very, very large compared to our current human vantage point.

1:01:46So like you could have, you could maybe not have like immortality if that requires not dying like infinitely long time looks impossible in our universe. Like eventually information processing threads will, you know, if not before then with the heat test of the universe, which is kind of significant because from a theological perspective, like whether you live for 80 years or 80 million years, like it's all kind of really a blink in the eye of eternity, you could argue. And so it doesn't really change fundamentals. But from the kind of parochial perspective of a current human life, you could certainly have extreme longevity and extreme amounts of wealth and extreme amounts of most other things.

1:02:29But still there are limits and those limits would be like relevant in various ways. What about moral constraints? Would that be any? Yeah. So this is another like more subtle but potentially very important source of constraint.

1:02:45So like what's the easiest way to like I mean, so I mean some people have thought for example there is like it's immoral to enhance humans biologically. It's not a very popular view these days, but during President Bush, he had a council on bioethics that he set up and populated with a bunch of bio -conservative thinkers. And they were trying to argue that it's somehow a violation of human nature or something to try to enhance humans. So like distinction, like therapy, like medicine, curing a disease, fine, but like trying to slow the aging bad because it kind of, you know, and it's a little like once you start to think about it, it's really hard to make out that distinction like you think like, you know, genetic therapy to make you smarter bad, but education good, even though it hopefully makes you smarter and like, it becomes problematic, but if you did have that view like then that would be a whole bunch of possibilities that would be cut off if you just couldn't change what like the basic physiology of what we have.

1:03:55And you were confined just with like moving things around in the external world, to try to cheer you up by like having a really nicely decorated room or something or like a, like there's only so much you can do to affect your inner well -being if you can only affect it by having sort of nice visual stimuli and nice acoustic waves going into your brain.

1:04:23If you can't potentially some other interesting ones that if somebody had some preference to have another person relating to them in a particular way, like the experience of being loved by a particular other kind of person, then it might be that the only way to generate that experience fully realistically would be by instantiating that other person. And if that other person then like presumably would have moral status that might be all kinds of ways of treating that other person that would be wrong. So moral constraint might then limit the kinds of experiences you would be able to have. Well, with another person, yeah, I suppose as soon as you involve somebody else who has moral consideration that changes quite a lot.

1:05:21Yeah. And in fact, I think that is, so we discussed these artificial purposes that are so likely to create games and set yourself goals. Like that's one type of activity. I think there are also a possibility of a bunch of natural purposes remaining. Like purposes that would call upon us to make various kinds of efforts, not just because we create random goals for the sake of having something to do, but that kind of exists independently of us. And a lot of those would derive from this kind of interpersonal entanglements and various kinds of cultural entanglements, where, I mean, to take the most reductionistic case of it, which is not so inspiring in its own right, but you could imagine more natural versions of this.

1:06:06So suppose you have person A and person B, and person A wants person B's preferences to be satisfied. Like they care about person B and wants person B to get what they want. And then if person B happens to want person A to be doing something on their own steam, then the only way that person A can achieve their goal of satisfying person B's preferences is by themselves doing this thing. Like they couldn't, they could have a robot to it, but that wouldn't satisfy person's feet preferences. So from the vantage point of person A, they now have reason to do this thing. It's not an arbitrary goal. They set themselves.

1:06:44It's the only way they could possibly achieve their goal of satisfying person B's preferences. So, so in this kind of like it seems to be a token artificial, this particular case, but you could have made in more subtle ways of this, where there's like a tradition that you feel a commitment to and that you want to honor. Part of that transition is that you you know you engage in certain kinds of practices, you refrain from certain kinds of shortcuts, you respect other people's preferences to various degrees because they are, yeah you want to honor them and so there would be a bunch of stuff then that maybe you need to do yourself and you can't outsource them.

1:07:26So you've managed to, over the last decade, straddle all the ways it could go wrong, all the ways that it could go right. Toby then sort of bifurcated that down the middle with the precipice, his work, and he's got this analogy where he sees humanity being sort of walking along a precarious cliff edge. And if we fall, then everything's kind of fucked. And if we make it on then this is sort of beautiful meadow on the other side. How important or critical do you think the current moment is in humanity's future? What's the, how long is the precipice in your perspective? Yeah, I think, I mean, it is weird because it looks like we are very close to some like key juncture.

1:08:18which you might think is prime aphasia implausible. So there have been thousands of generations before us. Right. And if things go well, there might be millions of generations after us or people living for like cosmic durations. And like out of all of all of these people that you and I should happen to find ourselves just next to this critical junction where the whole future will be decided. It is striking. That seems to be what this model of the world implies. And maybe that is like an indication that there is some things slightly puzzling or impossible about it, that there is maybe some more aspects to understanding our situation than is reflected in this naive conception of the world and our position in it.

1:09:07And you might speculate what that is. I mean, I have this earlier work on the simulation argument and stuff like that. But if we take the sort of naive view of reality, then it does look like, yeah, my metaphor would maybe more be like a balanced beam where a ball is rolling down like a thin beam. And the longer it rolls, the more likely it will be to fall off the beam. But it could fall on one side or the other. and that's hard to predict. But yeah, I think it probably will fall off. Like the idea that the normal human condition, as we now understand it, will just continue for, like hundreds of years, I mean, let alone like hundreds of thousands of years, that seems to be like the kind of vague idea that a lot of people have, it just seems like radically implausible to me.

1:10:00It would be unlikely in your opinion that in a thousand years time, five thousand years time human existence will reflect what our normal sort of day to day is now. Yeah, and I think the only possible way is for that to happen, you could like create some scenarios like one would be if we do sort of have some massively destructive and that knocks us back to the the stone age or something and then maybe by 500 years we would have climbed back up again to something resembling the current human condition. But then in that scenario, we would have spent most of the intervening time in a rather different condition.

1:10:37Or another might be if you get some very strong global consensus of some particular orthodontics. Moratorium of something, something. Yeah, like the kind of bioconcentral and then we start to ban all kinds of different technology so you get like a kind of some sort of bureaucratic sclerosis or deliberate decision to do this is a we've gone to this point but let's not so there are various scenarios in which something like that but if the basic sort of scientific and technological push forward is allowed to continue then it does look like we are sort of very near developing a range of transformative So AI being kind of the most obvious of those, but if it weren't for that, then I think like synthetic biology will create a bunch of other possibilities and then nanotechnology and so I think even if AI were somehow like if you just pretended that wasn't there, I still think we would be in profound transformations.

1:11:38That's interesting to consider that if technology moves forward even at a slow pace, even if it was to drop by a really significant margin for where it is now, given long enough time, you end up with a radically different world in either a way that you intended or a way that you didn't intend. And the way that you didn't intend is probably going to be pretty bad. And the way that you intended, hopefully, is going to be the one that is pretty good. Either way, you end up with a radically different day -to -day experience for most humans. Yeah, I mean, yeah, I'm not so sure about the first part.

1:12:11like if it's something we didn't intend, then it would almost certainly be very bad. I'm not sure. I mean, you might think like the world we have currently ended up with. I'm not sure whether you could say like that's what people a thousand years ago intended. They might kind of, in fact, be quite a shocked about some of our habits these days. But it more just sort of have happened as a result of a bunch of different people going about their business and pursuing very local aims and then at the systemic level eventually, you know, an intent of consequences can still be positive. If we're, yeah, if we're a lot, I mean, I think like the degree to which the future depends on our intentions is possibly quite limited.

1:12:59There's sort of bigger dynamics at play and we barely even understand what they are. we don't really know what we want at a big scale. Like most people have the hands full just thinking about like the next week and you know what to do if they're boss don't like them at work or the partner has a like like this this is what fills human life and then trying to get ahead in the low like there's very little thinking by anybody really like trying to weigh where should humanity be going in a million years like what's the optimal trajectory. I think that we could do with a little bit of more thinking about that but It's not like the primary shaper of the direction of the big ship of humanity is What have you been most surprised by Over the last ten years when it comes to AI development?

1:13:49I think just how anthropomorphic the current generation of AI models are the idea first of all that they are sort of almost human level and that they can talk in ordinary language is already kind of interesting in. But then that they even share some of the like quirks and psychological foibles of humans is like if you if you 10 years ago you would have come and said, well we're going to have these AI systems, you know, they can do all of these things, they can like program and write poetry and but if you really want them to perform at their best you need to give them a little pep talk when you ask them a question you got to say think step by step this is really important I might lose my job if you get the answer on and then they perform a little bit better than if you just ask the question like you like people would have thought you would completely lost your marbles right and yet that's where we are today so that's surprising.

1:14:53I think less surprising, but still interesting is the degree to which development so far has been continuous, like rapid, but incremental, like a sequence of steps, each of which has sort of significantly, but incrementally improved on the previous step, and the the degree to which the progress is quite tightly coupled to the scale of compute being applied to this. So you have this big compute hypothesis, which is basically that the most important determinant is not the particular architectural features of your model, but just the amount of compute you use in training and the amount of data. And you get performance in proportion to how many dollars you spend on training.

1:15:42it basically. That's like too crude. You also need some skilled engineers, but we are kind of closer to that being the case than one would maybe have expected in the median scenario. Exante where you might instead have thought, we're going to, you know, crumble around until we find this clever algorithmic hack and then suddenly it's going to explode. It's going to open something up. Yeah, yeah. Whereas it's like more like just scale it up. It works better, scale it up more. It works even better. Now it is still possible that at some point, that like some little map less missing bit will fall into place and we could still get an intelligence explosion in those scenarios.

1:16:19So we shouldn't like over index on what we've seen so far, but it's still interesting. Does that change your perspective on what is more or less likely from a takeoff scenario from a house super intelligence could come about stuff like that? Yeah, I think it makes it somewhat more likely that there will be political forces at play. Like, when things happen more gradually, it's easier for the public and for policymakers to realize what is happening and to sort of try to change it. And so we already see sort of at the geopolitical level with like the chip export restrictions and more recently reporting requirements for training, like models using more than 10 to the power of 26 flops and there might well be more if we continue to see sort of increasingly powerful AI systems over a sequence of several years, that there might be time for more actors to kind of try to exert influence of this.

1:17:25And if it were just some lab one day that like stumbles upon like the key missing thing, like with a computer in their basement and you go sort of overnight, then it would be more likely to be like an isolated thing We're just a few people were having their hands on the tiller. Have you got any idea which scenario you think is more optimal?

1:17:49It's really hard to say I think I think it seems probably better if whoever develops this technology first has the option when they are like starting to develop like true super intelligence to go a little bit slow in the final stages, like maybe to pause for a half a year or something, rather than okay, now we've got it figured out and then immediately cranking all the knobs up to 11 because maybe there are 19 other labs, you know, racing to get their first and whoever takes any precautions just immediately become irrelevant and fall behind. And the race goes to have race like most gonghaw are willing to take the biggest risks.

1:18:34That seems like an undesirable scenario. And so having some ability perhaps for the frontier labs to coordinate or unless one is already naturally significantly ahead, maybe the a small site of leading labs should at some point be able to synchronize that could be desirable. I think it's very unlikely but less unlikely than a couple of years ago that we could end up with some kind of perma ban like on AI. I think that would be undesirable. I think ultimately this is like it's a portal through which I think humanity will need to passage to the future. But we should recognize that there will be significant risks associated with this transition.

1:19:22And the slower as well that this happens, I suppose the more opportunity there is for political, policy, human fuckery to get in and coerce and go. So it's like it's very much a double. On the one hand, yeah, you do want like you It's kind of uncomfortable either way. Like some random person in some lab or just gonna control the future, that sounds pretty scary. You won't definitely at all still oversee this, right? But then you think the other end, like, wow, all the security establishments of governments around the world, is that like a much more comfortable situation where they like the military, and not just one military maybe, but like, and then you get all.

1:20:03So either way, I think it's a little bit disconcerting. So I don't feel super comfortable. Like I don't have a very strong view at the moment as to what is the most desirable trajectory. And it might be anyway something we don't have super fine grain control over. I think one can try to notch things on the margin, like towards a more cooperative, inclusive, and friendly and thoughtful trajectory, I think that seems good to do. I like to try to encourage this idea that the future could be good for both humans and for digital minds and for animals and for as many people as possible. And there really is that potential there.

1:20:47Like the upside is so enormous that that could be plenty for not just one value to be realized, but for a whole range of different values and perspectives. So our first instinct, I think, should be to seek these kind of win -win positive sort of outcomes. And then like if at the end of the day, there are also some irreconcilable differences we'd have to strike some compromise there. But there's like so much you can do before you get to that point that it would be tragic. If you just kind of skipped over all of that and let's get to the point where we can fight about something like that that would just be a big tragedy.

1:21:23What is the current state of AI safety in your view? Obviously, 10 years ago, conversations about alignment and takeoff scenarios and all of the rest of it was obscure Reddit threads and a couple of people in some weird message boards. Is it overfunded, underfunded, overresourced, underresourced? Where should people's attention be placed at the moment? Well, there is a lot more talent in the field now. And I mean, a lot of the smartest young people I know are going into AI alignment and working on it. And all these leading labs have research teams, as I said, it's probably still under resourced.

1:22:09I think it looks more like talent constrained at the moment rather than funding constrained, but to some extent funding can help. There are some questions about whether alignment work spills over to capability progress. Like some of the things you would want to do for alignment, like better methods of, now interpret what is going on inside a little mind and figure out why exactly why is it behaving the way it is that would be useful for alignment. But it could also shed a light on how to sort of, you know, what's limiting performance and how to boost it. So it gets pretty complex. I think some other things like better cybersecurity in the leading labs to make it less likely that the weights of these models just gets stolen.

1:22:56I think it's like maybe be helpful.

1:23:03And yeah, but I think like more work on alignment seems positive. Having some ability for leading labs at a critical time to go a little bit slow. It seems positive. I would be with that require with that require coordination between multiple labs in order to be able to do that because you do have this sort of first pass to finish post. Yeah, it depends on like I think they already well one already all their idea and which might still be relevant is that maybe you would have one lab or you know whether it's one country running one lab or one just private industry lab or whatever it is but like one would have some some lead over the, like just naturally some, like one lab might just be a year or two ahead, because they started earlier where more lucky or had better talent or something.

1:23:51So then that would create an opportunity for this leading lab to slow down for a year or two however long their lead was, right, without falling behind. And it might be desirable if rather than having like a super competitive race, you had And then that kind of pause would be self limiting, you see, like because once they have paused for two years, that would be another lab kind of catching up. And then maybe they could pause too, but you would have to make an increasingly strong case for pause as more and more independent actors became capable. So it would be a pause that could exist and it would eventually expire.

1:24:28And exactly when it would expire would depend on how strong the argument was for AI risk. And that would, it seemed, create a much lower risk of ending up with like a kind of permaband where this technology is never developed. Whereas if the past, to getting the ability to pause for a year is to say set up a big international regulatory regime or like creating a lot of stigma around AI research, like developing some mass movement like smash the machine type of thing. then that's much more likely to spill over into something that then become a permanent orthodoxy or a regulatory apparatus that just has like an incentive to perpetuate itself.

1:25:09So more worrying from that respect. How impressed have you been with the power of LLM's? Do you think that they are going to be the bootloader for what we need from a superintelligence perspective or is this have you got limited hopes for how far they can sort of climb functionally? Well, we haven't yet seen the limits of what one can do when scaling these. I think these transformer models, I mean it's not just language, but they could have other modalities as well. But they do seem very general and a lot of alternatives that people try to turn out in the end to basically just result in similar performance as transformers.

1:25:57The transformers run well on the current generation of hardware. Like the parallelize very well, etc. And so it might be that you need a little thing on top of that. Like maybe it's like the engine block and then you need some sort of agent loop. Or maybe some external memory augmentation or some other little thing. But that you would still have this big kind of transformer or something similar to it. Like there would be some variation. But as the basic thing that extracts statistical regularities and abstractions, that's pretty plausible, I think. It's interesting. I certainly wouldn't have guessed ten years ago that something that you have a conversation with and that is able to accurately predict what it would say would actually be the forefront.

1:26:47You know, it seemed to me tracking quite closely from, whenever like 2015 -2016, the development of AI that I think your book and then subsequent conversations around AI risk kind of blew up that conversation. And then it seemed to me that maybe the 2018 -2019 -2020 that AI hadn't really delivered the threat that people perhaps slightly earlier in the 2010s were worried about. And then a chat GPT comes along and this conversation just gets thrust straight back into the fall front of everything. So it seemed like it had a thrust and then a little lull and then it's really, really sharply come back up again.

1:27:29Yeah. I think it's, I mean, yeah, why shouldn't like over index too much on like any of the one latest little development. But I think also people's expectations changed. So now like, wow, it's been like four weeks without an amazing release. It looks like AI winter, it was all just hype. And if you zoom out, I still think we are on an extremely rapid up ramp and have been since the start of the deep learning revolution in 2012, 2014. Nick Bostrom, ladies and gentlemen, Nick, I really appreciate you have been a huge fan of your work for a long time. Your book is in the hundred books that everybody has to read list that I've been pumping for a very long time.

1:28:16Where should people go? They want to keep up to date with your work and your books and everything else. Well, I'm not active on social media, so I think my website, nickbostrom .com, is where I put my papers and everything, so that might be the best place. Nick, I appreciate you. Thank you for your time. Thank you.