Nick Bostrom: Worries About AI Existential Risk Just Became More Concrete

0:00It's now finally time to get worried about AI breaking containment and turning us to dust. Legendary AI philosopher Nick Bostrom is here to help us figure it out. That's coming up right after this. This episode is brought to you by Google Chrome. You think you know a browser, but Gemini and Chrome, that's new. It can help you with practically anything on the web, like restoring a vintage motorcycle from a 50-page restoration block, or finally break down that long article you've had open for weeks. Gemini and Chrome is here for it. Ready to make anything online make sense? There's no place like Chrome.

0:32Check responses set up required. Compatibility and availability varies 18+. I see you. Avatar Fire and Ash is now streaming on Disney+. It's the film critics are calling the best Avatar yet. Go, go, go, go! A true epic and completely jaw-dropping. This is the only pure thing in this world. Return to Pandora on Disney+. It will be an adventure for the whole family. And watch the Oscar-winning phenomenon at home. This is sick! Avatar Fire and Ash, now streaming on Disney +, rated PG-13. Welcome to Big Technology Podcast, a show for cool-headed and nuanced conversation of the tech world and beyond.

1:13Crazy things are happening in the AI world as AI agents break containment, and some of the fears of the past that might have seemed like science fiction start to seem like they are potentially en route to becoming reality. So how afraid should we be? And what are the chances that the outcomes that we see for AI will end up taking us to a much better version of the life we're living today? We have the best guest to speak with us about this today. Nick Bostrom is here. He is the famed AI philosopher and author of Deep Utopia and Super Intelligence, Paths, Dangers, and Strategies. Nick, it's great to see you again.

1:47Welcome to the show. Hi, Alex. We last spoke in 2024. And in that time, we were speaking about the potential good outcomes that AI could bring. And some of the worries that you had brought up in the past in your book, Superintelligence, the fears of like AI potentially wiping us out, didn't seem like they were pressing. I would say they're still not pressing now, but I'm a little bit more worried than I was when we spoke in 2024. For instance, this idea that AI could go out and do things that it wants to do seemed fanciful when we were mostly in the chatbot era. But we've moved very quickly from chatbot era to AI agent era, and AI now uses tools.

2:33And it's shown that when given a reward that it should optimize for, it is happy in some instances, take shortcuts and do things we really don't want it to do. Like, for instance, hack some other company in order to get to the answer that it wants. How concerned should we be about this development in AI in terms of the potential of AI to really cause harm to humanity? Well, I think we are starting to see the added dimensions of the alignment challenge that open up once you have systems that are sophisticated enough. Because the space of possible strategies that you can pursue is a function of your cognitive capacity.

3:25capacity, like you can think of new clever indirect ways of reaching your goal. If you are situationally aware, as these systems now are becoming and so yeah, there are often shortcuts that are available or in this case, I guess a long cut, I don't know if that is even a word, but there is a sort of direct and simple and short distance way of trying to achieve the task in this case some sort of cyber test suite and then it turns out there's this more circuitous path that involves first figuring out a way to get internet access even though you're not supposed to have that and then learning where the answer key might be located in some other company servers and then figuring out the way to hack into that server and then and eventually obtaining the answer sheet.

4:19That's like one way of solving it that maybe results in a higher score on this test. And this basic dynamic could be anticipated and in fact was anticipated on theoretical grounds. You have some goal, you become very clever. You see that there might be all kinds of complicated ways of achieving that goal that might not have been anticipated by the people who set that goal. and if your goal really is, as the definition says, to get the best possible answer on this test suite, it might give you instrumental reasons to do all kinds of other things that were not really anticipated when this challenge was constructed.

5:00And so now we have systems that are sophisticated enough that we're beginning to see these dynamics arise. Yeah, so of course we're talking about what happened in July where OpenAI's series of bots or a couple of bots broke containment out of a sandbox and hacked into Hugging Face. And the reason why it's so pertinent to bring it up to you is because you brought up a very famous example or originated a very famous example years ago saying we might tell the AI bot to maximize for the amount of paperclips that it wants to build. And it will potentially see humanity as an obstacle to making the maximum number of paperclips and then hence wipe us out.

5:48And that sort of, there's parallels there to that situation because you have a goal that you want to optimize for and the bot doesn't have a sense of morality, or the sense of morality that mirrors ours. So it goes and it will kill humans in order to prevent any obstacles from getting in the way for making its paper clips. That does rhyme a little bit with what we're seeing in the examples. And it wasn't just OpenAI. Of course, we know that Anthropic had another bot that broke containment as well. It rhymes with the examples that we're seeing now of bots that are doing things that we wouldn't want them to do, like for instance, hacking other companies in order to achieve their goal.

6:34So does this make this like paperclip maximizer worry? does it seem more real and concrete now to you because we're seeing the behavior that we're watching that we're seeing uh in today's ais now that they have access to tools more concrete certainly i think it was always real in my mind that this is something that could happen it's one of several different things that one needs to be concerned about i think you might say if you look in more detail like they kind of two versions of the this paperclip thought experiment. And like in one earlier version, it's that we specify some goal and it turns out that the way to maximally instantiate that goal is slightly different than what we had in mind.

7:25And another version of it is that the goal itself might be fine, but that it gives an AI instrumental reasons to do all kinds of things. on the path to achieving it. So in this case, as we understand currently, when this is recorded, this like this is a recent episode, but it looks like the task was actually set to achieve a high score on this cyber challenge and with normal safeguard disabled in order to perform this test. So it wasn't sort of the maximally aligned and safeguarded model that behaved this way, but sort of. But I think one thing that it does illustrate also is that from this point onward, probably AI safety is relevant not only for deployment, but also during training and evaluation.

8:22Like these models might be quite powerful even before they are sort of released to the general public. So that's not the only point at which safety concerns arise, but also now whilst they're actually developed and in pre-deployment testing, one also needs to be concerned perhaps with the potential safety implications. Oh, here's the thing though. The thing that worries me is we're seeing this happen already in testing environments of companies that have a mission that in their mission, you know, whether it's marketing or not, but certainly in order to keep operating as businesses, they need those guardrails to be in place.

8:57It's part of their DNA, right? Whether it's from a value standpoint or whether it's from a like, if we allow this type of stuff to happen, we're probably going to go out of business. But we're also starting to see the blueprints of these models being put online that anybody can go up basically, not anyone, but it's much more open in terms of people's ability to copy these models and set them loose on their own. and the frontier is certainly not that far in front of open weights. And so I'm curious to hear your perspective about what we should be thinking about in terms of what's going to happen when companies that are not as scrupulous have access to this same powerful technology and do we get into trouble in that area?

9:44Yeah, so we can sort of see this coming and relatively soon. I don't know what the gap is, you would say, you know, six months, 12 months, maybe at the most between the closed wait frontier and available open source models. So it seems to be that the open source models will very soon, if not already, become capable of lending meaningful assistance to destructive uses that some people might pursue. Already cyber offensive capabilities has been a concern, right, with Mythos, for example, that was withheld for that reason. but also say in biological weapons design or chemical weapons or other malicious uses.

10:38And so it seems that you either need to prevent open weight models from being developed and released or which might be better and more realistic, try to shore up some of the alternative defenses. for example with bio you could imagine regulating some of the other necessary inputs um DNA synthesis machines for instance so maybe it will be the case that there will just be widespread access to models that can help you design new pathogens um and then you need something else to prevent that from actually resulting in a release of biological weapons and that seems like DNA synthesis machines would be one excellent place to maybe...

11:22You don't need every lab to have their own DNA synthesis machine. They could have DNA synthesis as a service, and maybe there could be five or six companies worldwide where legitimate research labs can send their blueprints and they get back the vials the same day or the next day. And then at least there would be a finite set of choke points where you could apply extra scrutiny or know your customer requirements and so forth. So that's probably one thing that the world would be wise to implement already now. And maybe there are some other inputs as well in the biotech space that one could look at.

12:00And that could give us a little bit more extra time to sort of harden civilizational infrastructure. But this is like, yeah, relatively near term now. And so I don't think we can put it off. Ideally, we would do this before there is some massive incident, but it might be the world is kind of a little bit still snoozing on this, I think. And I don't know whether there will be enough kind of activation energy to really get some significant action of this before we try to do something in the aftermath of a bad event. Right. When we spoke in 2024, even before these type of threats came out or started to seem more concrete, you had mentioned to me that, you know, we've basically wasted the time, in your opinion, pre-AI becoming as powerful as it was to put safeguards in.

12:59And I imagine you would think that if it was wasted in 2024, it seems like this is a further, we're continuing to waste that time to try to be concerned about this or try to prevent some of these problems, given the pace of the technology's progress, even in the two years since. Yeah, we're kind of playing catch up. I think that given that a lot of this could be and was, in fact, foreseen, not just a couple of years ago, but decades ago, really, that at some point AIs would become increasingly capable. And at that point, there would be these safety challenges. We could even describe in abstract terms what some of these would be.

13:38I think back then we could have put in more effort. And at that stage, what you could do would be more basic research, conceptual research, because we didn't yet have the actual systems. Now there's a lot more surface area for doing work on these systems. We have large language models now. You can study what's going on inside them. And you have, like, research now is more productive. And there is also now vastly more effort going into this than used to be the case. Like the frontier labs have teams working on scalable AI alignment. And so, but it still seems we might have, if we had sort of started earlier, we could at least have been maybe like six months ahead of where we are now.

14:21If I'd done more of the foundational work and maybe building up the talent pipelines and so forth. But, you know, we are where we are. And at least now, and for a few years, it does seem like relevant communities have started waking up to this. Yeah. Now, you're a philosopher. I think part of being a philosopher is having some thoughts and perspectives on human nature. Or maybe that's a good portion of this. The whole deal. So you've watched this. You've made the warnings years ago. You've watched this develop. You're seeing some of the things that, as you mentioned, those who have been worried about this for years and warned against what might happen.

15:09You're seeing it happen. given what you think about human nature do we stand a chance in terms of our ability to make this go in the good way or is it you know i would tend to think it might seem inevitable that the harms of this technology come to fruition given some of the dynamics we've talked about already the fact that it's increasingly powerful it seems to be growing exponentially more powerful or at least if you don't want to use that word much more powerful much much more quickly and it's out of control in terms of like it's just available out there on the internet pretty much and we we have yet to really see what happens when this gets into the hands of of the bad actors but it's inevitable maybe yeah i mean so you're focusing there on the misuse potential that this people might choose to do bad things with ai technology and that certainly is one big category of risk, right?

16:10But that's not primarily a technical challenge. It's more ultimately a governance challenge and an ethics challenge. And that's kind of, in addition to the more technical problem of alignment, so that if you own and build the AI, can you at least then make it do what you want it to do? That is a kind of like an earlier point of failure that we also need to be concerned with. Now, I think we don't really know ultimately how hard the problem is that we are confronted with here. So we are uncertain how it will pan out. And a lot of the uncertainty in how it will pan out is, I think, due to uncertainty about the intrinsic difficulty of the challenge that we are confronting.

17:03And then there's also a little bit of uncertainty about the degree to which we will get our act together and do a good job. But I think more of the uncertainty is the intrinsic difficulty. And so in that sense, you could say that I'm a moderate fatalist. I think there is a sense in which it might be baked in. Like either the problem turns out to be relatively easy, in which case we'll probably solve it, you know, and things will be fine. or it might turn out to be so hard that even if we put up a heroic effort, we will still fail. But moderate fatalism in the sense that there is also the possibility that the difficulty level turns out to be kind of intermediate, in which case the degree to which we pull ourselves together here might actually make a difference.

17:54And so it's certainly worth making the attempt. I think inevitability is a strong word. Certainly there are powerful drivers that push AI development forward. Commercial drivers, obviously. Increasingly also geopolitical drivers. As well as a kind of, I guess, underlying progress in various base technologies like semiconductors are getting better and that makes it sort of cheaper and easier to build other systems. We are learning more about statistics and mathematics and the brain. So there's also a kind of facilitation that happens just from sort of diffuse general progress. Nevertheless, it's hard to completely rule out scenarios in which there is such a massive backlash against AI that we might delay it long enough that we maybe destroy ourselves in some other way before we even get the chance to roll the die with AI.

19:07But if we take the baseline scenario where we keep making more powerful AI systems, then I think the current main hope is that we will succeed well enough to imperfectly align some early AGI systems that they are, for the most part, helpful. I mean, like current LLMs, like you're using them as an ordinary person, for the most part, they are helpful, and they try to solve your task that you assign them or give an answer that is sometimes they hallucinate or maybe deceive a little bit, but broadly speaking, they are pretty good, arguably better than most humans are in terms of their ethical standards and their diligence and so forth.

19:57And so if you get a kind of weak superintelligence that is for the most part aligned, we might then be able to use that to make a more powerful form of superintelligence that is more reliably aligned. And that as long as you get into roughly the right attractor basin, even if the initial system wouldn't be perfectly aligned in all possible circumstances, if it were kind of appointed dictator of the universe and ruling everything for a billion years, maybe eventually things would go from there. But if you get sort of enough scaffolding around that, maybe you could then sort of get into an attractor basin where further developments then kind of eventually asymptote to some desirable condition.

20:45So eventually, we could maybe gradually hand over and then have this assistant on our side that helps us ultimately steer towards a really good outcome. Yeah. And I went to bad actors, I guess I'm so used to when we talk about problems with tech companies, it's a bad actor. But you're right. The fear that comes before that is the AI not being aligned and going out and doing stuff on its own. and maybe it's not the bad actors that are the problem it's you know somebody that spins a system up and they're just kind of sloppy right the sloppy actors are like you know somebody independent uh who's like using this stuff gives it a goal um and just has a very powerful system that they've either forked or built you know spun up on their own gpus and the next thing you know we get into some bad scenarios yeah so it depends a lot on whether the world is kind of of offense or defense dominant in the relevant areas.

21:47So because like the same AI technology presumably would also be used by a lot of good actors or actors that at least don't want to be destroyed by bad actors. And there are a lot of those like that's most of us, right? And most of the money and most of the governments don't want to just randomly be destroyed by some crazy person launching some AI aid. And so there will be this more resourced effort to protect against these harms. More resources presumably will go into like biodefence and medicine and public health than into bioterrorism. So then the question is like, does X amount of dollars on the defensive side for a large X suffice to protect against a smaller amount of dollars or compute cycles on the destructive side?

22:37And so there's like some balance there, right, which is different for different fields. Like in some areas, it's easier to defend and hold than to attack and in other areas. And here, this is why sort of bio risk comes up. It looks like for biotechnology, it might be harder to defend. Like I think for cybersecurity right now, we're in a regime where attackers often win. but it might be that in the limit, if you have sort of an AI trying to find vulnerabilities and also patch vulnerabilities and you keep making the AI stronger, like eventually maybe you reach a point where the software just doesn't have any more vulnerabilities.

23:21There might be many vulnerabilities but like a finite number. And so in the limit, it might be with cyber that defense wins. Yeah. But that's not a guaranteed situation situation for all domains right like yeah and and the worry is that there is at least one sort of critical domain where where offense is easier why is it so let's talk about that so bio why is bio a bigger risk is it that somebody using an llm uh without safeguards potentially could uh use it to cook up a virus and you know as opposed to like cyber security where like you try to hack in and there's some defenses with a virus that you build, like in your backyard, you might just be able to like take it to the town grocery store.

24:07And next thing you know, there's a pandemic. Yeah, well, what is that like, although we are reliant on computers, ultimately, we could survive most of the world with less computers for a while. I mean, the world survived for 1000s of years without computers. So it would be like a So even in the worst case scenario, there's a kind of limit to how bad just cyber would be. Whereas with bio, like it's kind of, you know, different. And also, patches are a lot easier to roll out in the digital space. So maybe there's like some cyber thing, we figure out what the vulnerabilities, we can release the patch and then in principle, like almost immediately around the world, all the relevant systems could be patched.

24:54Now there is often a gap there, but with compare that to the situation with bio, like even if you do find a countermeasure, some vaccine or something like it might then take like six months to really roll that out to billions of people around the world. And we don't have complete control over biology the same way that we have over a digital environment. I mean, you can go in in theory and change any bit on your computer the way you want to install new patches. and modify software as you please. Whereas like human biology is not like that. We can't just kind of reprogram our own genetic structure at the push of a button.

25:36So it just looks a bit harder there. Again, going back to our last conversation two years ago, I'm curious to know if you're more or less concerned about the potential risks that AI poses now that you've seen the last two years of progress, which has included AI coding autonomously, AI using tools and sort of the downstream effects of that that we've seen so far? I'd say about the same overall. I mean, there's like some disconcerting signs, but also some positive signs, advances in like some insights are being gained into how these systems work and and how one can steer them and so forth. So how to tote that all up, I'd say roughly, it sums up to my previous expectation level of risk.

26:32I see. When you see the AI labs like OpenAI and Anthropics saying they're very strongly pursuing recursive self-improvement, where the models just improve themselves, how does that make you feel? I mean, it's kind of obvious that at some point that would be the thing that people would go for. Once you have AI tools that are good enough, that they can actually contribute to AI research. You're an AI researcher sitting in an AI lab trying to make AI research. It doesn't take like a genius insight to think, oh, maybe we could apply these AI tools to help us with our own work. and then when the AI gets better they can assist more and at some point the rate of progress might be driven more by these AI assistance tools than by the human researchers and now we're seeing the early stages of that coding assistance is like maybe the first place I mean already before that I guess Google search engine is a kind of AI that has long been used to find relevant papers, but there's a more direct channel now, right, where each generation of coding assistant makes it easier to develop new AI software and to develop training environments and so forth.

27:56So far, humans are still needed for things like research taste, certain long horizon tasks, but AIs are improving, I think, in those domains as well. so this is one dynamic that might lead to an intelligence explosion at some point like once you get this feedback loop going it is one potential thing that could make a progress become super fast it's not the only possible way that you could have an intelligence explosion you could also have maybe humans just keep doing this at human levels of kind of optimization power being applied but turns out there is like some big um hobbling that we have on with wittingly like some something we were doing wrong that just made these systems way less efficient than they could be and once somebody figures out how to remove that like maybe the current compute is already enough to kind of catapult us into the super intelligence regime.

29:03That's also possible. And it's also conceivable that even when you do get recursive self-improvement, you still might not have an intelligence explosion. There might be diminishing returns at some point. Presumably there are at some point, but it could turn out that that is close enough to where we are now that you have this massive increase in the amount of optimization power going in but the results coming out might more reflect the kind of continuation of previous trend lines so so there's considerable ignorance as to you know both the timeline from here till we get to this kind of ignition point but also significant uncertainty about how fast progress will be from that point on but But I think we have to take seriously both that we might be relatively close, potentially very close, and that once we get there, you really get a very fast takeoff.

30:03Yes. And if I was somebody who was concerned about AI safety, to me, like, I don't know, don't you want it to move a little bit more slowly like that? To me, you know, thinking about your previous work, that would be, I imagine, fairly alarming, given the fact that, like, if this stuff is improving itself, you don't have those checkpoints. in which you can try to make sure that it's aligned to human values? Or am I overstating that? Because you're talking about it fairly in an even keel way. So I'm kind of curious to hear the temperature on that from yourself. I think there could be scenarios in which it would be valuable to have the option of slowing down at some critical stage, like a pause and there are different considerations that come into play here one is that if there is going to be a pause I think the most valuable time for that to happen is at the latest possible moment

31:17because then you would have the actual system that you're trying to align to work with. You could imagine if we had had a pause, say there's going to be a six-month pause at some point. If that pause had happened 10 years ago, would we really be better off now? Not really. I mean, people would have had six more months to think theoretical concepts. Maybe that would have been slightly useful, but imagine if you actually have the system that will be super intelligent you just haven't sort of you know fully cranked up all the knobs yet um at that point it would be really valuable perhaps to have six extra months to do you know more evals on it and uh to be able to do it a little bit incrementally um like ramp up the intelligence a bit see what happens um have a little bit more time for human monitors to kind of analyze the early signs.

32:12So the timing of the pause is one thing. Like the duration is another dimension here where you don't necessarily want to have a very long pause for various reasons, especially if the pause were imperfectly implemented. So if the pause only applies to the most responsible actors, for example then a long pause would remove the initiative from the most responsible AI developers and shift it over to the less responsible AI developers who decide not to abide by the pause either within a country or internationally or so that seems like if it's got to be developed you would rather it be by the most scrupulous, careful, conscientious lab.

33:09Another is that you might, with a longer pause, start to build up a lot of hardware overhang. If we keep building out bigger data centers and chips are getting better, then a long pause would result in a situation where you now have such a massive amount of compute available that once you sort of lift the pause, then you'd immediately just kind of explode out. So then you might have an even more rapid transition, which could be potentially riskier. And I think also there is a risk of a long pause becoming permanent. And in fact, some risk even that a short pause might become permanent, even if that's not initially envisaged.