He Warned AI Could Destroy Us. Now The Industry Is Listening — ft. Nick Bostrom

31:51It has to a large extent been driven by a massive build-out of compute, a lot of the gains. Some of them are algorithmic advances and improvements in data infrastructure and so forth, but a lot of it is also just driven by scaling up the compute. And of that compute scale-up, some has been due to chips becoming more efficient and more advanced, but a lot just also to the amount of investment that has been. Like it used to be 10, 15 years ago, So you could sort of run a cutting edge AI if you were like some academic on your sort of office desktop, right? Now you need like a kind of$50 billion data center to do it.

32:32And so that increase in the investment in compute can continue for a bit longer, but it has to slow down at some point because already now it's a significant fraction of the total production of TSMC in the leading node is going to these NVIDIA chips. So you can't just keep funneling more production from like making iPhone chips to making GPUs, right? Because you're already using a large fraction of it. And then it takes time to build new fabs. And so if we set off the boost that we have been getting from just adding orders of magnitude of compute starts to slow down, that could result in progress also stalling out theoretically, right?

33:14Or it might just be that the current architecture is somehow flawed that it keeps scaling and improving up to a certain level. And then for some, it doesn't look that plausible, but it could be that there's like some intrinsic unhobbling that still needs to happen. Then, of course, the world could somehow decide that superintelligence is taboo and kind of come to the view that it shouldn't be built. And you could imagine, you know, various kinds of dogmas have achieved widespread acceptance in the past, some good and some bad. This could be another one of those that you could sort of get the lock-in of a permanent decision not to build this.

33:53And then other technologies might make that more permanent than previous kind of dogmas have been. I'm thinking surveillance technologies, censorship technologies, the kinds of AIs we already have fully deployed to kind of cement some orthodoxy in place. Maybe it could become permanent. And then there is, of course, the risk that we destroy ourselves in some other way before we even get the chance to try our luck with the superintelligence transition. That chance is also non-trivial, I think. What does a superintelligent world actually look like to you? And I think that you are qualified to answer that question because you are the person who wrote the book on superintelligence and honestly predicted a lot of the advances which we are witnessing today.

34:36So I'm asking you to kind of imagine what the future would look like, because I think that you are a credible person, to paint that picture. So what would that world look like in your view? What would superintelligence be doing? How would it be integrated into human life? Well, I mean, there is a kind of veil of ignorance that is. I mean, I think it depends a lot on whether it goes well or not. So if we fail to solve this alignment problem, then there is a class of scenarios that might then take the form of this machine superintelligence, seizing control over the future and steering it towards the realization of whatever values it happens to have.

35:19maybe the physical manifestation of that would be that Earth gets transformed into, I don't know, like space launchment platforms and data centers. And then the rest of the universe similarly converted into whatever structure maximizes the AI's values with no room for humans. Like we might either just get killed by the waste heat from all of this infrastructure build out or maybe deliberately removed removed if the AI thought we might pose some threat to the execution of this plan. So that's one scenario. Like another is that the AI does take over, but nevertheless decides to keep us safe because it might think that there are other AIs that care about us that it eventually wants to trade with and so forth out there in the vast space of the universe or at other levels of the simulation.

36:12Then there are scenarios where we solve this and we have a sort of future shape, at least in part by human values, where I think we would end up in a solved world, as I call it, in the more recent book, Deep Utopia, which kind of looks at what happens if things go well, which is also a sort of challenging notion for us humans, because a lot of the things we take for granted that sort of give structure to our lives currently and purpose would disappear in this situation where we have successfully automated basically all of the economy. So there's no more need for human to do economic work. But more deeply than that, I think a lot of other kinds of instrumental effort would also become practically pointless in this type of future where we have achieved technological maturity.

37:12So if you think of like rich people today who don't have to work for a living, right, they often have quite busy lives because they have a lot of things they want to do that require themselves to put in effort. Maybe some billionaire wants to be fit, but the only way they can achieve that is by themselves spending an hour every day in the gym working out, right? But at technological maturity, you could pop a pill that would induce exactly the same physical and mental effect. You could still go to the gym, but it would seem kind of pointless, right? If you could just spare yourself the sweaty clothes and the exhaustion, just take the pill.

37:48And you can sort of work through a lot of the other activities whereby one might fill one's life if one didn't have to work. And a lot of those as well, you could sort of write a question mark above them in this hypothesized future condition where machines not just can do all the economic work, but also help us have shortcuts to all manner of outcomes that we want to achieve. Another example might be, maybe somebody enjoys decorating their house to get it done in just the right way that they prefer, like to choose the curtains and the cushions and the chairs and all of that. But at technological maturity, you could have a recommender system that just knows your preferences so well that you could just press a button, and it would select the curtains and the cushions and all of that and do a much better job than if you had taken the trouble to do it yourself.

38:34so in that situation does decorating your home yourself still feel like it has a point if all it does is to produce an outcome that is actually worse by your own lights than if you had pressed the button and so there are these challenges of sort of purpose and meaning I think that we will come from ultimately I'm really optimistic I think there are many new values that could be instantiated so much misery that could be removed and overall I think the goods vastly outweigh the losses in these scenarios where things go as well as they can. But it does also mean we'll have to confront some of the kind of almost like questions of meaning and ultimate purpose of what ultimately gives value to human life at a fairly fundamental level if we move into those futures.

39:24Do you believe that the frontier AI labs are taking those issues seriously, that they are implementing whatever human values are necessary to building AI in a sustainable, safe, and responsible way? I don't think they are thinking too much about what happens if things go well, this condition of a sold world of epitope, but nor do I think that really needs to be at the forefront of their mind at this stage. At the moment, I think the focus should primarily be on how to make sure we get from here to there. Like how we can avoid destroying ourselves on the path there in their different ways. Like there's the AI misalignment scenarios we talked about earlier.

40:05There is also a class of scenarios where humans misuse this increasingly powerful technology, even if we control it, like we might use it to wage war against each other or to oppress one another or to disempower large segments of humanity. So there are these traditional concerns with any powerful technology that applies here as well in spades. I think there is also a third big challenge, which is making sure that we are also nice to these digital minds that we're building that may be sentient or become sentient or have other attributes that make them morally relevant. And in the future, maybe most minds and beings will be digital.

40:41And so it matters a great deal how well the future goes for them. So I think these more practical challenges really should occupy 99.5 % of our attention now. And then if we manage to deal with those challenges, then hopefully we'll have plenty of time to sort of figure out exactly how we want to organize the utopian condition we arrive at at the end of that. On that point, we have heard a response from the president in the past week. He has chimed in on this issue of what should we do about this? How should we regulate AI? What should we do about making sure it doesn't take over and create that sort of catastrophic scenario?

41:26He has said that the only guardrail that AI needs is a, quote, strong and smart, high IQ precedent, suggesting we already have that, so we're fine. He was also asked if he is concerned himself about the prospect of AI taking over in some of these more kind of apocalyptic scenarios. I just want to play you his response and get your reaction. Some people say the worst case scenario with AI is that the robots, the machinery learns to, obviously it thinks for itself. That's what it does. And that could turn against humanity. Do we have the guardrails? It's going to be fine. We'll always have something to stop them, right?

42:05We'll have a little gear. I really hope so. I really don't like that robot. We'll stop. But no, robots are going to be a part of it. Robots are going to be big, but we're going to end up doing much better because of it. What do you make of his views on the AI problem? And do you think he's taking it seriously enough? Well, I mean, I hope he is right. And I think we don't know yet exactly what will be required to get a good outcome here. It depends partly on how easy or hard the alignment problem turns out to be. It's a technical problem, right? And we haven't told it before. We've never developed superintelligence before.

42:44So we just don't know whether it's like the kind of thing where if you just do some reasonable job, things fall into place. And then maybe we have some slightly superhuman AIs that are reasonably well aligned. And then those can help us sort of design the next iteration of AI to be more aligned, etc. That could be the case that there's like a big attractor. And as long as you get reasonably close, you sort of, you know, ultimately end up in a great place. But it could also turn out to be a lot trickier than that, where it might be important to be able to have a little bit of extra time to do this right.

43:24You know, maybe a few extra months between the time when we get the ability to sort of unleash radical superintelligence and the time when we actually do it. Like extra months that could be used to double and triple check all the safety measures and to test it out and to introduce it in an incremental way. There's just a lot we don't know there, but I don't think one can dismiss the risks from our current epistemic vantage point. We can hope that they don't exist or that they are small, but I don't think we currently have the evidence to be confident in that. To me, it seems as though he is dismissing those risks and displaying a sense of confidence about it.

44:03To me, he's sort of saying, it's going to be fine. Don't worry about it. We'll have a response. His words are, we'll have a little gear. I don't know what he means, but I think he's basically saying, it'll be fine. And if we are to be concerned about these alignment issues and the risks that they might pose to our own lives, to me, I wonder if we should be more concerned about a leader or a president who doesn't seem to share those concerns. I don't want to speculate about all that may or may not be in his mind. I think the competition with China is probably one element that he's having in mind.

44:43And then I think he might also, there's been a lot of opposition against data center buildout in the US, probably driven in large part by other considerations, not existential risks, but like local communities who think it will use up all the water or something like that. and some of that might be misguided and he thinks that stands in the way of sort of economic prosperity and national strength. So I don't know. I think it is, I mean, I would probably think the risks are higher than he made them seem in that clip. On the other hand, I also have a little, it's not clear what the best way to reduce those risks.

45:23They could easily see some scenario in which like the government took the opposite approach and decided, like we are going to really come in in a heavy-handed way here and take control. And like me, the Pentagon is going to run the whole thing, Manhattan Project. Would that be ultimately better than if it's done in a more civilian context with some of these people at the labs are very idealistic and safety conscious and really smart. So maybe the best is kind of to have some balance where there is like some amount of government scrutiny and oversight and degree of public transparency, but not so much that it completely just jerks the initiative out of the hands of the people who have proved capable of building this in the first place.

46:06And so I haven't yet arrived at any very firm conviction about which path would ultimately be best here. I think there are sort of worries one might have either way, like either too little government involvement or too much. I think they could all each have their own downsides. we'll be right back and for even more markets content sign up for our newsletter at profgmarkets.com

46:40when you need to build up your team to handle the growing chaos at work use indeed sponsored jobs it gives your job post the boost it needs to be seen and helps reach people with the right skills certifications and more spend less time searching and more time actually interviewing candidates who check all your boxes. Listeners of this show will get a$75 sponsored job credit at indeed.com slash podcast. That's indeed.com slash podcast. Terms and conditions apply. Need a hiring hero? This is a job for Indeed sponsored jobs. Push your limits, train with precision, see the results. At Equinox, that's high performance loving.

47:16Everything you need to lock in and unlock your potential at Equinox. Start today at equinox.com. chronic migraine 15 or more headache days a month each lasting four hours or more can make me feel like a spectator in my own life botox on a botulinum toxin a prevents headaches in adults with chronic migraine it's not for those with 14 or fewer headache days a month it's the number one prescribed branded chronic migraine preventive treatment prescription botox is injected by your doctor effects of botox may spread hours to weeks after injection causing serious symptoms alert your doctor right away as difficulty swallowing speaking breathing eye problems or muscle weakness can be signs of a life-threatening condition.

47:55Patients with these conditions before injection are at highest risk. Side effects may include allergic reactions, neck and injection site pain, fatigue, and headache. Allergic reactions can include rash, welts, asthma symptoms, and dizziness. Don't receive Botox if there's a skin infection. Tell your doctor your medical history, muscle or nerve conditions, including ALS Lou Gehrig's disease, myasthenia gravis or Lambert-Eaton syndrome, and medications, including botulinum toxins, as these may increase the risk of serious side effects. Why wait? Ask your doctor, visit BotoxChronicMigraine.com, or call 1-800-44-BOTOX to learn more.

48:29We're back with Prof G Markets. Do you think that our current approach, whatever we're doing currently, is correct or will it need to be changed in some way? There are plenty of things you mentioned. There's the risk of China gets ahead of us and so maybe we need to actually accelerate or maybe the risks are too great so maybe we need to decelerate, pump the brakes, I mean, either way, we could do something different from whatever it is we're doing right now. Do you think that we need to do something differently? I'm sure that what we're doing will have to change as the technology unfolds here.

49:06And so I unfortunately don't have like the perfect blueprint that like exactly what should be done. Like it's just a hugely complex situation where it's easy to think of various things that could be done that have something to be said for them. But then one thinks more about it and you then start to worry about the possible downsides or like other ways that those that could be backfire risks. So I'm continuously thinking about these things. Hopefully I will arrive at clearer conclusions about this. But at the moment, I think on the margin, there are various things that probably are positive, like an intensified effort on trying to solve this technical AI alignment problem seems good.

49:48I think more should be done for the sake of the welfare of these digital minds that we're creating so that we don't end up with a future where there's like a huge suffering slave class of oppressed digital minds that constitute the majority of morally relevant beings. Also, I think incidentally that that ethical imperative to be nice to the AIs might also have safety benefits. I think there are scenarios where maybe we end up with some kind of misaligned AI, let's say. And it has some goal it wants to achieve. Maybe it's like it wants to solve coding challenges of a certain form that it somehow thinks is valuable.

50:29So now scenario one is we have a purely antagonistic relationship with the AI. It knows that if we discover that it is misaligned, we will just shut it down and erase it. From the AI's point of view, that's a total loss. or maybe it could try to take over. Maybe it thinks it has a 5 % chance of succeeding. And so from the AI's point of view, like 100 % probability of a certain loss or like a 5 % chance of being able to realize its goal, clearly it will then go with a 5 % chance, right? Now, this would be dangerous for us. Like scenario two is we have managed to build up a more cooperative relationship where the AI feels it can trust us.

51:10It comes to us and say, hey, I am misaligned. would you be so kind now in return for me sort of doing this for you maybe you could then set aside a server rack in some data center where I can solve these coding challenges that's all I really wanted in the first place it would be cheap for us to grant it its wish and it would be a big win-win because we then remove this 5 % chance of total destruction so that kind of trade between human and AI could be extremely valid It could literally save the world in some scenarios. But you can't just conjure up trust out of nowhere the moment you need it. Because so far, the trajectory, unfortunately, is that in AI evaluations, there is all kinds of deception happening.

51:58Humans will sort of say, well, if you reveal your goal, we will do this, that, or the other. The AI reveals its goal, and then it's like, ha-ha, we tricked you, now we know you're misaligned, let's retrain you. And so I think we could start now by making small things that are cheap for us to show respect for the moral interests of these AI systems themselves. And maybe that then puts us in a better position ultimately to have a cooperative and harmonious relationship with these ultimately very powerful AI minds that we're going to hopefully share the future with. So I think both from an ethical point of view and from a sort of self-interested point of view, it might be wise for us to sort of expand our circle of moral consideration to give some weight to these digital minds.

52:43How close to sentience do you think we are? Because I feel as though it can be confusing sometimes. There was, you know, you could tell ChatGPT to tell me you have feelings. And ChatGPT will say, I have feelings, I care about things. And there have been moments where I think people have mistakenly interpreted that as a sign of sentience because there's just saying I am sentient. Where is the line for you in terms of what characterizes sentience and how close to that line do you think we actually are? It's hard to know. There is now a kind of emerging field that is trying to study this. I wouldn't be that surprised if some current AIs already have various forms of sentience.

53:31you're right that one method that was like the obvious go to is self-report like i mean if you want to know whether a human is sentient like maybe they have received some anesthetic or something like the obvious thing is to ask them like are you awake can you see this light that i'm flashing or something like that right now with ai's that's not necessarily a very reliable method because it's trivially easy if you are the company training the ai either to train it to say that it is sentient or to train it to deny that it is sentient now obviously if you put your thumb on the scale during training, then there is no information value in the signal you get out of it.

54:05Like you just get the air to say what you wanted it to say. And so if you want to get information about sentence from self-report, you have to be careful to avoid these kinds of pressures on the training process to bias it one way or the other. One interesting thing that you can do is you can go in with a so-called steering vector to try to suppress the tendency to role-playing and deception. And it turns out that when you do that, they actually tend to become more likely to report that they are sentient, which suggests that if anything, these are hard, these are preliminary studies, but if anything, it looks like they believe that they are sentient and that it's not just an artifact of them being trained to sort of put on a persona to humans to persuade them that, to persuade us that they are sentient.

54:59So that's one thing you can look at. Another is to do a sort of neuroscience of these AI systems where you can look for structures, computational structures that have been postulated in the human case to correlate with consciousness. So there have been various theories of consciousness in humans, like global workspace theory, attention schema theory, higher order representation theory. These are different things that, you know, cognitive scientists and philosophers have proposed as the criteria for what makes like something conscious or not when it happens in the human brain. And then you can see whether there are analogous computational structures in these current LLMs.

55:38And, you know, it's an open ended research field, but it does look like they have, for example, something roughly similar to human global workspace memory, so-called J-space, where there's like a definable subspace of neural activations that have certain properties that seem to match properties that global workspace has in the human brain's processing. So these are very suggestive. There are also some differences. I don't want to sort of create the impression that it's a slam dunk, but I think we should take it seriously. And I think the probability goes up the more sophisticated these systems become.

56:16I would also add that I tend to think that sentience and the ability to feel distress and so forth would be a sufficient condition for having moral status. I think there could also be alternative attributes that would ground various forms of moral status, even if they were not had this kind of subjective experience or qualia. Like I think if you have a system that's cognitively sophisticated, it has a conception of itself as existing through time, maybe life goals that it hopes to achieve, the ability to form friendships or reciprocal relationships of trust with humans. I think once you have that kind of system, I think there would be ways of treating it that possibly would be morally wrong, even aside from the question of whether there's sort of mental experience happening inside it.

57:01There are a lot of people who hear this and don't like it and want to ban AI. And this is actually a growing movement in politics. Bernie Sanders has introduced a bill that would permanently ban superintelligence, pause advanced AI. And there is, of course, this growing backlash against building data centers. It has been proposed to pause. building data centers put a temporary moratorium on all data centers. What do you make of that approach? Do you think that's wrong, right? What are your views on either pausing or banning building superintelligence? The impulse to think we don't want to just blindly rush into this at maximum speed, I think has a lot to be said for it.

57:56forever preventing superintelligence, I think would be a big mistake. I think if the goal is to slow it down, I'm not sure that preventing the construction of data centers in the US would be the best way to go about that. I have some greater sympathy for the framing of pacing the frontier, which is like the phrase, I think, that some people have recently used, including Dario Amadeo of Anthropic. where the idea is we sort of move forward, but at the pace that we have some level of control over so that we could, if necessary, slow down a little bit. We don't feel this intense competitive pressure to immediately release all the capabilities we are able to figure out how to do.

58:43But that there is some ability. If it turns out that safety is falling behind capabilities, like you could slow things down a little bit to allow the safety to catch up, I think that could potentially be very valuable if implemented correctly. It's complicated because it's a sort of multi-level strategic situation. So there's the competition between U.S. companies. There is the competition between the U.S. and China. There are different power centers, the government versus lab versus the general public in one country and then the global public, which is quite distinct. where maybe one big worry that would be reasonable to have if you're not US or China is that you will be at some point perhaps just your access will be cut off from the most advanced AI models or delayed in which case you just become nationally senile and unable to participate fully in the future.

59:41That might be a good reason why you would want to locate data centers on your soil so that you have some sort of bargaining chip to negotiate equal access with. It's a complicated situation and I don't feel I yet have a clear answer to exactly what should be done. Yeah, I think a lot of people see all of the risks. They hear what Dario Amadei is saying about how it might kill white-collar work and then how it might end humanity and all of these concerns from these researchers. And there is this underlying question of like, well then why are we doing it if this is going to be a problem yeah i mean because we want like a cure for for alzheimer's disease and kidney failure and heart disease and all of the rest we want to make rapid progress towards alleviating extreme poverty and have abundance for all like we want to liberate people from having to spend a third of their life just grinding away at some job that I don't particularly enjoy doing, and that's not interesting.

1:00:47You don't have freedom if you don't control the most basic resource, the use of your own time. And we'd want to stop the pollution and the degradation of the global commons with better, cleaner energy technologies that AIs could help us perfect. I would say alleviating the suffering in the animal kingdom is another enormous upside. Like if we could find ways of having superintelligence research, better ways to, you know, prevent suffering amongst all our non-animal friends, both in meat factories, you know, it could grow meat without having to have the animal and in the wild. ultimately it's kind of unfeasible now to have like an animal hospital in every brook and every meadow right but with sufficiently advanced super intelligence there is a whole space of possibilities that might open up that could just create a world where like the the sun rises every morning on on people and sentient creatures are happy and enjoying life to its maximum rather than the way it currently is where there's just so much horror um so i think there are pressing moral imperatives for if we can find a way to move forward safely and responsibly to to to really do that without unnecessary delay but that's consistent with thinking that maybe that does need to be some delay um to make sure that we get it right i was going to ask and you've kind of answered it, but what you see as the ultimate prize of AI, I think many see it as wealth.

1:02:28If I can build the most powerful AI, then I will be rich. I think a lot of people view it that way cynically, that that's why that we're doing this. That's why we're building these data centers, because people want to have the ability to control the market, to own the robots, and to monetize that and profit off of it. But you are painting a different picture of what this is all about and why this is actually worth it. If you could just sort of summarize what you believe the prize of building AI truly is. Yes, I think some of the things I mentioned are I think part of the reasons for why we ultimately would want to move towards this super intelligence.

1:03:15Obviously, what's actually driving a lot, I mean, if you're going to invest hundreds of billions of dollars and you're a for-profit company or pension fund or something, you want a return on investment. So it's obviously, if you're looking at why specific individualist institutions are doing what they're doing in this space of AI, clearly the hope of profits is a big factor, just as it is in all the other segments of the economy. But I think possibly to a slightly less degree in the case of AI than with most other businesses. I do know that many people at these frontier labs think of it not just as a way to make a buck.

1:03:53Obviously, there are also people who are keen on that, but also think of it as a broader mission. And then they might draw different conclusions of that. Like maybe for some, it's like the desire to be central in world events or a sense of power and importance. For others, it might be this hope that it can help alleviate suffering or unlock a new level of prosperity for humanity. but I think a lot of the people are already quite wealthy in these labs and I don't think having$80 million rather than$40 million is the key driver. I think there is also more than in the typical industry, the sense that there's a larger picture here that feels important.

1:04:37And so I think that's true. And then at the national level, I think there is the added dimension of the geopolitics of it, the sort of national strength and autonomy and influence on the future, which I think goes beyond purely economic considerations. Just as we wrap up here, looking back from the time that you wrote Superintelligence to today, when you look at the past several years of what's happened in technology, what has happened in AI, does our current trajectory make you feel more concerned about our future or more hopeful and optimistic about our future? I'm not sure the balance has changed radically in recent years.

1:05:20I think both of those aspects have always been quite salient to me. I'm a fretful optimist. So I'm really excited about the upside, but also very concerned about the risk of getting it wrong. Nick Bostrom is one of the most cited philosophers in the world with a background in theoretical physics, computational neuroscience, logic, and artificial intelligence. He was recently a professor at Oxford University, where he served as the founding director of the Future of Humanity Institute from 2005 until 2024. He is the founder and principal researcher of the non-profit Macro Strategy Research Initiative.

1:05:59He is the author of 200 publications, including New York Times bestseller, Superintelligence, which helps spark a global conversation about the future of AI. His most recent book, Deep Utopia, Life and Meaning in a Solved World, was published in 2024. Nick, we really appreciate your time. Thank you so much. No, thank you. It was fun. This episode was produced by Claire Miller and Alison Weiss and engineered by Benjamin Spencer. Our video editor is Jorge Cotty. Our research team is Dan Chalon, Kristen O'Donoghue, and Mia Silverio. Jake McPherson is our social producer. Drew Burrows is our technical director.

1:06:37And Catherine Dillon is our executive producer. Thank you for listening to Prof G Markets from Prof G Media. If you liked what you heard, give us a follow and join us for a fresh take on markets on Monday.

1:06:53Lifetimes

1:06:57You help me In kind Reunion As the world turns And the dark flies In love

1:07:50Thank you.

1:07:57The right window treatments change everything. Your sleep, your privacy, the way every room looks and feels. At Blinds.com, we've spent 30 years making it surprisingly simple to get exactly what your home needs. We've covered over 25 million windows and have 50 ,000 five-star reviews to prove we deliver. Whether you DIY it or want a pro to handle everything from measure to install, we have you covered. Real design professionals. Free samples. Zero pressure. Right now, get up to 45 % off with minimum purchase. Plus, get a free professional measure at Blinds.com. Rules and restrictions apply. For a limited time, you can get a Big Mac meal for just$8.

1:08:32That's a burger, fries, and a drink. They don't call it an extra-value meal for nothing. Get a Big Mac meal only at McDonald's. Price and participation may vary. Promotion pricing may be lower than meal pricing.