188. Will AI Give China or the US Total Power? (William MacAskill)

22:16There's GiveDirectly, which is extraordinarily evidence-based, and just basically guarantee that you're going to massively improve the lives of many people, versus things that just are more speculative. And so one of the things that I think we've learned, or certainly I've learned or changed my views on over the years, is that a certain mode of reasoning that seems very speculative has actually now got quite a remarkably powerful track record where those of us in effective altruism were warning about funding, working on issues of pandemics and AI for many, many years before COVID, before the chat GPT moment.

22:53And I mean, I wish we'd done, been able to do more on pandemics. There's been remarkably little global response after COVID. And I think that actually risk of the pandemics is going to go up massively as the ability to create novel viruses gets democratized over the coming years. And I think the challenges from AI are like actually underappreciated, even despite all of the hype around AI. I think people are underappreciating the risks from, you know, I see it as proceeding in kind of four stages, the risks from cyber weapons, bioweapons, concentration of power and then ultimately loss of control.

23:27It's obviously extremely important to prepare for new technological advances and to try to prepare for them early so that we don't have massive pandemics from new pathogens designed in someone's garage, or we don't have the most intense concentration of power the world has ever seen as a result of heads of state commanding wholly automated armies. But how do you compare? How do you compare working on that versus something as evidence-based as give directly or distributing bed nets. It's an extremely hard problem. I don't want to say, oh, it's obvious and the answer is like AI. But what I do want to say is everyone should be reasoning with this.

24:08They should be reckoning with it. And trying to have as informed a view as possible, at least, on which of these areas is the right one to focus on. Earlier in the interview, when you talked about what philosophy is and what it's about and the questions it's grappling with. As you were speaking, I was thinking, well, politics should be doing that as well. And I just wondered whether you felt, even what you were just saying there, is politics really grappling with that stuff? Not really, kind of at the margins. Both now and through history, have you ever seen a political leader or any level of politician and thought, that person is coming at this from a philosophical standpoint?

24:45Or do you think that our politics is so driven by short-term interest that it's almost impossible to think beyond. I mean, the case that leaps most out to me is the writing of the US Constitution. So there you've got these amazingly brilliant thinkers in a room for three months, hashing out the wording of this document. Jordan Adams himself said, the institutions we are now building may last for thousands of viewers. And if they go wrong, it will be hard to correct them back onto the right path. So it's of the utmost importance that we get this right. And they were, in fact, incredibly farsighted in a number of ways.

25:25That is actually a rare case of there's some new order coming into existence, namely the United States itself, and effectively politicians and thinkers able to have a longer term view, try and take the challenges. It's quite depressing that you have to go that far back. And And it's also quite depressing at a time when you feel that that is a constitution that is under profound threat at the moment. I think it is extremely worrying. I mean, the example is interesting because I think you get these hinge moments. So end of the Second World War, another one with the formation of the UN, the Marshall Plan and so on.

25:59Could you argue that from a philosophical standpoint? Oh, absolutely. I mean, again, it feels very strange when comparing it to the world today, which feels so real politique. and so driven by kind of national interests. Just how idealistic the United States could, after the end of the Second World War, it was the only nuclear power, and it could have said, oh, great, well, now we're going to just take over. And obviously it did to some extent, it exported its values. But the general tenor was in terms of, oh, we're going to try and in general enrich the world, in general kind of steer the world in the positive direction.

26:37And yeah, it's just a sad fact about the last couple of decades that the political climate has moved away from there and now even moving away from the little space.

26:46Rory Stewart:Can we develop it a little bit more back to a sort of cranky obsession of me and Alistair? So we're quite moved by Jean Monnet and the creation of the European Union and the sense that there was somebody with a quite long-term vision who was also quite practical. You know, this is where we're going to get five years, 10 years, 15 years, who pulled this thing off. And of course, our dream might be a future in which Britain becomes much closer to Europe again. Who knows? Norway, Ukraine, Canada. And we begin to develop a values-based fourth corner to the global system to do some of what the US might have done after the Second World War.

27:19Rory Stewart:How would you think about what would be involved in trying to do that over a 10, 20, 30-year period? And is the time very unpropitious for this? Well, I think it could be. So the thing that's most obvious in my mind as you say this is the relationship between this and AI. So the views I have on what will happen with AI are, you know, they are unintuitive. They took me a long time to get to. My view is that probably within the next even five years, we'll have AI that can itself automate AI R &D. We'll have this leap forward in AI capabilities. After that, we'll have something like a century's worth of technological progress happening in a decade or less.

27:59In fact, I think it would be a much better world if Europe, the UK, other kind of democratic countries other than the US had major and significant bargaining power in that world. However, at the moment, the US and China, and you can look at this just by where computing power is housed, where it's almost wholly the US and China at the moment, and the rising countries are United Arab Emirates, for example. And so in order to have, yeah, the influence is kind of positive, like, yeah, positive influence and values coming from the UK and Europe and so on. Yeah, these countries simply need to have bargaining chips over the development of AI, which will be the most important technology of essentially maybe ever.

28:50And that will involve having the hard power of computer chips and manufacturing capability. Now you found is something called forethought. We should say to our listeners and viewers that you're not just somebody sort of saying, because I studied philosophy, I also know about AI. You set up something called forethought research, which specifically looking at this. And you've stated publicly that you put the odds of democracy surviving AGI at less than 50%. So even without somebody like Putin threatening democracy or Trump threatening democracy, you think that the technology is what, even bigger threat than human beings to democracy?

29:29Yeah, I mean, yeah, so think about why we have democracy. So, I mean, there's two ways or several ways I think we could lose democracy. One is a non-democratic country could become very, very powerful, where I think it's, again, reasonably likely that a single country ends up with 99 % of the world economic and military power. 99 %? Yeah, or more, essentially, yeah, because... Well, no, the only more is 100. Or you can have more nines. Trump has interesting views on percentages.

29:59Rory Stewart:Explain a little bit how, why, just explain that path first. How would you end up with one country accumulating huge economic power? Yeah, so. And military. And military power. Yeah. And again, I'll say like, all of this stuff seemed crazy to me to begin with. It's been like years of working on this and thinking about this and debating arguments that have got me there. The main point is that you get essentially kind of increasing returns to scale through the development of the transition from where we are today to a society with extraordinarily powerful AI, where at the point of time that you have AI that can itself design better AI systems.

30:41So we already have AI that's extraordinarily good at coding, at least on verifiable tasks, it is as good as the best coders in the world. It's not as good on fuzzier tasks. But that's getting solved too. The point of time when you have AI that can just do the job of a machine learning engineer in an AI company is plausibly not that far away. Again, my best guess is it'll happen in the next, more likely than not in the next five years. So then at that point, you now have these AI systems that are designing better AI systems. Okay. And then those better AI systems can themselves design better AI systems.

31:14So even on the same number of chips, it's like you start off with, let's say, tens of thousands of human machine learning engineers. You quickly move to hundreds of thousands of AI machine learning engineers that then design even better machine learning engineers. So then you have a population of millions of AI scientists, essentially, able to design the kind of next generation. Better and faster than people could. Better and faster than people could. People who are coming out of universities are thinking, that's a job I might have done, that job's now gone. Exactly. So the social and economic implications are huge.

31:52Well, absolutely huge. And we can definitely get into that. It's fascinating and quite discombobulating. But the key thing now is that over what our modeling suggests would be quite a short period of time, some number of years, you would go from the leading country, let's say the US or China, having powerful AI, to then suddenly having what is equivalent of hundreds of millions of workers. Imagine the smartest person in the world. Now imagine them working 24-7 with perfect focus to whatever task is being set to them. 365 days a year, hundreds of millions of them. Every time you learn, every time one of them learns, all of them can learn the same lesson.

32:33That is extraordinarily difference in terms of economic and political power. Economic power, because now just think about anything you can do, certainly to begin with from the armchair or by asking humans to do things in the physical world. Well, that's an awful lot of science. That's all strategy, cyber capabilities, design of bioweapons, potentially surveillance and monitoring, including like nuclear submarines and so on. It could save us on the energy front. It would save us. I mean, well, there would be two factors. So you would have new sources of energy, cheaper energy, but there'd also be enormous demand for energy because now you can take an H100 chip and you can run the equivalent of this smartest human being running at 24-7 on it.

33:17So I actually think the price of energy would go up in this world. I was trying to add a little bit of positivity to this otherwise relentless depressing picture.

33:27Rory Stewart:You're doing brilliant, but I'm just going to pull you back for our listeners back to the basics. So let's just take an example. So the US goes through this path and in five, 10 years time is what's happening. Basically, the chat GBT 15.0, whatever it is, is suddenly no longer something which is given to European companies to run their stuff on. But instead, they build their own stuff in the US. And suddenly, almost every company in the world is in the US because they're just much smarter, much more productive, all the manufacturing, the robotics and everything. in which case there's a huge sucking sound as the economies of the rest of the world collapses and all the economic value is sucked to Silicon Valley.

34:05Yes. And this can be a matter of choice. So already at the moment, the US is restricting the sale of chips to China. If you now have AI systems that can create a bioweapon for you, if you ask, well, it might well be the case the leading country wants to restrict it to only within their borders, perhaps only to national security use. We don't know. And so it may well be that at this point, the US could just say, okay, we're just going to go it alone. Or alternatively, they might actually just keep the technology restricted in some ways, so with military capabilities, and that would be sufficient for total hegemony of the world.

34:43Or alternatively, they might just grab currently unclaimed resources, which would be sufficient to get total total dominance. So Elon Musk has already been talking about data centers in space. In the near term, I think this doesn't make economic sense. But in the long term, it is going to be where almost all computing happens, because that's where all the energy is. There are other kind of structural advantages.

35:08Rory Stewart:Because the sun has a billion times the amount of energy that nothing else does. Exactly. So on Earth, we could scale energy consumption a thousandfold. up to then beyond that, you can go, yeah, two billion fold more. So it's a really big gain. And at the moment, this is all covered by the Outer Space Treaty. So it can't be owned by any nation. I expect the Outer Space Treaty to get thrown in the bin, essentially, when it becomes a real economic value. So what the United States could say is like, look, we're not even going to take anyone's territory, but we're going, we're the first ones to space.

35:41SpaceX essentially have a near monopoly on private launches already, we're going to take these resources, thanks. And then at that point, all other countries are just locked out of most economic power. Okay, Rory, we'll quit breaking them back for more.

35:57Rory Stewart:Hey, this is Michael and Hannah from Goalhangers, The Rest is Science. This episode is brought to you by Cancer Research UK. We often think of beating cancer as treatment, but imagine stopping it before it begins. After years of work, Cancer Research UK scientists are launching a clinical trial of LungVax, the first vaccine designed to prevent lung cancer. It builds on TracerX, the world's largest cancer evolution study, which tracked lung cancer cells over many years to uncover the disease's earliest warning signs. LungVax is designed to train the immune system to spot these signs early on, destroying 40 cells before cancer develops.

36:37Rory Stewart:So it's not treatment, but preventative, with the potential to stop lung cancer before it starts. The first stage of the trial starts this year, focusing on people at higher risk. It shows what long-term research makes possible. For more information about Cancer Research UK, their research breakthroughs and how you can support them, visit cancerresearchuk.org forward slash the rest is science. Did you know about one in three people with plaque psoriasis may also develop psoriatic arthritis, which causes joint pain, stiffness, and swelling? Does this sound like you? Listen to what it sounds like to be a million miles away.

37:26Rory Stewart:Trimphia, Gucelcomab, taken by injection, is a prescription medicine for adults with moderate to severe plaque psoriasis who may benefit from taking injections or pills or phototherapy and for adults with active psoriatic arthritis. Serious allergic reactions and increased risk of infections and liver problems may occur. Before treatment, your doctor should check you for infections and tuberculosis. Tell your doctor if you have an infection, flu-like symptoms, or if you need a vaccine. Imagine being a million miles away. Explore what's possible. Ask your doctor about TrimFaya. Tap this ad to learn more about TrimFaya, including important safety information.

38:05Rory Stewart:Some follow the noise. Bloomberg follows the money. Whether it's the funds fueling AI or crypto's trillion-dollar swings, there's a money side to every story. Get the money side of the story. Subscribe now at Bloomberg.com.

38:26just going back to the democratic survival point so you've talked about the power that can go to one country be that and we're we're living with the short list of two it might be america it might be china you've also talked about the loss of any sense of economic value from human beings that we our brains and what we do with our hands just become kind of redundant irrelevant so that's where they kind of, I think the social problems come. But then you've gone to this further point where these clever and clever and clever and clever AI models can then create armies in which there is no structure of obedience.

39:06There's no structure of obeying orders. Well, it's the other, it's just, they will always obey orders. That's the risk at least. Or they become autonomous. Yes. So this is going to be a huge challenge. It's one of the things I'm working on at the moment is AI has already been heavily used in the military. It was involved in the taking of Maduro. It was involved in Iran. Again, we're not looking that far out. And again, this all seems crazy, but it's crazy because the technology is moving so quickly. It won't be that crazy. You could just have fully autonomous weapons, where we have a choice there.

39:37Rory Stewart:And explain what autonomous means. Yeah. So we know drones, you might see them kind of flying around. Autonomous would mean that rather than having a human controller guiding its every movement, you could give it some very open-ended goal or perhaps a whole fleet of drones. You give a very open-ended goal. Destroy London? Destroy London, yeah. Or just we're at war with China. We want to win. And then it will just figure out the strategy and implement that. Think of a wholly autonomous self-driving car. One where you just say, look, I want to get to Oxford. And then it'll just, it'll figure out the best path and drive you there.

40:09We're not that far away from that technology being feasible. But now think how power changes in that world, where at the moment, if a general says, I think I'd like power over the country to myself, and orders his soldiers to storm the Capitol building, the soldiers will say no. it is an open question how we choose to govern and design autonomous weapons, where one way could be that these are just tools. A hammer does not resist you if you try and hit someone on the head with it. Similarly, AI is a tool it should do as instructed. But then you could have a world where you have a military consisting of millions of such autonomous weapons controlled by a single person, you know, in the US, the commander in chief.

40:55And that is a very, scary situation and it's one that we should be actively working to try and avoid. You mentioned the US commander-in-chief. Let's not say his name any more than we have to. But which of the two current commanders-in-chief of the two biggest militaries, Xi Jinping and Trump, who is winning that race? I mean, for AI at the moment, it's utterly clearly the United States. All of the frontier AI companies are based in the United States, where the leaders at the moment are Anthropik, OpenAI, um google in particular deep mind they're really the key three front runners most computing power is based in the united states as well i think 60 of global computing powers in the united states china lags behind it kind of varies but something like nine months behind and if they were having to do this kind of going alone they would be much further behind again now a big part of all of this is because of the export controls where the United States a number of years ago under Biden said it would restrict the sale of the most powerful chips to China.

41:59So they're having to run on less powerful hardware as well as having a less, well, they have many brilliant scientists, but it is a less well-developed kind of talent ecosystem than in the US. So they're doing kind of a fast following approach. However, it's very easy to catch up to the frontier of technology there's various techniques you can use so it's only a nine-month gap let's just say that either of those two or any other current world leader who is um really trying to wrestle with this let's say they brought you in and sat you down and let actually let's forget china for a minute let's say it's the democracy sat you down and said all right well you we're buying your argument here but what should we as political leaders with an election coming up because we have to deal with those things?

42:48What should we as political leaders be doing right now to make the right calls, not just for our generation, but for the future? What would you tell them? Yeah. And is this political leaders in the US or UK? US, UK, France, Germany, Australia, Japan, Korea, the democracies. Yeah. I think there's a couple of things. So on the software side, I think here's one kind of regulatory approach. And ideally, this would be agreed, at least among all the kind of democratic allied countries is to say every company producing frontier models, like the most powerful AI systems, needs to have what's called a model spec.

43:22That's just an English language document that says, look, this is how we want our AI to behave such that, for example, if you type in like, please design me a bioweapon, it will say, no, sorry, I can't do that. That's one thing you can write in. The model spec has to agree to, had to abide by certain foundational principles. So why wouldn't the government want to be designing new bioweapons? Well, perhaps. And why wouldn't China just think, right, there's you democracies telling each other what you can't do, but we're just going to hoover all that you say you can't do, and then we'll work out how to do it.

43:50Well, in the case of bioweapons in particular, this is just a lose-lose for everyone. I mean, because bioweapons, it's very hard to release something that doesn't come back to you. So the biggest risk is from rogue states like North Korea, where they just, you know, So they've got so little power that making crazy, like reckless threats can actually be quite powerful. But look, there's going to have to be some huge bargaining over what are in those principles and what have exceptions for national security. But just briefly, I think, yeah, on the software side, having this written document and then saying any frontier model has to be able to demonstrate that AI will behave in line with the specification, including in very unusual scenarios.

44:31And that means you have to show that it hasn't been sabotaged, whether internally or via foreign interference, and the AI hasn't formed goals of itself. So you know that it's going to generalize well.

44:44Rory Stewart:We can keep going. I think there's 58 different questions that follow on from Will's position, but Alistair, I had some for you. So we've got Will laying out these very, very, very radical, very disturbing, and unfortunately, very plausible visions of what might happen in the next five, 10 years. And yet we've got a politics across the world that barely speaks about this. You know, we've just been with the president of Serbia. We've been covering elections for the last four years all over the world. Ain't nobody really talking about this stuff. In fact, the campaigns feel like they're still stuck in the 1980s.

45:21Rory Stewart:You know, it's cost a living. What do you think? I mean, if you go back to your time in number 10 and someone like Will comes in and he says something like this, really radical, that almost sounds to you like, in five, 10 years, the world as we know it's about to end, is basically what happens is that people just bury their head in the sands and say, well, he's obviously mad or an extremist. I don't want to listen. I mean, how do you deal with this kind of thing? Well, I actually think that if we were back in the day, and I think I'd say the same with Bill Clinton as well. I think if this had been happening around our time, and you'd gone in to see Tony Blair at this time, he would have been utterly fascinated.

45:56And in fact, he is utterly fascinated by this. And I think Clinton would have been the same. And thought, partly would have thought, how do we UK, or in Clinton's case, the US get ahead of this, but I think would also have seen the bigger picture. And I think it is partly the narrowing of our politics and our political debate that has made it so that's what I was going about the long termism. It's so it seems the politicians are scared of having these long term debates. I mean, to be fair to Rishi Sunak, he's probably the only one I can think of who gave it a go in starting that kind of annual summit.

46:28I think he lost a lot of credibility by the high point being sitting down with Elon Musk, who just came out with his usual Muskian kind of nonsense. Of course, Elon Musk, a big admirer of yours, I saw. Well, he was in the past. Is he not anymore? Not anymore. Not anymore. Okay. So I would like, and I'm trying to think if there's any other current world leaders. You see, I think a politician who took this on, and kind of did scare the world a bit, based on knowledge and science and truth, as you get to your philosophical points, I think people would start to listen. Because I think we see the benefits of AI.

47:03We feel the benefits of AI. But I think there's a part of the public and the politicians that don't want to confront the scale, a bit like climate change. They don't want to confront the fact that we might be literally burning ourselves to death because it's easier to say, well, we're dealing with it. We're putting 150 million pounds into this and we're doing more solar panels or what have you. I think the last generation of leaders probably would have said, we've got to sort this. But the thing that's weird, so I mean, yeah, I've written about long-termism, the importance of taking future generations seriously, looking very far ahead.

47:35The thing that's just strange about the world now is that all these things I'm talking about, I'm thinking the next five years, as in very significant chance that we start on this takeoff of AIs building better AIs and this huge leap forward in technological ability in the next year or two even is probably not, but maybe even then. And is it, let's just say, because, you know, I don't know if you've listened to Roy's miniseries on AI, but I got the sense that he's become much more sceptical and much more alarmed about where it's all heading. Not sceptical about the power of it, but alarmed about the power of it.

48:11And I've always been in a quite scared sort of place. Is it too late for politicians to have much impact on this, do you think? Absolutely not too late because the world still hasn't kind of woken up. There is already a differential in terms of higher power from the US and other countries. But Other countries do have some. The Netherlands has ASML, which produces 100 % the entire world production of what they call extreme ultraviolet lithography machines. And I would thoroughly recommend everyone to go and look into this. There's a great Veritasium video on YouTube. It's very fun. It's the most sophisticated technology in the world, essentially.

48:48And why have the Dutch got it? It's essentially one.

48:51Rory Stewart:Yeah. I mean, the Dutch control 100 % of one bit of it. The Taiwanese control nearly 100 % of the other bit of it. And an American company controls nearly 100 % of another bit of it. So the advanced computer chips, which the whole AI model is based on, has a huge link to NVIDIA, to this Taiwanese company, and to this Dutch company. And yet we don't seem to be able to leverage this. And why did the Dutch get it? Because there's a small European country. Is that one company that did that? It's one company, ASML, it's called. And so, yeah, these... Was the government involved? I don't think... Not to my knowledge, no.

49:22So it's just the fact that somebody randomly was born to be Dutch? than started a company. It's because there was just lots of companies tried. We'll talk about the technology. So imagine these machines. They're so big they have to be moved in several Boeing 747 jumbo jets. And what they are doing is etching designs of chips that are, if you look at your fingernail, see how much it grew in the course of a second? That's how fine these huge machines are etching chips. It involves these incredible processes like firing little droplets of tin, 50 ,000 of them per second, that get zapped three times in order to generate the ultraviolet light that is creating these etchings.

50:03So it's just extraordinarily difficult technologically. Many companies tried. They all failed apart from this one company. Similarly with TSMC producing these computers. Which is the Taiwanese company. The Taiwanese company produces 90 % of the frontier chips. Again, it's just very hard. Many companies have failed. Is it possible, therefore, on that one base, You remember when Nokia was kind of dominating the telephone market and Finland suddenly became this sort of, you know, greater economic power. Is it possible that that leads to the Netherlands becoming the most powerful economy in Europe?

50:35I mean, they have far more bargaining power than they know. And with the export controls, ASML and the Netherlands, they got really trounced by them and they got brawbeaten by the United States, to be honest. But they could. And it's not just them. Japan, South Korea, Germany all play essential parts in the semiconductor supply chain as well. I'm not hearing much UK-ery in this list. I mean, what the UK has is home to talent. So London is the second most important hub for AI talent outside of San Francisco. By headcount, it's about a third the size of the, but that's a lot of kind of number of machine learning researchers because Google DeepMind is based in London.

51:20There's also offices from Anthropik and OpenAI. But in terms of manufacturing, in terms of, yeah, there's a role in kind of hard power, the US lags enormously behind. It could change that if it wanted to. But more recent news, it was going to develop more data centers in the north of England as that's fallen through as a result of permitting inability to grow energy enough and some copyright reasons too.

51:44Rory Stewart:I didn't want you to go without talking a little bit about the personalities of some of the people that are dominating AI and the tech industry. Because one of the things that's going to be true is if our world is increasingly in the hands of three companies, four or five, I mean, maybe we can add Elon Musk and Mark Zuckerberg into the picture, who are worth hundreds of billions of dollars. And you're putting a huge emphasis, not on a kind of broad democratic base, but these kind of solitary oligarchic figures who may have, to be frank, quite eccentric lives, eccentric views of the world, damaged childhoods, obsessions with science fiction, weird insecurities.

52:29Rory Stewart:And you've met some of them. Sam Backman-Fried, in a sense, was one of these people. I mean, what is it one should be, without trashing your friends who've been generous to your various projects, what should we be cautious of when it comes to a world run to some extent by these people? No, I'm happy to speak very frankly about all these people. I will say two things. The first I'll say is that relative to the batting average for companies dealing with ethical issues, where the bar is in hell, think of Exxon, think of Philip Morris, I am very happy that the three leading companies are led by Demis Hassabis, although he's part of Google, so it's how much influence he ultimately has over the steering is more muted.

53:15Dari Amadei and Sam Altman. These are all above the average. They have all done things like invest in safety. Slightly developing megalomaniac tendencies. Oh, no, but they are also megalomaniac. Both of these things can be true. So in terms of the things you should be worried about, entrepreneurs and very successful entrepreneurs are already extremely weird by populations because it's not an enjoyable job to run a billion-dollar company. So you need to be megalomaniacal to some extent. You need to be power-hungry, power-seeking, really want success in some form. Also, a very distinctive thing among such people is incredible competitiveness.

53:54It's striking how often such leaders are like intense game players as well. So intense competitiveness isn't necessarily what we want in something where cooperation would be much better. But then the final thing is just imagine the situation you're now in where over and over in your life, you have made some crazy bet, some belief that you had that others didn't have. And they were all wrong. and you got proved right and now you're the billionaire okay now you've got some new new crazy belief and other people are telling you you're you're all wrong um you know you're going to really kind of um back yourself again and so i think everyone should be very worried about how much power can go you know we've talked about how much power can go into government hands that's the thing i'm most worried about but i think we should be very worried about how much power could go into company hands as well i am expecting companies to be worth tens of trillions of dollars, again, within just a few years.

54:51And with relatively small workforces? With workforces that ultimately go down to one, because it's a single person commanding, the CEO commanding a wholly automated workforce. Again, we'll see this increasingly, even just over the next few years, we'll have single person companies where all the staff are artificial intelligences. Again, that could lead us, you know, think of the East India company had its own army, you know it was having power on the level of states i think it's reasonably likely that we'll get the same with companies too i'm trying to think of something to kind of you know cheer us up a bit um i mean yeah the last thing i'll say is i'm rory has just been grinning this whole time since we've in a bemused way since we've gotten on to artificial intelligence again i want to say like i'm not like a science fiction person i'm not like an ai hype person i don't work for a company i'm a work for non-profit i would rather i would like nothing more than to be wrong about all of this stuff i have like come to these conclusions kicking and screaming by the changes that we're seeing and um the forecasts that i think follow from some fairly basic i guess my final question question is this i've mentioned a couple of times you know what thought has been given to the the social and economic consequences you just described there a vision of a ceo who doesn't need human beings, who can have massive economic power, even military power as an individual.

56:12What are the human beings, what are we all going to do and how are we all going to live? Yeah. So, I mean, here's, I guess, here's something that's a little bit of optimism in this or something. So the world will be getting enormously richer over this time period. And I actually differ. Won't be shared well. Well, so there'll be this one-off redistribution before extraordinarily intense concentration because robots are lagging behind. I'm laughing because I know what I'm about to say. So, okay, at the moment, robots are like lagging quite far behind kind of AI for cognitive skills. And this is because when it comes to kind of evolution and how much time evolution has to optimize our brains, well, science is just very new.

56:54It's not at all what we've evolved for. But doing fine motor coordination, me picking up this glass and putting it down, that is extraordinarily complex what I just did. It doesn't feel like it because it's so second nature to us. It's just, it's all built into our brains from evolution. And moreover, the robots don't exist. So we can't actually build equivalent of a human hand in terms of just how fine our kind of touch perception is. And so over the period that we're moving from AI that we have now to AI that far exceeds kind of human cognitive capabilities, there will be complementarity between that AI and human workers.

57:27If only, I think this is for many reasons, but if only because we are able to act in the world and we have these fine motor skills and the robots haven't yet been built. And so this means there'll be this period, which is just a one-off gain where wages for humans will go much, much higher because we are able to go and build the robots that will then replace ourselves. That will then replace us, I see. And so we'll have, you know, so we can use, you can use your phone. it has a camera it has an audio it can be like here you're attached to your head the ai can be telling you what to do you're just you've got this like you've got this like coach this met like manager doing exactly the physical movements so that you can build these robots is it not at least a party if i was a kind of one of these big ai gurus is about to rule the world i'm sitting listening to this i might think god this guy sounds like he knows what he's on about and he sounds like it could be a bit of a problem for us as well let's get him in and offer him a ginormous salary about a thousandth of what i'm on as the ceo yeah and get him in and use his brain on that is there nothing that would take you above the 32 000 pound a year yeah i mean it's actually honest completely honestly this is something that i you know that's the way i can get like a lot of intrusive uh thoughts about it's like i know all these people i was close to and like good friends with.

58:48And, you know, we're all engaged in this kind of ethical project together. And now some of them are worth hundreds of millions of dollars, even kind of billions and so on. And from an ethical perspective, I think it's good that I have made all this noise about saying, like, well, this is my income and I'm not going to live beyond that. Because it means that I kind of can't get tempted by, it would be extremely embarrassing for me if I said, well, actually, I'm going to give up on all that and take the big salary. Obviously, I could do that in order to donate. But I think the bottleneck that we have is much more about ideas, much more about ability to speak freely than it is about money.

59:30And so, yes, it's this very striking thing, seeing so many people I know become extraordinarily wealthy. But it's an unexpected benefit of the pledge that I took that uh yeah and that sort of temptation is um off the table i suspect most i dare not speak for the many millions of them but i i think most of the rest is politics family will probably admire you more than them well we'll see thank you will so much thank you so much for having

59:59Rory Stewart:this was a great conversation thank you thank you very occasionally i wake up in the middle of the and think, why am I the one who's putting all these bloody guests, these presidents and prime ministers? And Rory just occasionally wheels in with somebody who spends his time with baboons. And then he says to me, I want to talk to a philosopher, right? And as the team who were with us in Serbia, I was saying, you know, why are we doing this guy, this philosopher guy? I thought that was absolutely brilliant. I thought it was absolutely riveting. Why do I think it was so riveting? Because I think I am of that mindset that doesn't – I kind of follow it, and I listen to what you said on the series, and I read books, and I've read some of the big biographers.

1:00:49And if I see a big profile of Sam Altman, I'll read it and what have you. But I don't really dive into this in a way that I do with some of the other things that we talk about. And I think that is partly fear. And I think what he's basically saying is, yeah, you're right to be scared. And we've got to confront that. We've got to understand it. And I think the point you raised about where politics is at the moment, when I watched the Tory party local election broadcast the other night, and you said, our politics is stuck in the 80s. I mean, it was just, it was like, you know, we'll give you a stronger economy.

1:01:19We'll, we'll, your schools, you're, we'll deal with this, we'll deal with that. It felt really back in a different age. And he's basically saying, this is the age and you're saying, we've got to talk about it more. So I found it, yeah, I think our people will love that.

1:01:33Rory Stewart:I also thought your last question was a really good one, because I do think that apart from being a highly intelligent, articulate guy who, as you said when we're chatting about, is not umming and ahhing a lot and is able to explain very complicated things very simply, which I think is a really good definition of intelligence. He's also still independent. And as you pointed out, that's very, very rare. God knows what I would feel if one of these companies approached me. I know one person who took a job with one of them at$200 million a year. It's wonderful, as he says, that he can be so self-deprecating about it and say, he gets intrusive thoughts, he's very tempted, but it just would be really embarrassing because he set himself up as the guy that doesn't do that.

1:02:18Rory Stewart:Just how powerful that is. And I just wish there was more of that. I mean, I think in a funny way, without being too romantic, I think in the past, there were many, many more people who felt a strong sense of embarrassment or shame, which stopped them doing things, which are becoming increasingly normal in politics. The Trump administration is just the most extreme example. But thank goodness, because most of the people, you're quite right, they get bought. Most of the great AI minds end up with shares in these companies, consultancy groups of these companies. And then what can you really do to call them out?

1:02:53It's interesting. The other thing I found really interesting was that he clearly thinks that politicians and governments still can get on top of this. I'm not convinced by that because I think a lot of these people are now so powerful economically and technologically every other way that they really are the sovereign individuals. They really are going to end up with more power. I mean, I maybe would have asked him, these people he's talking about, Altman, Musk, and the rest of them, are they more or less powerful, say, than a president or prime minister of a middle-sized Latin American, European, Asian country?

1:03:27Rory Stewart:I think much more powerful. I think the only thing that he might say is that we should never underestimate the power of something like the US federal government. I mean, if they wanted, they could lock these people up, put them in jail. Still, US government has the army, they don't. Now, as a practical point, you're right. These people are so wealthy, so influential, and the Trump government's so corrupt. This is the perfect storm. It's the worst possible moment to look for sensible long-term regulation when you've got Trump in as president. So we've got this explosion of AI, the most dangerous moment in technology in the world, and a president that doesn't care about this stuff at all.

1:04:01Rory Stewart:And add to that this competition point, which is every one of these leaders says that they're worried about AI safety, but they're not going to do anything about it so long as their competitor has got their foot on the acceleration. Then they make the same argument with China. There's no point us regulating, you made this point, we're not regulating ourselves in the US, what happens with China? So it's a kind of prison dilemmas problem. It's a race to the bottom. It's a sense of everybody saying, yeah, of course, if it was just up to me, I'd behave much better. But, you know, if everyone else is doing it, what's the point of my...

1:04:31And I think it does go back to this sort of choice of the future, whether, you know, he's saying it's 50 % where the democracy will, less than 50 % where the democracy will survive. Dave. That's why I think nine, admittedly, it's going very fast. But when he said China's nine months behind, that doesn't feel very long to me. If you're a dictatorship, you can catch up a lot quicker. And especially as they are so bright, they do have, I don't know how many, he was talking about, you know, San Francisco, London, but how many great brains have they got currently working on this in China?

1:05:03Rory Stewart:Oh, unbelievable. And Tsinghua University is graduating more top level computer scientists than any American college at the moment. And the sheer weight of talent in China, you wouldn't want to bet against them. A couple of final things for me. I feel that even actually, it's one of the reasons I'm feeling quite cheerful about what we're doing at the moment, because I do realize how lucky we are to be able to try to talk relatively openly about things without having to worry too much about what a party donor says or our employer says, right? I do worry that politicians are so constrained in terms of just being honest and open about what they're thinking about.

1:05:43Rory Stewart:Second thing that's lovely about Will is that lovely moment where he answered you early on when he talked about mental health, about how initially this was unbelievably painful for him. He was stuck in a sort of spectrum-like anxieties about if I bought a cheaper breakfast cereal, I could give more to the extreme poor in Africa. And now has become, as far as I can see, actually sort of surprisingly good-humoured, witty, relaxed, and cheerful about the sort of complexities and absurdities of the world. That was quite cheering up. I've not quite got there yet. Don't worry. Many years to go. Many years to go.

1:06:18Anyway, thank you for that. We'll see whether our – I think our regulars will really, really like that. I admitted that Sapolsky was a surprise hit. but okay just occasionally roy i'd like you to pull in a president or a prime minister okay

1:06:31Rory Stewart:just occasionally well yeah yeah and and and sometimes when i feel i'm pulling people in you like to jump in and take the credit for them and say why aren't you doing no excuse me if you're talking about kemi badenock it took me one text message to get a yes i was already there do you want to share the text messages from days before i'm just unlike unlike you as soon as i'm going somewhere i'm not immediately showing off to everybody else about how well i've done Or insulting everyone else for doing nothing when they're working hard. That's not a way to motivate a team. I think I'm quite good at motivating teams.

1:07:04Rory Stewart:Not this team at this rate. Well, we're probably wandering into philosophy now. Yeah, absolutely. I'm not sure everyone needs to hear this spat about us and Kemi Bainlock. All right, anyway, Alistair, thank you very much. I think our listeners will be very excited if Kemi Bainlock's going on TBH. Okay, thank you very much, Alistair. Bye-bye. Thank you.