What If the AI Revolution Isn’t Real?

0:00Thanks for listening to The Rest is Politics. To support the podcast, listen without the adverts and get early access to episodes and live show tickets, go to therestispolitics.com. That's therestispolitics.com. So good, so good, so good. Spring styles are at Nordstrom Rack stores now, and they're up to 60 % off. Stock up and save on Rag & Bone, Madewell, Vince, All Saints, and more of your favorites. How did I not know Rack has Adidas? Why do we rock? For the hottest deals. Just so many good brands. Join the Nordy Club to unlock exclusive discounts, shop new arrivals first, and more. Plus, buy online and pick up at your favorite rack store for free.

0:38Great brands, great prices. That's why you rack. Spring is here, and the shopping list is long. Time to make a Lowe's run. Buy three bags, get three free of stay green one cubic foot garden soil. Plus, right now, members can earn four times the points on an eligible purchase. Start spring off strong with these deals and more. Our best lineup is here at Lowe's. Ballot to 325, while supplies last. Soil offer excludes Alaska and Hawaii. Loyalty program subject to terms and conditions. See lowes.com slash terms for details. Subject to change. Point boosters subject to exclusions and more terms apply.

1:11One-time only offer. Kayak gets my flight, hotel, and rental car right, so I can tune out travel advice that's just plain wrong. Bro, Skycoin, way better than points. Never fly during a Scorpio full moon. Just tell the manager you'll sue.

1:28Rory Stewart:Oh, instant room upgrade. Stop taking bad travel advice. Start comparing hundreds of sites with Kayak and get your trip right. Kayak, got that right. Hi, Rory here. So Matt Clifford and I are back with another episode in The Rest is AI. And this week we're back to a simple but pretty uncomfortable question, which is, might AI just be a normal technology? something that maybe really doesn't deserve great fear and frenzy. And our guest today is Arvind Narayan. He's the director of Princeton's Center for Information Technology Policy. And he's a really interesting voice. He's a challenge to Joshua Bengio, who you heard last week, because he thinks that it's not very likely or that actually we're over-focusing on the end of humanity existential threat.

2:24Rory Stewart:But at the same time, he's a challenge to a lot of the people that are hyping the technology because he's saying, in many ways, this stuff is much less reliable than we think. And even once it gets more reliable, it's going to take decades for some of these things to be adopted, not a few months. And therefore, the change that AI is going to bring is going to be much more gradual. So it's probably a very, very sane, thoughtful, challenging voice, one that maybe isn't heard as much because it doesn't necessarily suit either the end of the world is nigh people or the AI is going to change the universe tomorrow people, because he's suggesting that a lot of the issues are just around how humans do or do not adopt technology.

3:12Rory Stewart:Here's a taster. And to listen to the full episode, sign up at therestispolitics.com. There's a really interesting thing as an outsider that one observes, which is one will get Elon Musk saying there's a 20 % chance it's going to destroy humanity. And Sam Altman says it's going to end the world, but in the meantime, it's going to lead to some great companies with great machine learning. And then suddenly you'll get Gary Marcus will pop up and say, no, this is all completely overblown, there's only a 1 % chance it's going to eliminate everybody, right? But of course, if one thinks about this, I think Joshua's point was, it doesn't matter if it's a 1 % chance, a 0.5 % chance, a 0.1 % chance, what the hell are you guys thinking about?

3:53Rory Stewart:I mean, you're gambling, right? At some level, you're taking a risk that you wouldn't take with nuclear waste in your back garden. I mean, you wouldn't be reassured for me to say, don't worry, there's only a 0.1 % chance that this nuclear waste is going to wipe out your family. You'd be like, what the hell are you doing? Stop, right? One thing I strongly believe is that we should not be thinking about this in terms of probabilities. At the moment you're arguing about what the probability is, you're already down a very confusing path that can only lead to, I think, misleading guidance. So I've looked at the most sophisticated efforts that we have for estimating these probabilities.

4:34It was led by the Forecasting Research Institute. I actually work with these folks. I know them well. They're incredibly smart. And they did a really well thought out effort to get dozens of expert forecasters to discuss, try to change each other's minds and provide these probabilities. and they put out a 753 page report. And I've read that 753 page report. And, you know, you could have a room with these so-called super forecasters debating and you could have a room with a bunch of people who are high and debating what the future of AI is going to be and you can't tell the difference. And this is no slight to the super forecasters, some of whom I know, they're incredibly smart people.

5:20But the thing is, we have no empirical basis for predicting what these probabilities might be. The arguments that people are giving, it's things like, oh, you know, AI might decide to colonize space instead of Earth. So even if we had super intelligent AI, maybe the probability is not as high as we think. Or someone else thinks, oh, AI might decide that killing all humans will make the planet cooler and it helps computer chips work better. And so maybe it will decide to kill all humans. And so, you know, they're listing a bunch of reasons like this, assigning some numbers to each of them. and then multiplying all of them at the end.

5:55Right. So this is the best method that we have. So these probabilities are all bogus. That's that's my strongly held view. We should not think in terms of probabilities. I do think the risks are potentially real. I'm not advocating for ignoring the risks. But I think the right response cannot be, let's try to stop all this. There are two big problems with that. One, it's just not going to work. The only way it could work is if you have an authoritarian world government that can control every AI developer everywhere.

6:26Rory Stewart:Can I pause on that, Arvin, for a second? Because that's an empirical claim. I mean, at the moment, these large language models basically can only be run by some of the largest, wealthiest companies on Earth with enormous data centers with enormous compute power. So, it seems plausible at the moment that if President Trump, driven by Christian nationalists, and Xi Jinping wanted to simply shut down the large language models of two Chinese companies and a small handful of American companies, we would cease to have these LLMs operating pretty quickly. This is not true at all. It might be true that absolutely the most powerful frontier models can only run on powerful GPUs, but you have slightly smaller models that are maybe one step below that can run on consumer-grade hardware.

7:20And the cost of running these models is dropping by something like somewhere between a factor of 10 to a factor of 100 every year. The cost is dropping very rapidly, both because the hardware is getting faster per dollar and because algorithmic improvements are allowing us to squeeze more juice out of smaller models.

7:39Rory Stewart:Again, I suppose to push this argument to the next phase, the claim only needs to be that the existential risk is posed by the frontier models. And if one could shut down the full infrastructure that powers the frontier models, then one has less to worry about from the models that you're talking about. Now, again, I completely disagree with this. I think historically, this is very easily falsified. When OpenAI built GPT-2, which was two generations before ChatGPT, which was when the world started noticing, they thought that model was so dangerous that they were not going to release it for people to download and use.

8:16And that's something that my grad students can build today, you know, just for fun and learning in a day or two, right? And so historically, you know, when we look back, our ideas around what constitutes the threshold level of danger have kind of been comically off. And I don't think there is really any clear relationship between the power of a model in terms of how computationally heavy it is and what dangerous things it might potentially be able to do in terms of enabling cyber attack, in terms of various things that worry about, in terms of bio-risk. Actually, some of those models can be much, much smaller and faster than these large language models because those biological capabilities are not about language at all.

9:03Rory Stewart:To understand this first argument, this is a the cat is out of the bag argument. It's too late. The stuff's built. If these models are going to become deceptive killer robots, they're already out there and there's nothing we can do about it. It's not just that, that you can't even prevent these models getting even better in the future unless, again, you have this authoritarian world government that stops even small teams of developers from acquiring consumer-grade hardware and doing research to improve the efficiency of those models and their capability. I want to say I want to go on the record as being against authoritarian world government.

9:42I mean, I'm very interested to sort of use this as a way to think about what we should do. And, you know, I'm a little bit biased because I'm one of the people in the UK who, one of many people who helped set up the AI Security Institute, which does these like pre-deployment evaluations in the UK government on security adjacent risks on a voluntary basis. There's no regulation that requires the companies to do that, although they have agreed to do that. And my view has been, to your point, I guess in agreement with what you're saying, Arvind, that it's very, very hard to know. When people say, we should pause now, we should stop now, I get the impulse, but I'm like, well, how do we know?

10:19And I think governments having better knowledge of what capabilities actually are, what it can and can't do, feels like a very valuable thing. There's a lot of people that look at what we've done with that institute and said, It's amazing that government's built that capability, but it has no teeth. But what do you think we should do? I mean, it sounds like you actually referred earlier to the idea of having transparency requirements. Do you have a sense of what the right regulatory framework is, given both your skepticism of some of these scenarios, but also your concern clearly about some of them as well?

10:52For sure, yes. And I think the AI safety and security institutes have been a wonderful thing. I've strongly supported the work that they do. No question on that. And yes, I think transparency and knowledge are the most important thing for now. But I do think there is a lot more that can be done that should be done. One big area is developing AI for defense against these very risks that people are worried about. And historically, this is always how it has worked. We're acting as if AI is a new thing, but AI has been gradually improving out of the public eye, admittedly. for decades now. And the reason that things have worked out is because of the attacker defender balance and the efforts that we've taken to shift that balance in a healthy direction.

11:39So when it comes to cybersecurity, for instance, automated tools have been in many ways superhuman at hacking for decades now. And yet the world hasn't ended. In fact, things have gotten a lot better. And that's because companies who are developing software use these very same automated tools to find and fix bugs before they put software out there, before attackers have a chance to take a crack at them. And that's the critical thing. When these AI models are now becoming even better at automatically finding vulnerabilities, do we have the right incentives in place to ensure that software developers, as well as mom-and-pop businesses who might be deploying software, have the know-how, have the funding to use this to protect their cybersecurity?

12:23Those kinds of things can benefit from government incentives and can be enormously beneficial.

12:28Rory Stewart:This moves us on to a new thing. So the first disagreement you've clearly had with Yoshua and Jeffrey Hinton and others who were calling for a stop is your cat's out of the bag. It's too late anyway. The earlier models can do a lot of damage. Stopping the next frontier model is not going to make much difference. Argumentary. The second argument that certainly people like me would make who are anxious about this is, for goodness sake, what is it about the Trump administration or the financial incentives or the psychological personalities of these people running these models that remotely convinces you that they're incentivized in any way to really put proper safety regulations in place, use the AI to counter themselves?

13:12Rory Stewart:Every time I meet them, they are in an arms race. They're saying, government back the F off. Government doesn't understand what we're doing. They're never going to understand what they're doing. They don't know what they're regulating anyway. Get out of our way. Here we go. I've got to get there before. And then insert the name of somebody running one of the other models who they think is unbelievably dangerous and is going to destroy the world. And I've got to build my model before, insert Elon, Larry, Demis, Mark, right? And if not, I've got to build it for the Chinese. Because I don't know the Chinese can build it, right?

13:46Rory Stewart:None of this gives me any sense that we are in an environment in which these people are, either they or the US government are getting remotely behind safety. I am not saying that we're in a good place right now with regard to regulation, both with regard to safety regulation in the sense of these catastrophic risks, as well as the kinds of safety issues we've already seen on a massive scale, some of the psychological issues that we've talked about, the use of AI models for making non-consensual nudes. These are problems that we knew were happening, the nudes one at least, at this point for something like six years.

14:23And it took forever for policymakers to start acting on these things. And so the pace at which policymakers are responding is definitely inadequate. But I think at the same time, the arms race narrative is a little bit oversold. I think what we're seeing in a lot of cases is that when regulators and policymakers in one company are falling behind, the harms, for the most part, are felt locally. So, for instance, in the US, people are realizing that with respect to the safety harms around the psychological impacts of large language models, it's affecting Americans. And now policymakers, both at the federal and the state level, are actually taking this very, very seriously.

15:13And I think we're going to start to see a mindset shift and we're going to start to see policymakers stop buying this arms race narrative. And they're going to recognize that a lot of these harms are actually going to be felt within their own countries. and the way to regulate is not just at the level of the model developers but how they're deployed and not just on the companies such as open ai and google but also everybody else who is using these models within their own companies and try to get ahead of risks like people trying to hand off the running of their companies to to large language models for instance one observation i have on that from both my own time in government in the UK, but also spending a lot of time in Washington talking to people is that to paint with very broad brushstrokes, the executive tends to see this technology through a security lens.

16:07And there's lots of both institutional and other incentives that make that happen. And typically legislators tend to see it through a sort of, what you might call like an online safety type lens. And certainly when I was in number 10 here, we would have these high side conversations about AI and security, and then we'd go and see MPs who were worried about their kids or the kids of their constituents talking to chatbots. I think I may have mentioned this on a previous episode, but in 2025, I went to a very interesting conference of faith leaders talking about AI who totally rejected the security framing and really wanted to talk about what is this doing to kids and their minds.

16:46I suspect that over the course of 26 and 27, we might see this sort of like quite executive security dominated framing kind of actually diminish as some of these cases in particular that you've referred to and kind of come to the fore and actually see legislators sort of be a bit noisier voices in this talking about these sort of more sort of online safety type harms. That's exactly my prediction. Yes, as these models get more diffused into society, we are going to see an increase in the salience of that set of concerns. And I think the good news is that a lot of the interventions, not all of them, but a lot of the interventions for both sets of concerns are similar.

17:26And so, I think the momentum around one kind of safety issues can also be used to address the other kind of safety issue.

17:32Rory Stewart:Europe, if I was speaking on behalf of the commission or European leaders, they would feel they have been making a good faith attempt to try to work out how the hell you talk about these different types of risks and how you present regulation. And the response that they have got from Silicon Valley, they feel, is one of unmitigated contempt that, you know, Europe's stuck in the Middle Ages, this is grotesque regulation, this is just another sign of a kind of sclerotic, aging, useless economy, and we will use every tool in the power of the American executive and corporations to punish Europe for trying to regulate us, and we will use every lever we can to break those regulations open to let us in.

18:17Rory Stewart:And again, that doesn't make me feel that this is a culture that is very sympathetic towards people trying to think about safety or security. Yeah, I agree. I think Europe's approach is one that is democratic. It's not necessarily the only right approach. I understand why the U.S. has chosen a different approach, but I think the hostility of American companies and the government to Europe's approach has been really problematic and it's not one that I support.

19:03hearty, satisfying, and built for life on the go. From craveable sauces to satisfying textures, they're designed to keep you going without slowing you down. So put that fork down. Try the new wraps today in app or at order.sweetgreen.com. Available at participating locations only.

19:24Rinse knows that greatness takes time, but so does laundry. So Rinse will take your laundry and hand-deliver it to your door, expertly cleaned. And you can take the time pursuing your passions. Time once spent sorting and waiting, folding and queuing, now spent challenging and innovating and pushing your way to greatness. So pick up the Irish flute or those calligraphy pens or that daunting Beef Wellington recipe card and leave the laundry to us. Rinse. It's time to be great.