314. Why do the tech bros really want more regulation?

0:01Steph McGovern:We often talk about who is winning the AI race, but now the tech bros are saying we should slow down. What's this all about? Is this really about safety? Trump thinks it's a hoax. So who should we believe? And what does this mean for AI regulation? Plus, given the stock market reaction so far, is this a potential burst of the AI bubble?

0:24Robert Peston:This is literally the biggest story of the moment. And the questions are, will China cooperate with America? Will we essentially get an agreement that's a bit like the pact that we had to stop the proliferation of nuclear weapons? And also, can Andy Burnham, can the government play a role or are we just destined to be powerless and vulnerable spectators?

0:50Steph McGovern:We're proud to say that The Rest is Money is powered by Octopus Energy this year. And Greg Jackson, the founder and CEO, is with us. Greg, I wanted to ask you about heat pumps. Octopus Energy now make them. Why not just import them from China? You know what? We want to make heat pumps that are designed for British homes and that are cheaper to run, cheaper to install and actually cheaper for the hardware. And that's what we did. So, for example, octopus created heat pumps that run up to 70 degrees C, as hot as a gas boiler. So you don't necessarily need to worry about having excess insulation and changing the heating system.

1:30That sort of thing makes much cheaper for British homes.

1:33Steph McGovern:Nice one, Greg. Thanks for explaining that. Right. We're going to go to the episode. This episode is brought to you by Google Chrome. You think you know a browser, but Gemini and Chrome, that's new. It can help you with practically anything on the web, like restoring a vintage motorcycle from a 50-page restoration block, or finally break down that long article you've had open for weeks. Gemini and Chrome is here for it. Ready to make anything online make sense? There's no place like Chrome. Check responses set up required, compatibility and availability varies 18+.

2:14Steph McGovern:Well, it's been a dramatic week in the world of AI. I am sure you've all seen the apocalyptic headlines about how AI is going to kill us all. And then off the back of that and the debate around it, we've seen the AI company share prices fall. So we're going to talk about what this all means. And I'm just going to give you, first of all, a quick reminder on what happened. So the story in terms of the news headlines starts with hugging face. So this is a major platform where researchers and companies experiment with AI models. So OpenAI was using the platform to test how good its systems were at finding and exploiting security flaws.

2:56Steph McGovern:So it deployed around 1200 AI agents in separate what we call sandboxes. So these are like isolated test environments. So these agents got put into these sandboxes to do their thing where they wouldn't be able to talk to each other or get into any real production systems. But they managed to escape from these controlled environments and find ways to secretly communicate with each other, to collaborate. And then they basically work like a digital swarm to break into Hugging Faces' real servers, like a team of rogue hackers, basically, but without any human intervention whatsoever. So when OpenAI found this out, it obviously set off alarm bells.

3:42Steph McGovern:And since then, there've been other examples that have come to light. And I'm sure you've all seen that viral clip of the computer scientist and Nobel Prize winner, Geoffrey Hinton, who's basically like the godfather of AI, who said that a 10 % chance of AI killing all humans was not unreasonable. So that has been one of the things going on. But Robert, there's also, as well as this evidence of AI going rogue on its own without humans, there's also been some worrying stuff around how humans are using it badly too, hasn't there?

4:13Robert Peston:Yeah, that's totally right, Steph. I mean, you know, since we have seen the development of AI at a cracking pace, there's been two sets of fears about potential harm to humanity from AI. Taiwan has been AI becoming godlike and autonomous, self-improving. And, you know, one of the things that the AI leaders are currently saying is we're very, very close to AI becoming self-improving in a way that they would find very hard to control. And that's why the incident that you reported about these agents cooperating and then swarming and breaking through all the guardrails put in place by OpenAI has caused so much anxiety.

5:06Robert Peston:Just on that, I was talking to some government ministers last night and they said, oh, my God, we've been told about even worse stuff than that. Now, they wouldn't tell me what they meant. And I have to say, if there is worse stuff, given that these AI giants are committed to transparency, it doesn't really augur very well that the rest of us don't know about this worse stuff. But then let's talk about the fears about the harm that, you know, existing AI can do in the wrong human hands. Anthropic, published an incredibly long report, essentially going through all the bad actors around the world who'd been trying to use Claude to do really scary things, among which, for example, was I think they referred to them as a tribe in Yemen trying to perfect their missile guidance systems using Claude.

6:04Robert Peston:Now, they say that they prevented some of what they were trying to do, but not all the modifications this tribe was making. You know, Anthropic doesn't say that these were the Houthis. But if these were the Houthis, of course, that brings us to one of the world's clear and present problems at the moment, which is the Houthis are targeting with missiles the Saudi pipeline. And that's one of the reasons, as part of Trump's war in Iran, why the oil price is spiking so much at the moment. We're also frightened about rising inflation and rising interest rates. So all of this stuff is connecting up at the moment in a somewhat anxiety inducing way.

6:46Robert Peston:But one of the things, again, that is also striking, quite apart from the whole range of risks posed by AI, is the range of responses to this threat. because some surprising people, or perhaps they're not surprising, are basically playing down the fears, aren't they?

7:03Steph McGovern:Yeah, I mean, Trump is one of the people who's trying to play this down. So he's saying it's a hoax and he's compared these warnings about AI to what's happening with global warming, which he says is a scam. He said it's being perpetrated by the radical left Democrats, as he had called them on his social media account. And he reckons that the only guardrails needed for AI is a strong and smart president called Donald Trump. But I don't think this is a surprise to hear him say that, because if you think about what this might mean, you know, off the back of these warnings that have come out, we saw, as I mentioned right at the start, that the share price of these big companies start to fall.

7:44Steph McGovern:And obviously Trump's really built a lot of his success, a lot of what he's talking about when he says things are going well in America is around what's happening in AI. So any fears that threaten the development AI and potentially the speed at which it's being developed, putting a pause on that could mean the US not only seeing their big companies' valuations fall, but also China taking over the AI race, which is, you know, his big thing is beating China. When you think about what Trump cares about, it's just beating people and China is one of them. So, yeah, he's been playing it down. And interestingly, his AI advisor, so a guy called David Sachs, he's a venture capitalist, he's a tech entrepreneur.

8:32Steph McGovern:He's been saying that the fears of AI are overblown as well. And again, he mentions China in this when he's been doing various interviews about it. And he keeps talking about how this slowdown in the AI development is not a good thing because it would mean the US would fall behind China in the AI race. One of the founders of Anthropic, Dario Amadei,

8:51Robert Peston:has had quite a lot to say about this as well. In terms of where the debate is now, it's really been focused by a long essay written by Dario Amadei and he made three recommendations. One was that independent experts should come into the likes of Anthropic and OpenAI and Musk's X and Alphabet Google, who would monitor the development of AI and in particular monitor the movement towards total self-improving AI. This is when AI is able, through its own abilities, to improve its capabilities. And just to repeat what I said earlier, the worry about that is essentially AI becomes a completely, you know, some people describe it as an independent life form, autonomous.

9:54Robert Peston:Essentially, the only controls on it would be, in a sense, the morality that has been programmed into it in the so-called alignment process, aligning with our values. And of course, there are just endless examples from both people within the industry and philosophers about how this could all go badly wrong. The sort of classic example of this, actually, which I've recently been talking to Nick Stern, the climate expert, about is, you know, if you asked an AI of that sort to solve climate change, its answer might be to exterminate humans, since we are the cause of climate change, which is obviously not what you want.

10:35Robert Peston:But equally, the fear of the real doomsters, those who are obsessed with what's known as P-Doom, that's the probability of catastrophe, the real fear of these people is you get this godlike AI and they basically just take the view, oh, well, we're not terribly interested in humans, they're a bit annoying and let's get rid of them, which is the sort of extreme version of the so-called doomsters' fears. Anyway, you mentioned the godfather of AI, Geoffrey Hinton, saying there's at least a 10 % probability of us being wiped out. I mean, he's been, frankly, very worried about these risks for years.

11:15Robert Peston:That's not really new. What was sort of striking is the pronouncements of possible doom from the likes of Dario Amadei himself, from the senior people within Google Alphabet, OpenAI. There was a scientist within Anthropic who also said a more than 10 % likelihood of doom, as it were, of us being wiped out. So Amadeus has come up with these three proposals. One, invite outside experts in, give them all the information that employees have so that they can warn the likes of Dario Amadei, you know, when things look as though they might be going wrong. And Dario says, if we get those warnings, we will react accordingly.

12:08Robert Peston:Sam Altman at OpenAI has said that they will do the same and, you know, they equally will take precautions. Then there's a second request. This is the one that Trump has basically stuck two fingers up at, which is for a regulatory body, which would consist of mostly people from within the industry, but with government backing. So it would be sort of self-regulation, but underpinned by the authority of the state. And then thirdly, Dari Amadei makes the utterly obvious point that none of this would work unless there was global cooperation. And given that the only other country in the world that has made really serious progress in AI and, you know, regarded as perhaps six months behind America in this field is China.

13:06Robert Peston:Right. Yes. Yes, of course. You know, France has mistrial and there are, you know, other large language models of a sort of smaller, less significant sort all over the world. But basically, When Amadei says there needs to be international cooperation on safety, you know, that what that means is detente, you know, some kind of understanding between America and China. But it runs, of course, against two problems. One is that Amadei himself was unbelievably rude about China in his essay, basically describing them as the, you know, this appalling, irresponsible enemy. quite hard. You know, that wasn't exactly wooing China to come to the negotiation table.

13:44Robert Peston:So if you if you wanted China to cooperate, you might not have been quite so aggressively rude. And of course, you know, there's the other big point which you referred to, which is President Trump does feel more competitive against China than he does against any other nation in the world because it is the other superpower. And his his whole approach is to basically get to, the most powerful AI long before China does. And in his hands, it would be a sort of weapon against China, which is pretty terrifying.

14:18Steph McGovern:Yeah, and the chances of an agreement, as you say, between US and China to regulate AI is near on impossible. And similarly, the Chinese government have already been out saying how ridiculous they think that these AI warnings are. They've said that, OK, yes, they need to make sure there's regulation, But they call these the narratives of threat and confrontation and malicious competition. So they only serve to disrupt the process of global AI governance and are not in anyone's interest. So they've been pretty hard on the attack back. But there's one thing before we go back onto China, I just I think it is worth just pointing out, as you say, about this this rare display of unity between all the tech bosses about this.

15:02Steph McGovern:Because as you say, we had like Sam Altman come out, the OpenAI CEO saying, yeah, we need to slow the pace of the frontier and, you know, committed to this independent and evaluators with employee like access in the business, as you pointed out. And then Google DeepMind saw Demis there has said this, you know, this is pointing towards the right path forward, what Dario's saying. And then Elon Musk came in, a man of few words, and just said, Dario is right. But interesting that there's that unity, but it makes you suspicious, doesn't it? So Trump saying there's not a problem, China kind of saying there isn't a big issue, but all these tech bosses, these AI bosses, the leaders in all of this are suggesting that they need to be regulated by government, by cooperation and collaboration, as you say.

15:57Steph McGovern:but it rings alarm bells, doesn't it?

16:00Robert Peston:Yes. I must just say, I love the idea of Elon Musk as a man of few words,

16:05Steph McGovern:but although he was obviously in this particular instance,

16:10Robert Peston:many of us would wish that he'd just shut up more generally, to be perfectly honest.

16:15Steph McGovern:Loads more to come. We'll catch up with you after the break. This episode is brought to you by Workday, the AI platform for HR, finance and IT. Now, unless you've been living under a rock, you would have heard a lot of chat about AI and its relationship to business, both on this podcast and everywhere else on the internet. Now, the thing is, you would have also heard about AI getting things wrong, leaving you guessing whether it's done something right or not. Guessing is for game shows, not your business, especially when your margin for error is zero. Workday is the enterprise AI platform with a difference because it has a deep understanding of your organization's context and guardrails.

16:56Steph McGovern:So every AI action is permission aware, giving you accuracy 24-7 and the ability to not just get work done, but get it done right. From HR to finance and IT, Workday's AI makes it safer to go faster. It's a new Workday. Search Workday to learn more. This episode is brought to you by Google Chrome. You think you know a browser, but Gemini and Chrome, that's new. It can help you with practically anything on the web, like restoring a vintage motorcycle from a 50-page restoration block, or finally break down that long article you've had open for weeks. Gemini and Chrome is here for it. Ready to make anything online make sense?

17:37There's no place like Chrome. Check responses set up required, compatibility and availability varies 18+.

17:47Robert Peston:if you just look at the behavior of big tech over the last 20 years what you see is an industry basically dominated by businesses that want to be monopolies right i mean essentially google what became a monopoly in search and there is a fair amount of evidence that a bit more competition earlier on would have been, you know, in all of our interest. You talked about Sachs, the economic advisers to Trump. You know, one of the things he said that I have a degree of sympathy with is there is one interpretation of what anthropic open AI in particular have asked for, which should ring alarm bells, which is they are effectively looking to be categorized through regulation as the safe regulated companies.

18:46Robert Peston:And in history, when that happens, when you have the sort of authorized players, that is a very significant barrier to competition. What you're essentially doing is you are entrenching the market leaders in their market leadership. Now, it is also the case, to be frank, the opportunity for others to compete with these two is comparatively restricted simply because of the hundreds of billions of pounds that are being invested by them. It's very difficult at the moment to see anybody else in these closed AI systems raising that sum of money. Even so, if, for example, you look at, you know, an industry I know incredibly well, which is the history of banking, you know, when we got to a world of certain institutions and in the UK, they were, you know, NatWest and Lloyd's and Barclays and, you know, HSBC being, you know, the authorised regulated leaders.

19:52Robert Peston:it then became incredibly difficult over decades for anybody else to challenge their control of the industry. And so if there wasn't this genuine risk, and I think it is a genuine risk of catastrophe from human catastrophe from AI, I would be genuinely very anxious about them asking to be regulated in a way that preserves their market leadership.

20:18Steph McGovern:Yeah, it's a really important point, Robert, that, you know, how difficult it is to become a bank in the UK because of the regulation around it. And it's taken years for some of our now biggest companies to have got the banking license, like a really classic example is Revolut. So this is obviously, you know, the largest digital finance app. They applied for a banking license back in 2021. And then they had a five year approval process before finally getting that full authorization to launch as a true UK bank. So that didn't happen until March this year. And yet Revolut is one of our most valuable companies.

20:59Steph McGovern:And then you've got other ones which have failed when they've got to the hurdle. So Bank North is one. So this was this, again, like another challenger bank based in Manchester. And it was a financial technology startup founded in 2018. They managed to get the first bit of the license, but then they struggled to raise enough money to survive and eventually gave up their license. So this is the whole strict requirements, as you say, put off and can easily exclude those challenger businesses or take them years to get through the process and the licensing requirements. Yes.

21:36Robert Peston:And look, if you were going to be cynical about what Anthropic and OpenAI are up to in saying the pace has to be slowed, is there is a very clear implication from them, which is we will only slow the pace if our competitors slow the pace. and, you know, they want to keep their competitive advantage. And so, you know, there are some who just say, this is what oligopolists, very powerful big businesses that want to exploit their market power to charge all of us more, this is what they tend to do. As I say, I do take the risks of extinction or at least very significant damage to us seriously. But again, if you were going to be cynical, you would say, actually, there is a system, a set of laws that can deal with this.

22:30Robert Peston:And they are basically product liability laws. I mean, look, obviously, if Anthropic or OpenAI wipe all of us out, we can't sue them for doing that. But any harm less than wiping all of us out, you know, we can basically, as customers, sue them to kingdom come. And so the other, as I say, if you wanted to be a little bit more hard nosed about this, look, these are two businesses, OpenAI, Anthropic, which have raised are raising colossal of sums from investors. They want to get a lot of that money back from floating on the New York Stock Exchange or on Nasdaq. They want a list. They want to IPO.

23:16Robert Peston:And, you know, this uncertainty around essentially product liability, around whether it's humans abusing it, whether it's AI going rogue, the potential cost to these businesses would be off the charts. And truthfully, unless they are able to reassure not just all of us, but their investors that they have these product risks under control, then the idea that they're worth a trillion dollars is a joke. I mean, you could actually argue, given the kind of risks they say that are associated with their development, their net value is currently zero. So there is an enormous amount of commercial self-interest for these companies in saying that they need to be both self-regulated and regulated by governments.

24:11Robert Peston:And supervised is the other important term in these, because there is a difference, actually. One of the things I should just make clear, because this is always the case, when it comes to making any company, whether it's a bank or an AI company safe, if you're going to go down the regulation route, you both have to set rules, and then you've got to have watchdogs who keep an eye on the companies to make sure they follow the rules. One thing I do want to come back to, maybe not now, but we must come back to in the course of this conversation, is whether it is impossible to see an agreement between America and China, because I'm not 100 % sure it is impossible.