Why the US Must Engage China on AI Safety Before It’s ‘Game Over’ 

0:00The thing about AI for business, it may not automatically fit the way your business works.

0:05Stephanie Flanders:At IBM, we've seen this firsthand. But by embedding AI across HR, IT, and procurement processes, we've reduced costs by millions, slash repetitive tasks, and freed thousands of hours for strategic work. Now we're helping companies get smarter by putting AI where it actually pays off, deep in the work that moves the business. Let's create smarter business, IBM. When you're running a business, the best days are the ones where priorities stay on track. For midsize and large companies, that isn't always easy. Risk can touch multiple parts of an organization at the same time, often in ways that aren't immediately obvious.

0:43Stephanie Flanders:It might involve property, liability, or cyber. It could stem from regulatory requirements or challenges tied to a specific industry or the scale of an operation. At that level, managing risk becomes an ongoing discipline, not a one-time decision. At The Hartford, the focus is on helping businesses manage risk before it turns into something more disruptive. That means working with companies to identify where they're exposed, decide what matters most, and put practical standards in place so risk is managed as part of day-to-day operations. And when losses do happen, The Hartford can pair that risk control work with insurance coverage grounded in underwriting, risk engineering, and claims experience developed over time.

1:24Stephanie Flanders:Learn more at thehartford.com slash risk mitigation. Certified plant genius here. Most people see a busy plant shop, but I see a perfectly balanced ecosystem. Thanks to genius from Global Payments. Inventory, tracked, payments, seamless. Reviews in one place, absolutely genius. From sold-out crowds worldwide to running this shop, genius grows with you. Your Monstera's potted? Healthy roots, strong growth, just like this shop. Big League reliability for your business. That's genius.

2:01Bloomberg Audio Studios. Podcasts, radio, news. The fact is China has cutting-edge AI labs. They're only six months behind the frontier in the U.S. And if they get a really powerful model that can do cyber hacking, for example, and they put it in their systems and their systems are open source, it's game over. This is just going to proliferate all around the world. So that is a very bad outcome. So let's try and be creative about at least trying with China to talk to them. You're going to need both sides to slow down. But my point about the discussion in China, which I came into touch with when I was there, is that there is at least a discussion and openness to think about safety.

2:42The door is open just a crack. And so it's up to American negotiators to go there and try and give it a nudge and open it more.

2:57Stephanie Flanders:I'm Stephanie Flanders, Head of Government and Economics at Bloomberg, and this is Trumponomics, the podcast that looks at the economic world of Donald Trump, how he's already shaking up the global economy, what on earth is going to happen next.

3:13Stephanie Flanders:Last week, we looked ahead to this week's summit between the US and Chinese presidents. I'm recording this on Tuesday, May 12th, and by the time you listen to this, President Trump should already have touched down in Beijing, ready for what feels like a high-stakes encounter on Thursday with President Xi Jinping, even though the expectations for concrete policy outcomes, as we heard last week, are firmly under control. Now, along with trade, one of the big issues hanging over that conversation will be US-China competition in advanced technologies and the race to dominate AI. Tesla's Elon Musk and Apple's Tim Cook are in the group of top executives Trump's decided to invite with him to China.

3:56Stephanie Flanders:And there is some mild speculation that President Trump might further loosen the export controls on US cutting-edge semiconductors for Chinese companies. But the safety of this race, protecting the world from the misuse of these increasingly powerful tools both countries are now capable of producing, well, that doesn't seem to be high on the agenda. This week's guest on Trumponomics thinks that is a big and potentially dangerous missed opportunity. Sebastian Mallaby is a senior fellow at the Council on Foreign Relations, author of multiple award-winning books, and most recently of The Infinity Machine, Demis Hasevis' Deep Mind and the Quest for Superintelligence.

4:40Stephanie Flanders:I sat down with him in our New York studio a few weeks ago now to talk about his new book, The Future of AI, and the role of China, which he had recently visited. I think you'll find what he had to say fascinating. Whether hopeful or pretty alarming, well, I'll leave that to you.

5:04Stephanie Flanders:Sebastian, welcome. Thank you very much for being here in the flesh in the New York studio, which is always fun. There's so many themes coming out of your book, The Infinity Machine. I wanted to start with something that we talk about a fair bit on Trumponomics, which is the impact of AI on the economy, society, I guess, especially jobs. But I think given the themes of the book, and I know things that you've been thinking about since it came out, we'll move on fairly quickly to the question of safety. How worried should we be about the pace of the improvement in AI's capacity? But also, how likely is it that governments or anyone else will be able to bend it to their will?

5:44Stephanie Flanders:Humanities will. So first thing first, we had a conversation with Darren Osamoglu about it recently. I mean, economists, I've noticed, they tend to divide into those who say AI is going to have a very radical impact on business models, on the economy, on society. But that's going to take a long time and we will have time to prepare. There's another group who say, no, no, this is all going to happen much sooner, but it will probably be less seismic. You had all these years researching this book, really up close looking at the evolution of this technology. What's your best guess on that? Yeah, I think the second view, which is that there'll be an explosion in the power of the models and it'll immediately affect the economy, is put about by the technologists who are just focused on the algorithms and on the models and what the base model can do.

6:37And it completely underestimates the rollout challenges. If you think about this idea of recursive self-improvement, which preoccupies a lot of people in Silicon Valley, once you've got a machine that is good enough to write the code for the next iteration of the machine, and then it recursively self-improves, it'll just explode upwards. And that is true so far as the computer science goes. It's not remotely true as far as actually applying it to the real economy goes, because you need to build the computing capacity to serve those models. So you need all the chips, and there's a shortage of chips.

7:10You then need all the power to fuel all those chips. The people who are in the center of this are talking about having to launch data centers in space. That just gives you a sense of how difficult this is going to be. And that's not to say anything about the institutional frictions of integrating AI into, say, a law firm or a bank or whatever. I mean, that's a non-trivial challenge. You've got to get all your clients to sign off on the confidentiality of their information and so forth. So I do think that rolling it out such that it makes a difference in terms of job displacement or productivity is more kind of a decade plus than it is a kind of three to five year thing.

7:46Stephanie Flanders:And on your point about the exponential improvement, I guess the other thing, and the colleagues who are much more focused on this in the economics team have also said this to me, that when you think about a sort of pie chart of all the kinds of tasks that humans do that are part of the workplace, that exponential improvement is really only in a certain subset of those tasks. And there are other areas where they're really quite far behind that are quite important for jobs. Yeah. Another way of saying the same thing is that there are sectors which are probably going to be quite difficult to disrupt.

8:17And the public sector is pretty slow at adopting tech. And that's a pretty big chunk of some economies, right? Right. Construction, healthcare, there's a whole slew of economic sectors which don't seem to be very right for disruption at all. And then the ones that are right for disruption, the archetypical people working on a screen doing knowledge work and analysis, which is very similar to what AI does. As I say, you've got to get the customers okay with it. And that's quite a challenge.

8:47Stephanie Flanders:Railways is the example that's often used and it takes a long time for business models to adapt to that. If I think of something like the smartphone, which is extraordinarily new by historical standards, less than a decade old, day to day, all of us have been transformed by that. Is it comparable to that, that it's not necessarily having a big change on the economy or on productivity, but day to day, we're all being affected really quite quickly? Yeah, I think there's a lot of truth to that, that people will be using it. They already are using it. It'll occupy quite a lot of their headspace. but it's not necessarily making them more productive.

9:23One of my favorite stories about the anthropic agent, agentic system, which was out in January of this year. This is before Mythos, but the earlier one, people got very excited about, oh, this agent can organize my email, it can create a bespoke news summary for me in the morning. And so a friend of mine was implementing all these productivity enhancing hacks. And finally, after doing about four of these, she said to the agent, Would you please now sync up my calendar such that you check that the trained timetables, on average, are not too late or some sort of extra productivity thing, to which Anthropic answered or Claude answered, are you sure you're not just giving me this task because you're trying to avoid your own work?

10:08You know, so I mean, I do think there's a difference between the excitement around it and the adoption of it to actually make you more productive. And then even in science, AlphaFold, which was the DeepMind system published in 2020, end of 2020, for which Demis de Sabeis got the Nobel Prize, which unraveled protein folding. This was said at the time to be the breakthrough for faster drug discovery. And it probably will be over the longer term. But now we're six years later. There isn't a single drug that's been discovered thanks to AlphaFold. Not to say that there hasn't been progress, because now there's a much more sophisticated build-out of AlphaFold with other aspects of the drug discovery process being automated by AI.

10:53So I think on a kind of 10-, 15-year horizon, yes, it will massively accelerate drug discovery. But on a six-year horizon, it hasn't.

11:00Stephanie Flanders:That's a really interesting example. There is a wonderful, one of the good documentaries about that whole period. It's almost a sort of companion to your book or a part of your book. But there's a wonderful bit of footage where they have decided to just identify all of the proteins and then just put them onto the internet and allow researchers anywhere to access them. And they're watching in real time the uploads from all over the world on a map of the world. And it's very moving because you're sort of seeing in India and Africa all these researchers a lot more than they thought in the space of a few hours uploading all this information.

11:31Stephanie Flanders:But it's fascinating to hear that they have not actually produced any new drugs on the back of it. I mean, that goes to something and we move into the broader sort of policy and safety territory a little bit. But Darren Assamoglu was on the show recently and I was checking in with him because we talked to him over time on this very issue. And his focus and his advice to governments had been to think about you can't alter the nature of the technology, but you can change the way it's adopted. and that adoption was going to be the main thing that determined how jobs were affected, whether you're enhancing productivity of humans or merely just allowing businesses to replace humans.

12:17Stephanie Flanders:I later had a conversation with Jason Furman about that, the former head of the Council of Economic Advisors, and he thought that was just completely wishful thinking, that you can't tilt the technology even to that extent, that you can encourage it to be more complementary to labour as opposed to just substituting for labour. So what do you think about that? You know, I think in a race dynamic where you have multiple labs in multiple countries producing this stuff, the idea that you're going to control all of them in some direction that makes the technology complementary to labor, not replacing, does strike me.

12:48I'm kind of on Jason's side on that. The other thing to say is that the labs are going to create what the consumers want. And if the consumers, meaning in some cases the business consumers, the enterprises, have an opportunity to replace labor, they want to take that. And if they don't, their competitors in some other country will. And so the laws of competition and capitalism, I think, are very much on Jason's side.

13:10Stephanie Flanders:I think part of that is this dynamic, which is also very relevant to the book, of the race with other countries, particularly the race with China. But if you're a country, if you're the UK, for example, or could be any European country, that's not necessarily going to be at the forefront of the technology, but clearly does want to be at the forefront or close to the front in adoption in a way that also helps the economy, helps their workers. is there any advice from your book or from the research that you've done that you would have for governments who are trying to do that? Yeah very much I mean I think there's a lot of commentators especially in Britain draw exactly the wrong lesson about the story of DeepMind so the classic complaint is DeepMind is a British company founded in London main founder is British she's very patriotic about it he refuses to move to Silicon Valley and it turns out to be the most consequential sort of pioneer AI lab in the world.

14:09That's amazing. But these idiots sell it to Google in 2014, and it becomes a subsidiary of an American company, and that's terrible. And I think that's wrongheaded in the sense that really the sale to Google by DeepMind in 2014 was a trick, one could say, to get the American parent to pour nearly a billion dollars a year of research and development money into London, with enormous spillover effects for the tech ecosystem in London. So I think that's a win for Britain. The company stayed in London, the founder stayed in London. That's great for Britain. What was terrible for Britain, and about which Britain should beat itself up mightily, is that my book also tells the story of DeepMind trying to help the National Health Service.

14:50And both Demis' mother trained as a nurse, his co-founder Mustafa Suleiman, his mother trained as a nurse. For this reason, they really wanted to help the NHS. They were willing to do it pro bono for the first five years. And they started helping and rolling out AI algorithms that recognized early signs of blindness, early signs of cancer. This was just super positive, helpful stuff that was going to save the NHS time and help NHS patients. And it got blown up by a stupid backlash over privacy or privacy, depending on which side of the Atlantic you're on, which I mean, I really kicked the tires on this, the complaint about leaks of patient data were fictitious.

15:31And so Britain, because of this political climate of suspicion of tech, particularly suspicion of the subsidiary of an American tech company, created such a backlash against this collaboration between the NHS and DeepMind, that both the doctors and the computer scientists at DeepMind got cold feet and the collaboration collapsed. And Britain could have been really the cutting edge country in the world on medical AI. And they blew it.

15:58Stephanie Flanders:But putting together the two pieces of what you said, they are related because one of the reasons that there was a lack of trust was the foreign ownership, the US ownership, and particularly the association with major tech companies in Silicon Valley who have not necessarily earned our trust, many people would say, over the last 20 years. They've made promises about privacy, about sharing of information that have been broken. Every single major tech company has broken them, including Google. They have changed the terms on which even they were developing AI. In different cases, there were promises even that had been built originally into the DeepMind charter around safety and other things, which then got overridden as they needed just more and more money from the parents.

16:45Stephanie Flanders:So it seems to me, yes, in principle, it would have been great to have the UK on the cutting edge, but also completely understandable that having given tech companies the benefit of the doubt of various other times and rather regretted it, that they wouldn't want to do it in the case of this very sensitive subject. Yeah, that's all true. At the same time, it would have been nice if the critics looked at the specifics of what was going on in this collaboration between DeepMind and the National Health Service. And if they'd look more carefully at the specifics, they would have seen that this was positive for Britain.

17:15And if you just have a blanket view that anything to do with American tech is ipso facto a matter of suspicion because there's been bad behavior in other instances. I'm not in favor of that. I think that Britain did miss an opportunity there. But I think the issue that arises out of this is that it's in the interests of the tech companies, especially now with this super powerful technology, AI, to get governance right, to be proactive in inviting governments with democratic legitimacy to hold sway over the technology. Because if you just take it onto yourself as a youngish startup and you say, this is how healthcare is going to evolve, I'm doing it.

17:55It's not legitimate and people won't accept it.

17:57Stephanie Flanders:I mean, some of this gets to that sort of broader question of our degree of control and also the degree to which any of the people involved in this can shape the outcome in line with what they themselves want when they're in a competitive context. You have a very striking phrase in your book towards the end. It's inventors dream of shaping the technology that they create. Often the technology shapes them, the technology plus the business, political and geopolitical currents that it unleashes. There's a bit in your book where it starts to feel, I felt like some of the historical analysis that you see of the start of World War One.

18:33Stephanie Flanders:You have everyone individually in your story had potentially the right incentives, was very focused on being careful, identifying risks ahead of time, managing them ahead of time, not inflicting things on society before we were ready for them. The end result has not been that at all. That's right. The central character in my book, Demis Asabis, is a good person, has good values. But whether he can actually do good, even if he is good, is the central conundrum. And he really wanted to make things safer. That's why he fought a secret three-year war with Google, trying to force them to put more safety guardrails around AI.

19:17That's why he suggested to Rishi Sunak the idea of a Bletchley Park AI safety conference, which happened in late 2023. So he does care about safety. But when other labs are willing to go fast, there isn't much a single AI leader can do. And what, of course, that points you to is that the collective action problem needs to be solved by government.

19:40Stephanie Flanders:But what we've seen in a lot of these cases is the researchers involved, the companies involved, say, yes, we think it's very important to have a collective agreement. But if a government comes along with concrete rules, whether it's the, I'm sure you probably feel strongly about it, but the EU internet safety rules, AI safety rules, President Biden had a rather different set of rules. There tends to be the response, how could governments possibly understand this technology? We want some agreement, but not this one. So how do you solve that? Yeah, look, you're right that, you know, at the moment, there is lots of state level initiatives to regulate AI in the US.

20:18And for example, OpenAI is actually funding the opposition to those regulatory efforts. So there is hypocrisy sometimes.

20:25Stephanie Flanders:And after the fact, they often say, yes, actually, it's quite good and we've adopted it, but they never want it in advance. Yeah. On the other hand, there is a counterexample, which is that in the Biden administration, when they set up the AI Safety Institute, and it included a mandatory requirement for all the frontier labs to submit their models to this institute for review, the institute didn't have a veto, which it should have done in my view, But still, it was mandatory to submit. And I've spoken to the Biden people who created that executive order, and they say that there was zero resistance from the labs.

21:02All the labs were more in the mode of saying, look, you need to be aware, something's coming down the pike, it's very powerful, you need to get ahead of it. And when they did get ahead of it, or they started to at least move on it, there wasn't resistance. That's what the Biden people say. So I think it depends. It's a bit like with Google and DeepMind and the National Health Service in Britain. There can be good instances of technology company behavior. And again, if we paint, this is true of anything, if you paint the Chinese as uniformly bad, you can never collaborate with them on anything.

21:34And that's actually a bad outcome in itself. Same with the tech companies.