Anthropic boss calls for AI slowdown

0:00This BBC podcast is supported by ads outside the UK.

0:30and more options to their lineup can shop at machinefinder.com. Booyah! Saving money on everything you need for your next project at Menards. Add energy efficiency to your home with Johns Manville Foam Board Insulation. These panels combine a reflective facing with high-performance foam that creates the ultimate layer of insulation around your home to save on energy bills. Save big in all insulation products now at Menards. Plus, check out our weekly flyer on Menards.com for more great deals happening now.

1:11Hello and welcome to NewsHour from the BBC World Service, live from London. This is Owen Bennett-Jones. And coming up a little later in the programme, as the singer Celine Dion takes the stage after her treatment for stiff person syndrome, we'll hear from a fellow sufferer. What we've got to manage, if you're lucky enough to have those resources, can really make you look great while you're out. It's almost like it's a double-edged sword in many ways. But if you look good when you go out because you rested, you took the right meds, you planned, people think in a way that you don't have this disease.

1:52So managing a terrible condition. We'll hear about that a little later. But first of all, US President Donald Trump said this week he has no concerns about AI leading to human extinction. His concern, rather, was that China would win the AI race. But some of those actually involved in making AI are saying they're less relaxed than that. The boss of Anthropic says the industry should slow down. And other big names are also agreeing with that, including OpenAI's Sam Altman and Elon Musk. On top of that, a couple of Anthropic execs resigned this week. One of them, Jacob Coxon, has told the BBC, there is a strong chance we could all die in the immediate future.

2:37Well, Gina Neff is a professor of responsible AI at Queen Mary University of London, joins us live now. It does seem more and more people, you know, who are at the top of this really are coming out saying, regulate me. Is that right? They're saying regulate me with huge caveats. Dario Amadei is calling to, quote, pace the frontier of AI development, just as the company are floating a$1.6 trillion IPO. His message is clear. Anthropics say they have such powerful technology that only they should be entrusted with it. The sentiment is both undemocratic and also cynical. But I think we all need to be heeding Amadei's warning here.

3:22the technical development of AI right now outpaces the social, legal and regulatory advances that we're going to need if we're going to use AI safely in society. So are you saying that Amo Day, the head of Anthropic, is basically saying Anthropic's, yeah, he's thinking Anthropic's in the lead. So it's fine if we freeze developments just now, so leave me ahead. Well, they absolutely want to pace right at the moment when they're in the lead. So for example, OpenAI just announced that they're putting a pause on their own IPO because it would be unsafe to do so at this particular moment. Anthropic is going ahead.

4:07What they're calling for in this regulation though is thin when it comes to regulation. It's like regulation proposed by a high school dropout of Model UN. They would hire independent evaluators to sit in-house with the company. Now, we've seen that story of businesses doing self-regulation before, and it always fails. Well, just to understand the threats, I mean, because all these statements are made. Now, Amadeus said rogue AI agents could take over the internet in six to 12 months, causing hundreds of billions of dollars in damage. I mean, that's a slightly vague way of putting it. What does he mean would actually happen?

4:45Yeah, this is coming on the heels of one of Anthropik's regular reports around the cybersecurity abuses and other harms that they've detected within the company. And what they can see is that bad actors around the world are using clawed tools like ordinary people do to do some bad things. And they're worried that as we move to what's called the agentic AI moment, that these agents might not be able to be controlled. They want good people, and in this missive he published, democratic people and people in the US to control AI and not other countries in the world. The report has lots of examples of China, Iran, and others, Yemen, using clawed tools to do bad things.

5:41We're just going to hear about the Yemen example just now. Just before we do, you've used this word agents. It's slightly confusing. And basically what that word means, rogue, not rogue agents, really, rogue computer programs. Is that right? Agents aren't rogue. Agents are simply computer programs that can work autonomously without being specifically programmed to do a particular task. So they're given a goal and they're said, crack on, go for it. And the challenge here is that those agents don't necessarily, they haven't been to school, they haven't been to church, they don't have human discernment.

6:17So the kinds of values that AI agents have to go about those goals aren't human values. We might understand if something is illegal or immoral or unethical or could really get us into trouble. AI agents don't have that sense of right and wrong. Yeah, and you can't just tell them to obey the law. That's right. You can put in guardrails and you can put in safeguards that make sure these agents aren't stringing together very complex tasks. And that's partly what Amadei is calling for here. Remember, Anthropic was founded five years ago by computer programmers who broke away from OpenAI because they thought that company was not doing enough for AI safety.

7:05And now they're in the lead. They're saying they should be the ones to create safe AI. OK, let's just hear about the Yemen example that you mentioned, because basically Anthropic said users in North Yemen, they didn't say who, but really this has to be the Houthis, carried out sustained effort to develop guided weapons with the AI platform Claude. Well, Sarah Eldib reports on Yemen from Lebanon for Associated Press. So what details did Anthropic give about this? They told us in a lengthy report that came out on Thursday that they have been able to identify users in northern Yemen that were operating in cells, basically like software engineers that have worked to develop and design weapons programs, three different programs to develop and design weapons.

7:58basically all of them sophisticated, precision, long-range ballistics and missiles with hypersonic capabilities. So the attempts were picked up. The company tells us that it was picked up because their safeguards blocked many of the requests, but not all of them, because they used tactics to evade detection, such as hiding their goals or splitting their work into different sessions and so on. It is a very striking report. And from your knowledge of Yemen, I mean, when they're saying users in North Yemen, that does sound like the Houthis, right? They don't identify the users. They don't tell us who they are.

8:37But we know that North Yemen is the territory held by Houthi rebels and it's controlled by them. And we do know that the rebels have a wide array of weapons, sophisticated as well, long range as well, cruise missiles, ballistic missiles. I think what I've understood talking to weapons experts and people who know about the weapon capability of the Houthis is probably while everyone, as we've seen, everyone is trying to understand how to use AI for their daily life. It seems that the Houthis are using, experimenting with the models to see how they can probably produce their own weapons domestically and kind of be self-reliant.

9:17One of the attempts, as we understand from Anthropic, is to use a mobile phone software, something that you can find off the shelf, to develop a guided long-range missile. So this is, in a way, or what was one of the interpretations was, it seems to be their attempt to kind of use domestically available components to produce weapons. They have a lot more tech savviness than we give them credit for. Yes, it's interesting you say that. So I remember bin Laden, when he was leading al Qaeda said, we're not going to use email, we're not going to use any of these messaging systems, because it's not our technology.

9:53And we don't understand it. And we're better off using paper and paper messages carried by messengers, because, you know, we know that's secure. We don't know what we're doing with this high tech stuff. But maybe that's changing. And the Houthis are at the front of that change. Well, when we spoke with the official, he actually dismissed the report and said we are developed and sophisticated armed force and we will not be using open source to develop and produce our own military capabilities. So I think the line is the same, but it seems to be available for everyone. I mean, everyone is experimenting with how to use AI.

10:26I don't see why fighters in Yemen wouldn't be. Yeah, it's free. It's free and it's there. it's real it's there i mean i think what anthropic is telling us and and i think this is the other thing is i tried to reach out to them to get a bit more clarity on the period and a bit more information but i haven't heard back but i think what they're telling us is that their safeguards managed to block many of their requests but not all of them so i think that's also what they what the company wants to say is that it's putting out the attempts that have happened using its ai models, but we don't know what else is going on there.

11:05I mean, we know as much as they're telling us, basically. And that's Sarah L. Deeb from AP, Gina Neff, Professor at Queen Mary University of London, still with us. I mean, this may sound naive, but isn't it possible for Anthropic to say, don't help anyone make a missile system, don't help anyone make bacterial weapons? it's it's absolutely not naive in fact it's what companies like anthropic should be doing and to be fair anthropic is stepping into that leadership role i think many people would also want to see governments able to step into that role for example the united states dismantled their national ai security capabilities when the trump administration came back in they threw the work that had been done through the AI Summit series internationally.

11:59The UK, on the other hand, is now seen as having one of the world's leading government AI security institutes that can do those evaluations. Those kinds of choices of who should be able to use this powerful technology should not be in the hands of a few dudes in Northern California. It should actually be much more regulated. I have read that China is actually probably ahead of the US on this. Is that true, do you think? China's certainly ahead in terms of thinking about regulation and safety. They've, from the beginning, wanted to align AI very closely to Chinese values and Chinese development.

12:38In terms of where we're going at the frontier, China has access to what's called open-weight models. These are ones that we might call open source. they're slightly different and less powerful for now than the ones that are being developed in the US. The gap is closing, however. And so being able to say that one country can have the monopoly on AI may not be the future we're going down. But I guess the point is on the regulation side, that it's very difficult for the Americans to accept regulation if they think they're going to slip behind China in terms of the AI development. I mean, that's, yeah, that's that's a tension underneath all of this, isn't it?

13:18Well, absolutely. I mean, the idea that going full speed ahead is going to lead to harm of people is also going to lose the race. We can't lose the race and lose our lives at the same time. Okay. Well, thank you very much. You're incredibly clear in describing all this. That was Gina Neff, Professor of Responsible AI at Queen Mary, University of London, on some of the risks that even the bosses of AI companies are now beginning to highlight. You're listening to the World Service of the BBC.

14:07In Paris this evening, Celine Dion has taken to the stage, performing once again after her appearance at the Paris Olympics, this time in a series of full live concerts, and tonight's is the first. She was diagnosed with a rare neurological condition four years ago. It affects her singing voice. It forced her to cancel shows and left her struggling to perform. There has been a lot of excitement from her fans about her return to the stage, including those who waited outside her hotel ahead of the show.

14:43It's no work. It's unbelievable to stay here and to go to the concert and to see Selene so close, so kind. And she's feeling good, she's happy and it's unbelievable.

15:04We're incredibly excited because we know almost nothing about the show. No set list, no details about the stage and hardly any information has leaked, which is unusual. We can't wait to hear those first notes and witness her big return. So she has a really big following and a lot of talent, but she does have this terrible condition, stiff person syndrome, SPS. It's rare, it's incurable, autoimmune disorder, affects only around 8 ,000 people across the world. And it means basically your muscles get very stiff and there are spasms. and she's described it as a feeling of though you're being strangled.

15:41Aliyah Khan is a doctor who's got this. She was diagnosed at the age of 31 and now is forced to work mostly from her bed. I asked her whether she's surprised that Celine Dion is able to perform with this condition. I'm not. If you told me a person has stiff person syndrome many years, she's been struggling with it. I might say, yes, being on stage, that's going to be, whoa, too much. However, having lived it and discovered that this disease is all about personal management, it is a spectrum disorder that can have some people walking and some people bedridden. It can have, you know, some people working, some people not working.

16:26So those big swings, I think, are what are so confusing for people that don't have this condition. You're bedridden, right? I'm in the bed yet 20 hours a day. And does the condition fluctuate? You know, do you have good days and bad days? Right. So she might be able to predict that to some extent and perform when it's good. Yeah. And even when we're having bad days, if we know that there's something that evening, we're preparing. I'm going to take these meds at this hour. I'm going to put on these clothes at this time. And with our medicines and immunomodulating chemotherapeutics and what we've got to manage if you're lucky enough to have those resources can really make you look great while you're out.

17:11And it's almost like it's a double edged sword in many ways. But if you look good when you go out because you rested, you took the right meds, you planned, people think in a way that you don't have this disease. And that's why there was concern in the patient group that Celine Dion going on stage in a way was making people think this is not such a bad condition to have. Or that it's very manageable if only we find the right thing. And if that was the case, we would say, what do you want? And then we'd all go on it and we'd all be better. But so I like to give her some credit that, you know, it's not all better.

17:50It's that she knows she's got the schedule and she can make it work, you know, as a medical director for eight months, making it work with a central line catheter in and hiding it with turtlenecks. So simply, I would put it that this is an autoimmune disease. So your own immune system is crashing down on the enzymes that put brakes on the nerves. So you're fast in all of your nerves. That's why we're rigid motor. and then also why our throats can tighten up and we need Botox around our vocal cords, etc. But it's also a fast mind disease. So, you know, your mind is nerves and so that's where the psychiatric symptoms come in that keep us inside.

18:34And so it's not just the falls and the physical rigidity, but it's also all of that psychological damage that makes you not want to be your best. The thing is, it strikes me that with just 8 ,000, people suffering from this awful condition, it means there must be very little economic interest in a drug company working out a solution, right? I mean, if it was 8 million or 80 million or 800 million, there'd be a lot of work going in on it. Exactly. I understand why there's, you know, like you said, the dollars aren't going there. And that's another thing, you know, Celine Dion has to deal with is why hasn't she given more?

19:15And I'm always shocked by that. But yes, those are the kind of questions that we field as people that are like, you know, in the advocacy spotlight. Hey, Aylea, why isn't she giving more to that clinic or why that clinic and not, you know, and they want to know what is RT therapy or what is this immunomodulation? So I think there's a lot of confusion in the community as to if there's something that can fix us, why aren't we all on it? You know, you sound so reasonable in your approach to this. And it's obviously a really awful situation you're in, in bed 20 hours a day. But given your attitude to this, just tell me, what is your feeling about Celine Dion performing tonight?

19:58I mean, do you celebrate it? Do you worry about it? What do you think? I celebrate it because I think that the team around her is probably gangbusters. So I'm not as worried about her falling or something. I've lost two adult teeth. Like we smash our faces, all kinds of things. So I worry about her less in that regard. Like she's got a good support structure. A lot of this disease is about your support structure, which leaves many in the cold because it's a mysterious disease. Nobody wants to really deal with it. So I feel a little bit celebratory in that, like, yeah, like, heck, yeah, one of us is going out there and doing our thing, because even if it's just for one time, yes.

20:40On the other hand, I feel bad for my counterparts that have to hear from their spouses, like, she's doing it, how come you can't? And that's a very typical one. You know, if she's doing it, why can't you find out what she's on and get on it, and then we could have our life back? Because this doesn't just take away your own life. It takes away what you thought you'd build with your partner, your parents, your children, whatever it is. I can't walk my dog. I can't go to the oceans two blocks away. So those things, I celebrate it because it's life coming out of one of us, you know? And I love anything that is, you know, creative or intellect coming out of us because we don't know how much, you know, how much time any of us has, which fall is going to give a brain bleed not to be so dark about it, but that this is the kind of thing we go through.

21:34Suicide as a psychiatrist is something that is not discussed in this disease enough, but I know that, you know, plenty of evidence of it. So the pain and what you, and the rigidity and all that, if she can get out there and perform because all of the planning's been in place. I love it. I just, I wish it came with a, I had to do all this to get to here, or I'm on this therapy and it's look what I did. But I, I don't get that. Nobody gets that vibe. You know, that was like, how come she's not talking about us? I get that a lot. And I'm like, I think it's because she's trying to sing. you know that sounds silly maybe i didn't you know say it right but i think if you have a calling and you have and she's got millions of people and jobs and that depend on her there is a fastness of the mind with this disease that i think can give you a sort of jedi kung fu mind control to separate yourself from your body.

22:43And that was Dr. Aaliyah Khan, who, like Celine Dion, is living with stiff person syndrome. You're listening to the World Service of the BBC. We'll see.

23:22performance and long-term potential and get this these veteran machines can pick up the latest precision tech with ease you know it and fans ready to add john deere certified pre-owned equipment and more options to their lineup can shop at machinefinder.com booyah john deere is putting in work and putting up highlights oh man look at those passes tillage planting harvesting three years in and these pre-owned machines are still at the top of their game. We're talking reliability, performance and long term potential. And get this, these veteran machines can pick up the latest precision tech with ease.

23:59You know it. And fans ready to add John Deere certified pre-owned equipment and more options to their lineup can shop at machinefinder.com.

24:11Once again, Donald Trump has got to the top of the news agenda by making just a brief statement. This time he said he would love to see a unified Ireland. He's in Dublin. He was speaking to reporters there. And the question of Irish unity has been one of the most consequential in the UK for over a century. It's caused decades of violence. Well, I don't want to cause any problems, but I will value it. I'd love to see it unified. I have friends on both sides, great people. They're Irish people. How can they be bad, right? I'd love to see it. I think it would be a great feather in everybody's cap if that happened.

24:52It's going to happen eventually. Well, later he said the UK would have something to say about it, but I think it would be a great idea. Now, just for those who are not across all this, the island of Ireland is divided now between British Northern Ireland and then the sovereign country of Ireland. There was a peace agreement. There was decades of violence and then a peace agreement anchored by the so-called Good Friday Agreement. And amongst other things, that Good Friday agreement, lots of compromises in it between those who want to stay with Britain and those who want a united Ireland. But it laid out a process by which a united Ireland could be achieved.

25:27And that was one of the reasons that it was agreed to. Well, Katrina Perry is a BBC journalist currently in Dublin covering this Trump visit, but also reports on Washington politics. So just tell us, you've watched it through the day, because this was said this morning and we've had the whole day. what kind of impact has it made? It's sent shockwaves really I would say across the island zone and indeed on that to the US where I'm normally based President Trump was in Dublin as you say making those remarks. He's since relocated down to Doombag here which is where I am as well. I've tracked him all the way here so if you hear some loud noise behind me it's because there's a true party festival scene going on in this village here for the occasion of the president's visit and more so the Irish golf tournament that's being held here at the Irish Open.

26:20But no one would have anticipated that President Trump was going to make these remarks on this visit. Any nervousness from the Irish government would have been more centred on would there be any sticky issues around trade, around tariffs, around the situation in the Middle East, because the government of Ireland has taken quite a different approach to the government of the US when it comes to recognising the state of Palestine, when it comes to a new law that's just about to be signed in, in fact, in just two days' time, which would ban imports from the occupied territories, from the Israeli settlers, an issue, this issue of Irish unity that anyone would have anticipated President Trump raising.

27:04And indeed, when he heard those remarks, there were gasps amongst the press pack who were watching it. Yeah, it takes something to shock an Irish press pack. We've heard it all over the years. But do you think it was premeditated? I mean, you know, because apparently it was the sort of penultimate question, wasn't it, in a press conference, and he was asked about it and then made this remark. Do you think he planned to do it? I don't think so, no. I mean, the question came actually from a British journalist who had travelled over here for the visit of President Trump. And it's not something that we've really heard anything about from this administration.