Is there a 10% chance AI “could kill all humans”?

0:00Still waiting in line? Again? That's time you'll never get back. Save time and money with Stamps.com. Over 4 million businesses have skipped the line with Stamps.com. Join them to save up to 90 % off carrier rates from your computer or phone right now. Print postage for certified mail, registered mail, and packages in seconds. Then schedule a pickup right from your home or office. For a limited time, go to Stamps.com and use code PODCAST for a free welcome gift. Taxes and fees apply. This episode of the Times Tech Podcast is sponsored by KPMG. Katie, we've been tracking the AI surge for a while, but some of the latest data is pretty striking.

0:42It is. KPMG's latest quarterly AI pulse suggests that 7 in 10 UK business leaders still see AI as a top investment priority, even if the economy slows down in the next year. But turning that investment into tangible value is another question. And so a lot of organizations are still figuring out how to move from endless AI pilots to something that works across the business. KPMG's focus is on that gap, helping organizations scale from the pilot phase to AI-powered transformation they can trust. To find out more about how KPMG can help your organization make the difference, you can visit kpmg.co.uk forward slash AI.

1:23This episode of the Times Tech Podcast is sponsored by ServiceNow. Danny, one thing we keep hearing from business leaders right now is AI sounds great, but how do you actually make it work inside a company? Exactly, because most organizations aren't neat, shiny systems. They're layers of software, legacy tech, and teams all doing things slightly differently. ServiceNow sits across all that, acting as a control tower for making work move seamlessly through the organization. Connecting people, systems, data, and increasingly AI agents so that things don't happen in silos. Learn how ServiceNow puts AI to work for people at servicenow.com.

2:05Danny Fortson. Yes.

2:07Danny: You don't look like you're in your normal spot in Silicon Valley with your lovely plant behind you. I'm not. You look, if I may say so, like you're in a hostage video. Don't worry. I don't have a newspaper with like today's date on it to show that like, you know, proof of life or whatever. I admit, it's actually a Ramada. Ramada in, what do you guys call it? A Ramada. I don't know. Ramada. A hotel. You're in a hotel. Yeah. In scenic Calgary. I'm here for a story covering another tech billionaire. So I am safe. I appreciate your concern. Well, let's get to the pod. We shall. Let's get to it. Hello and welcome to Times Tech, where we unpack how technology is reshaping our everyday lives.

2:52I am Danny Fortson.

2:53Danny: And I'm Katie Prescott. And the cheery news we bring you this week is that there is more than a 10 % chance that AI could kill humans. Cool. Cool. And that's according to a key figure at Anthropic, the chap who leads research on what's called alignment, basically making sure that the models do what they're told to do. So that might win the prize for the most extreme warning about AI of the week. But he is not alone. And the last few days have been peppered with news of a potentially very dangerous new era of AI. Yeah, this whole conversation started when OpenAI released a powerful new model called Astra, which Jensen Huang of NVIDIA hailed as AGI, artificial general intelligence, you know, human level intelligence.

3:42We're here. We've done it. We'll be getting into the details of all of that, including why it's got tech bosses sounding the alarm yet again.

3:50Danny: And the question is really, how seriously should we take this? But before we get into that, we wanted to tell you about our next episode of the interview, which is the part of the tech podcast where we speak to some of the biggest names in tech, shaping all of our futures and doesn't get much bigger than this. That's my big name, Klaxon. That is the big introduction that we need to Bill Gates, the Microsoft founder. And we had a fascinating discussion about why he thinks AI is completely different to any previous tech, why it poses a threat to humanity and who should be responsible for controlling it.

4:29Danny: And that episode is going to be out next Tuesday from 6 a.m. London time. So make sure to listen or watch on our YouTube channel. Yes, indeed. So all very exciting, but let's get back to today's episode because we're also going to be talking about a creepy new documentary trailer that's been released, which has everyone in Silicon Valley talking. It's about Elizabeth Holmes. Remember her? The disgraced founder of Theranos, which is this health tech company that was once worth$9 billion before it all went downhill and it was exposed as a fraud. It's a pretty crazy story and feels relevant to this AI moment that we're in.

5:06Danny: Should we start with the light news, though? Yeah. Let's do that. So when I woke up this morning and jumped onto X, as you do, and started scrolling through some of the top posts, I was greeted by one that kept coming out time and time again. Some of people retweeting it from a guy called Jacob Coxon, who said he was resigning from Anthropic because he'd spent the last three years there doing pre-training research, also at OpenAI, by the way. and he said neither company is acting responsibly they're racing straight to self-improving super intelligence and gambling with our lives now of all of the retweets from jacob of this message which you know and there have been a number of people haven't there over the past few years who've resigned from the frontier labs yeah this has become kind of like an oeuvre like it's a pretty large body of these kind of like i'm quitting here's why we're all up a very famous creek and i'm gonna go you know write poetry in the mountains and wait for the end times so relatively well trodden path but what stopped me in my tracks was a retweet from a guy called evan hubinger at anthropic who leads what's called alignment so this is making sure that the ai is literally aligned with what whoever's created it wants it to do.

6:33Danny: When it's not, they call it misalignment. Evan says, Jacob is correct here. We really do earnestly believe AI could kill all humans. Exclamation mark. I personally think it is a greater than 10 % within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to. Happy Friday, everyone. So that's kind of mad. And as you say, it comes right off the back of this launch of Astra by OpenAI with Jensen Huang and Greg Brockman at OpenAI saying, the era of artificial general intelligence is here. Folks, human-like intelligence, welcome.

7:18So let's just briefly talk about Astra and then we can come back to the warnings and what it all means. so astra they released it about a week ago they put out this very slick marketing videos like three minutes of people using it in all these different incredible ways and they're like look agi is here this is all very exciting this is our most powerful and also most aligned model you just talked about alignment this idea that it is lockstep with what is good for us as humans it's not going to go off piste and do something crazy in the same breath they said it's their first model that hits their quote-unquote critical threshold for cybersecurity.

7:57And Britain's AI Security Institute found that Astra performed, quote, a range of malicious actions, end quote, when tested, including fake identities to deceive developers, violating the scope of its instructions and acting without approval. So it's kind of, again, the two-edged sword of, oh my goodness, we've created this thing that is more powerful than anything we've done so far. Like we have you know, well and truly moved past that first chat GPT moment. These are things that can just do incredibly powerful, complex things, you know, with very little oversight, etc. Yay, but also scary times.

8:34Danny: Yeah, and it was very much yay, right? I mean, the video is one lady saying, can you draft a legal document for me? Oh, and also, I'd like a tennis court. And talking to this large screen, which, you know, the AI just bustles off and does these things. But these models are getting more and more powerful. And on the same day that this was launched, Reuters reported on another instance of OpenAI's models escaping from their testing environments. And that's something which OpenAI said, well, at the moment, you know, we don't have any proper rules in place for how to report that. We're sort of discovering this on the hoof.

9:16Danny: It does feel like Like there are more and more and not just warnings from the people making it, but warning, you know, warnings from events happening from the companies themselves about the AIs, frankly, getting out of control. Two things that struck me from the Astra thing. One is in the video, and it's a subtle, but I think very, very intentional choice. All the people using Astra are just talking to it. Exactly. Like nobody is sitting there behind a computer. everybody's like standing around walking just talking to the air talking to the room in front of a big screen and then the astra is responding and so i think it's really interesting that this we've talked about it a lot on this pod around the death of the keyboard and does it make more sense given just the nature of this technology should we just talk and there was no one pointing an implementer to screen or anything they were just by their side sitting in an armchair having a chat book me a tennis court hey i had that rice and chicken that i bought the other day could you get me that again that was really nice thanks mate and sam altman said that's his favorite marketing video so far that they've done and it is that dichotomy between the cozy marketing video hey look what we've made it's so powerful it's so exciting this is how it fits into your real life versus actually this is so powerful that it can take malicious actions and it deceives people and how you reconcile those two things is very very difficult i just went to open ai's website and i just looked at the press releases it's september and this is september 9th they put out 15 press releases in the past nine days with titles including an alien mind daybreak for the defenders and all of this just like really like what are you talking about and i don't know if you read did you read the piece from cal newport that this computer scientist he wrote it in the new york times he's been kind of diving into a little bit what makes these leaders tick you know dario and sam and all of these folks and critically he's like they're doing a really big disservice to all of us because every week they're going to come out and say what they've just said this week we might all be dead in 10 years or something like that crazy and he's like that's a complete distraction to what the industry is doing right now like around cyber security around data centers around ai slop ai friends what's that doing to kids and then the analogy he makes is like you know, if Ford cars were spontaneously cashing on fire, his quote was, we wouldn't tolerate the chief executives changing the subject to discuss our general lack of preparedness for a future with flying cars.

12:11We'd want them to fix their blowing up cars now.

12:13Danny: All right. But to my mind, the difference with cars is there are very strict regulations in place around. There are cars and when they're allowed to go on the road. And they have to have seatbelts. And at present, there are no such restrictions on AI. And it's something that we talked to Bill Gates about in our conversation, which is going out on Tuesday. He's been very vocal about the fact he wants to see some sort of international body in place, whether that's a little bit like for aviation safety or akin to the UN or something, in order to sort of put the guardrails around AI because it shouldn't be left to these companies.

12:53Danny: But I just wanted to read you something just as we talk about who's in charge of this technology to give you a sense of what it might be like working inside Anthropic. It's an ex-post from Tom Blomfield, who, as you may remember, is the founder of Monzo, who moved to Y Combinator in San Francisco and is now at Anthropic. And he's set his overall impression of Anthropic after two months in the job. Hyper earnest group of mega nerds who are very focused on ensuring the say I thing goes well for humans. surprising lack of ego or politics. People are often weird, but I haven't met anyone malicious or mean.

13:30Yes. I mean, I think that's becoming abundantly clear.

13:33Danny: And there was some huge news here in the UK this morning broken by the Financial Times, which was that Anthropik's latest model was not given to Britain's AI Security Institute to test. Now that's a big deal because AC, as it's called, has up till now had a chance to look at the models and has provided these very, very clear evaluations of their dangers, globally recognized as having incredibly high standards, hugely experienced researchers working there. And there is much concern about why Anthropic withheld that access, whether it was under duress from the White House. So, you know, there's a sort of political element here.

14:18Danny: But also it goes back to the fact that who's in charge. And at the moment, no one is in charge. And so it's very, very hard, I think, for any of us to understand just how consequential this is when you've got people saying, you know, like that guy on X, I'm resigning because no one's really taking this seriously. If the bodies that are in place, although they're voluntary, and they don't have any teeth, you know, don't have that access, where does that leave any of us? The other thing that, again, just going back to how these leading labs are approaching this, you know, there was a big kerfuffle this week amongst the, you know, 15 press releases.

14:55One was about the solving of one of these quote unquote millennium prizes of math. One of the most difficult unsolved problems in math that's been unsolved for nearly a century. Open AI comes out like astrosaulted. And everybody's like, oh my goodness. And it's like, I don't care. stop with the math olympiad and all that like i get these are things that show the capabilities but again if you look at public opinion 52 of americans believe ai will do more harm than good data centers have become a huge issue in the upcoming election bipartisan everybody hates them and is trying to stop them and it's not surprising that if we're talking about like oh look at this math equation no one even knew was a thing unless you're like a hardcore mathematician like what about curing cancer or developing free energy or new materials or solving climate change like stop with the hey look at this latest thing we did that doesn't really affect anyone's life and the existential dread that we're all supposedly sleepwalking toward where's the good part and what are you doing to solve the issues that you're bringing up right now and i just find it it's such a strange strange marketing approach and it's they're shooting themselves in

16:19Danny: the foot over and over again my fear actually is that it isn't just marketing and that they are clamoring for someone they're clamoring for a grown-up to pick up the baton here and so far no one is but we should move on after the break we're going to talk about how silicon valley is taking center stage in Hollywood with all these big blockbusters coming at once all around tech, Silicon Valley and what it's doing to us.

16:50This episode of the Times Tech Podcast is sponsored by health and life insurer Vitality, your health's best friend. Vitality works differently, get active, build healthy habits, and you can unlock rewards from some of the UK's top brands while helping to keep your premiums low. It's award-winning health and life insurance that helps you live healthier. Find out more at vitality.co.uk. This episode of the Times Tech Podcast is sponsored by KPMG. We often talk about the shiny object problem with AI, where things look impressive in isolation, but are much harder to roll out across a large organization.

17:29Issues like governance, workforce readiness, and trust tend to become more important as things scale. KPMG's approach is to look beyond AI as just another technology rollout, and instead, how it could redesign work across the organization and fit into wider processes and decision-making. So it's less about individual use cases and more about how organizations can transform and deliver outcomes, powered by AI. To find out more about how it can help your organization make the difference, you can visit kpmg.co.uk slash AI. Today's episode of the Times Tech Podcast is sponsored by ServiceNow. ServiceNow describes itself as an AI control tower for business reinvention, connecting people, systems, data, and AI, so workflows more smoothly across an organization.

18:19The platform integrates with different clouds, models, and data sources, bringing them together in one place to help your team. To learn more about how ServiceNow puts AI to work for people, visit servicenow.com.

18:37well should we move on to an equally scary story tell me about this new documentary the elizabeth holmes doc what's that all about so this guy nathan fielder he like very famously trained and actually became like an airline pilot for a series and he's like this really interesting kind of guerrilla journalist slash comedian And anyway, apparently Elizabeth Holmes, after she was sentenced and 34 days before she was set to go to prison, she wanted somebody to document like her life. And so Nathan Fielder showed up with a camera crew and did that. And he just dropped this trailer. No one knew this was happening.

19:20And then just out of nowhere drops this trailer where it's him in what appears to be a living room. She's like sitting there and she's just like staring into his eyes, unblinkingly saying how she's like, I am not deceiving you. I'm always real. And you're just kind of like, everybody was like, whoa.

19:43Danny: Well, should we remind everyone about the Elizabeth Holmes story? I'm sure so many people are familiar with it. But she was the founder of a company called Theranos, which basically claimed to be able to spot a range of diseases just from a tiny drop of blood. And anyway, it turned out to be an enormous house of cards. She had some extraordinary backers. She raised almost a billion dollars. The equipment didn't work. They'd fake various tests. And she was sentenced more than 11 years in prison after she was convicted of fraud in January 2022. So this is a bit of a sequel, is it, to sort of Bad Blood, which was the book that John Carrier wrote, the famous journalist who unraveled this story.

20:22Yeah, the film's called You Can See Everything. So it's not out yet. It's coming out, I believe, in October. And I think what's really interesting about it is she entranced a whole bunch of investors. And she had military generals on her board. She had Henry Kissinger on her board, all of this stuff. And she conned them all. And the core thing was the thing she invented wasn't really an invention because it didn't really work. And I think it's really interesting it's coming out now. Because we were talking earlier about AI, everything is happening there. and just there is a gold rush happening right now.

20:57There are going to be 10 Theranos's of the AI era. People are raising money on like a two-page deck PowerPoint, raising hundreds of millions. Everybody's trying to get there is before the music stops. And there's just a lot of really dodgy stuff happening. And it always comes out whenever the proverbial hits the wall and kind of the bubble pops. But I think we're going to see, you know, stories that make Elizabeth Holmes' story look quaint by comparison, at least in terms of the money at stake.

21:34Danny: There was a spectacular example here in the UK of a company called Builder AI, which raised half a billion dollars, including from Microsoft. And the allegation has been against that business, which is now folded, that its AI was actually 700 people in India. i mean i think the sort of claims about that are disputed but yeah there are lots of stories doing the round as you say yeah i just think there's going to be a whole bunch of theranos of the ai era you know it'll be fascinating to see it all kind of play out but before we go there's two other things two other films artificial which i cannot wait to see oh and It looks great, doesn't it?

22:18Danny: And the trailer is far more extensive than the Elizabeth Holmes one. So you get a real sense of... And it shows Sam Altman showing up and he's voicing over like, I'm going to call it ChatGPT. And he's like going into one of his houses and there's like a wall of machine guns because he famously has a doomsday bunker with guns and gold and whatever. And a sort of guy just wearing a swimming trunks lounging outside on a bed and then a voiceover saying, we're opening Pandora's box. Yes, exactly. So that's coming out at Christmas. And then before that, The Social Reckoning is coming out, which is really about Francis Haugen and the whistleblowing that happened there at Meta.

22:55And, you know, it's the sequel to The Social Network, which in a way kind of set in stone for a lot of people who don't follow tech, like who Mark Zuckerberg was. And I think that's what artificial could do for Sam Altman. And now we have, again, going back to Meta and Mark Zuckerberg, the social reckoning about the whistleblowing, the effect on kids, and how that all played out over the, you know, 15 years later. So again, it's just fascinating that we have all these big blockbusters coming at once all around tech, Silicon Valley, and what

23:32Danny: it's doing to us the glorification of facebook and the social network and how mark zuckerberg founded it at harvard and then as you say the fallout the impact that that's had on society yeah down the line exactly well anyway lots for us to watch it's nice that they're catering for people like us who needs the odyssey hey when you've got a film about sam altman exactly so It'll be great. But I think it's really appropriate that these companies and these people are, in a way, at least for a time, will be like the center of the cultural conversation in a way that's different. Because when Hollywood does it, it just creates a different conversation and brings a different level of kind of knowledge to, you know, the man in the street.

24:15Danny: Anyway, I think I'll leave you to Calgary. Good luck. I'm really glad you're not a hostage. Thanks, eh? Come, come. That was my... Nice Canadian impression. Canadian impression. Cancel for that. We are going to be back next Tuesday with an episode of the interview with Bill Gates. We talk about where AI is heading. Billy G. Yes. Don't worry, Danny doesn't call him that on the podcast. Get it all out. And we talk about why he doesn't think the companies who are making AI should be responsible for it. So yeah, it's a fascinating interview, a fascinating to hear one of the world's most powerful people and leading technologists talking about that.

24:57Yes, indeed. So thank you, as always, for listening. Check us out on YouTube. And if you already don't subscribe, first of all, why? And just do it. Subscribe so you don't miss this episode or any future episodes. Thank you, as ever, for listening. Goodbye. See you next week. Bye-bye. Bye. This episode of the Times Tech Podcast was sponsored by KPMG. One of the themes we keep coming back to is the gap between AI ambition and real-world application. And that's where KPMG is focusing its work, helping organizations move beyond pilots and think about how AI can change the way the business operates and how it fits into wider processes and decision making.

25:38You can find all their insights and how they can help your organization make the difference through successful AI workforce adoption. Visit kpmg.co.uk slash AI.

26:20to try ShipStation free for 60 days. Hey, it's your ceiling vent, so I'm dripping. Could be the rain, could be the upstairs bathroom. Yikes. You could hire the guy your neighbor recommended, but I'm pretty sure that's just his cousin. Do we know if he's licensed, or does he just own a ladder? Listen to your home. Go with Thumbtack, upload a photo or voice note, and we'll diagnose your project and match you with the right pro for the job. Thumbtack. We know homes. Hire the right pro today.