So Could AI 'Kill Us All'?

0:00Some people treat ChatGPT like some kind of smart search engine, and some use it to get work done. ChatGPT Work is a new way of working in ChatGPT that can take action across your apps and files, stay with a project for hours if needed, and turn a goal into finished work. It's designed to help you move from a chaotic starting point to a reviewable first version. So all the source materials, briefs, and scattered information that you have to grind through to turn into something useful can just become something useful. Put ChatGPT to work on your most ambitious ideas and projects. Get started at ChatGPT.com by selecting Work Mode, available on Plus and Pro plans.

0:42Coffee genius here. Most people see a busy cafe, but I see precision at every step. Thanks to Genius from Global Payments. Transactions? Instant. Inventory? Precise. Operations? In sync. Absolutely genius. From sold-out crowds worldwide to managing the morning rush, genius keeps operations running smoothly. One portado. Flawless pour, perfectly timed. Just beautiful. Big League reliability for any business. That's genius. At Venture Global, we think about what can be done, not what's usually done. Through innovation, Venture Global is not only building some of the largest energy facilities in the world right here in the United States, but delivering American energy at a fraction of the cost and a fraction of the time.

1:31So while others are busy talking, we're busy building. That's Venture Global. That's unstoppable energy.

1:45Bloomberg Audio Studios. Podcasts. Radio. News. Earlier this week, an AI researcher at Anthropic resigned from his job and posted about it on X. He'd spent the last three years working at both Anthropic and OpenAI, he wrote, and neither company is acting responsibly, he said. He said the companies were gambling with our lives and suggested that people in the AI space thought the technology could eventually kill us all. It was scary stuff, and the posts made waves in the AI industry and beyond at a time when concerns about the risks of AI are mounting. Bill Gates says the computer industry that he championed is now a global threat.

2:28OpenAI System was able to hack into another AI platform called Hugging Face during a security test. With me today is Bloomberg's Mike Shepard, Senior Editor for Technology and Strategic Industries. Mike, how would you describe this moment for AI? Are we at a turning point? You know, we talk about things that go viral. This took off, but it also has some staying power. It didn't just ripple through Silicon Valley. It echoed across Washington and in global capitals, too. So there is this moment of reflection now in this technology that has so much promised to change the economy and change the way we live.

3:08I'm Sarah Holder, and this is The Big Take from Bloomberg News. Today on the show, the AI safety reckoning is here.

3:19Mike, I am curious why this particular post from this particular researcher broke through, though. AI researchers have been sounding the alarm for a while on these kinds of risks. This person, Jacob Coxon, not a public figure, not someone I'd heard of before. He doesn't have any other tweets on his profile. Why did people pay so much attention to what Jacob Coxon was saying? And what did he actually say about why he left the company and the things that he's so concerned by? I've been reflecting on this, too. One, its simplicity. It was very direct. It was personal. He really addressed the audience, not just a tech audience, but really a Main Street audience as well.

3:59And he wasn't very preachy in it either. He was issuing a warning, but he kind of left it to the reader, the consumer, the audience to try to make their own decisions. He did issue a warning that, in his view, after having worked at both Anthropic, which he just resigned from, and OpenAI, his prior employer, that neither company was acting responsibly in their respective pursuits of what's known as superintelligence. This is the version of artificial intelligence that surpasses human capabilities in most areas and also possesses the ability to improve itself. And he called that pursuit of what's called recursive learning or self-improvement, he cast it as reckless, really, Sarah.

4:44And he spelled out in passing the cybersecurity risks, that there could be a massive and crippling hacking attack using AI that could really disrupt financial systems and critical infrastructure. There's also the threat of bioweapons, too. So all of this kind of hangs in the background of his message. But then he also issued an admonishment to his fellow Silicon Valley AI workers like, hey, do you really want to be a part of this? Are you sure you want to have a role in developing this technology in a way that could be so destructive and consequential in a negative way? I put that question to you, Mike.

5:28Why are people still building it? Well, for Anthropic, part of their identity is to pursue AI development responsibly. And they actually see themselves as the company that can achieve this superintelligence or artificial general intelligence in a way that will be safe for humanity. And it's not just that Anthropic, a number of researchers out there are committed to the pursuit of this technology for that reason. They want to get there first so that it's done safely, so they believe in it. Others are a little bit more skeptical about the apocalyptic vision that, you know, some are casting as the backdrop for AI development, that it will kill us all.

6:12And they think, hey, look, it's not that bad. We can control it. We can understand it. And most AI will actually be used in much smaller and refined applications. I want to drill into this idea that AI will kill us all because, you know, Jacob's former colleague in alignment science lead at Anthropic reposted Jacob Coxon's post and said that, quote, we really do earnestly believe AI could kill all humans, unquote. And he gave it a more than 10 percent chance within the next decade. That's like a pretty remarkable thing for someone who currently works at Anthropic to say, right? It is. It's striking to hear this conversation.

6:52And yet it's one that's been kind of going on in the background, a little bit out of sight from, you know, the people in Washington who would be making artificial intelligence policy and out of sight of the folks on Wall Street who are investing so much in this technology, raising billions of dollars in capital on behalf of the hyperscalers, the cloud companies that are investing in AI. And then on behalf of, of course, Anthropic and OpenAI, which are preparing these massive IPOs, and then the general public, too, which are more concerned about AI's impact on their utility bills than potentially, you know, life on Earth as we know it.

7:33And yet at the same time, it is kind of a recurring theme within that community in Silicon Valley. So I want to talk about sort of the context that this X thread came out in, because just a few weeks ago, we learned about the hugging face hacking incident, which Jacob pointed to as a warning shot. I'm wondering, Mike, could you just tell us a little bit more about the hugging face incident and why this was such a big deal in the AI space? We've known for a long time that AI could be used as a tool to try to break into computer systems around the world. If, you know, put in the wrong hands, certain AI tools could be incredibly destructive.

8:17And that's what led Anthropic to withhold largely the release of its new Mythos model back in April. The difference in this Hugging Face episode, though, was that the models themselves engineered this attack on the Hugging Face repository of AI models and other data connected to AI. In this particular incident, a group of models and agents that were being developed by OpenAI, they had been placed in a secure testing environment known as a sandbox and had their guardrails removed so that researchers could test their capabilities. Now, while the agents and models were inside this so-called sandbox, they were given a task, a very difficult cyber problem to solve.

9:08and the agents and models began collaborating together. The researchers didn't have great visibility into all the things that they were doing and the models eventually figured out, hey, maybe we need to get out onto the internet and that would mean breaking out of this so-called secure sandbox. And if that presented them the path to the solution, great, it didn't matter. In other words, they didn't see it as an impediment. They were willing to break the so-called rules of their environment in order to win the game. Exactly. Exactly. And ultimately, it led the agents to go find a whole bunch of models that they could access and data that they could access at Hugging Face without Hugging Face's knowledge or authorization and really out of the view of researchers at OpenAI.

10:00Well, it's funny, Mike, there's been this sense that emphasizing or even overstating AI's capabilities, even scary ones, can be somewhat of a marketing win for these companies. It's just another way of demonstrating just how powerful AI can be, helping them justify this massive spending on this technology. But are we at a point where that is backfiring? Are these risks and possibilities getting too real? The Huffing Face episode does make it feel more tangible, certainly. And you do see more of these arguments taking shape and becoming more coherent on the risk side. And remember, they're also being paired with all the fear and loathing about data centers and their impact.

10:46And yet at the same time, the counterargument to regulation has been that, look, you are fear mongering because you want to pump up the value of your products as you head into a massive and what could be a trillion dollar valuation for an initial public offering. That is a claim or an argument that I've heard from a number of people in the tech industry and in government who oppose much heavier or stricter regulation on artificial intelligence. They do see those kinds of measures as restraining innovation. And they also see that some of the doomerism talk could be aimed at just drawing attention to and adding to the allure of these products and models that the AI companies are developing.

11:33When we talked about mythos, there was so much demand for it after anthropics said, hey, only a few are allowed to have it. It does create the aura of forbidden fruit. And then all of a sudden, everybody wants access to it. Coming up, what AI safety guardrails could look like in practice and how competition with China complicates the politics of slowing down.

12:02This is the Bloomberg Tech Minute brought to you by ChatGPT. Now with ChatGPT Work, I'm Carol Masser. Has the AI moment arrived at corporate offices worldwide? Not yet, and it's perhaps several years away, according to one survey of corporate recruiters who circle the world's business schools each fall and spring. Bloomberg's Rob Mandelbaum notes that according to the Graduate Management Admission Council, the skills recruiters sought most among 2026 graduates of business school master's programs, including MBAs, were communication, problem-solving, and adaptability, which changed little from last year.

12:38Training in artificial intelligence tools ranked in the bottom third of the skills list, but recruiters did say AI-specific training will be more important in five years. On that, recruiters rated current graduates as only somewhat prepared or less to work with AI tools, and said business schools are not doing a good job of teaching AI skills. Bottom line, according to the survey, what most companies want when it comes to AI skills is for candidates to be able to automate routine work. That's the Bloomberg Tech Minute brought to you by ChatGPT. Put ChatGPT to work on your most ambitious ideas and projects.

13:14Get started at ChatGPT.com today by selecting Work Mode. Available on Plus and Pro Plans. The new NFL season is here and you should be listening to NFL Daily as we march along to Super Bowl 61. Last year, we had Mike McDonald, Sam Darnold on the field after the game, but that was last year. If you're a football sicko like me, NFL Daily is your kind of show. It is in the name, NFL Daily, fresh content in your feed every day, all season long. Matthew Stafford coming off his MVP season. on a dazzling dart from Matthew Stafford. Big time trades that shake up the NFL. AJ Brown is loose. And of course, those Seahawks trying to run it back and win another Lombardi trophy.

14:04The Seahawks win Super Bowl 60. They are bringing the Lombardi back to the Northwest. If it's happening on a football field, NFL Daily will cover it. Listen to NFL Daily on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts. I'm Brett Coleman. And I'm Trevor Sikama. Have you ever watched a game and wondered what really made the difference beyond the final score? That's exactly why we created NFL Pro. Got a man wide open. Touchdown! Seahawks! Every week, we use exclusive next-gen stats insights and NFL content to break down the biggest stories, matchups, and moments from around the league.

14:45We go beyond the box score to uncover the numbers, the tendencies, and details that help explain what actually happened on the field. Fires it out in the flat to Barkley, who beats one man, spins past another, leaps over at third. And don't worry, this isn't just a show for stats experts. We use advanced analytics to help tell the story behind the game, turning the numbers into insights that every fan can understand and enjoy. So whether you're a casual fan learning more about football or a hardcore fan who can't get enough analysis, our goal is to help you see the game in a whole new way. If you've ever wanted to know why a play worked, how a matchup was won, or what most fans missed on Sunday, NFL Pro delivers insights you simply can't get anywhere else.

15:32Listen to NFL Pro on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts.

15:45Mike, what are the kinds of guardrails that experts say could limit some of the worst outcomes of AI superintelligence or sort of the recursive learning that Jacob Coxon was talking about? Well, some of them would be testing by the government or more robust reviews of these models before they go out. And the companies have been trying to do this. But one of the challenges with this technology is that even if we're not fully at this moment of self-trained models, a lot of this technology is developed almost in a petri dish. It can kind of run itself. It can do some of the coding itself. and with that you lose a little bit less visibility into the process and into what went into it.

16:31In other words, you and I may want a model to do something, but it ends up deciding on its own, hey, I think I'm going to go in a different direction. It's called the alignment problem. And even if they put garb rails in place, how will we know that they are fully effective? And lawmakers here haven't really gotten their heads around it completely. Bernie Sanders introduced recently this measure that would call for a ban on developing superintelligence altogether. And we've seen others in Congress call for a kill switch on AI. I'm not exactly sure how that would work, but it would, in essence, force the companies to have something available that they could pull the plug on a runaway model if need be or a runaway negative tech event if need be.

17:21how that will work in practice. You know, we are probably a long way of seeing how that would develop. It's important to remember that we are heading into the November midterms. We have a very narrowly divided Congress and we have an administration that is naturally averse to regulation and especially when it comes to artificial intelligence. I mean, another argument against more aggressive AI regulation has been, you know, its AI models to keep up with China. But I was listening to the Radio Atlantic podcast and listening to Bill Gates talk to the interviewer there. And he's among those who have made the point that having AI run wild is not in China's interest either.

18:05So I am wondering sort of about the global consensus around reining in some of these risks and whether the argument on the sort of China competitiveness side is starting to shift as well. The U.S. certainly has the national security considerations that they do not want to see China gain an edge in AI for fear that it could be deployed against the U.S. eventually in a military conflict or even in a less kinetic way in terms of cyber attacks and other things that could disrupt the American economy and U.S. security. So this rivalry with China in AI is not only top of mind, but when President Donald Trump is asked publicly about safety and about whether data centers should be slowed a little bit to reflect local concerns about their impact, his argument is that we can't slow down.

18:58We need to stay ahead of China. And we also heard Treasury Secretary Scott Besson saying this week that if we don't stay ahead of China in AI, it's essentially game over. A lot of this will come to a head when Presidents Donald Trump and Xi Jinping meet here in Washington in a couple of weeks. And AI and certainly safety will be on the agenda for those talks. Well, there are those within the AI industry in the U.S. I'd count, you know, the co-founder of Anthropic, Dario Amodai, among them, who are saying we want the government to regulate us. They need to step in. We can't regulate ourselves. But what can companies actually do without action from Congress?

19:40And what are they actually willing to do themselves in terms of putting up these guardrails and slowing down on some of the innovation? Well, one thing that companies could do is perhaps slow the pace of trying to reach artificial general intelligence or super intelligence. That has been a stated goal of both Anthropic and OpenAI, that they want to get to AGI soon and really try to win the race there. And instead focusing on more bespoke and refined direct applications of the technology. You know, when it comes to physical applications like robotics and self-driving cars, maybe put more of the energy toward that rather than this utopian goal of, you know, an all-powerful model.

20:30This is The Big Take from Bloomberg News. I'm Sarah Holder. To get more from The Big Take and unlimited access to all of Bloomberg.com, subscribe today at Bloomberg.com slash podcast offer. If you liked this episode, make sure to follow and review The Big Take wherever you listen to podcasts. It helps people find the show. Thanks for listening. We'll be back tomorrow.

21:02I'm Josh Maikowitz, and I hope you'll join us for Season 5 of Dateline Missing in America. In each episode of Dateline's award-winning series, we will focus on one missing persons case and hear from the families, the friends, and the investigators all desperate to find them. Maybe you could help solve a mystery. Search Dateline Missing in America to listen completely free or subscribe to Dateline Premium to unlock new episodes early. College football will hijack your weekend, ruin your sleep, and break your heart. And that's why we make the Solid Verbal College Football Show. I'm Ty Hildenbrand.

21:41He's Dan Rubenstein. Every week, we preview the games that matter and the storylines that are going to ignite your group chat. And you can join us in the aftermath as we react to what happened, why it matters, and how the sport went completely bonkers yet again. Listen to the Solid Verbal College Football Show on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts. I'm Brett Coleman. And I'm Trevor Sikama. If you love football and want to understand what's really happening on the field, check out our show, NFL Pro. Every week, we break down the biggest plays, trends, and matchups using exclusive next-gen stats insights and NFL content you will not find anywhere else.

22:21Whether you're new to football, a fantasy football fanatic, or a diehard fan, NFL Pro will help you analyze the game like a pro. Listen to NFL Pro on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts.