Anthropic Whistleblower Says AI Could “Kill Us All”

0:00Exchanges on the M &A and IPO landscape. Exchanges on the dynamics affecting global trade. For the sharpest analysis on finance, business, and the economy, count on exchanges. The Goldman Sachs podcast. Listen now. This episode is brought to you by Google Chrome. You think you know a browser, but Gemini and Chrome, that's new. It can help you with practically anything on the web, like restoring a vintage motorcycle from a 50-page restoration block, or finally break down that long article you've had open for weeks. Gemini and Chrome is here for it. Ready to make anything online make sense? There's no place like Chrome.

0:36Check responses set up required. Compatibility and availability varies 18+. Star Wars The Mandalorian and Grogu is now streaming on Disney+. What are you waiting for? Hop on. The must-see galactic adventure comes home on Disney+. This is the way. Our only chance is to work together. I like this kid. Always wear your seatbelt. Star Wars, The Mandalorian and Grogu. Now streaming on Disney+. Rate PG-13.

1:16Money market's bad. If money is evil, then that building is hell. The show goes on! The folks are never watching so soon! Welcome to Prof.G Markets. I'm Ed Elson. It is September 10th. Let's check in on yesterday's market vitals. The major indices fell as fighting in the Middle East escalated. Brent crude topped $101 per barrel for the first time since May. The yield on 10-year treasuries hit its highest level since 2023 after the Treasury announced it is tripling debt buybacks to$6 billion. On Calci, the odds of a rate hike next week climbed to 57%. And finally, Metashares rose more than 6 % after the company released its Muse AI agent.

2:04Okay, what else is happening? A mass extinction event could be upon us. A former Anthropic researcher announced his departure from the company on X late Tuesday night. He revealed that employees at both OpenAI and Anthropic believe that AI could, quote, kill us all by the end of the decade. He went on to say that at OpenAI, many have not, quote, deeply internalized the civilizational stakes, while at Anthropic, the stakes are, quote, well understood, but they are locked in a race. He ended the message calling for an increased amount of safety cooperation between labs, urging other researchers to do the same.

2:44In response to this message, another current Anthropic researcher chimed in saying that he is, quote, correct, and that anthropic researchers, quote, really do earnestly believe that AI could kill all humans. They also put the chances of that happening within the next decade at greater than 10%. So, is AI really going to kill us all? Here to help us break it down, we're speaking with Alex Kantrowitz, author of the Big Technology newsletter and host of the Big Technology podcast. Alex, Thank you for joining us on the show. This tweet has gone insanely viral. It now has about 100 million views, and that's in less than 24 hours.

3:25This guy is basically saying that Anthropoc and OpenAI are developing technologies that are going to kill us probably, maybe, in the next few years. What do you make of it? Do you think he's right? Well, I don't think he's right. I mean, I would say that it obviously has garnered a lot of attention because some of the AI activity we've seen might lead you to that conclusion. We've obviously seen over the past couple of weeks swarms of AI agents doing things like solving Millennium Prize problems. And OpenAI had thousands of bots that attacked an infiltrated hugging face against its wishes. So we're clearly seeing some crazy activity from bots right now.

4:03But as to the question of will AI kill us all, I am strongly on the side that it will not kill us all. Now, could it sort of take on financial institutions and hack into companies? Absolutely. But at least for now, AI needs us to exist. Like right now, we're taking so much of the economic activity that humans create and using it to build data centers and power the data centers and feed these bots. So I think offing us would be a terrible decision on their part. And I think they know that. So this idea that they're about to kill us with now or within the decade is wrong. And unfortunately, it really muffles the message of the AI safety advocates because there is a there there.

4:41It's just not this take all humanity and turn us into paperclips there that I think a lot of them believe in. Such a wild statement to say that there's a 10 percent chance that we're all going to die. And then also to have that endorsed by someone who's actually a leader at the company right now saying like, yeah, we actually a lot of us think this. A lot of us think it's actually even greater than a 10 % chance that this stuff is going to kill us all. Why are they saying that? I mean, and what are we supposed to make of that if they're kind of just putting it out on Twitter that the technology that they are building might destroy humanity?

5:21Like, is that something that we should even remotely be taking seriously? Well, I'm taking it more seriously than I did in the past. I think previously a lot of this sounded crazy to me. And I think the reason why these people are saying it right now is because many of them were early to this. So before AI could form coherent sentences, they saw that if you were to give it more compute and more data and make the algorithms, make the models bigger, that they would improve. And everybody else thought that that was ridiculous, that they would never get to the point where they could actually speak like humans.

5:50and now they are. And so if you were at this point where early on you thought that the AI technology was going to improve and keeping it aligned with human values was going to be a challenge, and clearly that's proven to be the case. I mean, the AIs are doing things we don't want them to do. Then sometimes in your brain, the next logical conclusion is, well, eventually they're going to want to take over the world and kill humanity, so we won't be in their way. But to me, there's a logical hole in that thinking, which is that if AIs want to take over the world and get all the resources that the world has to offer, you have to work hand in hand with humans.

6:27You don't take them out of the picture. So maybe there's a way where they co-opt our species and have us continue to do everything we can to build data centers and empower them. But we're already doing that. So why kill us? I just don't see where that ends up being the place where they land. A lot of politicians are using this moment to advocate for pausing AI progress. This has been advocated for by several politicians. Bernie Sanders has been one of the most vocal advocates saying that we basically need to stop or put a pause on this. And, you know, to be fair to him, suddenly this researcher says, yeah, this might kill all of us.

7:10So it's a pretty decent argument when you think about it. What do you think is the next chapter in the AI regulation story? A lot of people have been wanting data centers to be banned flat out, AI to be paused. Do you think that that is the next part of the story? Oh, it's absolutely going to be a major issue in the 2026 and the 2028 elections. There are going to be politicians that are going to want to say, listen, we can't live with this stuff. So let's pause it. I think for AI, if you're rooting for AI to continue, especially if you're on the business side of things, in good news, a lot of this is localized.

7:48So it's municipalities, it's states, it's cities. It's not necessarily something that's happening on a federal level. And I would be stunned if we saw a pause on a federal level. So there might be some obstacles that come out of this for the AI companies looking to build in certain locations. They should be able to overcome those obstacles. But that being said, I think it would be unwise to underestimate just how big of a political issue that this has become. And the fact that some of the staunchest allies of AI, I'm talking specifically about the Republican Party in Texas, they've now become enemies of AI.

8:25And if that continues to be the case, you know, this little thing could spiral into something bigger. But my prediction right now is that we won't see AI stop in the United States as some worry, but it definitely is going to become a little bit more difficult to build than it is right now. Alex Kantrowitz is author of the Big Technology Newsletter and host of the Big Technology Podcast. Alex, always appreciate your time. Thank you. Thank you.

8:58John Ternus opened his tenure as CEO of Apple with the company's biggest change to the iPhone in nearly a decade. At Apple's Surprise and Shine event yesterday, the company unveiled a$2 ,000 foldable iPhone called the iPhone Duo. It's about the size of a passport with a display 80 % larger than the most recent iPhone. Apple also announced the iPhone 18 and the 18 Pro, the AirPods 5, and the new Apple Watch series. The event was John Ternus's first product launch since taking over from Tim Cook on September 1st. Apple shares closed roughly flat. So, joining us to discuss the beginning of John Ternus's term and the foldable phone, we are speaking with Patrick McGee, award-winning journalist and author of Apple in China, the capture of the world's greatest company.

9:48Patrick, welcome back. Good to see you. Let's just start with your reactions to a$2 ,000 foldable iPhone. The iPhone Duo. I mean, look, I think it's a marvel of industrial engineering and hardware engineering. I mean, it looks fantastic. I've been able to talk to some people who've been playing with it today in Cupertino, and they do say it's fantastic. Obviously, the question is, are people going to pay$2 ,000 for this. And I think the two scenarios are either that it underwhelms or that Apple is already sort of baking in that it's not going to have high sales. And so it's going to meet low expectations, as it were.

10:25You know, the thing I think about is when Steve Jobs introduced the iPad in 2010, he was able to credibly market it as an entirely new product category, saying it's more portable than a desktop, but it's bigger than an iPhone. And that felt intuitive, and they've sold hundreds of millions of them since. Is the iPhone duo trying to position itself like that? That's the sense I got from the new CEO, John Ternus. But I'm not really convinced. I'm someone who has an iPad with a keyboard. And so if I need to get a little bit of work done that's not on the laptop, that's what I'm going to do. If you're in that situation, why would you also have an unfoldable phone?

10:59I just don't know what service it serves, even though it's cool. I think the question for Apple is this their next generational product? And that's something that a lot of people have been waiting for. It was supposed to be the Vision Pro, or at least that was the way it was kind of marketed. The Vision Pro, based on my understanding, has been something of a disaster. They don't really report it properly in their earnings, I think because it's not doing well. I mean, what legacy do you think this product will have? Is this their ticket? Are they betting on this thing being the next era of Apple?

11:37I don't think they are. I mean, to be clear, I think the 18 Pro, the new version of the best iPhone, will do just fine and people will buy that. I'm probably going to buy that. So, I mean, for them not to have market hits isn't to say that they're in any way failing. You know, the closest sort of precursor we have to this is the iPhone Air last year. Now, it didn't do well in terms of sales, but it sort of allowed them to really push what the iPhone Pro was doing, right? And so they could sort of have this compare and contrast and really sort of like, you know, if you were someone that complained that the iPhone Pro had just gotten too big and durable looking and you didn't need that many cameras, it was like, look, here's something that, as they would say, is lighter than air.

12:20And even though it didn't sort of work in terms of sales, I do think it worked in terms of the marketing pitch. And you're right. Apple has been so dogged by these reports that it doesn't really innovate anymore that coming out with this product sort of gives a halo to the company, even though I don't think many people are going to buy it. So by saying it's not going to be a big sales hit isn't to say that it's a mistake to have it in the market. What do you make of John Ternus' start? How do you find him as a communicator, as a leader? And what kind of direction do you think he's going to take this company?

12:52You know, I'll be shocked if anybody says they feel like they learned a lot about John Ternus, either as a person or as a communicator, let alone as a leader today. And that's because Apple, since COVID, has done away with keynote presentations, live keynote presentations. It was a big deal when Tim Cook had to take over after the passing of Steve Jobs and really do his first keynote. You know, you could tell he was nervous. It's still up there on YouTube if you want to follow. There's nothing like that for John Ternus. Who knows how many takes took place before he sort of nailed it, right? He wasn't talking in front of a live audience other than sort of waving a little bit at the start.

13:24So I get why Apple has these like cinematic experiences. But Steve Jobs never would have tolerated that. Right. And to be clear, that's not necessarily criticism. I mean, it's not Steve Jobs' company, obviously. And so they're able to pack in more information and they really de-risk things. They don't have, you know, hundreds of reporters around the world all talking about some flop. Right. Some error that took place on stage. Like there's all sorts of reasons why it makes sense to have a cinematic thing. But they've just sort of taken the like the oomph out of it. Right. You sort of want to have the live demo.

13:59And it's really interesting that OpenAI does do these live demos. Right. They're a little sort of a little quirkier. But it's almost telling you that like a startup can take those kind of risks in a way that a Goliath cannot. He has a lot to live up to. I mean, his predecessors were Steve Jobs and Tim Cook, two very different CEOs, one a legendary visionary founder, the other kind of, some would say boring operator, but who also created more shareholder value than I think anyone who took over as a CEO ever in history. I mean, what kind of CEO do you think he will be? Do you think he will be more of that visionary product person or more of an operator who just keeps the trains moving?

14:46I kind of think neither. I mean, to be clear, Apple is going to keep the trains running on time. I just don't think it's John Ternus that's going to be making that happen, right? Sabi Khan is the chief operating officer. He's been at the company since the late 1990s. You know, he's an accolade of Tim Cook. He knows what needs to be done and he can get it done. what's going to be more complicated is if they really do position out of China and grow their capacity in India, right? So far, that's more hype than reality in terms of what needs to be done. And Tim Cook is basically staying on, right, to be negotiating with New Delhi, Beijing, Brussels and Washington.

15:18And so to a certain extent, I don't think we're going to see John Ternus make his mark, certainly not in the geopolitical scope of things, for several years, either until Tim Cook has sort of just moved on a little bit or has officially retired from the company. So I know that everyone wants to sort of, you know, put the first performance of John Ternus on a pedestal and sort of, you know, ask what this means for the future of the company. And I just think anticlimactically, we're actually not really going to know what Ternus' impact is for several years, right? Not a very good sort of media headline answer, but it's still Tim Cook's company and tells something dramatic changes.

15:53Like, not to go off on a massive tangent here, but Tim Cook fired Scott Forstall, who was kind of like a mini-me of Steve Jobs, roughly 15 months into his term. That was his like new sheriff in town, you know, template or approach. And it's going to be tough to see John Ternus make any sort of move like that. In other words, something where Tim Cook wouldn't have done it, but John Ternus would, you know? I don't know that we're ever going to get the board minutes about that sort of decision. But until we have that, how do you really judge whether it's Ternus' company or not? Final question. Will you be buying the iPhone Duo?

16:25You know, to be honest, if we spoke yesterday before the before the product had even come out, I would have said yes, just as a tech journalist. Like I want to I want to be, you know, be familiar with how it works and everything. Right. And now I've seen it. And even though it actually sort of works better and looks kind of cooler, like the continuity feature of how you open it, and it feels like as impressive as the hardware. And yet when I'm really thinking about it, you know, I don't want the base model. I need to double the storage. It means I'm buying a twenty two hundred dollar device. and I just don't think I'm there.

16:54I just don't know what purpose it serves in my life. I think it looks really nice, but it looks too big. I'm not sure about the size of it. Putting that in your pocket every day, I'm not sure that would be worth it to me. But we'll see. Patrick McGee is journalist and author of Apple in China. Patrick, always appreciate your time. Thank you. Thanks, Ed.

17:21After the break, more evidence of AI's concentration risk.

17:46in the decision-making at all. Good news, Granola has solutions for both. Granola is the AI-powered notepad that makes your meeting notes useful. It captures what happens in your meetings and turns it into clean, structured notes with decisions and action items pulled out and made easy to find. The best part? Granola can integrate seamlessly with how you already work. So there's no setup, no configuration, no friction. You get to stay present in your meetings and still get clear, actionable notes without any extra work. Once you try it on a first meeting, it's hard to go without. You can try it totally free.

18:17Just head to granola.ai slash markets. That's granola.ai slash markets to get your time back. Go to granola.ai slash markets and try it on your next call.

18:30Support for the show comes from NetSuite. Successful business owners know how crucial it is to keep up with changing technology. So when it comes to adapting AI for your business, it couldn't be easier with NetSuite next. Next, you probably know NetSuite, the AI-powered business management suite that securely connects all your data. It unites your financials, inventory, commerce, HR, and CRM into a single source of truth, and it's trusted by over 43 ,000 customers. NetSuite Next is the next big leap in how business gets done because AI is built into everything you do. It automatically surfaces custom insights throughout your day.

19:03AI agents work alongside you to solve problems and handle routine work. And anytime you have a question about anything, just ask like you're talking to a colleague. Whether your company earns millions or even hundreds of millions, it's time for NetSuite Next, where your business meets AI. For the first time ever, you can try NetSuite Next for free. If your revenues are at least in the seven figures, go to netsuite.ai.com. Built for every industry, ready for every boardroom. netsuite.ai.com.

19:34Support for the show comes from AG1. Now that it's fall, everyone's calendar is about to fill back up again with back to school, sports, and earlier alarms. When things get busy, it's easy to let good habits fall through the cracks. But here's a suggestion for a healthy habit. You might actually enjoy AG1. Just one scoop and eight ounces of cold water in the morning can help you maintain energy, support gut health, and immune system all day long. And the NextGen formula delivers more than 75 ingredients clinically shown to support gut health, fill common nutrient gaps, and improve key nutrient within three months.

20:06I use their little sticks when I'm on the road just to kind of shore up what I think I'm missing. As when I'm on the road, quite frankly, my nutrition sort of falls down. AG1 can help you keep one thing consistent. High quality nutritional support every single day, no matter where you start your morning. For a limited time, save 20 % on your first subscription order of AG1 Next Gen or AG1 Pro at drinkag1.com slash markets. That's drinkag1.com slash markets.

20:41We're back with Prof G Markets. We're months away from what could be two of the largest IPOs of all time. Both OpenAI and Anthropic have confidentially filed with the SEC to go public, but still, we know almost nothing about these companies' finances. Until investors can see those filings, many questions remain about the business of AI. Our next guest, though, might have some answers. Corporate finance platform Ramp updated its AI index yesterday, tracking how much American businesses are spending on AI. And the results are quite staggering. According to the report, about 80 % of all enterprise revenues at OpenAI and Anthropic come from just 1 % of businesses.

21:23And those businesses are now pulling back on spending. So here to break down this data, we are speaking with Ora Karazian, the lead economist at Ramp. Aura, thank you for joining us on the show. You've just published this index and the report is titled, quote, Cracks in the AI Thesis. And this number, 80 % of enterprise revenues coming from just 1 % of the businesses at OpenAI and Anthropic, seems to be a big part of that. Could you walk us through that number, what it means and why it's important? In any case, when you're analyzing these kinds of new and nation technologies or analyzing anything with a data set, you want to avoid focusing too much on one specific metric.

22:05It's very natural to see some noisy data and you don't want to overly highlight one trend. What's different about this is that we are seeing a number of metrics now move in a negative direction for the AI model companies. And we are seeing a great amount of concentration risk as far as their customer base. So if you are someone who is involved in the AI trade, that might be something that you want to take a look at that I think the market is relatively underpricing. So you're right to point out a lion's share of the AI company's enterprise revenues is coming from a specific segment of customers, 1 % of customers driving 80 % of that spend.

22:42A bigger problem is that within that, those companies tend to be very correlated with each other. It's by and large high growth tech companies, other AI startups as well. Now, that's fine as long as AI spend continues to increase. But then our latest finding today is that the top 1 % of spenders in terms of per employee per month spend are also slowing down their spend. So came down about 10 % month over month from$8K per employee per month down to$7.2K. Still a lot of spend. But then within that, you start to see a couple other negative trends. People can see that metric and they think, oh, it's because of summer seasonality.

23:24And that is somewhat true, but it is not a full explanation. Token volume and usage of these models actually rose in the same period. And there are a couple other metrics that then inform, well, why would spend trend down? One is that open-ain and anthropic are in this price war. So both introducing frontier models and cutting prices for them, trying to get the best model at the best price, simultaneously introducing highly performant cheaper models at the standard and light tier. And then you see an increasing share of that volume of AI spend volume by businesses going toward the standard and light models as opposed to the sort of frontier models.