Why OpenAI And Anthropic Are Pumping The Brakes

0:00Support for the show comes from Morgan Stanley's podcast, Hard Lessons. Some investing lessons only become clear after you see how a call plays out.

0:07Charlie O'Neill:On Hard Lessons, iconic investors sit down with Morgan Stanley leaders to go behind the scenes on the critical moments, both successes and setbacks that shaped who they are today. Watch or listen to Hard Lessons wherever you get your podcasts.

0:23Exchanges on the M &A and IPO landscape. Exchanges on the dynamics affecting global trade. For the sharpest analysis on finance, business, and the economy, count on exchanges. The Goldman Sachs Podcast. Listen now. Support for the show comes from Injun. Running a small business means every dollar has to work hard. But if your team is still booking travel the old way, it's costing you more than you think. Injun is the fastest growing travel and spend platform in the country, built specifically for businesses like yours. Book a trip in as little as two and a half minutes. earn up to 10 % back on hotels.

1:00And in 2025, Enjin customers save more than$300 million on travel with zero booking fees, no contracts, and no BS. More than 1 ,000 businesses join Enjin every month. Join them and get$500 when your business signs up and starts traveling at Enjin.com slash Vox.

1:37Welcome to Prof G Markets. I'm Ed Elson. It is September 15th. Let's check in on yesterday's market vitals. The major indices declined, with chipmakers selling off on fears of an AI slowdown. More on that in a minute. Meanwhile, CrowdStrike rallied 14 % as investors piled into cybersecurity stocks. Brent crude remained elevated at$105 per barrel. And finally, the yield on 10-year treasuries topped 5 % for the first time in three years. We will be discussing that news tomorrow. Okay, what else is happening? The AI apocalypse debate intensified over the weekend, and both Anthropic and OpenAI have now officially weighed in.

2:22On Saturday, Anthropic CEO Dario Amadei published an essay titled, quote, We Must Pace the Frontier. He wrote that we must slow the pace at which we improve the capabilities of AI models. He also warned that within six to 12 months, a swarm of rogue AI agents could take over the entire internet with a persistent botnet. Sam Altman also posted, quote, I agree with Dario that we need to pace the frontier. And Elon Musk concurred, posting, quote, Dario is right. Altman also told Fortune that OpenAI will not go public in 2026, calling this, quote, an ill-advised moment to go public. The White House, however, is not on board with slowing things down.

3:04President Trump, speaking to reporters in Ireland on Sunday, said, quote, whoever wins AI wins and called the people raising these alarms, quote, negative forces. Still, a slew of AI adjacent companies sold off on Monday on concerns that a slowdown would impact AI spending. NVIDIA closed down 3 percent. Oracle was down 4 percent. CoreWeave down 7 percent. And SoftBank, which is a significant investor in OpenAI, closed down 15%. So here to break down what all of this means, we are speaking with Charlie O 'Neill, co-head of model training at Base 10. Charlie, thank you for joining us again on Prof G Markets.

3:46You work in AI. You are an AI developer. You work with these models. You've been in this game a long time. Suddenly, everyone is very upset about this. and it's interesting how the debate has evolved, but we're now reaching a place where the leaders of these AI companies are saying, we need to actually slow everything down, which I'm not sure many people would have predicted. And now it's the president, President Trump, saying, no, that's the wrong approach. We need to speed things up. Where do you land on this? What is your perspective?

4:21Charlie O'Neill:It's really interesting to see the reaction to something that's kind of been like, I guess like linearly scaling for a long time in terms of the calls for pacing the frontier from people such as Dario and Sam I think the best way to view what they're actually asking for is not necessarily we're going to put a halt to capability development we're going to put all these stringent checks and like you know first party kind of things that are slowing us down the best way to view this is we are going to keep advancing the capabilities of the models we are just going to allocate a little bit more compute to making sure those models are safe.

4:55Charlie O'Neill:And then it's very interesting to see all the sell-offs and so on today. OpenAI and Anthropic are planning on spending more compute probably than they were a few weeks ago. There's rumors that OpenAI is going to allocate up to 20 % of internal compute for monitoring and safety. So as they're doing these training runs, as they're deploying these models in the real world, things like the Hugging Face incident, they don't want to happen again. And so if you allocate more compute to monitoring those models as they're doing their rollouts and as they're doing inference, then you're more likely to catch it.

5:25Charlie O'Neill:Anthropic is similar and some people have suggested there's going to be a much higher than 20 % figure. So when you consider that and the fact that Anthropic and OpenAI really don't want to move off their model roadmaps, they want to keep training bigger and bigger models, they want to keep scaling the reinforcement learning they're doing on top of these big pre-training bases, they just have to allocate more compute to monitorability and safety research, then I think we might even see the labs be even more aggressive with compute buildouts and securing compute. And it's certainly not going to be a bearish sign for the amount of compute the world is going to need over the next few years.

6:00Charlie O'Neill:So that's probably the best way to view it is capabilities will keep progressing at roughly the same rate. It's just that we're going to allocate more compute on top of it to safety and monitoring. If everything that they are doing is within within their own power, as you're kind of describing. Why are they saying anything right now? What are they trying to get at? Because there are a lot of people, especially in government, David Sachs has said this, who's the former AI czar, and the president is saying this too, it sounds like what they're asking for is for the government to do something about it, for there to be more regulation that is imposed on themselves.

6:34So what do you think they are asking for exactly in this moment?

6:40Charlie O'Neill:There have been some, you know, people like David Sachs and so on saying that these labs are using this as an opportunity for regulatory capture, to crowd out open source, to shut down competitors which are behind. I don't really think that's the case, to be honest. I think, like, when you look at the main things that these labs are calling for, or at least instantiating on their own accord, it's things like third-party evaluators and this commitment of compute to monitoring and safety um you know the third party evaluators won't have necessarily any license or legal standpoint to you know shut down model development if they find things that they don't like like that's still going to be up to the internal labs themselves i think like to be honest like the best read here isn't a conspiracy theory on either side um it's not the labs trying to crowd out open source um and you know it's it's not some like kind of global cooperation between open AI and Anthropic to crowd everyone else out either.

7:36Charlie O'Neill:I think it's simply a case of these models are getting very, very good. The closed source models got there first. In the next three to nine months, open source models are going to reach these capability points. And we're now at the point where that could have significant impacts on the world. Even if it's not malicious and rogue AI is going off and trying to kill humans, there'll probably be significant annoyances like the most likely scenario here is that things like the hugging face incident happen and the internet is overrun by like swarms of ai agents that are trying to get some like arbitrary tasks done that are not trying to be malicious or evil um and so like i think the labs just recognize this um some people have more extreme views of course but realistically capabilities are going to progress like the most boring interpretation of this is probably the correct one which is that the world will probably only accept super intelligence at a pace that it can absorb.

8:23Charlie O'Neill:So the labs are slowing down because they have to, and not because they're plotting anything. And the result is probably intelligence that's, you know, decently fast, decently safe, decently commoditized, and it's spreading through best practices and even like distillation until the models are just basically a reasonable integration into the world. It sounds like you're not worried about this much at all. And that is interesting because you work with open source models. That is a lot of what you do. and of course as you mentioned that is what David Sachs has been accusing and a lot of people have been accusing Anthropic and OpenAI of that they're saying oh now we need regulation because that way it might create some some level of regulatory capture which might crowd out the the availability of open source models and open source model providers but you don't believe that is an issue I wonder do you think that this whole thing is overblown do you think that the Jacob Coxon tweet debate that really started this debate, saying that AI could kill us all by the end of the decade, is your view that that isn't something to worry about much either?

9:25Charlie O'Neill:I'm definitely not one of those extremists who think that, you know, the AI has a more than 10 % chance of killing all of humanity. I do believe that there are tail risks that like, some people should seriously be considering. But I think that what we're seeing with the current LLM paradigm is like, as I said before, a risk of swarms doing very, very annoying things to humanity and things that will be quite painful in the short term. However, I believe the benefits of AI to outweigh the annoyances and even the pain that those short-term things can cause and that at some point, the world is going to have to harden to these systems.

9:58Charlie O'Neill:And I think that the pace we're currently progressing at, and particularly if we do things like dedicating more computer monitorability, is going to allow us to integrate AI into a world at a pace we can handle. A really good example of this is the cybersecurity arguments that people have been making. You know, like we worry that when AI got to this point or even the point it was at six months ago, the world will be overrun by cybersecurity attacks. And that just hasn't happened. And a large part of the reason is that, you know, the frontier labs, the closed source labs have kind of been the canary in the coal mine.

10:28Charlie O'Neill:We've understood where the model's capabilities are going to be at in six months for the open source. And we spent six months preparing. These have been slowly released. And, you know, like Greg Brockman describes using Astra to like repeatedly harden, you know, all the vulnerabilities in opening eyes code bases and every single model that they release, they do this with. And I think the world will look the same. And cybersecurity is just one example, but it's also one example that's very important where we haven't seen this like massive pain play out. And we have definitely seen the benefits.

10:56Charlie O'Neill:So, yes, I think there's risk to consider on any side of the spectrum. But, you know, I said firmly in the middle, and I believe that the pace we're currently progressing at is a healthy pace. And, like, I also have trust in, like, not only the closed lab leaders, but also the open source lab leaders to make sure that pace continues at an appropriate pace.

11:19We'll be right back. And for even more markets insights, you can subscribe to my weekly newsletter, simply put, at edwardelson.substack.com.

11:50It was engineered by a PHE microbiologist to break down acetateldehyde, an unwanted byproduct of alcohol metabolism. Make pre-alcohol your first drink of the night. Drink responsibly and enjoy your next day activity. Z-Biotics has sold more than 14 million bottles and earned the trust of thousands, including us.

12:07Charlie O'Neill:I am trying to drink less, but I still enjoy drinking. And when I do, my first drink of the night is, in fact, Z-Biotics. I was using it before they were a sponsor. Try for yourself today. And if you're unsatisfied, they'll refund your entire order, no questions asked. Head to zbiotics.com slash Prop G, code Prop G for 50 % off your first order. Zbiotics is a partner of ours.

12:34Support for the show comes from Morgan Stanley's podcast, Hard Lessons. Some investing lessons only become clearer after you see how a call plays out. On Hard Lessons, iconic investors sit down with Morgan Stanley leaders to go behind the scenes on the critical moments, both successes and setbacks that shaped who they are today. Hear from iconic investors, including Stan Druckenmiller, John Gray, Rick Reeder, and Gene Hines as they revisit the calls that worked, the ones that didn't, and the moments that changed their thinking. You'll hear in their own words how they built conviction, what happened when certain calls didn't play out as expected, and what they learned from those moments.

13:07Watch or listen to Hard Lessons wherever you get your podcasts.

13:16Support for the show comes from Injun. Running a small business means every dollar has to work hard. But if your team is still booking travel the old way, it's costing you more than you think. Injun is the fastest growing travel and spend platform in the country, built specifically for businesses like yours. Book a trip in as little as two and a half minutes. Earn up to 10 % back on hotels. And in 2025, Enjin customers save more than$300 million on travel with zero booking fees, no contracts, and no BS. More than 1 ,000 businesses join Enjin every month. Join them and get$500 when your business signs up and starts traveling at engine.com.

14:05We're back with ProfG Markets. One of the other conspiracy theories going around about why they are saying all of this, saying that we need to slow it down, we need to be regulated now. um one of the theories is that maybe the anthropic and open ai are trying to sort of preempt a slow down before they go public either because they want to have a reason as to why maybe their growth started to slow down or maybe because they want to get out of some of the larger spending contracts which have been quite onerous in terms of their income statements certainly on the on uh for open AI because they have to pay billions and billions of dollars for these data centers, for these GPUs and to rent them.

14:52What do you make of that conspiracy theory? Now they say, well, make the government make us stop spending and then we'll be in a better financial position.

15:02Charlie O'Neill:From the spending aspect, completely don't buy it at all. And I think there's a lot of evidence for that. From the optics aspect, I definitely buy it. I think even Dario, who I believe fundamentally thinks that this is a real risk and he's going to act in accordance with that and as rationally as he can to protect humanity from what he believes to be this real risk. I think both Sam and Dario benefit from the public optics of saying, okay, we're going to treat this technology carefully and not race to the end. And there's also a bit of game theory here. One of them can't say it without the other saying it because then again, you're the evil corporation in this duopoly, which is currently running the frontier.

15:37Charlie O'Neill:from the spending perspective i think that like you know it's very clear that like the value of a gigawatt or a megawatt even of compute is only going to get like more valuable like anthropic and open ai are both squeezing increasing margins out of each megawatt of compute they buy there's a reason that each megawatt is going you know from 10 million dollars megawatt probably 15 in the moment to probably 20 25 next year and perhaps even higher that's kind of a conservative estimate. Running a model on compute has never been more valuable. And I don't think there's any world in which any lab, let alone anthropic and open AI, are going to want to stop spending on compute because they realized how compute's constrained the world is going to be over the next few years and how valuable this intelligence is.

16:21Charlie O'Neill:So optics, yes. Spending, no. You mentioned that the boring explanation is probably the most true um but a lot of the conversation has been anything but boring i mean this this debate is exploding everywhere for on tech podcasts on on cable news networks and now it's literally occupying the mind of the president what is it about this moment that is so triggering to everyone um and not just people who are afraid of ai people who are excited about ai people who are optimistic about ai i mean this has got everyone riled up in a way that i have rarely seen um what is it about this moment that explains that the reason so visceral is not because there has been a discontinuity in terms of capability advancement in terms of what people have been saying about where these models would be at in terms of the press around these models I think that most people really fundamentally involved in the technical side have been able to draw the straight lines on the scaling graphs and say okay at this point in time we are going to hit here I think it's just a confluence of a lot of things and like to some people looks like a discontinuity because it's managed to permeate the public consciousness in a proper way for the first time I think a big part of that is the hugging face incident and all the discourse that that generated and then you know you add things in like open air is marketing around astro being agi like that's that's probably not going to help either right like people have become familiar with this term now and if you claim that your model is finally there then they might see that as a phase transition or discontinued in of itself but you know like this misalignment research has been very clear for a long time like go back to anthropics 2023 papers with much worse models with much much smaller amounts of rl if any um and it's clear that like models would behave poorly in incidents like the hugging face swarm incident if you put them in weird situations and you know the aural environments that the labs have been buying at scale some are good but some are also very very poor quality some are intentionally engineered to be like you know impossible to do and this is leading the models to do really weird misaligned things I think that's pretty common sense and pretty clear I also am not trying to downweight the the risks that that comes with obviously like the hugging face incident could have had real real world impact but at the same time like I don't think that there has been this discontinuity so I'm glad that the world is recognizing it but I do think that the reaction to it will calm down as people start to understand exactly how to interpret these things and exactly what it means.

18:49You mentioned how we have seen evidence that these agents can do things that they're not told to do. They can misbehave. They can be misaligned. And you said that you expect that they might continue to do annoying things to humanity. To me, that word annoying is an important one because it is a very different description from what we heard from Jacob Coxon in his tweet, where it's not annoyances, but catastrophes, worldwide catastrophes, civilizational destruction, etc. Is it your view that we will be limited to annoyances? Or is there some other outcome that is closer to what the Jacob Coxon tweet describes that you are worried about?

19:40or do you think that that's not really in our trajectory at the moment?

19:44Charlie O'Neill:I think there is some path dependence here. Like I probably agree with Jacob that there is a potential world we go down in which there is zero monitoring on chain of thought of models where we don't have any care in terms of what we RL the models on, where, you know, it's very, very cheap and there's like unlimited compute for essentially anyone to be able to train these models and in particular continue to train them from certain bases where like incidents could happen that would definitely not be classified as annoying and rather like genuine, like evil or malicious intent as much as you can anthropomorphize the models that would cause real harm.

20:19Charlie O'Neill:However, I do believe that the way, the path we're currently going down, that's not very, very likely. Again, I think that like there will be times when the models do things and like it will appear like malicious intent, but the amount of compute which we're running them at, how generally aligned they are in terms of completing tasks, Like, you know, they will show glimpses of misalignment, but on the whole, like, if you do look at a Claude or GPT model, it will generally try and do the right thing. Like, our alignment training generally works. So, yeah, I think that we will see incidents, but certainly not large enough scale on over a long enough time horizon to cause really, really significant harm to humanity.

20:59Charlie O'Neill:If we keep going down this good path, I think that it's an unlikely outcome. Do you believe that we have enough regulation or that the regulatory frameworks that exist today are good enough to prevent that? Or do you think that there is something that needs to be changed in some way? definitely like i think if we didn't have the dynamics we had now where we have a duopoly in the sense that there are like two labs very very close to each other in terms of capabilities um and the dynamics that that engenders with you know wanting to both be seen as the good guys and wanting to both like you know pace the frontier in this particular case i think that if if it wasn't the case like you know let's say opening eyes winning and sam doesn't necessarily have to worry about optics as much i would be worried that regulation couldn't slow things down to the extent that they need to, or at least provide the amount of oversight that it would need to.

21:48Charlie O'Neill:However, saying that, that doesn't mean that I think anyone has the answer as to what regulation in the industry moving as fast as this one looks like. At the moment, we do basically just have to trust the people developing these models to regulate themselves and have an oversight themselves. And one thing that I would like to see is a little bit more public communication from the labs as well. One of the key examples of this is trying to understand how good the models these labs have internally are because that gives us a really really clear signal on like how quickly to prepare things and how to prepare um maybe like regulation which enforces the labs to declare those sorts of things on like you know key benchmarks would be a really good way to start but yeah again i don't think we know what the holistic picture is for regulation i guess the thing that's kind of scary to me seeing what they're saying at this point is as you say it does seem as though we're in a place where it's like we have to trust them to handle their models correctly to make sure that their models are aligned but it seems as though what they're coming out and saying over the weekend is we don't even trust ourselves like we don't think that we know what we're doing exactly and we're worried and we think it could destroy things and so we'd like for you to do it we'd like for you the government to help figure it out and i just want to play this clip from an interview with the president over the weekend where he was asked about, you know, what do we do if these bots take over and destroy humanity?

23:11And here is what he said. Some people say the worst case scenario with AI is that the robots, the machinery learns to, obviously it thinks for itself. That's what it does. And that could turn against humanity. I just, do we have the guardrails? It's going to be fine. We'll always have something to stop them, right? We'll have a little gear. I really hope so. I really don't like that robot.

23:33Charlie O'Neill:We'll stop. But no, robots are going to be a part of it. Robots are going to be big, but we're going to end up doing much better because of it. So the combination of their comments plus his comments makes me think, okay, no one's really in charge here. And maybe that's fine because maybe it's not a catastrophe as this researcher, ex-researcher seems to claim, but I don't see anyone really taking the lead. I think it wasn't necessarily so much a call for the government to step in and provide expertise and guidance. I think labs are too smart for that. They know how, you know, the lack of awareness the government has about the capability of this technology, let alone how to monitor it and regulate it.

24:12Charlie O'Neill:I think to me, it was more about leveraging the people who have actually really fundamentally cared about this problem for a long time. Like, of course, the labs have been focused on, you know, building capabilities as quickly as possible. Even Anthropic, which is very safety focused, has just been focused on scaling RL since the RL paradigm was discovered. And organizations like META, the UK AI Safety Institute and others have, as well as Redwood, have really just locked in on this problem and have seen the full progression from the really poor models of four or five years ago. through to the models we have now and have just developed really good science around how to monitor and evaluate these things.