Pacing the Frontier: What’s next for AI?

0:00Hey, everybody. It's Ronnie. It's Thursday, September 17th, and I'm here in our New York studio with a very special guest, our internet analyst, Ross Sandler, who's joining us from our podcast studio in Silicon Valley. Ross, welcome back to the Barclays Brief. Hey, Ronnie. Very excited to be here. Thank you. I'm glad to be back. This is, I think, our round two of the AI debate. Well, it's great to have you back on the podcast, especially with everything happening right now. And wow, has it been a busy and confusing few weeks in the world of AI. Look, we're all scrambling a bit to get our arms around what it all means for this critical technology and market driver.

0:37Can you help frame what pacing the frontier is and how we got to this point? Yeah. Yeah. So pacing is just sort of a fancy word for adding a bunch of extra kind of safety and what the AI community calls alignment monitoring of what's going on in mostly the training process where they build these AI models, but also in the inference process, which is once a model is released to the public and we're all kind of using this stuff. It's basically just making sure that both in the training and in the inference stage, that these AI models aren't doing things that they're not supposed to be doing. So pacing is essentially just a fancy word for slowing it down a little bit to then add all this safety and monitoring on top of what they're already doing.

1:25So let's spend a little bit of time on how we got to this point. I think everybody has read about the OpenAI hugging face incident. Can you just tell our listeners what they need to know about that? and what happened and maybe how it helped catalyze this recent movement? Yeah, and I think it's worth like kind of stepping back a little bit even before that. So what's happening is you've had multiple kind of huge moments in AI over the last, I call it four years. First, there was ChatGPT being released in late 2022. And then you've had kind of a series of like pretty big breakthroughs, like the reasoning model breakthrough.

2:00And more recently, these models and these products have gotten very good at code writing and particularly at cybersecurity. And so part of what's going on is that we're just on this continuum of AI improving, and you're getting to a point sometime earlier this year around the release of Anthropics Mythos model, where we reached a new plateau, a new level of capability, whereby these AI systems are far more capable in the field of code writing and cyber. And so Hugging Face and what happened here was in the process of training a model, which they haven't even yet released, basically the model was being tested on a certain benchmark.

2:44And it is supposed to be somewhat contained in this secluded environment called a sandbox. And what happened was during these tests, which were designed to kind of test the limit of the model cyber capability, it escaped the sandbox, went out onto the Internet, broke into Hugging Face's infrastructure, which actually had the answer to the test that it was supposed to be completing. And in the process of doing that, it was like kind of self-replicating and it created a bunch of agents that helped with hacking into Hugging Face. And so the incident was surprising on like many vectors. It wasn't supposed to do any of this.

3:23Were these the swarming agents everyone's talking about? Yeah, there's a couple things going on. The agent swarm, like, you know, the beehive swarm. Swarm. Yeah. So apparently there was like 700 agents that were involved in breaching Hugging Faces infrastructure and finding sort of the answers to this test that it was supposed to be completing. And so, yeah, that's, that's the agent swarm. It begs the question of the sandboxing that OpenAI set up, how tight was that? And, you know, clearly there was like some exploits that, that happened here that the model figured out and broke containment. Hugging Face is another AI company, which is in the process of being acquired by NVIDIA.

4:05So it's sort of like a no harm, no foul. Everybody like says they're sorry and hey, we'll get it right next time, which is sort of what happened here. And so that's where the pacing and guardrails come in. Yeah. So it isn't just the Hugging Face incident. There's been about, I don't know, maybe a dozen or so of these of various levels of severity for both OpenAI and Anthropic that have happened, you know, somewhere, like I said, it started in basically the spring or summer of this year where these models got much more powerful, much more capable, particularly around cyber. And so you kind of have to like prepare the world.

4:44You have to kind of like get everything a little bit more tightened up in both the training process. And then also once you release these models, because they're far more capable than they were a year ago, you know, we're just in a new level here in terms of AI capability. And hence, like the bar is being raised in terms of what these AIs can do and also what the world needs to do to prepare for this. No, I definitely feel that in the quality of the products and the models that we're using now. Let's move it back to what this means for the AI labs at the frontier. From a growth rate perspective, does this mean lower growth rates or slower growth rates off a highly accelerated pace?

5:23How do you see this all playing out as we pace the frontier for these AI labs? Yeah, I think the revenue growth shouldn't be impacted at all, really, in the near term, because most of what's happening, like there's the training process, which is like, you know, these models that are doing these, you know, these breaches were sort of things that have yet to be released. The ones that are out in the wild, which would be an OpenAI's case, Astra, which is like their GPT-6, in Anthropik's case, Fable-5, these are kind of 5.1. These are sort of the state-of-the-art models that are out there. And those are mostly like pretty locked down.

5:59And, you know, us as the end user are kind of going about our business and kind of using these products. They're not really causing any real problems. So it's just in the training process for this next generation where you're starting to see some of these incidents. And so the revenue that the labs are generating today is mostly just a function of the diffusion that's happening with AI products kind of getting out there. Most of it is probably for models that are either Fable 5 in Anthropics case or Astra in OpenAI's case, or maybe even like the generation one step prior to that is currently what's being set up inside of these big companies.

6:36And so the revenue is sort of lagging in terms of the revenues supported by models that are lagging, the ones that are causing problems. I think if you kind of play this out, what pacing could mean in the future is that the cost of training and inferencing next generation models goes up for all this extra safety monitoring that needs to be done. And then the labs will either have to absorb that cost or they'll have to like kind of pass it on to the end customer sometime next year in the form of like higher token prices. And so could actually mean revenue goes up once you get to these next generation of models.

7:10But yeah, revenue seems to be doing quite well. I think we crossed the$100 billion mark sometime in the first half of this year for AI Lab ARR. We're going to end the year probably close to a little over$200 billion of ARR. So the revenue seems to be up and to the right. In three years? That's anybody's guess. So who wins and who loses here, just in general in your eyes, or because of this new pacing dynamic? Yeah. So I think this is an important point. If you look at like where the incidents are occurring, it's right now mostly just open anathropic. One could argue, OK, those guys are a few months ahead of the other Western labs and a few months even further ahead of the kind of open weight community coming out of China.

7:56And so maybe because they're training models that are like far more powerful and far more capable, they're the ones that are kind of the first ones that you would see running into some of these issues. They also have the most compute to do these big training runs. And so you're seeing it where you should be seeing it. It is also interesting, though, that, you know, Google, who has DeepMind, Meta, who has like completely rebuilt their internal AI lab called MSL, like their training models right now that are pretty much neck and neck with where OpenAI and Anthropic are. And yet they're not really seeing incidents.

8:31It's interesting to me that Google and Meta and even to a lesser degree, SpaceX are training models that are pretty much close to what OpenArientropics are doing. And they might just be taking a few extra steps to have the alignment, the safety, the monitoring up and running because you're not seeing those training runs kind of break containment and have all these incidents that we had with hugging face. And so it could be that if, you know, the two leading labs have to pace and maybe slow down a little bit and kind of implement a bunch of these new safety measures, that could mean that like Google, Meta, SpaceX, even the Chinese could catch up for some short period of time.

9:11That's interesting. So look, a lot to digest in terms of events in the last few weeks. Does all of this make you more optimistic, less optimistic? We spent a lot of time on the power of AI for positive economic transformation. Does this change the pacing of that in your eyes in any way? Well, I'm definitely in the AI pilled camp, which means I'm very positive at all times. But I think like stepping back, I know the hugging face and all this pacing and safety and security, etc. are important topics that we need to kind of work through. But the other thing that's happening is as models get more powerful and as we bring more compute online, it's just able to do quite a bit more, like some of the big breakthroughs that we were hoping AI would kind of deliver are starting to happen.

9:54So it kind of got lost in the shuffle, but OpenAI solved this like millennium prize math problem a couple of weeks ago, Navier Stokes, that if you look at what happened here, they put the swarm that we were talking about before, 10 ,000 concurrent agents working on this math problem. They put them on that for 88 hours, so about three and a half days. And the swarm was able to solve this historic, incredible math problem, like a big breakthrough in about three and a half days. And if you add up what that would mean in human years of kind of like nine to five work by a mathematician, it's like 5 million or something human years of work being done in just three days.

10:36I think this is important because what's going to happen next is that as these models get more capable, as we as an industry bring on more compute, you're going to start pointing the AI systems at huge problems within science, math, medicine, biology, robotics, like some of these new categories that are cropping up. And you're going to start to just see breakthrough after breakthrough after breakthrough happening. And these are things that can have like a huge impact on, you know, creating a new industry, creating all sorts of GDP, et cetera. And so just look at where we've come from. Like two years ago, we're talking to chatbots, then we, you know, kind of go into this like code writing agentic mode.

11:18Now we're about to step into like breakthroughs that actually start to change the world. So that's what I'm pretty excited about. It's very exciting. Why don't we end on some of those breakthroughs or just some of the things that you'll be focused on in the months to come around AI to make sure that your highly AI pilled excitement remains intact? Yeah, I mean, it's It's hard to know where they're going to come from. I think if you look at what Demis from DeepMind is working on, he's got this whole group working on various different problems across medicine, biology, kind of protein mapping, et cetera.

11:51So you're going to get a bunch in that area. It's all the things that like academia has been working on for decades that you're going to start to see these AI systems kind of pointed at these problems. You're going to just start to see more of this happening over and over. compression of innovation cycles. A lot of exciting things could come from that. Yeah. And that's the reason why you want to be bullish and not lose sight of that. When we get into the hugging face and some of the cyber issues, like, yeah, we need to engineer solutions that are safe and that everybody agrees upon and are set up the right way.

12:23But let's get the AI to have more of these breakthroughs. And then let's see where that takes us as a society. Every time I talk to you, I just walk away more AI bullish. This time is no different. Ross, thank you so much for being on the podcast with us. Thank you. So in conclusion, this pacing dynamic in AI has been the topic of conversation for market participants since the weekend. Ross helped me decipher the signal from the noise around this argument. And his view is that this was a natural part of the evolution cycle of this technology. If anything, it will just provide more safety without slowing growth rates for the industry and slowing the innovation cycle down, leaving him in an even more bullish place related to AI and breakthroughs to come in the future.

13:07One thing to note, we mentioned some private companies during this podcast that we do not cover out of Barclays Research. Thank you for joining and please remember to hit subscribe wherever you listen to your podcasts to be notified when new episodes of The Barclays Brief come out.