Is AI Scaling Dead? — With Gary Marcus

0:00Is the AI field reaching the limits of improving models by scaling them up? And what happens if bigger no longer means better? That's coming up with AI critic Gary Marcus right after this. Welcome to Big Technology Podcast, a show for cool-headed and nuanced conversation of the tech world and beyond. We're joined today by AI critic Gary Marcus, the author of the book Rebooting AI and Markets on AI on Substack. And he's here to speak with us about whether the AI industry is hitting the limits of scaling generative AI models up and what it means if we're truly seeing diminishing returns from making these models bigger.

0:38Gary, it's great to see you. Welcome to the show. Thanks for having me. So the genesis of this episode is that I did an episode with Mark Chen from OpenAI about GPT 4.5. And you come into my DMs and you say, listen, I want to give a rebuttal. Scaling is basically over and it's not exactly what OpenAI has said. Now, for those who don't know about the scaling laws, basically the idea is that the more compute and data you put into these large language models, the better they're going to get, basically predictably, linearly. Well, exponentially was the idea. Right. And so the context here is now we've seen almost every research house all but admit that that has hit the point of diminishing returns.

1:21I think Mustafa Suleiman was here. He pretty much admitted it. Thomas Kurian, CEO of Google Cloud, said that their diminishing returns are happening. Jan LeCun has also talked about the fact that you're just not going to see as many returns from AI scaling as you would beforehand. So just describe the context of what we're seeing right now. How big of a deal is it? And then what are the implications for the AI industry? Because this is the big question. I mean, how much better can these things get, right? That is the big question with AI today. Well, I mean, I have to laugh because I wrote a paper in 2022 called Deep Learning is Hitting a Wall.

1:55And the whole point of that paper is that scaling was going to run out, that we were going to hit diminishing returns. And everybody in the field went after me. A lot of the people you mentioned, I mean, Lacoon did, Elon Musk went after me by name, Altman did. And they all, like Altman said, give me the strength of a mediocre deep learning skeptic. So people were really pissed when I said that deep learning was going to run out. So it's amazing to me that a bunch of people have conceded that these scaling laws are not working the way they used to be. And they're also doing a bit of backpedaling.

2:26I think that that Mark Chen interview, I can't quite remember the details, but I think it was a version of backpedaling and redefining things. So if you go back to 2022, there were these papers by Jared Kaplan and others at OpenAI. And they said, look, we can just mathematically predict how good a model is going to be from how much data there is. And then there were the so-called chinchilla scaling laws. And everybody was super excited. And basically, people invested half a trillion dollars, assuming that these things were true. They made arguments to their investors or whatever. They said, if we put in this much data, we're going to get here.

3:01And they all thought that here in particular was going to mean AGI eventually. And what happened last year is everybody was disappointed by their results. So we got one more iteration of scaling after 2002, after 2022, that worked really well. And we call that GPT-4 and all of these models that are sort of like that. So I wrote that paper around GPT-3. We got another iteration of scaling. So three was scaling compared to two. It was much better. Two was scaling compared to one. It was much better. So much better meant, sorry, much more data meant much better. But what is much better? Well, I mean, one way to think about it is you didn't need a magnifying glass to see the difference between GPT-2 and we didn't call it GPT-1, but the original GPT.

3:46And you didn't need a magnifying glass for GPT-4 as opposed to GPT-3. It was just obviously better. A lot of people thought is that we would pretty quickly see GPT-5. And a lot of people raced to build it. So OpenAI tried to build GPT-5 and they had a thing called Project Orion and it actually failed. and eventually got released as GPT-4.5. So what they thought was going to be GPT-5 just didn't meet expectations. Now, they could slap any name on any model they want. And in fact, lately, nobody understands how they're naming their models. But they haven't felt like any of the models that they've worked on since GPT-4 actually deserve the name GPT-5.

4:26And it didn't meet the performance that these so-called mathematical laws required. And what I said in that paper is they're not really mathematical laws. They're not physical laws of the universe like gravity. They're just generalizations that held for a little while. Like a baby may double in weight every couple of months early in its life. That doesn't mean that by the time you're 18 years old that you're going to be 30 ,000 pounds. And so we had this doubling for a while. And then it stopped. And we can talk about why. But the reality is it's not really operative anymore. So there's been efforts to kind of misdirect and shift direction.

4:59so i think everybody in the industry quietly or otherwise acknowledged that hey we're not getting the returns that we thought anymore and nobody's been able to build a so-called gpt5 level model that's a big deal right i'm a scientist and as a scientist i was originally a scientist as a scientist we have to pay attention to negative results as well as positive results so when 30 people try the same experiment it doesn't work nature is telling you something and everybody tried the experiment of building models that were 10x the size of gpt4 hoping to get to something they could call gpt5 that was like a quantum leap better than gpt4 they didn't get there so now they're talking about scaling inference time compute that's a different thing so the original before we get there i just want to talk to you about i want to test your your um theory here so it's not that scaling is over right i don't think anyone in that we're talking about say scaling is over basically what they're saying is if you want to make the model better and i think that makes means more intelligent more conversational even um more personable you can still do it by scaling i think what they admit the what the thing that they admit though is that it takes much much more compute and much more data to get the same results that you would in the previous loops.

6:20So let's clarify two things. One is that what people talked about about scaling originally was a mathematically predictable relationship between performance and amount of data. You can go back and look at the Chinchilla paper, the Jared Kaplan paper, and lots of things that were posted on the internet. There were papers that saying, or t-shirts saying scale is all you need. And you looked at that t-shirt and it had equations from the Jared Kaplan paper. And it said, you know, here's the exponent. You can fit the equation. If you have this much data, this is the performance you're going to get.

6:50And there were a bunch of papers, a bunch of models that actually seemed to fit that curve, but it was an exponential curve. And what's happening now is, yeah, you add more data, you get a little bit better, but you're not fitting that curve anymore. We've fallen off the curve. That's what it really means to say that scaling isn't working anymore is you don't, you know, if I drew a curve for you, it was going up and up and up really fast and it's not going up as a function of how much data you had or how much compute you had. So we added a bunch of compute and you got this much better performance.

7:22And this is how people justified running these experiments that cost a billion dollars. They're like, I know what I'm going to get for the billion dollars. And then they ran the billion dollar experiments and they didn't get what they thought they would. Yeah, you get a little bit better, but that's what diminishing returns means. Diminishing returns means you're not getting the same bang for your buck as you used to. That's where we are now. So anytime you add a little piece of data, the model is going to do better on that piece of data. But the question is, does it generalize and give you significant gains across the board?

7:53And we were seeing that, and we just aren't anymore. So is there still a path for these models to become much more performant? I mean, let's say you do supersize these clusters to the point that they are insanely bigger than they were previously. Let's talk about Elon Musk's 1 million GPU cluster. Well, let's look at what Elon got for his money, right? So he built Grok 3, and by his own testimony, it was 10 times the size of Grok 2. It's a little better, but it's not night and day, right? Grok 2 was night and day better than the original Grok. GPT4 was night and day better than GPT3. GPT3 was night and day better than GPT2.

8:34Grok 3 is like, yeah, you can measure it. You can see that there's some performance. But for 10x, the investment of data, compute, and not to mention cost of energy to the environment, it's not 10 times smarter by any reasonable measure. It just isn't. Okay. And so this would be the point where I say, well, then this entire AI moment is done. However... Well, that's this moment. There will be other AI moments, but this one... I'm setting it up to say that it's not because, like you mentioned, you're talking about test time compute. That's another way to say reasoning, I think, which is these models.

9:11Well, I'm going to give you a hard time about that. But people do do that. But with reasoning or test time compute, you'll help me figure out the finer details. What these models are doing is they're coming to try to find an answer, and they're checking their progress and deciding whether it's a good step or not, and then taking another step and another step. And we've seen that they have been able to perform much better when you put that reasoning capabilities on top of these large models, which has enabled these research houses to continue the progress. And give you, but it's not really you, it's these companies some pushback on that.

9:47So it is true that you can build a model that will do better if you put more compute on it. But it's only true to some degree. So I don't get to whether it's actually reasoning or not. But it turns out that on some problems, you can generate a lot of data in advance. And for those problems, adding more test time compute seems helpful. There was a paper this weekend that's calling some of this into question. By the way, just to explain to folks, test time is when the model is giving an answer. That's what test time is. That's right. So you have these models now, like 03 and 04, that will sometimes take like 30 seconds or five minutes or whatever to answer a question.

10:29And sometimes it's absurd because you ask it like, what's 37 times 11? And it takes 30 seconds. You're like, my calculator could have done it faster. But we'll put aside that absurdity. In some cases, it seems like time was spent, sometimes not. But if you look carefully, the best results for these models are almost always on the same things, which are math and programming. And so when you look at math and programming, you're looking at domains where it's possible to generate what we call synthetic data. and to generate synthetic data that you know are correct. So for example, on multiplication, you can train the model on a bunch of multiplication problems, and you can figure out the answer in advance, so you can train the model what it is that you should predict.

11:13And so on these problems, in what I would call closed domains, where we can do verification as we create the synthetic data, we can verify that the answer we're teaching the model is correct, the models do better. But if you go back and you look at the O3, sorry, the O1 paper, Even then, you could already see that the gains were there and not across the board. They reported that on some problems, O1 was not better than GPT-4. It's only on other problems, these cut and dry problems with the synthetic data, that you actually got better performance. And I've now seen like 10 models and it always seems to be that way.

11:53We're still waiting for all the empirical data to come in. But it looks to me like it's a narrow trick that works in some cases. The amazing thing about GPT-4 is that it was just better than GPT-3 on almost anything you could imagine. And GPT-3, the amazing thing is it was better than GPT-2 on almost anything you can imagine. Models like O1 are not systematically better than GPT-4. They're better in certain use cases, especially ones where you can create data in advance. Now, the reason I wouldn't call them reasoning models, though you're right that many people do is what I think they're doing is basically copying patterns of human reasoning.

12:28They're getting data about how humans reason certain things, but the depth of reasoning there is not that great. They still make lots of stupid mistakes all the time. I don't think that they have the abstractions that we think, for example, a logician has when they're reasoning. So it has the appearance of reasoning, but it's really just mimicry. And there's limits to how far that mimicry goes i'll give you just one more example is o3 apparently hallucinates more than the models that came before it which is stunning like how does that happen i mean that's a good broader question which is our understanding of these models is still remarkably limited so the technical term or one technical term ability well i was going to give you a different one which is which is black box okay um but they're closely related those two terms you need interoperability to get figure out what's going on in the black box.

13:18If you can at all. I mean, I'd almost put it another way, which is that - But isn't the black box the thing in the plane that tells you what actually happened? Well, that's a different thing, right? So a black box in a plane is actually a flight recorder that records a lot of data. But what we mean in machine learning by black box is you have a model where you have the inputs and you have the outputs. You know how you calculate them, but you don't really understand how the system gets there. So in this case, you're doing all this matrix multiplication. Nobody really understands it. And so nobody can actually give you a straightforward answer for why o3 hallucinates more than gpt4 we can just observe it that's what happens with black boxes is you you empirically observe things and you say well it does that but you don't really know why and you don't really know how to fix it either another example just in the last couple days is apparently sam altman reported i forget the new model is is stubborn or what was it i forget no it's not stubborn it's a bro it's a bro cheap but that's gbt4o it's just like it became very fratty became very fratty and like you like right you would be like what's going like help me with this and it's like yo that's a hell of good question bro and they're like we don't know why this happened and they rolled it back completely yeah exactly or i thought they were partly rolled it or whatever no no sam said it's now the latest um iteration's been completely completely rolled back so right that was what i would call again empirical like they tried it out and it didn't work or it worked in the way that it irritated people right and so we don't know in advance like there's a lot of just like try it because that's how black boxes work and we have some things but those things are not very strong so the scaling quote laws were empirical guesses about how these models work and they were true for a little while which was amazing and they're not true anymore which is also amazing in a way um so we don't know what's going to happen from the black boxes.

15:11Right. Okay. So, but let me now sort of... And sorry, let me come back to one other thing quick, which is interpretability. So that's a very closely related notion. So let's say you look at a GPS navigation system. That's a piece of AI that's very interpretable. So you can say it is plotting this route. It says, you know, you can go this way, you can go that way. This is the function that it's maximizing. This is the database it's using. This is how it looks up the data. We don't have any of that in these so-called black box models. We don't really know what the database is that it's consulting.

15:39It isn't exactly consulting a database at all. And we don't know how to fix it. And so, you know, Dario Amode, who's the CEO. We just talked about this on the show. You actually praised his interpretability post. That's right. Call for interpretability. I'll be honest. I haven't read the paper yet. I just read the title. So bad on me. But the title of his paper was something like On the Desperate Need for Interpretability. That captures it. And I think he's right. I've said this too myself. In my last book, I talked about interpretability being really important. The only difference between Dario and me on this point is we both think that we're screwed as a society if we stick with uninterpretable models.

16:19He just thinks that LLMs will eventually be interpretable. And his company, to be fair, has done the best work on interpretability of LLMs that I'm aware of. Chris Ola, I think, is brilliant. But they haven't got that far. They've gotten further than anybody else. But I don't think we're ever going to get very far into the black box. And so I think we need to start over and find different approaches to AI altogether. Right. So, Gary, if I'm listening to what you're saying on this show so far, it is basically after GPT-4, we haven't made a lot of progress. However, a little bit. But let me just do the pushback here, which is, I mean, if you think about what it's like using these models after GPT-4, they are significantly better.

17:00I'll give you one example. I was using 03, this new reasoning model or test time model, whatever you want to call it. And I just, I'm in it and I'm doing crazy things and it's exceptionally helpful. So I put a photo of myself on a rock climbing wall and said, what's going on? And it like was able to look at the form, where my body was, where my, what my posture was, and like analyze all these things and give actually helpful coaching tips, which you never would have had with GPT-4. Then you think about what Claude is doing, the anthropic bot. I was with some friends last night, and this is what we do for fun.

17:35I vibe-coded a retirement calculator directly in Claude. It took like 10 minutes. We took a bank statement. We got a line graph of the person's balances, a bar graph of their expenses, financial plan, and then we coded a retirement calculator based off of the data that we had there. And then you also have PhDs that are now adding their unique insights into these models for training. They just basically are sitting and writing down what they know and the model is absorbing it. So we are seeing, I would call it, vast improvement over the GPT-4 models. So, I mean, there's a couple of different ways to think about that.

18:15So one is on a lot of benchmarks, there is improvements. But there's also issues of data contamination. Alex Reisner wrote an excellent piece in The Atlantic about the issues of data contamination. And we've seen a lot of studies where people are like, well, we tried it in my company. It's not really that much better. So they're better on the benchmarks. Are they better in general? Not so clear. It was a new benchmark released by a company called Val AI or something like that. Val's AI, the Washington Post talked about yesterday, where they looked at things like, can you pull out a chart based on a series of financial statements, SEC statements?

18:51from a bunch of companies. And these systems all claim to do it, but accuracy was under 10%. And overall, on this new benchmark, accuracy was at 50%. Would these new models be better than GPT-4? Maybe, but they weren't that good. So I think people tend to notice when they do well. They don't notice as much when they do poorly. And although I think there's been some improvement, there has not been the quantum leap that people were expecting. We have not moved past hallucinations. We have not moved past stupid reasoning errors. If you go back to my 2022 paper, Deep Learning is Hitting a Wall, I didn't say there'd be no progress at all.

19:28What I said is we're going to have problems with hallucinations. We're going to have problems with reasoning, planning until we have a different architecture in some sense. And I think that that's still true. We're still stuck on the same kinds of things. So if you have your deep research write you a paper, it's going to make up preferences. It's probably going to make up numbers. Like, you know, did you actually go back and check? So, for example, I think it's called, they all have similar names now, whatever Grok's version is, Deep Search, Deep Research. Yeah, some, I don't know. Deep Research Mini-06.

20:00I won't be convinced that we have AGI until these companies learn how to call Deep Research something other than Deep Research. They all use the same exact name. It's really bizarre. So whichever version Grok has, I asked it, for example, to list all of the major cities that were west of Denver. and to somebody who wasn't paying attention it'd be super impressive because I really wanted to know how well it was working I checked and it left out Billings, Montana so you got a list that looks really good and then there are errors this often happens and then I had a crazy conversation with it after that I said what happened to Billings and it said well there was an earthquake there on February 10th or whatever and I looked up in the seismological data I used Google because I want to have a real source or DuckDuckGo.

20:47And there was no earthquake then. And I pushed it on and said, well, I'm sorry for the error or whatever. So we're still seeing those kinds of things. We may see them less, but they are still there. We still have those kinds of problems. So I don't doubt that there's been some improvement, but the quantum across the board that people were hoping for is not there. The reliability is still not there. And there's still lots of subtle errors that people don't notice. And then if you want to talk to me about retirement calculators, there are a lot of those on the web. So the easy cases for these systems are the ones where the source code is actually already there on the web.

21:21Like Kevin Roos talked about this example of having, he quote, vibe coded a system to look in a refrigerator and tell them what recipe to make. But it turns out that app is already there on the web and there are demos of that with source code. And so like, if you ask a system to do something that's already been done, that's always been true with all of these systems. That's their sweet spot is regurgitation. And so, yeah, they can build the stuff that's out there. But if you want to code things in the real world, you usually want to code something that's new. And these systems have a lot of problems with that.

21:53Another recent study showed that they're good at coding, but they're not good at debugging. And coding is just the tiniest part of the battle, right? The real battle is debugging things and maintaining the code over time. And these systems don't really do that yet. But search has made them more reliable. when these bots are able to search the web and they are now starting to give you lots of links in the actual answers. I still get daily people sending me examples of it hallucinated these references. I'm not saying hallucinations have been solved, but for me, I will use it. It's an incredible research assistant.

22:29And then when it links out to things and I'm not sure of those figures, I'll then go to the primary sources and start reading. I mean, good on you that you go to the primary sources. source. I worry the most about people who don't. And we've seen countless lawyers, for example, get in trouble using these systems. Has it been countless? I just heard of one. Oh, no, no, no. There's many more than that. There's some in the US, there's some in Canada. I think there was just one in Europe. I mean, it's not really the countless one could sit there and do it, but it's got to be at least a dozen by now.

23:00And whether this is going to be, all right, I think we can both agree on this, that whether this is the end of progress or towards the end progress or whether there's a lot more progress. There's a real problem of people outsourcing their thinking to these bots. Well, Microsoft did a study, in fact, suggesting that critical thinking was getting worse as a function of them. And that wouldn't be too surprising. We have a whole generation of kids who basically rely on these bots and who don't really know how to look at them critically. In previous years, we were starting to get too many kids relying on whatever garbage they found on the web, basically.

23:33And I mean, chatbots are basically synthesizing the garbage that they find on the web. So we're not really teaching kids critical thinking skills. And nowadays, like the idea for many kids of writing a term paper is I typed in a prompt in chat GPT and then maybe I made a couple edits and I turn it in. You're obviously not learning how to actually think or write in that fashion. A lot of these tools, I think, are best used in the hands of sophisticated people who understand their limits. So coding has actually been, I think, one of the biggest applications. and that's because coders understand how to debug code and so they can take the system basically it's just typing for them and looking stuff up and if it doesn't work then they can fix it right the really dangerous applications are like when somebody asks for medical advice and they can't debug it themselves and you know something goes wrong okay so i'm going to take into consideration all the things that you've said so far and see if i can get a sense as to where you think we're heading it seems like there was a push to just make these models better based off of scale that could be things like the 300 000 gpu cluster i think meta used for llama 4 or it could be the million cluster gpu center that elon's built for grok and what you're saying is that's been maxed out pretty much like no one's it's not maxed out but it's just diminishing returns diminishing returns so the point that a point that i'm trying to make here is you don't believe that there's going to be anyone that's going to build a bigger gpu data center than that because if you're seeing diminishing returns from something that costs billions of dollars doesn't make sense to invest well wait a second i'm not saying people are rational i think that people will probably try at least one more time they'll build things you know probably elon will build something that's 10 times the size of Grok 3, which will be huge.

25:25And it will have a serious impact on the environment and so forth. It's not just GPUs, also it's data, right? Like how much more data is there? Well, let's come to the data separately in a second. So I think people will actually try. I think Masa has just bankrolled Sam to try. I just don't think they're going to get that much for it. I don't think they'll get zero. I mean, there will be tangibly better performance on certain benchmarks and so forth. But I don't think that it's going to be wildly impressive. and I don't think it's going to knock down the problems of hallucinations, boneheaded errors.

25:55So here's what I'm getting at. That's not going to feel much better than what we have today. It doesn't seem like you believe that reasoning is going to make the bot feel much better than we have today. Not the kind of reasoning we're doing. And there's no emergent coding. So are you basically saying that what we have in AI today, this is it? For a while. For a generative AI? For a while, I guess. I mean, look, I put out some predictions last year in March that people can look up that I had on Twitter. And those predictions included, I said there'd be no GPT-5 this year, or if it came out, it would be disappointing.

26:27It's supposed to come in summer. Well, this was last year. So I said in 2024, we won't see this. And that was a very contrarian prediction at that point, right? This was a few weeks after people had said, oh, I bet GPT-4 is going to drop off to the Super Bowl, like right after the Super Bowl. Won't that be amazing? So people really thought it was going to come last year if you go back and look at what they said on Twitter, et cetera. And it didn't. And I correctly anticipated that it wouldn't. And I said, we're going to have a kind of pile up where we have a lot of similar models from a lot of companies.

26:59I think I said 7 to 10, which was sort of roughly right. And I said we were going to have no moat because everybody is doing the same thing. And the prices were going to go down. We have a price war. All of that stuff happened. Now, maybe we get to so-called GPT-5 level this year. It keeps getting pushed back. I don't know if we'll get much further than that without some kind of genuine innovation. And I think genuine innovation will come. But what I think is we're going down the wrong path. Jan LeCun used this notion of we're on the exit ramp. How do you say it? Large language models are the off-ramp to AGI.