The AI Skeptics

24:10And if that sounds cynical, I don't know, I don't see any proof to the contrary. In fact, I see every single day more proof, more evidence that the only thing these people care about is money.

24:21Arvind Narayanan:All right. Well, before I let you go, I want to ask another question specifically about the technology itself and where it stands right now. Because when I look at the AI models that the public has access to, on the one hand, they obviously seem too unreliable to replace many human workers, right? The chatbots still hallucinate and make answers up sometimes. The image generators, they're not yet good enough to replace a film crew. But on the other hand, when I look at these products in light of where AI was just a decade ago, man, they seem amazing, right? You've got chatbots that can have deep conversations on almost any subject.

25:02Arvind Narayanan:You've got these AI systems that you can take a picture of your engine and they'll help you fix your car. You've got these image generators that can create these little movies that are so impressive that people in Hollywood are starting to worry about the future of their industry. And just in my own personal experience the other day, one of my producers uploaded the logo of The Last Invention into one of these video generators, and it made this moving graphic that, you know, 10 years ago, I would have had to pay thousands of dollars and wait days, if not weeks, to get work of that quality. and this thing just, boom, spit the whole thing out, right, in seconds.

25:42Arvind Narayanan:And it's not good enough quality for me to replace my actual logo with it, but it is impressive. And according to the folks at these AI companies, we are only at the beginning, right? This is just the pong on Atari stage and where they believe this industry is headed. And I want to ask you, when you see all of this investment going in to these companies, all of these talented people rushing to help them make their dreams come true, to create this wild new AI future, does it ever cause you pause? Do you ever wonder if maybe they can pull something truly transformational off or not? No. Like, I just, I'm sorry, I hate to do that.

26:26No, I think that we're not in the Pong stage. They want you to believe we're in the Pong stage. We are at the limits of what these things can do. We are. absolutely not at some early stage. They want you to believe that. They want you to believe that because they've sunk all that money in. They want you to believe that because if they have to admit that they've done all this for no reason, that would be incredibly embarrassing. So all the king's horses, all the king's men, every member of the media, every podcast, every investor, everyone, every PE person, every bank, they're all tied up in this crap.

27:03They've all put, all of the money and the attention in the world has gone into this. All of the smartest people bloviate about it every goddamn day. And here we are. The coolest thing we can come up with is, wow, it made a logo that was not good enough to replace your current one. Just to be clear, wasn't good enough. That's what it could do. Kind of interesting, not good enough to really do anything with it. And I think that is ultimately it. They need you to believe these are the early innings. Because if you start looking at this and thinking about this, even a little, and realize that the depth of hubris, everyone involved kind of looks stupid.

27:44It kind of makes you think, maybe these people aren't geniuses at all.

27:54Arvind Narayanan:We'll be back with more AI skeptics after a short break.

28:03Thank you.

28:33Gary Marcus:You can do that by clicking on the link in our show notes or by going to longviewinvestigations.com. Until then, here is a brief message from our sponsors.

28:50Gary Marcus:The Last Invention is brought to you by Quince. Every spring, I feel the same urge to clean things out. Closets, habits, routines. And with clothes, I keep coming back to the same idea. Fewer of them, but better quality. That's what I like about Quince. The materials feel elevated and the cuts are clean, and the prices, they aren't insane. They make everyday staples using premium fabrics like 100 % European linen and this incredibly soft flow knit fabric for their active wear. The kind of gear you end up wearing way beyond the gym. What surprised me most is the pricing. It's about 50 to 60 % less than what you'd expect from comparable brands.

29:29Gary Marcus:And that's because Quince goes straight to the source. working directly with ethical factories and skipping the middlemen. So you're actually paying for the quality itself instead of the markup. My favorite piece these days is one of their blue chore jackets. It fits great, it's really comfortable, and I know it'll hold up even though I wear it all the time. Refresh your wardrobe with Quince. Go to quince.com slash lastinvention for free shipping and 365-day returns. Now available in Canada too. Again, go to quince.com slash last invention.

30:27Sign up by April 27th. Visit everpass.com.

30:31Arvind Narayanan:Limited time offer. Terms apply. Welcome back. And now for Act Two, we're on the wrong path.

30:39Gary Marcus:I'm Gary Marcus. I'm a professor emeritus at New York University. And I've been, I think, correctly anticipating the limits of neural networks since my dissertation in 1993.

30:51Arvind Narayanan:So a few weeks ago, I was really excited because I got the chance to sit down and speak with Gary Marcus. And he is not only a scientist deeply steeped in the world of AI, but like many of his peers, he's been dreaming of a world shaped by powerful AI systems since he was only a 10-year-old child.

31:09Gary Marcus:And so I became interested in AI pretty much immediately. Like, how do I program this to play chess or tic-tac-toe? You know, I was an only child. I wanted to learn how to make this thing play games and things like that. When I was about 13, I wrote a database that you could ask questions in natural language. It wasn't very good, but I started to understand some of the issues. I wrote a Latin to English translator when I was 15 that allowed me to go to college two years early. So I was interested in AI from the jump.

31:39Arvind Narayanan:And also like many of the leaders in today's AI labs, he's not thinking of AI as just some new product, but as a super intelligence that might radically change humanity's future.

31:50Gary Marcus:In an ideal world, I would love to see superintelligence. There's a million questions about medicine, technology we could answer if we had a kind of all-knowing oracle that could think things through and simulate the body in perfect detail and so forth. You know, I would love that at some level.

32:06Arvind Narayanan:Where Gary Marcus differs from his peers is when it comes to the excitement over the large language models that have been fueling many of today's leading AI products.

32:16Gary Marcus:Like, I think AI has enormous potential to help. But generative AI, these chatbots, it's not a good technology.

32:23Arvind Narayanan:And in short, what is your criticism of AI in its current state and embodied in products like ChatGPT or Claude?

32:30Gary Marcus:It's inherently prone to all kinds of reasoning errors. It's inherently hard to control. I once called it a bull in a china shop. It's inherently prone to hallucinations, making stuff up. And that hasn't changed. I've been making these points for, depending on how you count, six years or 13 years or 30 years. And that hasn't changed. I keep over and over hearing, oh, just give us more data. It will solve these problems. But the reality is, as we're recording this, Gemini just came out and it hallucinates. It's the same story as in 2022 when ChatGPT came out, almost three years to the day when we're recording.

33:07Gary Marcus:And I said at the time, the problem with this technology is the way it's built. It's hallucinatory. It's going to be unreliable. This is not the droids we should be looking for.

33:17Arvind Narayanan:And it still isn't. I appreciate the Star Wars reference there. So going back to 2012, Gary Marcus found himself in a somewhat lonely place. As we covered in the series, Jeffrey Hinton and his grad students, along with these other contrarian AI researchers, They felt that after years and years of trying, their much maligned theories around creating AI babies instead of AI experts had finally won over the field. And then suddenly you had Google and all these other companies starting to take the race for AI more seriously. But Gary, he looked at all of that and he just sort of shrugged.

33:53Gary Marcus:It's a better ladder, but it's not necessarily going to get us to the moon.

33:57Arvind Narayanan:And then again in 2022, the new model of ChatGPT emerged and sent shockwaves throughout the world, Gary was not impressed. What were you thinking when you saw all of these people getting so excited about JetGPT and investing all this money?

34:13Gary Marcus:That it's a red herring. That these techniques are cool, but they're basically like a paper that Miller and Chomsky wrote in 1961 showing that as you get bigger databases, you get better approximations to English, but so what? You don't really have the deep semantics that you need, and it's not really going to get you to artificial general intelligence.

34:34Arvind Narayanan:Now, remember that one of the reasons we've been seeing this massive amount of investment and the building out of these data centers all around the world is partly because of the theory that the more we scale up the data and the compute power of these AI systems, the more capable they become. This is what made the huge jump from GPT-2 to 3 or 3 to 4 so exciting. But again, Gary looked at all that and said, this is going to hit a wall.

35:00Gary Marcus:Scaling, adding more data and more compute to these systems helps for a certain period of time. People have plotted graphs about it, but those graphs aren't going to continue indefinitely. You can think of lots of cases where if you extrapolated from the first few points on a curve, you'd get wrong answers. My favorite is now a joke on Twitter, the trillion-pound baby, which is you extrapolate a baby weighs so many pounds, eight pounds at birth, it's 16 pounds at one month, and so you extrapolate it by the time it goes to college, it's going to be a trillion pounds.

35:28Arvind Narayanan:So while most of the field was celebrating and investors were opening up their wallets to pour money into the AI labs, Gary wrote a paper saying that we need to tamp all of that excitement down because this exponential growth, it wasn't going to last.

35:43Gary Marcus:You might see another doubling or whatever, but this is not going to continue indefinitely. These are not physical laws of the universe, these scaling extrapolations. And you're going to continue to see with these systems, because of the way that they are built, hallucinations, problems with reasoning, problems with truth, etc. And people hated me for saying this. All the major figures in the field took shots on me on Twitter. Altman, Lacoon, Musk, and so forth. They were really mad. They really made fun of me. And that's actually what happened. We got another doubling to GPT-4. It was really a quantum leap forward compared to GPT-3, the way that GPT-3 was a quantum leap forward compared to GPT-2.

36:22Gary Marcus:But I kept warning. I said, GPT-5 may not be that. And it wasn't. OpenAI is rolling out its newest artificial intelligence model, GPT-5. And the reviews are mixed.

36:37Arvind Narayanan:What did we learn today about chat GPT-5? It came out. I watched it. I wasn't impressed by anything. I'll be totally honest. So last fall, after months of Sam Altman hyping up the new capabilities, ChatGPT5 was released. And while there were some definite improvements in some areas from GPT4, it was still plagued by hallucinations, and it failed to be an exponential jump forward.

37:02Gary Marcus:Early users describe it as less creative and more corporate, with some calling it bland or generic. Others report bugs and uneven responses. And what people realized was that I was right. Somebody wrote a book this year, this year using a chatbot, called Gary Marcus is Right. And it's become a common thing on the internet. People started writing Gary Marcus apology forms and so forth because I kept saying five would be delayed, take longer than you think, and it wouldn't be as impressive as he thought.

37:33Arvind Narayanan:I just want to pull back and try and get your take on the broader view here because it's one thing to say, look at what AI can't do now. And it's just a whole other thing to say, and it never will, right?

37:47Gary Marcus:Sorry, that needs a lot of clarification. The word it needs clarification there, right?

37:53Arvind Narayanan:Give me some clarification on it.

37:55Gary Marcus:Does it mean AI or does it mean large language models and chatbots?

37:59Arvind Narayanan:What I mean by it is the current path in the pursuit, for better or for worse, of artificial general intelligence. This thing that was at one time a pipe dream and that many investors and a lot of people who are excited and a lot of other people who are really worried think might be coming a lot sooner than we think. I'm talking about the current path more than I'm talking about, you know, one particular chatbot product or one particular strategy.

38:25Gary Marcus:I mean, already they're deviating from the path that I criticize. So what I criticized in 2022 was what I would call a pure large language model, which uses this technique called attention, predicts next word, does not use any symbolic systems, and you've talked about symbolic systems in your program. Those systems made limited progress. They made some, but they made limited progress and they ran out of room. They had reached a point of diminishing returns. So they're already doing what I said in 2022 you would need to do, which is putting symbolic systems in there. So they now have all the major systems have code interpreters that are symbolic systems inside of their large language model.

39:07Gary Marcus:So they have already adopted some of what I said we would need. Then there's a whole bunch of startups trying to add world models. You can go back and look at my 2020 paper, The Next Decade in AI, and I said that was one of four critical steps, was to add world models or find a system that can make use of world models. So slowly they are encompassing all of the things that I said. So, you know, Theseus's ship where you replace all the planks one by one, is this still Theseus's ship? So what they're going to do is they're going to replace every little bit of how the LLMs work with kinds of stuff that I've been talking about and maybe some other stuff that nobody's invented yet or maybe some stuff that hasn't been invented that I don't know about.

39:46Gary Marcus:And they're still going to want to call it a large language model, But it is not going to work in the same fundamental ways as the large language models that we talk about. When you say the current approach, it's going to be the current approach plus a lot of extra stuff that did not exist when I made these critiques. And they're going to want to take credit and say, you know, it's what we were saying all along. But the reality is what's going to actually make it work is having symbolic systems in there, having world models that are symbolic in there, et cetera.

40:13Arvind Narayanan:Now here's why labeling Gary Marcus and his camp as skeptics can be tricky. because the views that he holds are also held by some of the most enthusiastic believers in the AI revolution, including people like Demis Hassabis, who's the head of Google's DeepMind. And in recent months, even people like Ilyas Hescovar, one of the minds behind ChatGPT, who pushed really hard and really publicly for this idea that more compute and more data were going to lead to bigger and bigger breakthroughs.

40:45Gary Marcus:Up until 2020, from 2012 to 2020, it was the age of research. Now, from 2020 to 2025, it was the age of scaling. Because people say, this is amazing, you've got to scale more, keep scaling. The one word, scaling.

40:59Arvind Narayanan:Even he has come around to having a view much closer to Gary Marcus's.

41:04Gary Marcus:But now the scale is so big. Like, is the belief really that, oh, it's so big, but if you had 100x more, everything would be so different? is the belief that if you just honeydex the scale, everything would be transformed. I don't think that's true. So it's back to the age of research again, just with big computers.

41:25Arvind Narayanan:And all of these guys, including Gary Marcus, they believe that we are still headed towards an AGI future and that it might be here quicker than we think.

41:34Gary Marcus:I think that we will get to AGI. We might do it in the next decade. But I don't think we can do it with current techniques. We need new ideas.

41:44Arvind Narayanan:It's just that Gary and many of his peers think that we're going to need another AlphaGo or another GPT-like breakthrough before we get there.

41:53Gary Marcus:There's going to be several advances. If I had to guess, I would say there will be between three and 20 major advances before we get to AGI. Ideas that either don't exist yet or haven't really been put into production or appreciated yet. There will be a bunch of new things. and when we get all those pieces together, then we'll have AGI.

42:14Arvind Narayanan:And because of that, very unlike Ed Zitron and the AI is Griff camp of skeptics, Gary and his camp, they think that we need to take this technology seriously and get ready for its arrival. All right, so in our series, we mainly divided the big players in the great AI debate, as I think of it, between the AI doomers who believe that if anyone builds AGI, everyone dies, right? They are absolutely convinced that this is an existential threat that we're living through right now. Then there are, broadly speaking, the AI accelerationists who think that the benefits of AGI are so enormous that even though some of them think that there are risks, we should race towards the creation of a safe AGI as fast as we can.

43:07Arvind Narayanan:And then we have this third group that I call the AI Scouts. And they're the people who think that this could be great. This could be catastrophic. But what we really need to do right now is throw an enormous amount of resources into getting prepared for how transformative the AGI revolution is really going to be. I mean, mostly I'm with them. I'm not sure why you're not counting me as a scout. Well, maybe that's what I'm asking. Are you a scout? Should we understand you as not a skeptic, but a scout? One of the places I might differ from some of the people in that camp is that

43:46Gary Marcus:I'm very concerned about immediate problems as well as what might happen if we get to AGI. And I think I might be more concerned about that than some of the people who are in that camp. So I'm worried about, for example, discrimination and bias. We just saw a new study yesterday showing that LLMs, chatbots used in hiring decisions are racist and that humans who use them tend to pass along the recommendations without thinking. That's bad. That's bad right now. Non-consensual deepfake porn is bad and it's here right now. Cybercrime is increasing and that's here right now. There's a whole bunch of problems right now and I'm concerned with those.

44:26Gary Marcus:And I think a lot of the people who you call scouts don't talk about those things or don't talk about them as much as I might. But I do think we have, you know, at best 50 years to figure this out. I do think that there's a chance for AI to be positive. I think there's a chance for it to be negative. And we really need to get our shit together.

44:50Gary Marcus:and I would say that LOMs, chatbots, have been a dress rehearsal and it's a really depressing train wreck of a dress rehearsal in which we have found out that money and power influence the government to do nothing and we have a technology that is terrible at so-called alignment problem really not trustworthy and it's getting spread, it's getting investment We should not be putting a trillion dollars into a technology that is that poor at alignment. We should be putting a trillion dollars into building a better technology.

45:26And now, finally, Act 3, AI as Normal Technology.

45:32Arvind Narayanan:Arvind, thank you so much for doing this. Thank you, Andy. It's good to be here. And can we just start off by having you introduce yourself, tell us your name, and whatever credentials or title make the most sense for the conversation we're about to have.

45:45Gary Marcus:My name is Arvind Narayanan. I'm a professor of computer science at Princeton University. And, you know, like many people, I'm trying to bring more empirical evidence, more reasoning into the AI conversation. But I think what's so interesting about this is that many experts who are all coming at this in good faith have ended up in different positions in this landscape. And I perhaps represent one of those interesting poles of this debate.

46:14Arvind Narayanan:Yes. Well, that's why I'm very excited to be talking to you, because up until this moment, we've largely been speaking to different technologists and scientists, people who strongly believe that what's coming in the field of AI, maybe five years, 10 years, 20 years down the road, is going to be not just another new gadget, not just another new tech product, but something that ushers in a hinge moment in human history. And one of the reasons I was excited to talk to you is because you wrote this paper with your co-author called AI as a Normal Technology. And so I just want to ask, does that mean that you think what's coming is going to be more like the iPhone and how that changed the world and less like the discovery of fire and how that changed the course of human civilization?

47:03Gary Marcus:I would say somewhere in between iPhone and fire. All right. So AI as normal technology is not saying that it is just yet another gadget. We start out by saying that AI is a general purpose technology, and that means that it can be used for many different purposes, almost no matter what line of work you're in, but also in your personal life. So in that sense, it's different from a lot of different gadgets. It's obviously different from a toaster, which can do one thing. So right off the bat, that is one difference. But there have been many general purpose technologies before. We've had the internet, we've had electricity going even farther back, we've had the industrial revolution.

47:41Gary Marcus:And these are all technologies that yes, they did change a lot about human society, about work, about the rate of GDP growth, and those were all enormous transformations. but two things to keep in mind. One, they unfolded gradually. So the impact of the Industrial Revolution took many decades, many centuries to be felt. And even when technologies are very powerful, perhaps especially when technologies are very powerful, it actually takes us a while to adapt our institutions to this new way of doing things. And that's one reason that progress tends to be slow. So that's one thing to keep in mind.

48:19Gary Marcus:And I think the second thing to keep in mind is, yes, we're seeing claims that this could bring about the end of human civilization either in a utopian way or in a dystopian way. But I think the views that we hear tend to be the more extreme ones. And I think there is room in the conversation for more grounded views on that topic. And I, for one, think that those more extreme claims are pretty overblown.

48:46Arvind Narayanan:Okay, so just for clarity's sake, you are saying that you do believe that an AI revolution may be underway and that it will be transformative eventually. You aren't skeptical of the technology itself, per se, but you think that this revolution will be much more like electricity or much more like the industrial revolution, which slowly transformed society and less like, say, the awakening of a new digital species that in short order, for better or for worse, upends our entire world very quickly.

49:27Gary Marcus:That's exactly right. And my view is that we have a choice. It's not up to the AI companies alone to define what this technology is going to be and how we should think about it. How this technology plays out and, in fact, depends on decisions that we're all going to make about how we're going to integrate it into our various industries and occupations and workflows and what governments do with this, what elementary school teachers decide to do with it. So collectively, we have the agency to make the AI future take different possible trajectories. So that's one point of disagreement. And the other one is that, you know, I think my co-author and I, Sayaj Kapoor, and I think this is all going to take decades.

50:06Gary Marcus:It's not necessarily going to be a whole lot faster than the speed with which electricity disrupted various industries.

50:13Arvind Narayanan:All right. So I looked this up before our conversation today, and I learned that with electricity, even if you time it to when Thomas Edison built the first commercial power station in 1882, it still took roughly 50 to 60 years before electricity was widely available throughout the American population. or if you look at the telephone, Alexander Graham Bell, he patents that in 1876, but it wasn't until the 1940s and into the 50s before it actually reached a majority of American homes. So that's like 60, 70 years. And you're saying that the AI revolution underway right now is more likely to play out like that, that it's not going to be a sudden hinge moment in human history, like some people are predicting.

51:01Gary Marcus:That's exactly right. I don't think there's going to be a hinge moment. And to be clear, this is nothing to be complacent about. Even the speed with which the internet transforms society, yes, decades, but we weren't prepared for something happening on the timescale of decades. Social media has had enormous impacts on kids' mental health and things like that. And the research on that and the response from policymakers, all of that has taken a while to get going, arguably not fast enough. And we might end up making those same mistakes with AI. So it's quite possible that this will still be fast enough for things to go pretty badly, at least initially.

51:40Gary Marcus:But that said, that's still a very different story from some critical capability threshold at which AI is now writing the rules of history and it's happening at superhuman speed faster than humans can think or respond to. I think that is a sci-fi scenario. I don't think that's going to happen. And yeah, we lay out in our essay why we think that is. And to put it in a sentence or two, it's because we think change will happen at the speed of human behavioral change, at the speed at which organizations can adapt and transform. And it doesn't matter how fast capabilities continue to increase. We admit that capabilities might continue increasing rapidly, but that doesn't mean that it's going to be a force that transforms society with the rest of us having nothing to do but stand by and watch.

52:28Arvind Narayanan:Okay, so I want to get into your theory about what you think is going to happen to slow down change in the event that we really do discover or create a true AGI. Because when I was talking to one AI accelerationist on background for the story, he somewhat shared your view that it wouldn't be a sudden hinge moment, but he was telling me that he thinks that the technology is going to be able to be super transformative, but he was just worried about the state of our messy world. He was saying that there's just so many regulations out there. There's so much bureaucracy out there. There's all these issues with our supply chain.

53:06Arvind Narayanan:And that even if we had a true AGI right now, it would get stuck in this man-made bottleneck. And that would slow down the advancements in medicine and the advancements in science that he wanted to see in the world. And then on top of that, he also believed that many people are just afraid of artificial intelligence as an idea and that they would do all that they could to slow down or to stop the integration of this technology quickly in the economy and in society. Is your argument along those similar lines that essentially our world is just too messy and will not be able to quickly transform no matter what benchmarks the industry hits?

53:52Gary Marcus:I mean, that's exactly my point. And I think maybe this person sees that as a bad thing. And in my view, that's mostly a good thing that humans are not so eager to hand over control to AI systems. To be clear, in many cases, that can be a bad thing. So I find it pretty upsetting the extent of the backlash to self-driving cars, for instance. This is a technology that can save something like one million lives per year that are lost today in car accidents. in self-driving cars, at least Waymos, are already much safer than human drivers, and they're getting a lot safer very quickly. So this is an area where I would hope that there can be consensus that, yes, there are risks from AI systems.

54:34Gary Marcus:Yes, these cars sometimes can get into kinds of accidents that a human would have the common sense to avoid. But statistically, they're much safer. So let's figure out the right guardrails, and let's figure out how do we handle the labor displacement, you know, the gig workers are going to lose their jobs and the truckers and so forth. There has to be a way for society to compensate them. But with all that said, let's go ahead with this. The benefits are enormous. That would be the approach that I hope society would take. It's not the approach that society is taking. There are so many vested interests who are pushing back against this technology.

55:09Gary Marcus:And that's even, you know, in a case where an evidence-based look at the situation is just so compelling that we should go ahead with the technology. In other cases where there are genuinely multiple sides to the situation, you know, using AI in law or medicine or any other field, I think the extent of resistance that we're seeing and which we're going to continue to see is enormous. And there's just no way in which, you know, one day we're all going to wake up and decide to trust these AI systems. And so maybe that's a bad thing. Maybe we're going to, you know, lose out on some opportunities to make medical progress faster than we otherwise could.

55:46Gary Marcus:But it's also a good thing because to me, that's the main reason to have some comfort that these sci-fi scenarios of AI taking over one day are not going to come to pass because people are just not eager to give up power.

55:59Arvind Narayanan:All right, so give me your version of what you think is likely to happen. If, say, the AI labs are right, and somewhere in the next 5, 10, 20, 30 years, One of them is able to pull off the creation of a true AGI thinking machine. Speak to the listener out there who says, how would that not transform the world very quickly?

56:23Gary Marcus:The number of tasks in our economy today that people do that already could be automated is shockingly high. I wouldn't be surprised if it's more than 50%. Yeah. So that's the place to start. So why is it that there are so many things that can already be automated and yet we have people doing them? I think there are a lot of well-understood reasons for this. One of them is that in a lot of cases we prefer human interaction. Just, you know, going to a restaurant, we prefer interaction with a server who is human as opposed to always ordering from an automated menu. So that's a simple example. But in a lot of cases there are deeper reasons.

57:03Gary Marcus:Let me give you an example from the most recent election cycle. In Wyoming, in Cheyenne, Wyoming, there was a guy who said if he were elected, the mayor would be Chad GPT. He called it VIC for virtual integrated citizen and claimed he had built the spot. But from what I can tell, it was just Chad GPT behind the scenes. And he said it had an IQ of 155 and it can make decisions instantly. And not just decisions about how much to spend on infrastructure or whatever, but even contentious political decisions like book bans and so forth. ChatGPT is unbiased and neutral. And so we can have it make all the decisions and then we can trust it.

57:40Gary Marcus:That was his vision. And by the way, I learned about this because the Washington Post called me to ask, what are the risks of having an AI mayor? I was very confused by that question. I was like, what do you mean risks of having an AI mayor? It's like asking, what are the risks of replacing a car with a cardboard cutout of a car?

58:00Arvind Narayanan:You're saying that it just lacks the capabilities, like technically to be mayor? What exactly are you trying to say there?

58:06Gary Marcus:Well, not quite. I don't think it's a matter of capabilities. This is just to continue with this car analogy. I mean, it looks like a car, but the risk is that you don't have a car anymore. And so I think this look at capabilities is fooling us a little bit because the point of having human decision makers in politics, the reason that politics is so messy and inefficient is because that's the forum that we've chosen as a society for resolving our deepest differences. And all of that inefficiency, all of that messiness that happens is the point of politics. And try to replace that with an automated system is to completely miss the point.

58:47Gary Marcus:At best, this chat GPT mayor will reflect the views of open AI developers who have programmed it to act in certain ways. It won't reflect the will of the people. It's just missing the point of democracy. This might be a little bit of an extreme example, but one theme that comes up in our work is that so many of the decisions that happen in the course of our work actually have a moral component. We don't recognize this because that part is relatively easy for us and the cognitive component of our work is the hard one. And yet, if we're able to automate away a lot of the cognitive labor, we will still need humans making most of those decisions for accountability, for making sure that this is normatively in line with what society wants.

59:32Gary Marcus:And because with AI systems, at least as of today, and I don't know if this will change in 2050, we can never be sure that as the world changes, as these systems encounter a situation that has never arisen in the past, you can't necessarily be sure that they're not going to go off the rails in completely unforeseen ways. And so there are these unknown unknowns. So what is going to be the accountability when that happens? So I think for all of these reasons, even when you hit those capability thresholds, I think we're going to see nothing even remotely similar to an overnight replacement of human labor by automation.

1:00:08Gary Marcus:The effects are going to be much more nuanced and complex.

1:00:11Arvind Narayanan:That's so interesting. I wasn't thinking about this before talking to you, but as I hear you making your case, I'm reminded that before I was a reporter, I was a bartender for many years. And I worked at a lot of those fancy cocktail bars. and we used to joke that a machine could definitely do our job. You don't really need a human there shaking up the drink in front of you in a bar, but people just like it. They like having a friendly bartender. They like seeing us shake up their drink and put it into the little coop. And obviously this is also the case for a number of other occupations. I think about musicians.

1:00:49Arvind Narayanan:We love to connect with the humanity behind our favorite songs, right? People get obsessive about Bob Dylan or Taylor Swift or whatever. I also think this is a case when it comes to human teachers. There's a lot that could probably be outsourced to AI right now, but we like it when a human teaches another human. In a way, are you saying that this human preference that we have socially, that this is not going to go away, even if more and more capable AI systems are suddenly in our lives?

1:01:21Gary Marcus:I mean, we don't even have to extrapolate into the future. There are so many historical precedents that we can learn from. In the early 2010s, when everybody started having easy access to broadband and internet video, the big hope in the higher education system was that we have all of these professors at elite universities who are so close to the research, they're often the ones generating knowledge. surely it would be better for everybody in the world to be able to directly learn from them as opposed to their state college they go to where the professors are not necessarily at the top of their game, or so the thinking went.

1:01:59Gary Marcus:And so let's create these MOOCs, M-O-O-C, Massive Online Open Courses, where professors from Stanford and Harvard and Princeton are going to be the ones teaching everybody in the world. And, you know, you only need a handful of elite universities in the future. You don't need the rest of the university system. complete misunderstanding of what the point of college is. It turns out that a lot of what students want out of the college experience is not necessarily even hearing from professors. It's a lot of other stuff, such as the networking with their peers and so forth. But even to the extent that they're learning from their professors, it turns out that the critical ingredient is not information transmission.

1:02:37Of course, there are always books you can read

1:02:39Gary Marcus:on your own if what it takes to learn is just getting the information. But rather the fact that you're in a social environment that's conducive to learning. And that's, you know, I think at this point, at least well understood in the education community, the critical thing we're providing is that social environment. And so yes, there is a role for technology there, but I think it will have to be a matter of carefully integrating it. Replacing the social environment with a computer system is, I don't know, it seems pretty silly. It's a non-starter. It's just at the same level as your example of replacing the human bartending experience with It's something where, you know, you can certainly go to a liquor store and get your own alcohol.

1:03:17Gary Marcus:We've always had that. And so it's just missing the point of what the system is trying to accomplish.

1:03:22Arvind Narayanan:I love that. That is a very compelling case that you're making, this idea that we just like people. But let me push back on it a little bit here because there are a number of ways that even right now we are watching technology begin to replace different human interactions. And maybe this isn't a perfect example of this, but I think about young men and porn, right? Porn, it's probably as old as civilization itself. It goes back forever. But in recent years, with the rise of the internet, we have seen this massive increase in the amount of porn that men are watching. And we can compare that with the amount of time that they're spending actually engaging and investing in human romantic relationships or even in just having sex with other humans.

1:04:15Arvind Narayanan:And maybe you can extrapolate from that out also how social media has become for many people the dominant way that they're interacting with other humans. They're spending more hours interacting and looking at faces and reading the words or listening to the words on a screen than they are looking at the faces of people in the real world and interacting and talking with people in the real world, right? Isn't it the case that if we were to stay on the current trend and throw into that a super intelligent AI distractor or entertainer that we might, you know, wall-e ourselves, that we might sleepwalk into a world in which it is dominating us?

1:05:03Gary Marcus:This is exactly what motivates all of my research. The difference is I don't think this is the nightmare scenario that people are warning about. They're warning about a situation where AI itself is the adversary and it has agency and it decides to take the fate of humanity into its own hands. I think the harms we're seeing are all the result of our own collective bad decisions and the lack of policy guardrails and that sort of thing. And literally my whole career has been about trying to bring more empirical evidence into policy debates so that we can try to put guardrails into some of these systems before it is too late.

1:05:38Gary Marcus:So most recently, for instance, my colleagues and I have been spending a lot of time on the issue of AI chatbots and mental health. What we're seeing with people having suicidal ideation, some of the numbers coming out of this are really striking. OpenAI revealed that 1.2 million people per week are revealing suicidal thoughts to ChatGPT. And I think I saw a number from a Sam Altman interview recently that 1 ,500 people per week are discussing suicide with ChatGPT and then actually going on to take their own life. just absolutely striking numbers. And then so many people getting into delusional spirals because these chatbots are reinforcing, you know, they're sycophantic, they are overly eager to agree with whatever the user is saying.

1:06:27Gary Marcus:And then you get into a spiral where the user is expressing more and more extreme beliefs and the chatbot is validating and amplifying those at each step, sometimes with catastrophic consequences. There was a lawsuit recently in the case of a guy called Stein, Eric Solberg, who killed his mother and then killed himself because of this delusional spiral with ChatGPT. That's just one example of striking one tip of the iceberg, but this sort of thing seems to be happening on a massive scale. None of the people who were worried about existential risk came anywhere close to recognizing that this sort of thing might be a possibility.

1:07:00Gary Marcus:You only figure out what the actual risks are when you do the hard work of paying very close attention to the empirical data, data that's often coming from social science and this kind of on-the-ground research as opposed to theoretical speculations about hinge points at which society might change all at once in the future. So yeah, I'm very much on board. And indeed, I spent 90 % of my time thinking about and trying to mitigate exactly the kinds of risks you're talking about, which to me are categorically different from the ones that we started the episode with.

1:07:31Arvind Narayanan:So where do you think that comes from? Why is there such a pervasive belief among many of the people who are close to this technology? A belief that I think is a sincere conviction that what they're making is less like electricity and maybe more like a god or a new species, something that poses a real existential threat in and of itself. Do you question their sincerity? I've heard some people say that this is all about making money. I've also heard some people say that these technologists are not religious people, and maybe they are trying to fill a God-shaped hole in their heart by creating something like a digital God.

1:08:14Arvind Narayanan:Where do you stand on that? Why is this such a big part of the debate and the discussion among so many of the people who are close to this technology?

1:08:23Gary Marcus:I mean, like you said, these are sincere beliefs. I don't know about the God stuff, but look, I used to hold these beliefs. And so I do think I can at least share my experiences, even if I can't speak for everyone who holds these beliefs. So 25 plus years ago, the reason I decided to major in computer science was because I thought that the arrival of AGI would be a civilization defining moment. And I wanted some part to play in that. So first of all, there's a big, big selection effect among AI technologists. For a lot of folks, the reason we got into this in the first place is that we wanted to be part of something that's, I guess this is getting a little uncomfortably close to your God-related hypothesis, but we wanted to be part of something, you know, that's bigger than ourselves, that's humanity defining.

1:09:11Gary Marcus:So there's a selection effect there. So the people who are in this field are in it because they're kind of predisposed, perhaps, to think that this could be the greatest technology ever. So that's one thing. And the second is, it's true, again, like I acknowledged at the beginning, that the capability improvements have been repeatedly underestimated. I mean, think about it from the perspective of some of the folks you interviewed before, and I'm sure this came through in those episodes, to have this belief that neural network technology is capable of a lot more than people gave it credit for back in the 1980s, and to have persisted through that belief for decades while they were being ignored.

1:09:50Gary Marcus:And then to finally be proven right, I cannot even imagine what that must feel like, the idea that the world is constantly underestimating the power of these technical insights and that you and a few others alone are able to speak that truth and must cut through this kind of wall of skepticism that you're facing. I think that just intrinsically puts you in a position where it becomes very easy to ignore a lot of contradictory evidence because people have been skeptical before and been proven wrong, even though it took a couple of decades. But I think where folks are kind of unable to make a mental leap, and this is what I struggled with for a couple of decades until things started finally falling into place, is that there is not this linear technologically determinist relationship between tech capabilities and societal effects.

1:10:40Gary Marcus:There's such a long causal chain of steps through which technology gets translated to societal impacts, whether good or bad, and that those are all opportunities for collective agency over the direction of the technology's impacts. And it's only when I really started getting deeply into the writing and sociology and political science and policy studies and economics and so many other fields. And together with, you know, years of conversations with my co-author, Sayash Kapoor, that we felt that we had even a semi-coherent way of putting together what we agree is rapid progress in technology with a theory for how that is likely to affect society.

1:11:22Arvind Narayanan:And as we close here, would it be wrong for me to summarize my takeaway from this interview as essentially you are a skeptic, but not necessarily of the milestones that this industry might hit, but a skeptic of the fact that our big, messy, bureaucratic world can be utterly changed in a short amount of time. Is that about it?

1:11:45Gary Marcus:Exactly right. There's this funny example that Amtrak, with a lot of fanfare, launched its new train sets that were capable of going much faster than their old ones. But it turns out that, in reality, they got no speed improvements. Because the tracks are curved, that puts a maximum limit on your speed. And so, you know, you're improving a part of the system that's not even the bottleneck. And every time I hear a lot of these AI claims, I think about Amtrak's experience with its tracks and its new train sets. All right.

1:12:15Arvind Narayanan:So final question here. Ever since we started putting out this podcast series, we've been hearing from tons of people that it's freaked them out, right? People who are scared at some of the claims that people are making, scared about some of the predictions that people are making. and I don't like scaring people and yet I want to accurately report the debate that's happening in these increasingly powerful corners of society. I think that in this episode, people are going to hear you and they're going to breathe a sigh of relief. They're going to want to believe you. But how do you balance that with what you're also saying about how you still believe that we need to have a sense of urgency in how we investigate and regulate and study artificial intelligence and all the changes that you believe it will eventually bring to our world.

1:13:15Arvind Narayanan:I wonder, could you just speak to that listener? What is your message for them?

1:13:19Gary Marcus:I have a very simple message. If what one is worried about is existential risk, the vast majority of us can do nothing about it. The only two kinds of stakeholders who can do something are people working at these frontier AI companies and they can try to come up with some technical breakthrough that allows them to develop superintelligence without these existential risks, which we've written about we think is highly implausible. Or there can be a world authoritarian government that makes the rules for who's allowed to develop AI that surpasses some capabilities. So all the options that we have for superintelligence risk are fairly absurd ones.

1:14:01Gary Marcus:And I think that's the reason why there's a sense of panic. So it kind of leaves you with a sense of helplessness. But the kinds of risks that I'm talking about, every single one of us has a role to play in doing something about it. For a simple starting point is whatever your profession is, probably needs to come up with a normative framework for what are ethically acceptable and unacceptable uses of AI. When replacing human functions with AI is actually bringing great benefits, like saving lives in the case of self-driving cars, versus when it's the case that the human performance of something is the whole point, whether it's music or bartending or a teacher who's setting up a social learning environment in a classroom and where the incursion of AI would actually be terribly counterproductive and we need to collectively resist certain commercially motivated but ultimately socially harmful ways of deploying AI.

1:14:57Gary Marcus:That's just one example, but there are many other ones, right? So for the kinds of fears that I'm talking about, every one of us has a role to play in our own organizations, families, communities. You know, if you're raising kids, how should they be thinking about AI systems? What should they know about it? When is the right age to start introducing them to AI? So these are all things that we can start thinking about, and it's not something we should or can leave to the AI companies. And so hopefully the kind of fears that I'm talking about I'd leave you with a sense of purpose as opposed to a sense of helplessness.

1:15:35Arvind Narayanan:All right. Well, thank you so much for talking with me, for sharing your view. I really appreciate it. Thank you. This has been fun.

1:15:55Arvind Narayanan:Thank you for listening. The Last Invention is produced by Longview, home for the curious and open-minded.

1:16:02Gary Marcus:To learn more about us and to support our work, click on the link in our show notes or visit us at longviewinvestigations.com. We'll be back with more of the views shaping the debate around the AI revolution soon. We'll see you then.

1:16:24Thank you.

1:16:54Gary Marcus:available. Tap to learn more.