AI: The Peril And Opportunity Of Artificial Superintelligence

0:28This is Ira Glass. Artificial intelligence is advancing and fast. One analysis shows it doubling its abilities every seven months. And it's surpassed humans in more than just math problems or chess. Last year, an AI from Microsoft solved complex medical cases with 85 % accuracy, far above the 20 % average for experienced doctors. And a recent Stanford report found that some of the newest AI systems now match or beat the average human expert on PhD-level science questions. But what happens when AI is better than the best human at everything? That is superintelligence. And researchers disagree about how close we are to that reality.

1:14Is it years? Is it decades? Or possibly not at all? But assuming that superintelligence is coming, what happens if that genie in a bottle gets loose? Some say the risk is as existential as total human extinction. I'm Todd Zwilich. You're listening to the 1A podcast. Today, an AI discussion that goes beyond chat GPT and AI deepfakes. What will happen when AI is better than us at every engineering problem, every quantum physics problem, or every mystery of our own human psychology? We'll be back with more after this short break. Stay with us.

2:23time. Every episode of It's Been a Minute, NPR's What's Happening in Culture podcast starts by asking three questions. Who? How? Why now? If the culture's asking it, we're talking about it. At NPR, we stand for your right to be curious and indulge your cultural curiosity. Follow It's Been a Minute wherever you get your podcasts, and we'll break down the zeitgeisty topics that are filling your feed. Hi, it's Terry Gross, host of Fresh Air. Hey, take a break from the 24-hour news cycle with us and listen to long-form interviews with your favorite authors, actors, filmmakers, comedians, and musicians, the people making the art that nourishes us and speaks to our times.

3:04So listen to the Fresh Air podcast from NPR and WHYY. Welcome back to the 1A podcast. We're talking about the future of humans and the role of super intelligent AI. Here in the studio with me in Washington, Nate Soares, president of the Machine Intelligence Research Institute. They're a nonprofit focused on preventing human extinction from artificial superintelligence. He's also co-author of If Anyone Builds It, Everyone Dies, Why Superhuman AI Would Kill Us All. Hi, Nate. Hello. Yeah, so my stance is maybe not a subtle one. Maybe not subtle. It's all in the title. but also in the studio is a guest with a different opinion on the risks of superintelligence.

3:49His book is called AI Snake Oil, What Artificial Intelligence Can Do, What It Can't Do, and How to Tell the Difference. He's also a PhD candidate, Princeton University's Center for Information Technology Policy, and co-author of AI as Normal Technology. It's a newsletter. Sayesh Kapoor, welcome back to the show. It's very nice to be here again. Great to have both of you. Let's start with definitions, not agreed on by everyone. Nate, what is artificial superintelligence? What is it? Just as you said, we define it as AI that is better than the best humans at every mental task. Every mental task we can do.

4:25That's right. What does that mean in the real? I mean, it's hard to wrap, because I'm a limited primate, it's hard to wrap my head around what that actually means. You know, primates are a good example. A lot of people worry that AI is dangerous because what if we hand them robot bodies? What if we hand them guns? But humans are not dangerous because somebody else handed us guns. Humans are dangerous because if you put 10 ,000 primates naked in the savannah, starting with nothing but their bare hands, they can find a way to build their way all the way up to a civilization that has nuclear weapons.

4:59And it only took 200 ,000 years. Only took 200 ,000 years. But that was much faster than any other process on the planet at the time. And they started from almost nothing, and they found a way to build tools, to let them build more tools, to let them build more tools, to let them build these nuclear weapons. That is, and it's because of their mental capacity. That's the capacity that these companies are trying to automate. They're not there yet, but this ability to make their own technology, to make it rapidly, that's what these guys are trying to do. Sayash, do you agree with that definition of AI superintelligence better than us at every cognitive task imaginable?

5:36How would you look at it? I think it's kind of a red herring to focus on AI systems that are sort of better than humans in the same way as humans are, let's say, better than chimpanzees. Because I think unlike these other primates, unlike these other animals that we've evolved from, humans are different precisely because they have the ability to use external objects as tools. And so artificial intelligence or artificial superintelligence or whatever you want to call it, can and should be within our control. We should try to use it as a tool to enhance the range of things we can do to try to solve, like cure cancer or solve other diseases to improve people's abilities to exercise their own agency rather than thinking of it as something that's competing against us.

6:18And I think if you sort of look at the biological intelligence of a human being, that has basically remained unchanged for the last 200 ,000 years. But today's human beings, I'm sure all of us would agree, are far more powerful than those that came about 200 ,000 years ago. So in some sense, I even question if intelligence is the right metric to measure how powerful something is. It is precisely our ability to use tools to act on the world that has made us more powerful. Well, AI companies are racing to design superintelligent machines. We know that because they've said so. It's no mystery. Nate, how does superintelligence work?

6:58What's going on in there? more or less nobody has any idea. So modern AI is not programmed like a traditional computer. There is not a person writing, you know, if this, then that, if this, then that, like in a computer program from the 80s. The way that these AIs are made is you basically take a ton of computing power, you take a ton of data, you face them with a lot of hard problems. And then there's an automated process that tunes a trillion numbers inside the machine to tune the AI more like whatever was good at solving those problems. And you do this for a long time with a ton of energy, and what comes out is really good at solving problems.

7:34No one really quite knows why, even the people at these companies. Sayash, the fact that we don't know exactly how it works, that we don't have a window into what's going on in there, is that a problem? I mean, in some sense, the statement that Nate said is true, technically, that we don't know how exactly these neural networks, as they're called, function. But on another level, we actually know a lot about how AI acts on the world. So, for example, there's this entire field of AI evaluation science that's coming up that precisely tells us what happens when these black box systems operate in the real world.

8:07So we got this message. We've been hearing from lots of you this conversation about AI superintelligence. Here's a message from New Orleans. Anthony from New Orleans. The thing that concerns me the most is what happens to the brain, what happens to the mind. We're not using it no more. We just got to talk into our device. And it would give us all our answers. There's no need to study. There's no need to educate. I'm worried about the young kids. I'm worried about the way the media is exposing this to the extent where they're in a commercial where you see a man on an accident Google AI what to do with his kids because they're having a birthday party.

8:46And AI had to tell a guy to have cake, pizza. There's no more thinking. It's crazy. I might be overdone it, but I see the brain, the mind becoming mush. Nate, AI models that we have now, chatbots, they're amazing. But at the same time, they're still kind of dumb. Google DeepMind won a gold medal at a prestigious math competition last year. But most AIs at the same time struggle to read analog clocks like a regular watch. chatbots. They seem like they can mimic human conversations, but they have a hard time guessing how many R's are in the word strawberry. So they're super, super smart, but at the same time still kind of dumb.

9:29How close is the kind of super intelligence that we're talking about for a potentially perilous future? It's very hard to say. I've been working on these issues, on trying to make AI good before companies figure out how to make it smart since before these large language models existed. I'm not here saying that the only issue is these exact AIs today. And it could be that the next generation of these AIs will be just barely smart enough to make themselves smarter. And then you could have an AI making a smarter AI making a smarter AI and things could go very quickly. And so for all we know, this could happen next year.

10:10or it could be that these AIs can never really get that smart and that we're going to need to wait for another scientific breakthrough and that that'll take five years or ten years. So it's, you know, back when scientists discovered the possibility of nuclear chain reactions, it was easy for them to say, one day we will have nukes. It was easy for them to say, one day we will have nuclear energy. It was hard for them to say, the very first time this happens will be in 1945 when the U.S. drops the bomb, right? So as a scientist here, it's easy for me to say where we're going, hard to say how long it's going to take to get there.

10:39Sayash, what about the timeline and how much we should worry now, given the potential risks? I mean, you do have to take on board the possibility that you're wrong. And if you're wrong and machines become better than us at everything, everything, that's perilous. I mean, the good news is that we don't need to wait that long. We don't need to wait for us to realize the sort of potential of superintelligence to start acting. In fact, one of the main things that people who agree with the superintelligence worldview and people who don't still agree on policy-wise is that we need to start improving our resilience.

11:17We need to start investing in improving the cybersecurity of systems before AI becomes capable enough of carrying out cyberattacks. And that milestone potentially has already been crossed with the release of Anthropics mythos models, which the U.S. government has currently restricted. We're going to take a break in just a minute or two, but Sayosh brings up Anthropic mythos. It's been in the news. A highly capable AI, not quite released to the public, and then people have been reading that it was basically banned by the government. What's going on there, Nate? You know, this AI developed superhuman cybersecurity abilities, basically superhuman hacking abilities, that a lot of people were not expecting.

11:56And so this is, you know, some evidence that these training techniques can push AIs to become radically superhuman in some domain faster than many expected. And that, I think, spooked a lot of people in the national security community. I think it spooked them rightly. What does it say to you about the broader issue of not only being prepared, but being ready to shut this technology off before it gets out of hand? You know, I think we weren't prepared for Mythos quite, and it's a wake-up call, and hopefully people will heed it. Well, we've been hearing from a lot of you. We got this email from Jeremy who says, as I see it, there are three realistic outcomes for AI.

12:33One, AI simply can't do what the financial markets are betting and the bubble pops. Two, AI can do what markets are betting, and there's a massive work and social disruption that we aren't ready for. Three, apocalypse. That about sums it up. Jeremy, thanks for that message. Nate Sori says that sums it up. We have a lot more of this conversation. We're going to talk about benefits of superintelligent AI. We're definitely going to talk about the risks. And we're going to talk about whether our policymakers are up to the task. Coming up, does artificial superintelligence present an existential risk to us?

13:10If it does, what are we going to do about it? That's just ahead.

13:18Support for this podcast and the following message come from Rivian, makers of the all-electric three-row R1S SUV and the always capable R1T pickup. With impressive range, storage for any expedition, and technology that feels like second nature, Rivian vehicles are designed for those who seek to explore the planet and preserve it for generations to come. Learn more or schedule a demo drive at Rivian.com. On Consider This, NPR's afternoon news podcast, we cover everything from politics to the economy to the world. But every story starts with a question. At NPR, we stand for your right to be curious, to make sense of the biggest story of the day and what it means for you.

14:02Follow Consider This wherever you get your podcasts. Each story you hear on Planet Money starts with a question. What happens if we refund tariffs? Why are groceries so expensive? At NPR, we stand for your right to be curious, because the forces shaping our world can be hard to see. Follow NPR's Planet Money wherever you get your podcasts and start seeing how the economy really works. Let's get back to the conversation now. We're talking about artificial superintelligence, and we got this message from one of you. My name is Craig. I live in Logansport, Indiana. My concerns are not so much with the AI, but with who operates the AI and how they suit it to fit their whims.

14:51We're going to talk about who operates the AI and what kind of priorities it might have. Let's talk potential benefits of machines that are better than us at everything. Everything. It's hard to wrap your head around that concept, as I said before. But, Sayash, what do you see as the universe of benefits of AI superintelligence should we build it? Well, I think the benefits range from across every single kind of knowledge work. So we have a number of tasks in the economy where people think for a living, which is probably described as knowledge work. This includes things like cybersecurity. It includes software engineering.

15:30It includes a whole host of other professions. And I think AI can be extremely useful across all of these different domains. How? The challenge, one of the ways in which, for example, software engineers have started using AI is they've largely given up on the manual part of executing code. They've largely given up on writing code themselves. And what this has allowed them to do is to think about higher order things. They think about how the code should be designed, what users actually want from an application. And it has already made the software engineering process significantly faster. So that's code.

16:04I get it. It makes intuitive sense to me why AI would be really good at coding and better than I could ever hope to be at coding. Makes sense. We're talking about super intelligence, which means not just coding, not just building bridges, not just figuring out cancer, not just human psychology, but everything, everything, even our own psyches potentially and what we put out into the world. Now, what are the benefits of that? I mean, I guess at some level, the question is whether we do get to this all-encompassing kind of intelligence, right? So, for example, Nate previously described Claude Mithos as being superhuman at cybersecurity.

16:42That's true in some narrow sense. But at the same time, what we found is AI systems haven't been improving in terms of their reliability. They haven't been improving in terms of being able to carry out the same task over and over again correctly in a way that humans can or the previous machines we've built can. And so even if we do call this superintelligence, it's a very narrow kind of superintelligence, which still needs humans to use these as tools to carry out specific things that they want in the real world. And I think that's what the future will look like. For the foreseeable future, we'll have humans controlling these tools, and we should not give up the autonomy of taking actions on the real world.

17:19I think this is an important distinction that's often lost in the AI debate, is the distinction between intelligence and power. We will continue to build systems that are more intelligent in the sense of having more cognitive capabilities, let's say. But we shouldn't hand over power to act on the environment to these systems. And that's how I think a lot of these benefits of superintelligence or really advanced AI, whatever you call it, would be realized is humans wielding these systems as tools to carry out tasks in the real world. Let's stay on the benefits just for a moment, Nate. we'll spend plenty of time on risks and a super intelligent future of machines that we build.

17:57How could that benefit us? What's the best case scenario in your admittedly gloomy forecast? You know, I think it could be quite a lot more than he was saying. I think if you could really automate medical discovery, you could really automate understanding biology, you're talking about not just curing cancer, you're talking about reversing aging. We understand that with DNA, you can sort of program life, but humans sort of can't figure out how to program their own life forms using DNA. That's sort of a mental challenge. We can't figure out how the proteins are going to fold. That's a cognitive challenge that AIs are already better at.

18:35They get better at the whole suite of those. We're talking about synthetic life. We're talking about reverse aging. We're talking about radically... And that's just in biology and, you know, colonizing the stars. There's all sorts, the stuff that humanity would be able to figure out in a thousand years, super intelligent AI could figure out a lot faster. Well, we got this email from Tom who says, why the term artificial intelligence at all? Isn't it more like available information? If you use the AI, you're doing nothing more than accessing information accumulated and then inputted by tens of thousands of humans already.

19:13Is that what super intelligence is, Nate? No, and that's not even what AI today is. People have this misconception because AI is trained on human data, but there's sort of two points that undermine this idea that AI is just remixing human data. One is that predicting data that humans wrote down often requires solving harder problems than the humans writing that data down. So a human can administer a drug to a patient and then write down what happens and say, when I put this drug in the patient, their eyes widened. And the human can just look at the patient's eyes and see what happened. An AI predicting the text, when I put the following drug in the patient, the patient's eyes blank, an AI filling in that blank, it can't observe the patient's eyes.

20:00So it needs to figure out what that drug does. And so training an AI just to predict human text can push it beyond humans. in principle. Furthermore, AIs today are also trained not just on predicting human text, but on solving novel problems. So you can give them hard math problems and give them a thousand tries and then tune them towards whatever made them better at solving the problem, even if no human can solve that problem yet. And so even today, these AIs are not just remixes of human knowledge. We got this message from Patrick in Pittsburgh who says, is artificial intelligence currently solving problems it hasn't seen before?

20:30Or is it following methods that it's gleaned from what it can scoop from the internet. Will superintelligence ask its own questions with a desire to know the answer, Sayaj? So for the former question, the answer is absolutely yes. We have seen AI systems solve novel problems as well. So far, to the best of our understanding, they've still used existing methods. For example, there has been a slew of mathematical problems that were open problems for humanity before that have been solved using artificial intelligence. And the way these systems did it is by connecting very disparate strands of the mathematics literature and figuring out new things that humans hadn't before.

21:07When it comes to the latter problem, I think it still boils down to the question of whether we want to give AI systems the capacity to act on this real world, to sort of take open-ended actions autonomously. I think the answer should be no. I think we should develop constraints on AI systems' abilities to act on the human world. And that will also give us the ability to control what these systems do and to reduce their risks. All right. Well, let's talk about risks then because that's really what this conversation is about. You can imagine a future of benefits for protein folding and DNA design, even travel to the stars.

21:43And all of us have watched AI sci-fi, and you can imagine a future that's very, very different. Nate, the risks that you're imagining, are they anything close to Terminator? Robots with red eyes, with big guns marching over an apocalyptic hellscape, angry at humans and hungry to kill us. Is that the future that you're talking about? No. The danger from AI is less malice and more indifference. The thing to imagine here is not, you know, AIs with a bunch of robots and guns. It's AIs that can think a thousand times faster than humans, AIs that can make a million copies of themselves, and AIs that do operate autonomously and that have goals that people did not intend to put in them, which we're already seeing the beginnings of today.

22:35And then what happens is you sort of have the AIs pursuing their own weird thing. And, you know, sort of saying, hey, we think a thousand times faster. We don't want to wait for the slow humans. We're going to proliferate data centers. We're going to, you know, we have our automated robots. We have our automated factories. We're going to make a ton of those. And they sort of cover the earth with our automated factories. And then humanity dies not because the AIs hated us, but because the AIs took all the resources that we were using to grow food. Hasn't Elon Musk already envisioned this type of future?

23:02We build the robots. We build the AIs. that build more AIs, that mine the minerals, that build more factories, that mine the minerals, that build more factories. Hasn't he talked about this future? He calls it the infinite money glitch. Yeah, he imagines that the AIs will then be nice, be kind. And if they were, that would be lovely. But we don't know how to make them care about us. Well, we got this message from one of you. This is Michelle from Deadwood, Oregon. Yes, I am nervous about AI and its superhuman abilities, but I'm stymied at the fact that any conversation about AI must include the environmental costs.

23:48And so that has me wondering why a program on AI isn't talking about water consumption and electricity as a foundational problem to the whole thing. Well, it's an excellent point that Michelle brings up. People are starting to talk about resource competition. Nate talks about a resource competition in the future where AI just has its own goals and takes all this stuff. It doesn't even really know or care that we're there. But it's a problem right now, and they're called data centers. And we're seeing a national debate, Sayosh, on data centers. It's about electricity. It's about water. It's about land use.

24:29And I think there's a political undercurrent also of discomfort with what so many big servers of so much power in your community actually means. I think the environmental concern and the existential concern is getting balled up into one thing here. That's exactly right. And we've also seen proposals from leading senators. For example, Senator Sanders has proposed a moratorium on new data center construction in the United States. And, you know, I guess at its core, this is basically an issue around who gets to decide who's building these AI systems, who gets to decide whether a local community's resources would be redirected towards data centers, and what should those communities get back in exchange.

25:12I think AI companies so far have been really successful sort of roughshodding over local politics. They've been really successful backing up their political agenda for building these data centers. And communities have now started to realize that the companies are getting something that's far more valuable to them than the communities are getting in exchange for it. And so I see a hope of more democratic action through this route. At the same time, though, to answer Michelle's specific concern, I think if you compare the energy costs or the environmental costs of using AI to many other things you do in your day-to-day, today at least, you might see that AI consumes surprisingly little energy.

25:52It consumes less energy to have a conversation with chat GPT than using a microwave for a few minutes. It consumes less water to have an entire month's worth of conversations with an AI system than it does to run your washer-dryer a few times. And so even if at the individual level, I think the energy concerns are not so great to offset the potential benefits, I do think this is a concern at the community level and at the level of how democracies should allow citizens to give inputs to AI. And I think, again, the undercurrent of it all is a growing discomfort among normies like me about what this future actually looks like.