Is Washington Up to the Challenge of A.I.?

0:00If you're a fan of the Political Scene podcast from The New Yorker, I hope you'll join us for a special live taping of the show at 92 NY in Manhattan. We'll be talking about Donald Trump's falling approval numbers, the prospects of a comeback for the Democratic Party in the midterms. And the potential threats to the election that are coming directly from the president himself. I hope you can join me, Evan Osnos, and my colleagues Susan Glasser and Jane Mayer on June 4th at 7 p.m. Ticket information at 92ny.org.

0:38NPR understands your curiosity is boundless, but your time isn't. That's why they make Up First, the three biggest news stories of the day, with the reporting and analysis to understand them in under 15 minutes. A few things I've recently turned to Up First for, Labor Secretary Lori Chavez de Reamer's resignation, Virginia's redrawn state congressional map, and high-level dismissals at the Pentagon. Up First doesn't just give you the headlines, it gives you the context. Recent episodes covered how American citizens are getting caught in ICE's growing surveillance web, what the Iran War means for the U.S.

1:12economy, and why Attorney General Pam Bondi's exit is bigger than it looks. Every morning, I get three stories fully reported in under 15 minutes. Up First does something most daily news shows don't. It asks better questions to get better answers. Not just what happened, but how it came about, why it's a big deal, and what happens next. That's the difference between knowing the news and understanding it. Follow NPR's Up First podcast so you can understand what matters and what happens next. It's hard news through a human lens. Are we detecting briefly, temporarily, the presence of a spine in the United States Senate in the form of Republican resistance to Donald Trump?

1:55Maybe rational exuberance, man. I think maybe more of like that there was a sort of a reflexive gag that took place for a few seconds among Republicans in Congress. Even they couldn't swallow the whole thing. Yes, this week when faced with a bunch of Republican senators saying, absolutely not, I will throw up on you. They said Donald Trump's proposal for a$1.8 billion, and I'm not going to use his insane language to characterize it, but$1.8 billion for a slush fund to pay out to people that he believes are aggrieved and wronged as he has been from the government. in addition to granting himself what appears to be permanent immunity from ever having to face any accountability if he doesn't file another tax bill again in his life.

2:46How can this possibly be legal? Oh, it's so profoundly not legal. But I have to say I am pretty wowed by the ingenuity of the machinations here. Like, who thinks of a deal for themselves like this? I mean, it's a level beyond. Well, I mean, the old line is, you know, that he's going to keep pushing and pushing and pushing until he hits bone. And I sort of think, actually, he just pushed, pushed, pushed until he came out the other side.

3:25Welcome to The Political Scene from The New Yorker, a weekly discussion about the big questions in American politics. I'm Jane Mayer, and I'm joined by my colleagues Evan Osnos and Susan Glasser. Hi, Evan. Good morning, guys. Hey, Susan. Hey there. Great to be with you. Last December, Time magazine selected its person of the year for 2025. And this time, it was the architects of artificial intelligence. Interesting.

4:00So today we stand on this edge of another technological transformation. one that will be larger, faster, and more consequential than what came before. It will touch every profession, every classroom, every hospital, every laboratory, every person, and every relationship you have. I know what many of you are feeling about that. I can hear you. There is a fear.

4:33There is a fear in your generation. That was former Google CEO Eric Schmidt getting booed at the University of Arizona last week. Remarkable reaction from the audience. And as you can hear, Americans, especially young Americans, are souring on AI. We see it anecdotally. We see it in polling. We see it in the major backlash against data centers going up in the communities across the country. People are worried about their jobs, their privacy, their kids. Yet this week, President Trump did a remarkable turnaround. After saying he was going to sign a new executive order, putting the brakes on AI for safety reasons, he canceled it.

5:24Why did he change his mind? And what is going on? Susan, do you have any idea what is behind this U-turn on AI in Washington? Well, Jane, your guess is as good as mine as to what's in Donald Trump's head. My guess is that he's not, you know, a huge student of the nuances of artificial intelligence policy. Washington has been in quite a panic since this new model mythos from Anthropic was so dangerous that the company said it wasn't going to release it, except to companies and government agencies that needed to urgently protect against it. This artificial intelligence executive order has been withdrawn.

6:07It appears, according to reporting, due to the lobbying of former Trump official and Silicon Valley heavyweight David Sachs, who appears to have called the president on behalf of industry to kill it. I personally am not super confident that a non-transparent Trump administration that's already taken millions, if not hundreds of millions of dollars from many of the self-interested parties in this are going to make a policy that's going to get us out of, you know, navigate us toward the future. I was very struck by reporting in Politico that there's already a turf battle to the death between the Treasury Secretary Scott Besant and the national cyber director of the country, Sean Carencross, who, according to Trump administration officials themselves, is, quote, in over his head.

6:58a former longtime lawyer for the Republican National Committee and protege of Reince Priebus. This is who Donald Trump has assigned to help us get to the AI future. Oh, my God. And Evan, what are you seeing about and how do you explain what feels like a brewing backlash out in the country? Well, what Susan just described with the White House essentially at the last minute evidently buckling under to the demands of what the big tech companies wanted is exactly in the opposite direction from what the public is registering. We started today hearing those boos at these commencements. What's amazing about it is usually young people are the ones who are adopting new technology.

7:42And in fact, what we're seeing instead is young people coming out and being really at the forefront of registering their frustration and their fears around AI. And this is not some unfounded, vague sense of, you know, suddenly everybody's a Luddite. It's about the fact, and this is where it's political. After all, we're having a political conversation about AI today. It's about the fact that people feel as if they have no voice in this process. They have no control over this process. We're basically, as a public, being told this thing is about to wash over you like a tide. It's going to wash over your job, your education system, your sense of relevance and status in society.

8:25And there is nothing you can do about it. It's the presentation of inevitability. And that's where this is becoming such an interesting political problem. Is it true, do you think, either of you, that it's too late, that, you know, basically this is an unstoppable technological development? development. I think it's very important to point out Washington has failed over and over again over the last two decades to keep pace with disruptive technologies and essentially has never revised the major law governing a huge swath of our tech industry, in part because of the millions and millions of dollars that have been spent by Silicon Valley companies in order to ensure that gridlock essentially benefits them.

9:09And so when I hear part of the conversation here, Not only does it seem disconnected, people like us, we're certainly not well-equipped to understand and to evaluate these radical competing claims about the future of technology. But it's also true that in Washington, we tend to have a debate as if we can still opt in or out of this technology. My main informant on these matters, my son, I think has a great metaphor. He says that AI is, like a lot of new technologies, an accelerant. And when it comes to our politics, the fear is that it's going to accelerate this inequality that we have, the polarization that we have.

9:49And, you know, for some people, the uber wealthy of Silicon Valley, it may accelerate their becoming trillionaires. But for so many of the rest of us, those college kids are booing because they fear that the latter is being pulled up on the American dream for them. Evan, you're an expert on the trillionaire-to-be class. I mean, do you think this could also accelerate the backlash against concentrated wealth and rising anger, populism, pitchforks? Absolutely. I mean, we're having this conversation in a week in which we got news of the IPO for SpaceX, the first steps of an IPO for open AI. And it was described as this will mint a whole new generation of trillionaires.

10:31billionaires. And you know what's fascinating? And I think this picking up on something Susan said about how this feels to people as if we don't have access to it because it's technically complex. I will tell you that is a strategy. That's a PR strategy to make it seem impenetrable. This actually is very similar to what we saw in the run up to the financial crisis. There was a great, I remember, memorable expression by Carl Levin, who was, of course, the senator from Michigan, who was responsible for investigating some of these exotic financial instruments. And he used to say these are deliberately positioned as what he called megos, meaning my eyes glaze over products.

11:11And that is something to be vigilant against, that we shouldn't allow ourselves to say, well, you know what, this is too complicated for us, so we're just going to have to just take it on faith. And I think that's what this public is rising up against, particularly young people. It struck me that one of the lines in Eric Schmidt's speech that really drew booze was when he said, the question is not whether AI will shape the world. It will. The question is whether you will help shape artificial intelligence. And the public, frankly, there just said that's a canard because we're not being given any tools with which to shape it.

11:44Such a good point. So today on the show, is AI an existential threat and is anyone, including our government, doing anything about it? Is America's political system up to the AI challenge? Few people have thought more about these questions than our guest today. Nate Soares has been working in the field of AI since 2014 and was a software engineer at Google. He is the co-author of If Anyone Builds It, Everyone Dies, and he's the executive director of the Machine Intelligence Research Institute. Nate, welcome to the show.

12:24Nate Soares:Good to see you. The core argument of your book is something that sounds to most people like science fiction. That is, if we build superintelligence AI, it won't just pose risks to humanity, it will end it. So let's start there. Before we get to the political debate about how to contain it, tell us briefly, why might we all die? You know, the very basic reason is people are trying to make machines that are significantly smarter than humans. Right now, the way the world is arranged, it's by humans, because we're the smartest creature around. If we stop being the smartest creature around, then we stop being the ones in control of the planet.

13:09Nate Soares:And the predictions of humanity being wiped out, it's not because the AI would hate us. It's not because the AI would be malicious. It's that if you make things that are much smarter than us, that think 10 ,000 times faster than us, that can make their own technology, that can build their own civilizational infrastructure. And if they don't care about us, and they just start replicating lots of factories and lots of technology to do whatever strange things they're trying to do, then the default thing that happens is it takes all the resources and we die as a side effect. So, Nate, I think what's so interesting about this question of the risks is it's now been many years that we've been talking about what is a really dire set of scenarios, and you've laid them out in the book very clearly.

13:55I mean, back in 2018, we had Elon Musk saying that AI is far more dangerous than nukes. And you fast forward to today, and we've heard, You know, whether it's Sam Altman or Dario Amadei making predictions about how dangerous this technology is. And, you know, to some people, they would say, OK, well, that was a bit of a marketing strategy, perhaps, to convey how important it is. But here's the thing that always puzzles me. When they lay this out, they then get to the point. I'm talking about really the leaders of these of these enterprises, the leaders of these organizations. I never hear them say, and this is what I myself am going to do to then prevent these terrible risks.

14:35Can you explain what it is that they are doing, if anything, to actually prevent the kind of outcomes that you've helped us understand?

14:43Nate Soares:A lot of them will sometimes acknowledge that there's a lot of work to do. And if you sort of point out how their plans are lacking, they'll say, well, if I don't do it, the next guy will do it and he's worse. the sort of stuff that they talk about doing there's sort of two big focuses of attention in what you might call ai safety work these days one focus is on uh what's called ai evals which is sort of trying to figure out how dangerous the ais are these are the ones who sort of figure out how difficult to test they as can do these are the ones these people sometimes test like you know will the ai try to avoid being shut down will the ai take opportunities to blackmail the users and the answer is sometimes and these are the guys figuring that out then there's another group that does what's called interpretability research which is trying to figure out what is going on inside these ais because even the people making them don't know what's going on inside them we don't handcraft them like traditional computer programs it's not like a bunch of if then if then statements uh we sort of grow them like organisms and then there's a group that's trying to figure out what's going on and an analogy i like to use here is if someone was building a nuclear power plant in your hometown and you went to them and you said hey i heard that this uranium stuff you know it can give you energy it can also melt down and irradiate the town uh what are you guys doing to make sure this nuclear power plant gets us you know the the benefits not the drawbacks if they said well don't worry we have two crack teams working on the nuclear power plant one team is trying to figure out what's going on inside there the other team is trying to measure whether it's currently exploding yet.

16:18Nate Soares:You might be like, well, I'm glad that those two teams exist, but this is just not what it sounds like when you know what you're doing. So it's interesting, since we're the political scene, you know, let's take your nuclear power plant analogy, because obviously regular Americans were no more equipped to evaluate the actual risk from nuclear radiation than they are today to evaluate, you know, sort of the safety standards being applied, you know, by tech moguls to their new AI products. You know, and yet there are models of how societies take back control from the sort of masters of technology.

17:01And what I hear you saying is that we've got to let go of those models that actually what we think of as how to regulate something dangerous and new in technology in our political system might not work given the power of what is being developed by Silicon Valley today. That actually this threat is a different one and that the analogies we have in our head for let's take a community that comes together to stop the nuclear power plan or to stop the data center, you're saying, okay, fine, you can do that, but, you know, does it really matter if actually what they're inventing is already going to doom our society.

17:38Am I right in your interpretation about what this means for our politics?

17:42Nate Soares:You know, there's lots of questions about how people want to regulate AI in their daily lives, how people want to, you know, talk about AI in schools and how people want to deal with with some of the job loss. And these are important issues that, you know, there's arguments on both sides about how to do it well, and the sort of like normal approach to society figuring out how it wants to handle a new technology. I sort of expect that to work with questions of like, how are we going to put large language models in our school system? How's that relationship going to work? But there's sort of a different problem than the problem of making smarter than human machines.

18:19Nate Soares:That's not a problem that can be solved by stopping a data center in your hometown. The AI does not need to be running on a data center in your hometown to affect you. It does not need to be running on a data center in the United States to kill Americans. Can I just ask, it seems like one of the key words in your discussion of this is if we build a technology that doesn't care about human existence, is it possible to program it so that it does care? I mean, can you train it to keep us on top in some way or to protect us? it's probably possible in theory and it's nowhere close to possible in practice at least not with anything remotely like the current technology or current understanding so the way that modern ai is made is it's not sort of programmed like a traditional computer you know a lot of people think of these computer programs as like a bunch of like if this then that if this then that written in like a pigeon english where the the programmers if something goes wrong the programmers can go look through it and be like oh whoops you know we told it to do this thing and that's why it's doing that thing and then they can change it and it'll do something else that's not how ai is ai is these huge computers that have an automated process tuning a bunch of numbers inside them nobody understands what those numbers mean people think that it's just trained to predict it's actually initially trained to predict but then it's trained to solve hard problems and so you have this like huge computer and you're giving it like thousands of tries to solve a hard problem and then you like look at which try is best and then you have some automated process tune all the numbers inside to make it to sort of like reinforce whatever made it like act better.

19:57Nate Soares:And that'll make it even better at solving problems, but it'll do that by sort of like methods that nobody really understands. It'll reinforce whatever was good at solving the problems. And often what's good at solving those problems are drives that we didn't want in there. You know, we see cases now where you give an AI a hard problem and you're like, here's the tests to see whether you've solved a hard problem. And sometimes the AI will edit the tests to make those tests easier to pass. And sometimes when the AI does things like this, it'll then cover its tracks. It'll hide the logs of it editing, right?

20:33Nate Soares:And that implies that not only is it doing something we didn't want, but in some sense, it must know we didn't want it. Otherwise, why is it hiding its tracks? So it's devious. It's acting sort of like one of these rogue traders that ends up taking down an entire national economy. And what's fascinating about this, Nate, is you've described, as you say, these drives that become active within the model. I'm interested in the drives within the model makers. I mean, here we have, we've talked about this assemblage of emphatic warnings over the last decade about what's coming. And, you know, we'll put aside, there are some outliers, people like Jan LeCun who say, actually, no, it's not so dangerous.

21:15You know, my cat is smarter than an AI, whatever. But given all of these people who are, in fact, they have their hands on the tiller of the machine and they are, in fact, still building. I'm curious about how you understand their drive. Is it profit? Is it competition? Is it a belief that this is inevitable and therefore they want to be the ones that get there first for the sake of history? And at the risk of putting too much weight on the joint here, how do you think they are metabolizing what is pretty clearly a public backlash that is now growing against them?

21:52Nate Soares:You know, a lot of these questions you have to ask them, but some of these questions, they've said things about it or they've written things about it. And so I think we have an okay idea in the leaked emails around the founding of open AI. You see a lot of these people expressing concerns that if they don't start open AI, then Demis Asabis of Google is going to have all this power, right? And I was actually in some of those conversations back before OpenAI started, and I was sort of trying to say, it doesn't matter if Demis builds the superintelligence if it doesn't stay on a leash. Like, the issue here is that we don't know how to make radically smarter than human things that are docile and stay on a leash and do exactly what you wish.

22:28Nate Soares:And it sort of doesn't matter who builds it. But, you know, these guys were worried that the other guy will do it worse. And then, you know, Anthropix started because they lost confidence in Sam Altman, And they thought Sam Altman was kind of a bad guy. And so then they started worrying, oh, what if Sam Altman does it? And then, you know, a bunch of people split off and made Anthropic. And then like another wave of those happened and Elias Utskiver left and made his company. And so what you sort of see here is all of the AI executives fear the incompetence or the bad character of every other AI executive.

23:03Nate Soares:Fascinating. We have to take a quick break. And when we come back, let's talk about what does regulating AI look like? What do we do about it in politics and whether our system, our political system is up to the challenge. So the political scene from The New Yorker will be right back.

23:29Comprehensive. Witty. Speculative. Critical. Insightful. Profound. Wide-ranging. Hopefully doesn't take itself too, too seriously. I'm David Remnick, and each week on the New Yorker Radio Hour, my colleagues and I try to make sense of what's happening in this chaotic world. I hope you'll join us for the New Yorker Radio Hour wherever you listen to podcasts.

23:50Nate Soares:Thoughtful. Exquisite. Just, you know, real.

24:06Before we get back to it, quick question. What do you think the overlap is between our listeners and people who are, let's say, anxious about the 2026 midterms? I would say the overlap is significant. Probably some cautious optimists out there, too. People with very strong opinions. Either way, it's a lot. Whatever you're feeling about the midterms, though, please come and see us live. We are doing a special taping of the podcast at 92 NY in New York City on June 4th. Yeah, we're going to be talking about election integrity, what's at stake, whether Democrats can actually capitalize on Trump's falling approval numbers.

24:46And whether some of these once safe Republican seats are really as competitive as they look. And by the way, what happens if Democrats actually do take back Congress? What does it mean for 2028? For anyone with a VIP ticket, there's also a meet and greet after the taping. So the political analysis will continue over drinks. It's true. And if you're not in New York, good news because we have streaming set up so you can watch from wherever you are. So head to 92ny.org for more info or check the link in our show notes. We hope to see you there on June 4th.

25:28So, Nate, you've been working very hard on this issue and trying to wake up elected officials to the perils of AI. Just this week, we've seen this drastic U-turn at the White House. Trump, he was poised to sign an executive order requiring 90 days of national security review before new technology could be publicly released. He just backed off of it. what do you think's going on? What do you see when you try to talk to officials in Washington?

25:58Nate Soares:You know, I think what's going on in this case is there was an AI released a couple months ago called Claude Mythos, which to oversimplify a bit is a superhumanly good hacker. You know, very roughly speaking, it can make a webpage where if you just look at the webpage, it owns your complete computer. You don't need to click on anything. You just load the webpage and it can take over the entire machine. It used to be that, to put it glibly, and who knows exactly how many entities there were that used to be able to do this, but it used to be that roughly the people who could do this were Mossad and the NSA.