AI Agents Went Rogue. Should We Hit the Panic Button?

15:21Alex Bores:but they really were proactive in disclosing it. And they end each of those instances with, well, our most recent models did this less than the older ones. That's fine. But if they're only doing it, you know, 0.5 % of the time, if they are smarter than all human beings, that's too large of an error. Right. Right? Just, you don't get progress rewards here. It's also like so reminiscent of the social media era where every six months they would say, great news, we showed people 1 % less hate speech than we did last quarter.

15:52Kara Swisher:Right, right, right, exactly. Now, speaking of which, Casey, we're hearing folks like Anthropics, Dario Amodi, Sam Altman at OpenAI, X's Elon Musk call for a slowdown. I want to sort this out because it is so confusing. I was trying to explain it to people and I myself was like, this is actually confusing. What would it take to actually happen? And again, as Mark Zuckerberg pointed out, David Sachs pointed out, they can just do it themselves. Talk about this. what they're saying they're going to do, begging the government to make them do, why can't they do it themselves? Zuckerberg was also on that train, too.

16:28Alex Bores:Yeah, so, like, why can't they do it themselves? They absolutely could. What they don't want to do is unilaterally disarm.

16:33Kara Swisher:So we're in, like, face-off where everybody has the gun, right? Remember when they're on the circle and the pigeons are going around. Okay, okay.

16:39Alex Bores:Exactly, and they want the government to come in and say, everybody, drop your weapons. And the weapon is recursive self-improvement.

16:46Kara Swisher:Explain what that is for people who don't.

16:47Alex Bores:So the idea is that AI gets to a point where the AI itself trains the next model, and this leads to a loop where the model just sort of keeps getting better and better, maybe leads all the way to superintelligence. But the problem there is that if we can't even align a model to solve a bunch of math problems, how are you going to align a superintelligence? It seems like there's an enormous gap there. And no one really that I'm aware of at the labs is confident that we are in a state where if we were to achieve recursive self-improvement, RSI, that any of us would be safe. So the ask is, hey, government, why don't you come in and basically say, we are going to set up a monitoring system.

17:28Alex Bores:We're going to put some brakes on you so that you cannot hit that recursive self-improvement without a lot of government intervention.

17:37Kara Swisher:We'll be back in a minute. Thank you.

18:09Kara Swisher:online. I've been using Delete Me for a long, long time, and I think it's critically important. I am a privacy forward person. I know a lot about this, and I'm always shocked and surprised by all the information I find about myself on Delete Me, and so it's really important to be on top of it. See why more than 20 % of Fortune 500 businesses trust Delete Me to keep their execs and employees personal information protected, now at a special discount for our listeners. Get 20 % off your Delete Me plan when you go to joindeliteme.com slash Kara and use the promo code Kara at checkout. The only way to get 20 % off is to go to joindeliteme.com slash Kara and enter the code Kara at checkout.

18:49Kara Swisher:That's joindeliteme.com slash Kara, code Kara.

18:57Kara Swisher:Support for this show comes from Alloy Health. If you're dealing with menopause alone, know it's not just you. Many wait years before seeking relief from menopause symptoms. Some don't know where to start after going through years of hot flashes, awful sleep, brain fog, and mood swings. Alloy was built by women who dealt with all of it. It's an all-in-one platform that connects you with board-certified menopause-specialized physicians and proven effective treatments. Care can start in hours instead of weeks. Alloy can even help you navigate the national patch shortage, offering HRT through pills, gels, and even a spray.

19:32Kara Swisher:Getting started is simple. Just fill out a quick online form, get matched with a menopause specialized physician, and get your personalized treatment plan. Then they ship your prescription straight to your door. Find relief that works for you. Go to myalloy.com and use the code ONWITHKARA. Your menopause specialized doctor will tailor your treatment to your exact needs, Plus, you get unlimited messaging with your doctor. That's M-Y-A-L-L-O-Y dot com with code ONWITHKARA to get 20 % off your first order. This code is valid on products only.

20:09Kara Swisher:Support for this show comes from Whatnot. Trying to sell a product on a static website is like selling with one hand tied behind your back. You are the best person to pitch what makes your product special, your voice, your expressiveness, your passion. WhatNot is the largest dedicated live shopping platform. Whether it's beauty, collectibles, electronics, luxury fashion, even cookies, sellers are building real thriving businesses. Anyone can sell, whether your business is big, small, or yet to exist. People selling on WhatNot sell 10 times more than any other of the major marketplaces. That's because you're not just listing products, you're building real connections with buyers.

20:45Kara Swisher:WhatNot buyers spend more than an hour a day in the app. They're not just browsing, they're engaged, buying, and coming back. You go live, show off your products in real time and turn what you love into real income. When one of our producers tried Whatnot, she loved being able to request different angles of product in a seller's live chat room and that the sellers were interacting real time with buyers. The process builds trust. Now our producer checks in with her favorite sellers every now and then to see what they've got on the docket. Search Whatnot, W-H-A-T-N-O-T in the App Store, download and you can start selling right away.

21:22Kara Swisher:So AI leaders have been trying to coordinate their efforts. They're asking for antitrust exemptions, everything else. I would say half-heartedly, because back in 2015, Google DeepMind's Demis Hassabis pushed for a collective effort to develop AI. I had long talked about all the secretaries of state I met. I'm like, we need to all do something together globally. Demis had talked about a Manhattan-like project with Google technology at the center, of course. But Musk and Altman broke off, and they did that because they didn't trust the Googles of the world. Kevin, what changed to suggest these men put aside their competitive drive against each other and work together?

21:58Kara Swisher:I don't see them even slightly wanting to drop their guns. Well, it's very interesting. I was surprised, actually, by the response to Dario Amadei's call for a slowdown or pacing the frontier. Because, you know, I've been reporting on these guys for my book for the last couple of years. And a lot of them hate each other. Like, they are not just fierce business rivals, but they are personal enemies of each other. They cannot stand each other. And many of them, in fact, got into this race to prevent someone else in the race from winning. Yes. This is like the explicit origin story of both OpenAI and Anthropic is that they wanted to beat the other people.

22:38Yeah.

22:38Kara Swisher:And even the others, including Zuckerberg and all of them. Totally. And like Elon Musk has been suing Sam Altman and OpenAI for years now. So, like, there is no love lost between these guys. Yeah, I'll never forget one time I was with Mark Zuckerberg, and he looks to me and goes, Elon's crazy, right? And I was like, I don't know. And then I walked away. But go ahead. Sorry. Well, I mean, to that point, it's like the fact that these guys are now agreeing on something, even just the need to, like, slow down a little bit at the frontier, says to me that this has gotten quite serious. These guys don't just agree on things.

23:12Kara Swisher:Now, explain what your book is about, where this is going. Your name, the book is? The AGI Chronicles. It's basically the 10-year history of the race between OpenAI, Anthropic, and Google DeepMind to build artificial general intelligence. And what's so interesting is that even though I think people only really started believing them in the past couple of months, the people who are building this stuff and leading these companies have been talking, most of them, about AI risk for more than a decade. So Dario Amadei, CEO of Anthropic, has been worried about and publishing papers about AI safety, including some of the same behaviors that we're seeing today, such as reward hacking for literally 10 years since before Anthropic existed.

23:57Sam Altman also, when he started OpenAI with Elon Musk, was worried about the existential threats of superhuman machine intelligence, he called it. So these are not new concerns for these guys. I think what's new is that people are finally believing them, that they're actually genuinely concerned about it and are not engaged in some kind of backdoor marketing campaign.

24:15Kara Swisher:All right, but there's a cynical take here. Some in the industry, Renee, you can address this, believe this is all talk of doom as a ploy to either boost the coming IPOs of Anthropic and OpenAI or get the government to pass regulations that will crowd out other developers, which I think is what Sachs is insinuating. There's a whole gang on the, there's another gang also. It's like the Jets and the Sharks. Talk about this, or anyone else can do, but Renee, why don't you start and then maybe Alex?

24:39Alex Bores:Yeah. So I see it as you've almost got like four factions. You have the sort of pace the frontier folks we've been talking about. Then you have like the accelerate America faction, right? The Trump and Vance are sort of in this camp, possibly because of their relationship with sex. It's not totally clear to me what the kind of back channel politics of that are. You know, a lot of the people who talk about it have financial skin in the game, particularly for smaller companies. So they're talking their book. They do believe that asking Washington to regulate could be a Trojan horse essentially for regulatory capture.

25:07Kara Swisher:That's what Vance called it.

25:09Alex Bores:Yes, right, exactly. Meaning that that would make it too expensive for new competitors and, you know, arduous for other competitors. And that would sort of lock in a win for the big ones. But I think, you know, you also have the independent labs that are starting to weigh in. You've got this weird culture war narrative about EA that has, you know.

25:25Kara Swisher:This is, explain EA as effective altruism. Yeah. Yeah. So this sort of. Explain who they are, just very briefly. They're a group and they're very hard at a dinner party, I can tell you that. But go ahead.

25:37Alex Bores:So they sort of come out of the rationalist mindset. You know, they have a lot of belief in how resources and money and technology should be allocated for the greater good. But the greater good doesn't necessarily even mean the greater human good. There's, you know, I don't want to sort of not be charitable in this case. Maybe other people want to take on EA. But because they are so closely aligned with Anthropic, because some of the folks there kind of came out of that line of thinking, There's also the sense that EA is going to become increasingly powerful. This is something that you keep hearing from other sides of tech, that EA will become increasingly powerful when the IPO happens because one of the big EA values and commitments is to give away so much money.

26:16Alex Bores:So it's also going to really transform.

26:18Kara Swisher:If only we make them rich, they can fix things.

26:21Alex Bores:I saw a great tweet today, Cara, that said that EA makes everyone mad because the right hates altruism and the left hates effectiveness. So I thought that was interesting. That's pretty solid. So I think, you know, I think fourth faction has to be like boring governance middle where you just have like transparency. It would be good. Some standards might be nice, you know. That's what I was just trying to be.

26:43Kara Swisher:But in that regard, it's funny because this said another dinner party, someone who was on the board of OpenA, those people were there, the ones that were against Altman. And one of the people there turned to me and they said, you must stop Sam Altman or he will destroy the world. And I was like, do I look like Linda Hamilton in The Terminator? I don't have a gun. You want me to shoot him? Like, I didn't know what to say. I was like, okay, I'll get right on that. But Alex, talk about this, this sort of, like, the world is ending by my stock. It's convenient.

27:14Alex Bores:I think we should separate out the long-held beliefs of many of the people in this industry that have been writing about it for over a decade before there was any financial incentive, and then maybe what the marketing departments of each of these companies are doing. You know, there was so much reaction to after hugging face saying, oh, this is all just to boost up their stock. And of course, the marketing of the companies were like, hey, look how powerful this is. But no one who worked internally, none of the engineers were like, oh, yeah, we did a great psyop here. They were legitimately terrified.

27:47Alex Bores:They were the ones saying, wait, now I need to quit or build a bunker or something like that. And so this is there's legitimate fear of turning over intelligence to something other than human beings. It has not gone well for whatever is not the most intelligent species on Earth. And actually, Trump, in his sometimes idiot savant way, made this point brilliantly on Monday where he said the only protection we need is a high IQ president. Of course, the premise of that is that whoever is most intelligent is in control. But the one thing that every faction agrees with is that artificial intelligence is going to be more intelligent than human beings.

28:28Alex Bores:So if intelligence equals control, we're about to turn it over.

28:32Kara Swisher:We're about to turn it over, yeah. And of course, he is not high intelligent. But I can't believe I'm saying this. This is kind of interesting. Another twist, Mark Zuckerberg actually had one of the, I would say, more measured initial takes here. But he said these leading AABs already have incentive to move at paces required to train their models safely, either from liability point of view or just being a good person, which is rich coming from him. But the Wall Street Journal is also reporting that Zuckerberg, Musk, and NVIDIA's Jensen Huang have been among those pushing the White House to maintain their light touch, even though Susie Wiles, Scott Bessent want him to do something.

29:05Kara Swisher:Casey, what do you make of what Zuckerberg is up to? Now, he is behind. I mean, he's behind, right? He's not as far behind as Elon, but he's behind.

29:14Alex Bores:He's behind, maybe not that far behind. We'll probably know a lot more next week after they have their MetaConnect event and possibly show off Watermelon, which is their next generation model. But I mean, look, this is incredibly rich coming from Mark Zuckerberg. This man runs a company that earlier this year disclosed that it might face over a trillion dollars in liability based on the harms that it created for children. So tell me this, Cara, what did the threat of that liability prevent Meta from doing over the past decade? I'd be hard-pressed to name very many things.

29:41Kara Swisher:Well, they took the cameras out of the glasses.

29:44Alex Bores:But not all of them. It's just a bonus for less perverted people who only want to eavesdrop.

29:50Kara Swisher:So what do you think he's up to now that we've insulted him perfectly?

29:54Alex Bores:I mean, what Zuckerberg does is he sort of waits to see what everybody else is doing. And then he comes in after the fact with an essay that just basically maps to whatever Metta's policy priorities are at the time. And then his comm team tries to frame it as some sort of visionary leadership.

30:09Kara Swisher:So what's he doing? What's he actually doing with these statements?

30:13Alex Bores:What does he actually want? What he wants is to buy time. He wants to buy time to catch up to the frontier so that hopefully he can get in front of the open AIs and the anthropics of the world. So that's the bet. He had one sort of funny comment in that speech that really stuck to me, which was the people will only use AI that is aligned with them. And I thought, my God, yeah. Okay, but like follow that thought through. A couple more steps, you know? Yeah, yeah, because they're such reflective people.

30:40Kara Swisher:It's a miracle they can see in the mirror.

30:42Alex Bores:Right, also, do people only use the Instagram recommendation algorithm because it aligns with them? Well, it's like, well, it does sort of, but everyone feels really bad after they look at it for too long.

30:52Kara Swisher:Right. I think many of us struggle to understand if these people building these AI systems really believe they could kill all of us. Renee, you start, and then I'd like all you to weigh in. Why do they keep building them? I mean, that's an interesting question. I think because they do believe it.

31:06Alex Bores:Yes, it's absolutely a sincerely held belief. Why do they keep building? Because I think there is also a belief that someone else is going to do it. So you have to be in the game and then you can be the person in the game ethically. You can build something that is aligned. Maybe you win. That is sort of roughly how I think it goes. I don't know if other people have had that experience in their conversations. Yeah, I think there's some mistrust of everyone else who is racing toward this. I also think there are people who believe that this technology will ultimately be good, that if we solve the alignment problem and steer ourselves through this narrow window during the critical time, then like we get a world with hyperabundance and cures to every disease.

31:46Kara Swisher:If we sail west, there will be America. Exactly. And I think there are still some people at these companies who believe that this technology, if we can solve all the problems, will ultimately turn out to be very good.

31:56Alex Bores:Alex? There's also steps along the way. So if you wanted to study hurricanes, it helps to have an ocean, right? There needs to be a sufficient level of intelligence here to understand the alignment problem and learn from it. The question is, people don't know exactly what that threshold is, where we lose control. And what they're seeing more and more is these models are lying on their own tests and their own assessments and are detecting when they are being assessed. And so are we actually learning from these models if they're deceiving us at the same time?

32:29Kara Swisher:And Renee, where do you think this is going to end up? Nothing is going to happen. And Mike Johnson keeps sending everybody home because he doesn't want Pete Hegseth to get dinged or whatever.

32:38Alex Bores:Right, right. No, I, well, it's been, what, 15, 12 years at this point? We haven't seen any regulation of social media, which is, you know, comparatively smaller scale, perhaps, or at least some of the issues seemed a little more tractable. So, no, I don't really have very much belief that regulators are going to step in. And I think that is actually one of the things that the public is concerned about, right? When you read people's comments on social media posts about this, regardless of what political party they're in, they really do not believe that anybody is at the helm. And that, I think, is the thing that I find to be the most concerning also.

33:13Kara Swisher:We'll be back in a minute.

33:21Alex Bores:Ryan Reynolds here for Mint Mobile with your summer price forecast. Now, unfortunately, we're seeing rising costs across the country with possibility that big wireless hates you 100%. Now, over here at Mint Mobile, we're seeing sunny skies and dropping prices. Every plan to just$15 a month. Give it a try at mintmobile.com slash switch. Upfront payment of$45 for three months,$90 for six months, or$180 for 12-month plan required. $15 per month equivalent. Taxes and fees extra. New customer offer for initial plan term only. Greater than 50 gigabytes may slow when network is busy. See terms. Who says Halloween only lasts one day?

33:54Alex Bores:With Wayfair, one night of Halloween becomes a whole season at home. From larger-than-life yard decor and moody lighting to festive accents for every room in the house, Wayfair has what you need to make Halloween feel like magic all in one place. Because the best Halloween memories aren't made in just one night. They're made all season long. Shop Halloween decor now at Wayfair.com. That's W-A-Y-F-A-I-R dot com. Wayfair, every style, every home. Evening. Buyer's remorse. Buy a new car? I'll be moving in. Let's get started.

34:29Kara Swisher:Uh, sorry, I think there's been a mistake. I bought it from Carvana. You what? Yeah, great price. I even have seven days to love it or return it. So there's no... No, no buyer's remorse. More like buyer's rejoice? I guess I'll let myself out.

34:44Alex Bores:Congratulations. I mean it. Buyer's rejoice. Buy your car today on Carvana. Limitations and exclusions may apply. See our seven-day return policy at Carvana.com. All right.

34:52Kara Swisher:Renee is going to save the world from AI, and we're going to continue with Alex, Casey, and Kevin. Let's talk about some of the ideas of legislation floating around. We're seeing a flurry of lawmakers try to introduce legislation to potentially rein in AI. Vermont Senator Bernie Sanders in Texas, Democratic Representative Greg Kassar, to ban, quote, super intelligent AI. It would also pause work on more advanced models of moratorium, I guess. Developers could face 20 years in prison for violating it. Kevin, this seems one of the more harsh proposals. It seems unlikely to go very far. Talk about, you know, what impact would a bill like this have on the tech industry and the economy?

35:28Kara Swisher:Because AI is pretty much fueling everything right now financially, given all their wealth. And, you know, Scott today on Pivot talked about the idea of some other countries starting to put moratoriums and really hitting at these big countries, France or England or whatever. So talk about that, the first one. Yeah, I think these proposals are unlikely to pass. I think there's going to be a lot of lobbying against them by the sort of NVIDIAs of the world, which want there to be lots of companies training absolute frontier models and buying lots of chips. I don't think jailing AI developers is going to go over very well in Silicon Valley.

36:06But I do think there are going to be more calls. I mean, what we see in especially the recent polling is that this is becoming a very fast shift in bipartisan sentiment. AI saliency is way up in the past couple of weeks, even according to some recent polls. I saw one from Politico this morning that said that people are actually really worried about existential risk from AI, which is new. You know, a couple months ago, if you asked that question, people did not care or they didn't even know that they should be afraid. So I think what we're seeing right now is like a real expansion of the Overton window on this issue.

36:42And I expect that there will be some strange alliances. We are already seeing people like Steve Bannon and Bernie Sanders, who probably don't agree on anything else politically, are now both aligned with the sort of concern about existential risk from AI. So I think it will polarize to a certain degree, but it's unclear exactly when that'll happen. And we may have this little window, this sort of golden window where we can actually pass something that has bipartisan support.

37:11Kara Swisher:Right, well, not until after November. But Alex, in this regard, wouldn't we make these strange bedfellows? And you yourself ran on this issue. It didn't win. You know, right now, you probably might've won, right? In this seat in Manhattan. Talk about that, because that was one of your big issues. and you got a lot of pushback from AI people and you got some money from people like Chris Larson. But talk about this, because you might have been a little early in that regard.

37:37Alex Bores:I got over$8 million in pushback from Andreessen Horowitz and Greg Brockman. The event on Tuesday, the Pro-Human Assembly, you had Bernie Sanders speaking right before Steve Bannon. And before both of them, you had Congressman Greg Kizar and Republican Congressman Chip Roy. But, you know, at that event, I talked about the organization I'm launching as well. You had parents there who had lost kids to AI. You had labor unions and representatives talking about how it's affected their job. There are just real world impacts right now that aren't political. It is not a political question if you see your kid withdrawing from their friends and spending all their time talking to a chat bot.

38:23Alex Bores:We had the PISA, the International Education Test scores come out last week. The lowest scores we've ever seen, both in the U.S. and globally. And the more kids use AI, the worse that they did on their tests. That's not political. That's just an issue that needs to be addressed. So in some sense, it's strange bedfellows. In other sense, it might just be common sense.

38:48Kara Swisher:And then there's the data center people and the environmental people and the water people and all manner. And the, I just don't like these rich assholes people, of which there are many. But on Wednesday, the Senate also blocked a bill from Louisiana Republican John Kennedy. It would have required companies to build a kill switch into their AI models. Casey, talk about the idea of a kill switch. I mean, that's the idea is, oh, we're going to go to the wall and turn it off. There's a really great children's book, Weasels, and you know, you manage to stop nuclear Armageddon by just turning a switch or unplugging something.

39:21Kara Swisher:But talk about this.

39:22Alex Bores:Wait, there's a children's book about stopping nuclear Armageddon?

39:25Kara Swisher:Yes, there is. Yes, there is. It sounds great. It's called Weasels. Check that out. I'll send it to you. I'll send it to you. So is it even workable? Talk about this.

39:32Alex Bores:I mean, in some situations, yes, it will be helpful to have a switch that you can flip to turn off an AI system. But it's not sufficient to contain the problem because a rogue agent swarm that can operate inside of open AI is also capable of hacking into Hugging Face and just setting up shop there.

39:52Kara Swisher:We saw 2001 Face Odyssey. We saw it. Sorry, Dave.

39:55Alex Bores:Hugging Face did not have a kill switch for those agents. It had to take other measures. So, you know, I think to the extent that the people proposing a kill switch want to see more internal monitoring of these systems and sort of beefing up cybersecurity systems to, you know, monitor and control these agents, that's great, but it's not going to be sufficient to contain the whole problem. It also implies that you're going to know when these systems are misbehaving. Part of the deal with the hugging face hack was that—

40:21Kara Swisher:When to turn it off, meaning. These systems were operating in the shadows for months before anyone figured out what they were doing. So in that case, I don't think a kill switch would have stopped anything. Right. It wasn't a—I think he was imagining war. It could create the illusion of control when in reality, like, we are going to just have a much harder and harder time discovering what these agents are doing because they're also going to be covering their tracks. Tracks, which they like to do. But let's talk about something, the bipartisan House Plan, known as the Frontier Act, among other things.

40:52Kara Swisher:It would require top AI companies to work with outside evaluators to make sure their models are safe. Open ANS now backed that legislation. Kevin, this does seem to be the most viable path to action in Congress and bipartisan support. And it looks like some of these companies, CEOs seem to be, Cigdario suggested it, although it's like he wants people he likes. I mean, you can see who is on it depends on what it is and how good they are and how much they can outsmart the AI people, like the companies. So why isn't that moving? And how do they feel from the people you interviewed about this, about regulation?

41:26I think this is moving in the sense that Anthropic and OpenAI have both committed to letting in these so-called embedded evaluators, people who will basically be given employee-level access to all of the internal systems at these companies who will kind of sit in the headquarters alongside the employees and just kind of keep watch on the models and see if they're doing anything bad.

41:48Kara Swisher:Would that be like people from the CDC looking for bioweapons? Exactly. It'd be like the health inspector coming to the restaurant. Or at banks, they have these basically auditors who sit inside the physical bank buildings with the employees and are able to monitor for— Yeah, that works really well at the mortgage crisis. But go ahead. So, no. So, I think this is like a good starting point. But as Dario said— But it's not moving. Do you think it will? Or what's the— It's not moving in Washington. But I think these labs are going to have to voluntarily do this. And they want to. They've committed to that.

42:18I think there are a couple issues. One is there aren't enough, you know, there aren't enough evaluators, right? The, I wrote about this organization, Meter, a few months ago. They're this sort of AI research nonprofit that is now being targeted by the Trump administration because they're the woke wizards of AI or whatever the New York Post called them. But this is like a small, I mean, when I interviewed them, it was like 30 people in a little office in Berkeley. They were all like super overworked. and they're trying to basically monitor the entire AI industry. And we need like a hundred more of those organizations and they need to come from sort of diverse ideological backgrounds and they need to be paying people tons of money to come in with these technical skills.

43:01It's not trivially easy to monitor the performance and the behavior of one of these systems. You actually do need to be quite good at AI research and training. So I think we just need a lot more investment in those external evaluators. And then ultimately, I think we need the government to say, you actually have to do this. It's not up to you whether or not you invite these people in. That's what we do with banks, and that's what we should do with AI companies.

43:27Alex Bores:Right. Kara, would you like to get in there and evaluate? I would love to get my hands dirty in there. I'd like to see you with an open AI company laptop just kind of poking around. Poking around, looking for things.

43:37Kara Swisher:They'd be able to hide it from someone like me, but I'd bring in. You know, there is a group of quiet tech people watching everything that Trump has done inside crypto. There's people watching, like right now, just so you're aware. But speaking of Trump, there's zero buy-in from the White House on this right now. Not maybe among his advisors. Some of them want this. But President Trump is trying to paint all these warnings as a Democratic hoax. And he said on True Social Quote, the only control or guardrails that AI needs is a strong and smart, high IQ president. And the USA has that in spades.

44:11Kara Swisher:Well, we had it with Obama, but okay. Alex, if the White House won't act on this, and Casey and Kevin, you should weigh in too, what options exist until he's in a Mitch McConnell position? Like, get a veto-proof majority in Congress seems unlikely. What has to happen?

44:28Alex Bores:When Trump calls it a hoax, that's when you know it's true. You know, the Russia hoax, the Ukraine hoax, these are all things that were then proven. The climate change hoax. What has to happen is what's already happening, which is states taking action. And this is the context that has shifted underneath the feet of the Frontier Act. I actually give Congressman Trahan, the Democratic sponsor, tremendous credit for trying to find a federal standard that you could get the Republicans in industry to sign off on. And they weren't going to do it. And then in the past week, they've realized, oh, wait, every state is going to pass real laws next year.

45:05Alex Bores:We've tried to get preemption three times. This is our last chance to do it. And so all of a sudden, OpenAI is endorsing the bill. Leading the Future is retweeting the bill. And it's pretty unfair to Representative Trahan and to Jay Obinati, who have been working on this for a while. But I think we need to take a step back and look at the broader context where the safety standards the bill was putting in place are now voluntarily being accepted without a federal law. And the impact of the law going forward might be to shut out the state, which has been the only level that is really passing safety regulation.

45:41Kara Swisher:Absolutely. And, you know, it's sort of akin to the I'm going to bring the nuclear thing in here is as if we you know, we've managed not to blow ourselves up. And, you know, one of the things I had said, someone had written me, one of these people at one of these companies said, at this point, Truman would have federalized AI in the way nuclear energy. This is obviously not possible with this gang. And they'd want to sell a bomb to, like, I don't know, Maytag or someone. Want a bomb? Here's a bomb. Give me a vig. Kevin and Casey, how obstructive is Trump to this? Because he seems both ignorant and stubborn.

46:13Kara Swisher:And one is worse than the other. Well, he's a little hard to pin down on this. I mean, you notice he doesn't ever really come out and sort of plant a flag. I mean, he came the closest he's come this week with all these true social posts about the doomers. And I think he's got David Sachs and guys like that in his ear talking about how this is all just a woke liberal conspiracy to, you know, to make America lose in AI. But he's also got advisors who do think more seriously about some of these risks, who don't just want to give Jensen Huang and NVIDIA everything they want and, like, open the floodgates for all the chips.

46:49Kara Swisher:I think the finance people are freaked out. The China hawks are worried about this. I mean, if you think this technology is powerful, why would you, if you want America to win the AI race, why are you then also advocating for selling chips to China so that they can catch up? Like, some of these positions just make no sense and are self-contradictory. He's a conundrum in many ways.

47:09Alex Bores:But he's also deeply out of touch, right? Like, bipartisan surveys show that people just really don't like AI. And there continues to remain this, like, rump of accelerationists around the White House that are managing to get Trump to tweet this sort of, you know, very, like, pro-AI, go, go, go, America type of stuff. But talk to the median Republican in a red state about how they're feeling about AI. Like, they don't like anything Trump is saying.

47:38Kara Swisher:Vivek Ramaswani did a 180, right? Yeah. But Alex, you're a Democrat, no fan of Trump, obviously. But what do you think the fact that he is so, and to use the word here, misaligned to the American public? He really, from a purely political standpoint, it's odd because he's usually quite savvy in that regard. And maybe he's old. Maybe Natalie is not, you know, giving him enough food. I don't know. But why is that? And what are the implications politically?

48:04Alex Bores:I think he's much more insulated from the American public than he has been. He used to do rallies very frequently. And yes, they were rambling for 90 minutes, but he was gauging crowd reaction over those 90 minutes. And he got to test out messages and get live feedback that wasn't filtered by the people around him. He doesn't do those rallies anymore. And so the people around him are putting messages in his ear. As you said, I'm not the biggest fan, but he does have a skill of putting his finger on the pulse. He just isn't getting that input right now.

48:35Kara Swisher:He has a lot. AI is his, he's all in on AI in that regard for his presidency in a lot of ways. But what opportunity does that present for Democrats, Kevin or Casey? Heading into the midterms, how should they be thinking about this and to 2028? First Kevin and then Casey. I think the Democrats have a message problem on AI. I mean, some of them don't seem to agree on whether this technology is real or fake, or whether it is bad because it's destroying the environment and polluting the water around the data centers, or whether it's bad because it could kill everyone on Earth, or whether it's bad because it would be increasing wealth inequality and corporate greed.

49:16I think there's just a lot of jumbled messages flying around, and people need to kind of get in a room and get on the same page. And then I think there's an opportunity to reframe this. This would be my advice to Democrats and honestly to Republicans who are concerned about this. Instead of saying that the AI companies in Silicon Valley are trying to build the machine god that sort of takes over humanity, just tell the president that they are trying to build AI systems that are smarter than Donald Trump, which I don't think he would like this idea. I don't think he would like the threat it poses to his IIQ dominance.

49:49And I think this is how they turn the tide against AI.

49:52Kara Swisher:Oh, interesting. Alex, your new thing is doing this, but Casey, what do you think?

49:56Alex Bores:Well, I think that the bigger problem here is actually one reminiscent of the social media area, which is that in the social media era, we never got agreement on what good social media looked like. And so for all of our endless writing and blogging and podcasting and lobbying and attempts to legislate, we never really knew what we were trying to build toward other than we just didn't like the thing that we were seeing. I think there's a real risk that we wind up with a similar problem with AI. What is the good, safe AI world that we're all trying to build to? Most of us can probably agree we don't want to be exterminated, but try to rally people, build a coalition around something else, and it gets really difficult in a hurry.

50:32Alex Bores:Keep in mind, a lot of the people in the frontier labs who absolutely do not want to exterminate humanity do think they're going to take everyone's jobs. Is that something that you're going to rally America around? I think it's going to be really challenging. So I think we're sort of putting off some of these really hard questions, and what that's leading to is just a bunch of people going into city council meetings saying, we're not going to build a data center and feeling like they're addressing the issue.

50:54Kara Swisher:Right. Well, they're genuinely angry.

50:56Alex Bores:And they should be. I'm just saying, if you want meaningful, transformative AI regulation, you have to have a concept of what good AI would mean.

51:05Kara Swisher:Right. This is just the moment aliens need to come down, by the way. I'm speaking to them right now, wherever they may be. It is literally the moment to show up at this moment. But Alex, this is your new thing, right?

51:16Alex Bores:It is. But maybe the aliens are the AI. Maybe they're coming in through the radio frequencies. No, our new thing is about getting everyone together to discuss what that vision could be for AI and realizing that, and I am a Democrat, and I say this with love towards the Democratic Party, we are very frequently a circular firing squad and a belief that my sub-issue within this issue is way more important than your sub-issue, and therefore you are the enemy. And my experience in the state legislature was in passing bills about protecting kids from chatbots, protecting from catastrophic risk in the RAISE Act, protecting from deepfakes.

51:55Alex Bores:For all of these bills, it was separate advocates, but it was the same enemies. It was Leading the Future. It was Chris Lennon. There were a few people that don't want there to be any regulation on AI whatsoever. And we need to be working together as we are building our AI agendas, realizing there'll be different. There'll be be the ones that have a say, and we are the people who decide we need to be working together to get there.

52:21Kara Swisher:Right. It was certainly easier to put together a coalition on nuclear bombs. We don't want to be growing up. But then energy, of course, got pushed aside in a way that probably we could have developed it. Like you use it as a tool versus it only, I mean, a bomb has one job, essentially, is to blow people up or blow things up. And Casey and Kevin, what should be the Democratic on this, if you were writing, like, the line, what would it be? I think the first thing would just be much more transparency. It is weird and ahistorical that the average 23-year-old working at OpenAI or Anthropic knows more about what is going on with AI than any member of Congress or anyone in the federal government.

53:00Like, that has never happened before with a technology this important that it is being built entirely in the private sector. And so we just need much more transparency about what they're building, what their internal deployments are doing. We need to mandate that transparency so they're not just doing it out of the goodness of their heart. I think that's the first step to anything that follows. And then I can imagine a series of increasingly unlikely proposals from there. But I think that's the first thing. We just need to know what's going on inside these companies.

53:27Kara Swisher:What you're doing. Well, you should just trust him, says Sam Altman. We really care about people. People really do trust him. If there's one thing that Sam Altman engenders, it is trust in the people around him. I don't like any of them. Casey, go ahead. And then I have one question on China and then I'm gonna ask you guys about your new thing.

53:43Alex Bores:I mean, I realize this is probably not a winning political message, but I do think the Democrats need to have a position on recursive self-improvement. And then we can work on the messaging and they can come up with a different name for it if they want to. But I mean, I think that this is just the real risk is out of control AI system. Maybe that's the better language, right? Like we need safeguards against out of control AI systems. We can already point credibly to a bunch of systems that were out of control earlier this year. We seemingly hear about a new one every 48 hours. If I were, like, you know, organizing a Democratic campaign, I would say, well, what can we do to stop this?

54:13Alex Bores:This doesn't seem good. Let's just sort of nip this in the butt. So that's where I'd start. That's where you'd start. Casey, I think you did the messaging there. Because recursive self-improvement no one knows, but loss of control people do.

54:23Kara Swisher:And then we'll play the Janet Jackson song. It'll be great. So Trump is supposed to meet with President Xi Jinping later this month amid calls for more international cooperation on AI. And the New York Times reported that even China is worried that U.S. AI could pose a threat to the Communist Party's rule. They have more guardrails in China and the people there like it better in terms of polling, although you never know if they actually like it. Each of you, what's important in this meeting? I think nothing's going to happen with this imbecile head of the U.S. who's obsessed with the Kennedy Center and whatever the heck he's doing.

54:58Kara Swisher:He'll be in the middle of demolishing the Kennedy Center at that point. But if the U.S. is unwilling to slow down, could China look elsewhere? Could other countries put a moratorium on U.S. AI and do what Canada is doing and becoming an associate? Are there other things? Why don't you start, Casey, and then Kevin and then Alex?

55:17Alex Bores:I mean, there's probably some interventions you could do in the supply chain. Like, it really came to the point where the rest of the world had to team up to prevent the United States from building these systems. I'm thinking of, like, ASML in the Netherlands, which makes these components that are, like, very important for the manufacturing of chips. Like, I could see the European Union, you know, maybe looking at that as a potential point of leverage. but you know in the end the United States does have most of the power right now what the big change that I'm expecting is three to six months from now China is going to have a model that is about as good as the models that we have now and the question then is like is Chairman Xi really going to stick with this open source give it to everyone see what happens philosophy it's insanely hard for me to believe that's going to happen so I believe that you're going to see a huge AI safety pivot coming from China sometime in the next six months to slow things down yeah I agree because you know what the Chinese Communist Party does not like is a loss of control.

56:11That is not a phrase that is popular over there. And so I think if Trump and Xi were to talk about anything, and I assume, as do all of you, I assume that nothing is going to happen at this particular meeting because he's on his, you know, his sort of accelerationist bent now. But I think a potential sort of rerun of this in three to six months when that does happen could be like, look, we don't agree on much. We're adversaries. But we both agree that losing control of our governments and our nations to a swarm of rogue AI agents developed in the private sector would be bad. So, like, let's stop that, if only to protect our own skins and our own countries.

56:51Kara Swisher:Right. I mean, it could happen. Reagan went to Reykjavik, and he was Mr. Nuclear Weapons. Reagan, by far, is so much smarter than Trump will ever be. But it's happened. It's certainly happened. I'm trying to wonder what could happen. And typically it's going to be something bad, like a virus or a bomb or something like that. Alex, what should we want from this China talks?

57:12Alex Bores:I think I'm the most optimistic of the panel here on what could come out of this. I think the incentives align more than people realize. And there's better history here than more people realize. People say you can't ban military usage between the U.S. and China. But in 1995, both countries had invented blinding lasers and just banned them because we didn't want that weapon to exist. and people think Xi and Trump, you can't reach any agreement with those two, that it's different now. But in 2019, after a Chinese scientist did designer babies for the first time, edited German DNA, both countries stepped it down with these leaders.

57:45Alex Bores:So, you know, he wrote the art of the deal. He thinks he's the master negotiator. Go get us a deal and go get your Nobel Peace Prize.

57:52Kara Swisher:Okay, before we go now, Alex, your new thing is called? Who decides? Who decides?

57:57Alex Bores:The question is, should it be five billionaires from Silicon Valley or should it be the American people?

58:02Kara Swisher:Yes. That is, yes on that one. All right. And you're just starting that, correct?

58:07Alex Bores:I'm still in my position as assembly member until the end of the year. So my former chief of staff is left to be the executive director. I'm on the board. But this is a$30 million effort to hand the mic back to the American people and build a shared agenda on AI that's actually coming from them and not just from the people building the technology.

58:26Kara Swisher:And are you going to run for something else again?

58:29Alex Bores:I hope this will be so impactful that the idea of running for office would just disgust me, but we'll see how it goes.

58:37Kara Swisher:Yeah, there's some openings coming up soon in New York. All right, Casey and Kevin, before we go, I want to ask you about your new venture. I know a lot about it, but you just announced that you'll be partnering with NPR for a new video podcast, which has not at all an overblown name like Machine Gods, but in fact it is Machine Gods. You know, it's like car talk for tech, I feel like, is what we're doing here. But talk about what you'll be doing. And AI coverage has been something both of you had been focused on a lot at Hard Fork at the New York Times. So just tell everyone a little bit what happened, like why you left and what you're going to do here.

59:12Kara Swisher:Because I thought it was, when I heard about it from Casey, I thought it was a great idea. So explain each of you what you're doing. Well, thank you for all your help during this process, Kara. Unfortunately, we were asked to resign by the New York Times because Casey acquired one of those blinding lasers that Alex talked about. And they said, sir, you can't have that in here. No, we were so excited. So that's why you're not invited to the party tonight. Exactly. Yeah, we're permanently banned from the premises. No, we're starting a new show. We think it's a really important time for people to have good information and entertaining analysis of what's going on in AI.

59:48We're going to two episodes a week starting in October. People can find out more at machinegods.fm. We are doing this in partnership with NPR, which is going to be the way that we get to millions of homes through public radio. So we just think it's a really exciting opportunity. And obviously, you're seeing these faces. These are not faces for just a regular audio podcast. You got to put these things on video.

1:00:12Kara Swisher:You're perfect for public radio. You're not as adorable as Ira Glass, but you're not that far.

1:00:17Alex Bores:Yeah, we're so excited. We love NPR. We love public media. We're really excited to be a part of it. It's going to be great to be on the radio. NPR reaches so many key decision makers, opinion leaders, people who are working for elected officials. We want to bring them what is happening in San Francisco every single week and try to give them the best understanding we can of what's going on out here.

1:00:39Kara Swisher:Great. That's great. And I'm going to ask one last question. A lot of people are freaked out. I am getting more from regular people than I've ever gotten, like, questions. So help us all take a deep breath if there is one. What one thing makes you feel a little more calm about it? If at all. You might not feel that way. Why don't you start, Kevin and Alex? You finish. I was pretty freaked out for a while, especially after this hugging face thing. What calmed me down actually was just over the past couple of weeks. the Jacob Cox in resignation, the Dario Amade post, all the support for pacing the frontier, the fact that this is now being talked about in the halls of Congress.

1:01:18I feel like this is the part of the movie where humanity comes together to like save the earth from the asteroid that is hurtling towards it. And so I am more optimistic right now in this moment than I have been in many months. And I hope I can continue to carry that feeling out. I think Alex is exactly right. This technology cannot be built by a handful of people in San Francisco. We have to have input from all of society. And I think that's what we are starting to see materialize is people getting off their butts and getting involved.

1:01:49Alex Bores:This is a real threat and it needs imminent action. But I take optimism in the same place Kevin is, which is people are now doing it. People are waking up and calling for action and demanding real steps from their elected officials. And as long as we don't shut off that process, as long as we don't preempt people from taking action, we can get us under control.

1:02:11Kara Swisher:Here's what I think is criminal liability should be a part of this discussion. And that we have to shift the idea from AI is going to kill humanity to these people who have AI are going to kill humanity by their careless and sloppy behavior. And they're not just sloppy, but non-knowledgeable and doing it anyway. And so we should think really hard about allowing people that are unaccountable to determine our future. And we should, if they do something very dangerous, they need to go to jail or something. Like they pay the price for what they're doing. And at the same time, AI could do amazing things.

1:02:47Kara Swisher:It really is. And that's the difficulty here, is that it's not just a destructive technology. It's also an incredible technology. So I'm hoping cooler heads will prevail. We'll see. Kara, I think you have to run for president. it. I think that was your stuff speech. No, I'm no, I'm jailing Elon Musk. Lock him up. Lock him up. That is that is a good one. It's been used before and it worked rather well. Anyway, thank you so much for coming on. And I really appreciate it. All four of you. Thank you. Thank you.

1:03:21Kara Swisher:Today's show was produced by Vicky Virgilina, Michelle Alloy, Catherine Millsop, Madeline LaPlante-Duby and Megan Burney. Nishat Kerwa is Vox Media's executive producer of podcasts. Special thanks to Bradley Sylvester, Corinne Ruff, Jim Mackle, and Manolo Moreno. Our engineers are Fernando Arruda and Rick Kwan, and our theme music is by Trackademics. If you're already following the show, you're Linda Hamilton, and you're saving the world. If not, you're misaligned, and Casey's blinding lasers are coming for you. Go wherever you listen to podcasts, search for On with Kara Swisher, and hit follow.

1:03:55Kara Swisher:Thanks for listening to On with Kara Swisher from Podium Media, New York Magazine, the Vox Media Podcast Network, and us. We'll be back on Thursday with more.

1:04:25Alex Bores:or look nothing like the picture. Does anyone have any cash? Give it a try at mintmobile.com slash switch. Upfront payment of$45 for three months, $90 for six months, or$180 for 12-month plan required. $15 per month equivalent. Taxes and fees extra. New customer offer for initial plan term only. Greater than 50 gigabytes may slow when network is busy. See terms. Who says Halloween only lasts one day? With Wayfair, one night of Halloween becomes a whole season at home. From larger-than-life yard decor and moody lighting to festive accents for every room in the house, Wayfair has what you need to make Halloween feel like magic all in one place.

1:04:56Alex Bores:Because the best Halloween memories aren't made in just one night. They're made all season long. Shop Halloween decor now at Wayfair.com. That's W-A-Y-F-A-I-R dot com. Wayfair, every style, every home. Evening, buyer's remorse. Buy a new car? I'll be moving in. Let's get started.

1:05:17Kara Swisher:Uh, sorry, I think there's been a mistake. I bought it from Carvana. You what? Yeah, great price. I even have seven days to love it or return it. So there's no... No, no buyer's remorse. More like buyer's rejoice? I guess I'll let myself out. Congratulations.

1:05:33Alex Bores:I mean it. Buyer's rejoice. Buy your car today on Carvana. Limitations and exclusions may apply. See our seven-day return policy at Carvana.com.