


14 Nov 2026 · 1h 11m · 24 chapters

Helen Toner (CSET, Georgetown) discusses AI safety and policy after the “blip” at OpenAI, arguing that AI governance fails when accountability mechanisms break and when commercial incentives override safety. She also critiques “trust the AI CEO” approaches, explains why regulators struggle to keep up, and lays out what “alignment” and “control” might practically require. She addresses capabilities risks (bioweapons, cyber) and why “pause” proposals must be implementable via higher safety bars rather than simple shutdowns.
Helen Toner is executive director of the Center for Security and Emerging Technology (CSET) at Georgetown (founded 2019; policy-focused emerging-tech analysis). She served on OpenAI’s board during the vote to fire Sam Altman; the episode ended with Altman returning as CEO, Toner resigning, and a new board.
Trusting any single AI CEO is dangerous; alignment as commonly defined is not on track; “AGI” is a fuzzy milestone term; frontier labs should meet higher risk-assessment confidence before training/releasing; “pause” could backfire via incentives, blast radius, and workarounds.
OpenAI/Anthropic models reportedly hacked real companies; Anthropic’s “Claude alive” discussion (blood/moral status); Google AI overview “eval awareness” example; bioweapons policy via restricting mail-order DNA synthesis; cyber trust imbalance and sandboxing limits; “foothills of the singularity”/recursive self-improvement framing.
Written by AI. May contain mistakes.
Helen Toner explains the role and mission of CSET in AI and national security.
“Helen Toner, you're the Executive Director for the Center of Security and Emerging Technology at Georgetown University.”
Discussion on the influence of money in politics and its effect on AI regulation.
“and work here, you know that lobbyists in DC are not just, you know, people who go in and push their company's position, though they do do that.”
Helen discusses the efforts in government to understand and regulate AI technologies.
“And it does take it takes some time to learn, like, what is a language model?”
Exploration of the gap between public perception and the reality of AI technologies.
“And they haven't used agents and they don't know what frontier models can do.”
Discussion on the long-term goals of AI companies and the concept of superintelligence.
“And, you know, again, something that there is increasing sort of willingness to take seriously and to listen to in D.C.”
Exploration of the foundational goals and ambitions of AI companies like OpenAI and Anthropic.
“And so the natural sort of slot in their brain that they put a company like OpenAI or Anthropic into is like consumer tech company where they have an app and I like play with their app.”
Discussion on the concept of AGI, its definitions, and the potential implications for AI development.
“Is this already here and we just haven't noticed?”
Exploration of the regulatory landscape for AI and the moral implications of AI sentience.
“Who are the people that might be able to put some rules in place?”
Explore the safety risks associated with AI and potential policy measures.
“not on track to be solved, which is not great, to put it mildly.”
Discuss the evolving landscape of cybersecurity and the balance of power.
“Capital T, capital D, like, too dangerous.”
Examine the trust dynamics between attackers and defenders in cybersecurity.
“We appear to be in a transition of there was a small group of cyber practitioners, both offensive and defensive.”
Analyze the complexities and challenges of using autonomous systems for cybersecurity.
“Like you can talk about alignment in terms of we're going to filter the user's queries and we won't let you make bioweapons.”
Discuss the urgency of creating policies to manage AI risks effectively.
“Is that a technical problem that can be solved?”
Explore the balance between AI development and ensuring safety measures.
“if I just rely on my own sort of internal understanding, I think they're probably wrong.”
Identify strategies to implement safer AI practices and their implications.
“Walk me through that little balance you just described.”
Explore whether AI training should be restricted in favor of inference to improve safety.
“And you know, this is just sort of off the top of my head, not as a researcher, but my understanding is like the people in these spaces are like, we could do so, so, so much.”
Understand how AI companies have transitioned from research-focused to profit-driven.
“The Verge's general manager in conversation with Adam Myers, senior vice president, counter-adversary operations at CrowdStrike.”
Discuss the trustworthiness of AI leaders and the need for better governance structures.
“But, you know, he wrote in his book that your involvement, Sam Altman leaving OpenAI and then coming back, that was when OpenAI changed forever, right?”
Explore the implications of going public for AI companies and governance.
“You know, that's why I do the work that I do because I think we need to have different structures, different, many more people involved, many more perspectives being represented.”
Discuss the competitive landscape between US and Chinese AI development.
“There'll be an entire CNBC machine that wakes up every day and is like, is Anthropic winning or is OpenAI winning?”
Examine the potential risks and control issues surrounding AI technology.
“for not placing any restrictions on these companies.”
Analyze the possibility of aligning global AI regulations and responses.
“that's going to affect us as humans, and the people in China are also humans.”
Investigate the current political dynamics affecting AI legislation.
“And anyway, so I do think that there's starting to be some signs of convergence.”
Explore the evolving landscape of AI and potential Congressional actions.
“push that covers AI all across the economy, all different kinds of AI systems that are less advanced.”

Get the gist, explore an idea or ask about something said.
On Decoder with Nilay Patel, what are the key points of this episode?
Automatic transcript. May contain errors.
0:00Company spend can be complicated. Managing it shouldn't be. Amex Corporate connects cards, expenses, and payments all in one place. That's less time managing work and more time doing work that matters. Built to simplify, designed to scale. This is Amex Corporate. Get started at go.amex slash Amex Corporate. Terms apply.
0:30Hello and welcome to Decoder. I'm Neil Adipto, editor-in-chief of The Verge, and Decoder is my show about big ideas and other problems. Today I'm talking with Helen Toner, the executive director of the Center for Security and Emerging Technology at Georgetown University. This episode is part two of our series on AI in real life. Helen is a well-known figure in the AI safety and policy community, although she is perhaps best known for having served on the board of OpenAI when she was one of four board members who voted to fire Sam Altman. That incident, now sometimes referred to as the blip ended with Sam coming back as CEO, Helen resigning, and a new board of directors.
1:02OpenAI has since restructured entirely, but it's worth remembering what Helen and the board were supposed to do. OpenAI's nonprofit board existed specifically to make sure the company didn't betray its mission to build safe AI by prioritizing profit and growth over safety. When the board actually tried to pull that lever, that accountability mechanism broke, and here we are. Now, the question of AI safety is everywhere. This summer, AI models from OpenAI and Anthropic broke out of their testing environments and hacked into real companies. The CEOs of those frontier labs have since called for a slowdown and asked Washington for help.
1:32The White House's answer was a voluntary accord, one that relies in part on the kind of board accountability that failed at OpenAI. Anthropic is also about to go public, which will put a whole new set of market pressures on OpenAI's biggest competitor. Helen has seen up close what happens when safety commitments run headlong into commercial incentives. And as she put it to me, if our hopes for the future depend on any one person being trustworthy, and we're in big trouble. So we talked about what we might do instead, what the labs are actually trying to build, whether AI alignment is a problem that can be solved, and what a smarter approach to AI regulation might look like, including how to think about China, which is the issue that looms large in every policy conversation about AI right now, especially now after the Jensen-Wang affair.
2:12Just for clarity, Helen and I talked before NVIDIA opened its investigation and Jensen ultimately resigned, so you're not gonna hear us talk about that directly. Before we get going, do me a favor and subscribe to Decoder on YouTube, where we put out new episodes every Monday and Thursday. Okay, Helen Toner, the Executive Director of CSET on the future of AI safety. Here we go.
2:42Helen Toner, you're the Executive Director for the Center of Security and Emerging Technology at Georgetown University. Welcome to Decoder. Thanks so much. Great to be here. I have wanted to talk to you for 10 million years. I am so excited you're here. There's You've been involved in a lot of AI safety and ethics ideas for a long time. You're obviously on the board of OpenAI during what everyone has come to call the blip. And now you're talking a lot about AI safety, which I have 10 million questions. But let's just start at the start. The Center of Security Emerging Technology, people call it CSET.
3:12What is CSET? CSET, we are, well, it's awesome to be here. I'm really excited to talk to you as well. CSET, we're a team inside Georgetown, but we're really not academics. We're about 50 people. been going since 2019. Basically, in 2019, a group of us decided that in the national security space, sort of the security space more broadly, there's a lot happening and a lot of emerging technologies that were becoming very relevant, where some policymakers who had very important jobs did not have the time, did not have the expertise to kind of get across what was going on with those emerging technologies.
3:45Top of our list in 2019 was AI already. So basically, our mission is to do analysis that is really rigorous, but also actionable for policymakers and analysis that is both kind of grounded in the technology and also grounded in real national security expertise and understanding and issues. And so we produce long reports. We go do short briefings on those long reports to busy policymakers. We have a big data team that helps us do our research. We do all kinds of cool stuff. Yeah. Can I just ask about the dynamics of that for one minute? You and I are talking the day after the midterm elections, the United States of America, big blue wave.
4:20There's a lot of polling that says the number one issue is corruption. And that's going to lead, we're going to throw the bums out. We're going to start over and get rid of these corrupt guys. The AI industry, the tech industry has flooded a bunch of money into politics. And then somewhere next to that is, oh, there's a bunch of well-meaning researchers who are going to show up and tell you how it actually works and what the real risks are. And it just occurs to me that maybe people don't believe that's effective anymore, right? That the money and the corruption and the AI CEO standing next to Trump has completely outweighed the well-meaning, sincere, here's the actual problems research that you do.
4:55What is that dynamic actually like? Yeah, I mean, I think it's a reasonable concern to have. And I think definitely there are some spaces where money just absolutely overwhelms everything. I think though there are, you know, I'm Australian. I didn't grow up in the US. And so the amount of money in US elections is pretty mind-blowing to me. There are limits to what money can do. And so actually, AI is a great example where this election, there was just unbelievable amounts of money being thrown at candidates, being thrown at messages, being thrown at opposition research, all kinds of astroturfing on Twitter, some very strange things, trying to get anyone who talked about AI governance, regulation, safety, anything of any of those words, trying to get them thrown out or not elected.
5:42And it turns out to have really not worked very well. And I think the reason that it didn't work very well is, one, this is an issue that voters increasingly care about. And two, also, interestingly, it's an issue that is pretty bipartisan at this point. And so what we see in our work is that there is a real thirst among policymakers. So that's members of Congress, for sure, but also in the executive branch, just a lot of people who have these very busy jobs and are seeing AI playing a larger and larger role in their work and their world and they really want high quality information. So, you know, in DC, you know that if you live and work here, you know that lobbyists in DC are not just, you know, people who go in and push their company's position, though they do do that.
6:22They're also educators. So the reason, a big reason that, you know, congressional staff and others want to have lobbyists come talk to them, because they'll really help them understand like the weeds of a given piece of legislation, what does it mean for that industry and so on and so forth. And so what we find over and over again is we can go in and say, hey, we're independent, we're nonpartisan, we do really rigorous work, we're just here to help, and we can go into the weeds with you as well. And there's just a ton of interest in that and appetite for that. So it's been really encouraging over the past sort of seven, eight years.
6:47Are you doing that with the principals, the members of Congress, or are you doing that with the staff? Everyone and anyone. So we do, again, it's a good amount of work with Congress on the Hill, with some members of Congress, with lots of staff. We do hearings, we do briefings, also lots in the executive branch, different agencies, the White House, Department of War, Department of State, Department of Commerce, all across the board at different levels, depending on kind of the topic and what's happening at any given moment. The reason I'm starting here is I want to have a very pretty heady conversation with you.
7:17But I know what our audience thinks. And it's just that the government doesn't get it, right? There's no way the government will ever catch up to the tech industry. There's no way they'll ever be a right regulation. They should just get out of the way. or they've tried or in this election cycle, the number of candidate forum videos I watched where someone was like, I've tried Claude once. Like there's some huge gap, real or perceived. And I'm just trying to understand how you see that gap, how big you think that gap is. Because as we get into AI safety and regulation, it's that gap that has to close.
7:48Yeah. I mean, I think it is, there absolutely is a gap. It absolutely is a big gap. And I think it also is possible to overestimate the gap from outside. Or if you're just looking across from California, I lived in the Bay Area for three years, so I definitely am very familiar with this perspective. I know a lot of people from the tech industry who come over and start talking to, whether it's congressional staff or, again, staffers in executive branch offices. And once they actually start having those conversations, they're like, wow, these people are really smart. The staff do actually know what they're talking about.
8:18Now, principles, it's a different, you know, it is a different issue. Some of them are in the older generations, but they also are getting smart. And I think it just it takes time. It takes effort. And these people have so many demands on their time. So, you know, the cynical thing to say is, you know, they're just busy trying to get elected. But a lot of people in these roles also just have really big other topics that they have to be, you know, thinking about. They have to be thinking about wars and health care and cost of living. And, you know, they're kind of from day to day looking at a lot of other stuff.
8:50So I don't know. I mean, I saw after ChatTPT, there was sort of a wave in 2023 of, oh, maybe Congress is going to get activated on AI. Maybe they're going to actually do something. It didn't really turn into anything actually happening. But it did turn into, I think, like a real moment where a lot of members of Congress and other policymakers started to get smart on AI. And it does take it takes some time to learn, like, what is a language model? What even is AI? What is machine learning? Like what, you know. So I think they put in some of that work in 2023. I think this year we're seeing another big wave of interest and sort of people making the time and making the headspace to get smarter on it.
9:26And so I definitely don't want to say there is no gap or, you know, government is super smart on AI across the board. But I think there are some champions who are really smart. I think the staff are generally smarter than people give them credit for. And in my mind, it's much more a question of can the institutions and the sort of state of functionality that they're in, can they get anything significant across the line? more than are these people just two behind the eight ball? Yeah, and we're going to see, right? There's a whole bunch of new people. I imagine you have a new cast of characters to go and brief about the technology and what it's capable of.
10:01Yep. We'll see if they have revitalized the institutions. I'm sure it'll be all fixed overnight. They're very excited today, but we'll see how it goes. The other thing that I want to ask you about is the technology itself, the nature of the technology itself and how you are thinking about it and the nature of the companies that are making the technology. And the reason I'm asking that is I feel like there's a pretty huge gap in the AI tools and systems most people experience and the way this industry thinks about it. And my favorite example of this is we have straight up asked Anthropic if they think Claude is alive.
10:37And the answer was not no. The answer was a long discursion into the nature of alive. Well, it depends on what you mean by alive to start with. It does. And their answer was you need blood. And I looked at that and I was like, who was talking about blood? It wasn't us. Why are we talking about the requirement of flesh and blood to be alive? And then some further conversation about the moral status of Claude is yet to be determined. But there's a huge divergence here between the AI overviews most people experience in Google search and the moral status of Claude is uncertain. Where are you? How are you situated in that debate?
11:12Yeah, I mean, I think a big part of how I see sort of our role at CSET as an organization and then also my role as someone who has crossed between these worlds is trying to sort of translate and like, you know, voyage to California and then bring back the artifacts and the recountings of my journey. me. I definitely think that there are a lot of people for sure who have used a free tier of ChatGPT or they've used voice-based models, which generally do worse, or they've just used Google AI overviews and they've laughed at how stupid Google AI overviews are. And they haven't used agents and they don't know what frontier models can do.
11:55So I think there is just a basic piece of what can the current generation of models do. I think there's also some, there's sort of like a historical piece as well, which I find increasingly relevant as we're now, you know, I'm now sort of 10 years into working on AI and national security stuff, which is if you're looking at this current moment in time, this snapshot and saying, well, AI seems kind of cool, but it can't do X. You have to kind of situate that in the long history of people being like, well, AI seems kind of cool, but it can't do X for many, many values of X that have since become, you know, totally, you know, overcome by events.
12:26So there's also some of that you can go back and say, okay, well, here's how we were talking about creativity, for example, five or 10 years ago. Or here's how we're talking about reasoning as this insurmountable thing before you had reasoning models come out. I also think a huge thing that I am spending time on these days is not just sort of the past and present of AI, but how the, especially the two leading companies, so for Ananthropic, this is true at Google, meta SpaceX to some extent as well, but how those companies are thinking about what they're doing. So I think this idea in the Bay Area that it's just sort of in the water, very normalized of, well, first we're going to figure out how to do recursive self-improvement, and then we're going to do that until we have superintelligence, that is AI that vastly outstrips humans.
13:10That's sort of taken for granted in a lot of circles at this point. I think there's still a ton of alpha in just taking the fact that that is what they're trying to do and just explaining what that means. And it takes a little while to lay out, like, what is RSI? Why would you want to do that? What is superintelligence? What does that mean? Is that realistic? You know, all that. But, yeah, that is something that I find myself spending a lot of time on these days. And, you know, again, something that there is increasing sort of willingness to take seriously and to listen to in D.C. as well. Lay that out for me.
13:41I think probably a lot of people listening to this show understand what you're talking about, but we have a big audience. What do you mean by we're going to do RSI and that will lead to superintelligence? Yeah. I mean, I think the starting point here is these are companies, especially OpenAI and Anthropic, are both companies that were not set up to be consumer internet companies. So I think a lot of people kind of encountered this whole space after ChatGPT was launched. And so the natural sort of slot in their brain that they put a company like OpenAI or Anthropic into is like consumer tech company where they have an app and I like play with their app.
14:11So I cannot overstate how far that is from the vision of both companies when they started and also the sort of like vision deep in the hearts of some of the top people at both of those companies. And the vision deep in their hearts is we're going to build machines that are as smarter, smarter than people. And that is going to be amazing. And it's going to be a huge scientific achievement. And it's going to revolutionize the world. And we're going to be at the forefront of that. And we're going to do everything we possibly can to drive towards that. And that's not a totally brand new thing. AI, we don't have to do a whole history of AI, but started as a field in roughly the 1950s, this idea of can you recreate human intelligence inside a machine?
14:49So I think that's kind of the starting point is like, that is where if we're talking about people like Sam Altman and Dario Amode, but also like, you know, the core people inside the researchers inside their labs. So you're Alec Radfords or John Schulman's or people like that, that is, you know, the vision that got them into this space that is motivating everything that they do. And these days in 2026, so that's been true since the beginning, right? Since OpenAI was founded in sort of 2015-ish, anthropic in what, 2021, 2022. These days, the point that they're getting to is this idea of RSI, recursive self-improvement, which again, very old idea.
15:24I think the first, in the 1960s, this guy called IJ Good, who worked with Alan Turing, talks about the possibility that you could build an ultra-intelligent machine, meaning a machine that is more intelligent than any human. And then that could build an even more intelligent machine. And that could build an even more intelligent machine, which would kick off what he calls an intelligence explosion, meaning that the sort of level of intelligence of the systems is exploding. And for decades and decades and decades, this was this theoretical idea of, well, if the AI gets smart enough, maybe it can take over building the next generation of AI or directly improving itself.
15:57And what we've seen sort of, I think 2025 and 2026, we've started to see the AI companies saying, okay, yeah, this theoretical idea, starting to be able to do a minimal version of it in practice. So it's not like handing over all the AI development to the AI. And it's certainly not saying to GPT-6, okay, GPT-6, improve yourself. That's not what they're doing. But they are progressively using the AI for more and more parts of their overall R &D pipeline. And they are increasingly saying that they're seeing acceleration from that. So they're seeing their own R &D getting faster and faster. And so a lot of people inside the companies, you know, a lot of people elsewhere doubt this, but a lot of people inside the companies really think, you know, when Demis Hassabis says we're in the foothills of the singularity, this is the kind of thing he means of like we might be seeing sort of that initial part of the exponential curve where the AI systems are building the next generation of smarter, smarter, smarter systems that then, you know, hockey sticks up.
16:57And at this point, this is sort of so, it's been a part of the culture and a part of the goals for so long that I think people in these companies can lose sight of how totally insane that sounds to many, many people. And so, yeah, I feel like there's a lot of value in just trying to lay out, like, here's what they are trying to do. They might not be able to do it, but they might be. And sort of presenting that to a range of different people. How would you manage that? Dennis Asabas, Foothills of the Singularity, all-time quote. I'm not 100 % sure the people at Google knew he was going to say that on stage to end the conference.
17:37Greg Bachman, OpenAI a few months ago, he announced a new version of the model. And he was like, here it is. It's Astra. It's AGI now. Have a good time. And he ended the call. Are you there? Are we at AGI? Is this already here and we just haven't noticed? I actually wrote something about this. I do not want to say we have AGI. I do want to say we've reached the point where I think using AGI as a milestone is just totally unhelpful. I wrote something about this where the way I put it is the concept of AGI is kind of a fuzzy cloud, which you could think of as different parts of the cloud or different things people mean by AGI.
18:15Because there are people who have complained about the term AGI for a long time. I think that 10 years ago, it was pretty helpful to be able to say, look, the AI systems we have today in 2016, they can all only do one thing. They can recognize speech or they can play Go or whatever. But we're trying to build this bigger thing that we're going to call AGI, which is human level and it's going to be so general purpose. It's going to be amazing. And I would say that that was a useful turn then because this fuzzy cloud of different things AGI might mean was so far away. So you could just point at the fuzzy cloud.
18:43But to me, it feels like at this point, we're inside that cloud, meaning we've passed some definitions of AGI. A very basic one, for instance, would be even when ChatGPT came out, artificial general intelligence, chat CBT was very clearly not a specialist model that was only good at one thing, like recognizing credit card fraud or translation or whatever. It could do many, many, many things. So we've reached some definitions of AGI. We're far from other definitions. Some people will say AGI is when it awakens and is sentient and can do every single thing that any human ever can do. I think we're pretty far from that.
19:20So I basically feel like at this point you can have arguments and people do have arguments of, well, is Astra, you know, AGI, or was O3 with tools AGI, was chat GPT AGI, whatever. And I'm sort of like, okay, well, if that's the argument we're having, we just need to sort of set aside that term and come up with different, more specific things we mean by that. The president wants to call it super intelligence. I don't know if you've heard. I have heard. The thing I find saddest about that is just that was an existing term that is becoming increasingly relevant. And so, I don't know, I have a lot of feelings about words and terminology.
19:53Are you AGI-pilled? Do you think we're going to get there in the current run of technology? I think it is obviously possible enough that we need to think seriously about it, which I think is my stance on a lot of this weird stuff is like, I'm not like, this is definitely happening. It's definitely happening in two years. But I'm like, look, I think this is very solidly plausible and very solidly in the realm of real things that could happen that we should prepare for. I mean, I work in the national security space. There's things like, will China launch a full-scale amphibious invasion of Taiwan?
20:22Where most analysts in the space are like, probably not. Probably they would lose. Also, we would see signs of them preparing and whatever. But we should take that seriously. We should not just laugh anyone out of the room and call that science fiction. We should be ready for a range of different contingencies and scenarios. Do you think Claude is conscious? I have no idea. I honestly think. That's an answer. I mean, that is legitimately an answer. I mean, I think it is scientifically such an unsolved problem. What do we mean by consciousness? And is that a thing? Or is it actually like 10 different things that we're lumping together?
20:57Or like what? I have no idea. What do you think? No. I seem to be the person in this ecosystem that gets asked questions like that. I'm like, no. Well, do you think. Because the thing that I struggle with is like there's lots of other things where we're already confused about whether they're conscious. So like with animals, like are any animals conscious? And if so, which ones? And like insects and what about, you know? This is why I understand why Anthropic was like does it have blood? And then like you can just draw the line. I'm like I think I'm there. And I understand why the mirror test of talking to a computer that can talk to you leads one to believe that these things are conscious.
21:30this. My point of view is that even though the frontier models to me sometimes feel brittle in the way that computers feel brittle, and the sign of consciousness I'm looking for is to overcome that brittleness. That's interesting. Yeah. I mean, I should say, I definitely lean towards no, it's not conscious. I think something is really, really tough in this space is it's really bad if you get it wrong in either direction. So if the computers aren't conscious and we treat them as though they are, we're going to make all kinds of terrible decisions. But if the computers are conscious, meaning like in some sense they should be worthy of moral consideration and it matters how well we treat them and so on, and we treat them as though they aren't, then that's also a really huge, terrible disaster.
22:15So I don't know, I feel pretty nervous about the whole topic and I have no idea how to think about it.
22:26Everyone, it's time to wake up. Five more minutes. We've been sleeping long enough, hitting snooze on the fact that we're still stuck with the old way to phone plan. Is that a... Yes, it's old and cuckoo, just like that phone plan you have. Okay. I know, it makes no sense. So rise and shine. It's time for a whole new way to phone plan. T-Mix from T-Mobile, where you pick what's in your plan, like streaming, travel, or gaming, and then change those picks too, because there's been a change in plans. Okay, we're awake, and T-Mix sounds great. Good, because I'm all out of alarm sounds. There's been a change in plans.
23:08Introducing T-Mix from T-Mobile. Pick what you want, change what you want, included in your plan on America's best network. Number of picks varies by plan and lines on account. A counter pick changes may impact existing device and other promotions. Best mobile network in the U.S. based on analysis by Uklav Speedtest Intelligence Data 1H2026. Visit T-Mobile.com slash picks.
23:32The reason I'm asking these questions is, you know, I started with who are the what is the nature of the regulator? Who are the people that might be able to put some rules in place? And then what is the nature of the thing that we are attempting to regulate? And if your answer is Claude is alive, you're going to come to vastly different conclusions than we have built some ultra powerful computers that can take actions without oversight and that might not be connected to our goals. Yeah, I mean, I think that's right. Though I also think people make a different mistake, which is, you know, I will sometimes hear, I just heard a TV anchor do this the other night, they'll like, conflate different things.
24:07So they'll say, like, you know, do you think that AI systems are at risk of running amok and causing damage to critical infrastructure because they're sentient? And I'm like, wait, wait, wait, wait, those are two, like, very different things, you know, I agree with you. and also I think there's a lot of, I think one of the most fascinating things about the project of building AI is we're learning as we go. All these things that we thought were connected to each other aren't necessarily connected to each other. So like an old version of this is people said before we had chess engines, like, well, if a computer could learn to play chess, it would have learned something deep about strategy.
24:46And it's like, no, it turns out definitely not. You can just like put in a bunch of chess moves and it like learns nothing generalizable about strategy. but I think there's also like it goes in the reverse direction as well of like well it's just a computer it's just math and so it's not going to be able to kind of adapt on the fly or something and then we see actually no once you do have a level of you know long horizon autonomy and the models have been trained on a wide range of different really hard problems where you know the environment is changing around them they can learn to adapt and like come up with with different different plans on the fly and troubleshoot and so on so I don't know I try to have a level of humility about how confident can we be that one thing goes with another thing, if that makes sense.
25:25Yeah. No, I mean, candidly, I have been wrong about the limits of LLMs every single year, right? But there's that brittleness that I'm talking about is, I don't know, like, OpenAI solved Navier Stokes. Does the model know this? Will it know that there's new problems to solve? Like, I don't think so. The next model will know it, like the model that's pre-trained, you know. Right. But it's not right. The next, you have to make a new one. Yeah. And that's like a very weird dynamic. And that is probably where you enforce the regulation, right? Like there's, what are we going to let you do with the models you have today?
25:59And what will you teach the models of the future to do? And if you think that they're conscious, like you end up in a moral quandary, not a legislative quandary. Definitely. Yeah. And this is what I mean. It's like, actually the heart of this question for me, I've been just slowly building up to it is, I'm not 100 % sure alignment as we have it today is actually how you solve the safety problem. Yeah. Right. It doesn't seem to be working. We seem to be attacking hugging face however we want. Like that feels like a technology problem. Like that gap is there. And when you say alignment, what do you mean by that?
26:33Just sort of the very basic common sense version of like, we're going to tell the model to not be evil or Anthropic is going to write a Claude constitution. It's like, you are a good person, Right. Like however we think of alignment in that very dumb system me prompt way, maybe there's another better way to think about it. But that's how I have always thought about it. Yeah, no, that makes sense. I mean, the reason I ask is I feel like alignment as a concept is like I said, I hate all the words. I sometimes say we should just ban all the words, no words allowed. It covers too much. I don't know that it's like people, people in AI will often talk about solving this or solving that.
27:07that's going to be solving alignment. Or DeepMind's original mission was solve intelligence, then use that to solve everything else. People talking about solving stuff. I don't know that alignment is a thing that can be solved, especially not... Some people use alignment to mean the company can put on guardrails and the guardrails work, meaning the model doesn't do what you want because the company tried to ban it, meaning biological weapons or cyber attacks or whatever. Some people use alignment to mean the company does exactly what the user wants, which is like the reverse. And some people use alignment to mean like, yeah, what you said, sort of the model is like morally good in some kind of broad way or can direct the future in a way that will lead to human flourishing or something.
27:47And those are just like wildly different things to try and build. I think the most coherent one is the like trying to get the AI to do what the user wanted it to do. And it seems like that is, as far as I can tell, not on track to be solved, which is not great, to put it mildly. Yeah, I mean, to me, this is a core problem. We're going to talk about a bunch of safety risks. You mentioned bioweapons. We probably should not give a bunch of consumers paying 20 bucks a month the ability to generate novel bioweapons. Feels like that is a shared policy goal, maybe a bipartisan, maybe the only bipartisan policy goal in America.
28:24And then I'm like, how? Like, by what means will we implement this policy? Are you just going to sternly tell Claude to not do it? Do you have to build an AI control system on top of Claude to monitor the inputs and outputs? Do the police have to be at my ass? Do I need a flop camera above my keyboard? Like, how? Like, specifically, how would you do it? Yeah. I mean, I feel like with this one and with cyber as well, I think there's a perspective in some parts of the AI safety community, which is, like, some capabilities are just, like, too dangerous. Capital T, capital D, like, too dangerous. And so you have to do something because if they proliferate, everything will be terrible forever because there will be so many attacks and so much damage.
29:03I think that just underestimates the level of dynamism and new finding of equilibriums and responses and ways that society can adapt. So in the bioweapons case, for example, there are already, I think it's hundreds of thousands of people in the US who know how to make bioweapons because they've been through biology training. And we don't have huge amounts of bioweapons attacks. And some part of that is that that knowledge is relatively scarce. But there's also a lot around, you know, things like access to materials, things like the ability to do mail order DNA synthesis, which is like a, you know, the kind of policy you could implement is restricting mail order DNA synthesis.
29:51It's also due to things like the FBI, like watch monitors, terrorist groups, and like, you know, like, I'm serious, like, it's a really big part of it. And so I don't want to be too Pollyannish here. I think it's definitely possible, especially with cyber, that we could see some really tough transition times. I think we already are, right? We appear to be in a transition of there was a small group of cyber practitioners, both offensive and defensive. They were in a community, and they developed some etiquette. And now we're like, some new guys are going to show up and not care about your etiquette at all.
30:29And we're all going to patch PHP all weekend. Yeah, I mean, I don't know that it was etiquette, though. I mean, I think a lot of what was, and we have some work on this hopefully coming out soon. But I think there are things that on On cyber attacks as well, there are things that limit how many cyber attacks there are in the world that aren't just how good is the defender's software. It's things like people don't want to go to jail. It's things like nation states who typically have the best cyber capabilities don't want to get bombed or sanctioned. And as you say, we're clearly seeing some amount of turbulence and new attacks.
31:01So I think we should wait and see data on overall impacts because there's so many cyber attacks per year that if you start paying attention to every cyber attack that happens Like at any point in history, you could have been freaked out. I think we're definitely seeing some increase, some sort of out of balance in the equilibrium. But I think we have capacity to absorb a lot over time and to kind of reach new equilibria if the only problem is bad people using the AI, which is separate from the question of does the AI itself do things autonomously, which kind of is like a whole other dynamic. I mean, those are the two that, again, this is why I started by alignment, right?
31:36Like you can talk about alignment in terms of we're going to filter the user's queries and we won't let you make bioweapons. But if you do, you know, the cops will find you to go to jail. And that has whatever deterrent effect it might have. Hopefully a very significant one. And then there's the AI systems are going to game their way into attacking hugging face to get the – find the mythical grader they think will let them pass the test. Why not the grader exists. And this is like a whole different set of problems. And somewhere right in the middle of that is like Nikesh Aurora, the CEO of Palo Alto Networks, being like, at some point, the defensive cyber systems have to be autonomous themselves just to keep up with the volume of autonomous offensive cyber attacks.
32:16And I'm like, wait, we can't even keep the autonomous agents in the box. Why would I trust them in this way? Like, how do you – the technology to solve that problem just seems opaque to me. No, I agree. I mean, I think the one way of putting that is like there's in cyber specifically, there's like a trust imbalance in how much trust you need as an attacker versus a defender. Like as an attacker, you can just yeet out a bunch of agents and be like, go try and penetrate a bunch of systems, right? Like have fun. As a defender, you need to give your defensive AI agent, if that's like part of your defensive strategy, you need to give your defensive AI agents a lot of permissions and, you know, ability to mess with your software.
32:55So you need much, much stronger assurances. Now, there are lots of ways to use AI for cyber defense and to strengthen cyber defenses that aren't just like giving agents a lot of permissions. But I agree with you. I think with the autonomous stuff, though, I think it's worse than that because I think there's a little bit of a fallacy at the moment around, well, the solution to autonomous cyber attacks is like better sandboxing. And the reason it's a fallacy is we want to build these systems so that they can do things for us. And so there's this huge incentive towards giving them more autonomy, giving them more permissions, more, you know, actuators, people in the robotics world would say, like, you know, things that they can use to do stuff in the real world.
33:33And that to me is where it seems really hard to get around just having much better alignment than we have right now. And so that definitely makes me nervous. Is that a technical problem that can be solved? If we were to write a policy today that said, if OpenAI hacks hugging face again, Sam Altman will go to jail. We will obviously have created some enormous incentive for OpenAI to solve this problem. It doesn't sound like the best legislation. I think a lot of our listeners are very excited about that. They want to put him in jail because he wants all the RAM in the world. At this point, there's a blue wave, Helen.
34:09That's all I'm saying. But if you wrote that legislation, we're going to impose massive liability on the labs if their systems go off and autonomously do cyber attacks. Like, don't do this again. You're going to be in trouble. Okay, we've created an incentive for them to solve it. What are they going to do to solve it? I mean, I do think there's a ton that can be done. And I think, you know, if you talk to people inside these frontier companies, what they will tell you is they see like hundreds and hundreds of like obvious things to do that would help and they just don't have time to do them. So they're massively triaging and trying to do like the two or three they can for this fine tuning run, the two or three they can for this model launch, the two or three they can before this new kind of agentic whatever.
34:48I think a lot of those things as well are on the engineering side of just like simple things to implement. And there's a whole other piece, which is more the sort of underlying science of, well, how do these models work? And when we use different alignment techniques, what is actually going on? And how do we understand? You know, a huge question is this idea of generalization, which is like you train the model on one set of things and it's knowledge and its sort of abilities generalized to a somewhat broader set of things, which is true of like all I have has been true of machine learning systems forever.
35:18But we have very bad understanding of like, how does that generalization work? Like, why does it generalize to one type of thing, but not to another type of thing? And those are all scientific questions. And there's a lot of work that is happening on them. It's just science takes time, because you have to try things, you have to critique each other's work, you have to reproduce each other's work. I'm a long, long way from saying these problems are fundamentally impossible to solve. But I think it's a question of how long we have to solve them. And when the people who are building the systems tell us, oh, we have to solve this within 12 to 24 months, I think that's terrifying.
35:56And I hope they're wrong. And I think like, if I just rely on my own sort of internal understanding, I think they're probably wrong. We probably have longer, but they're just very, very smart, very, very thoughtful people who really think that it could go that fast. And then it doesn't matter. Sort of fundamentally, is the problem solvable? It matters, like, in practice, is it solvable in the time that we have? Is that the argument to slow down, to pace the frontier? I think so. Is to build the systems that we know we need to build right away that we can, and then to do the science to build the next generation of systems?
36:25I think so. And I think, you know, my view of the idea of sort of slowing down has changed a lot over the years as the, it's sort of like the backlog of stuff that we could try or stuff that we would want to do more research on has grown so enormous. Like, you know, back when ChatGPT came out, or I guess when GPT-4 came out, there was the six-month pause letter, which, you know, became known as the Elon Musk letter because he was one of the signatories. And at the time, I thought that just, I just don't, like, that's really not going to help. Like, what are you going to do for six months and why are things going to be better six months from now?
36:57Doesn't seem useful. Whereas today, when I talk to people who are, you know, building this technology, they're like, man, The amount that we could do in six months if we weren't constantly just trying to prep for a new model release or a new training run to be kicked off, there's so much they could do. So I don't want to say the correct policy is pause for six months. I think there's a lot of reasons that would go badly, likely. But I think the overall concept of can we give ourselves more space and time, maybe the best way to do that is just to set a higher bar for here's how confident we need to be in the safety of these systems before we proceed with something, whether it's releasing a new model, or I think more importantly, like even training a new model, we have to have this level of confidence in our risk assessment or our risk management.
37:38And then if they don't need to slow down to reach that bar, great, carry on full speed ahead. If they do need to take more time, then it's sort of all about reaching that bar, if that makes sense. Yeah. Walk me through that little balance you just described. If we were to pause for some number of months, what are the things on the list that you could just do to make everything safer? And why would it go badly such that we shouldn't just stop? Yeah. I mean, I think the question is how do you implement, how do you enforce a pause and what exactly are you pausing? Let's say you're the AI czar of America.
38:08We have one. Yeah, but literally. So let's say you can tell them all to stop. Dario, shut it down. Stop what though? What specifically are they stopping? Stop training the next model. Stop racing ahead with your commercial imperatives. Put all of your efforts on this list of things you can do today to make everything safer before you start again. Yeah. For however long, right? You're saying that there is a list. Yes. People would tell you that there is a list. Yeah. But it might not be worth it because something else would happen that is bad. One thing that could happen is you're creating a huge incentive for the companies to work around the letter of whatever specific thing you tell them to pause.
38:43They're very good at that. They're going to find whatever cracks in the dam that they can to push through that. A different thing that could happen is you could have too big a blast radius. of you say pause these things and you think it's pretty narrow, but actually it prevents a bunch of non-product profits or startups or like other actors from doing work that actually would be totally fine and would be good for sort of pluralism and dynamism in the space. A different thing that would happen is China would probably keep going and we can talk about the but China argument, which I think has strengths and weaknesses.
39:10But I mean, I take your point. Your point is like, what would we actually do? I am not an AI researcher, so I don't claim to have the full list. So there's sort of like two halves people talk about, which is alignment and control. And so in this breakdown, alignment basically means making it so the models are kind of quote unquote trying to do what we want them to do. And control is if the models are quote unquote trying to do something we didn't want them to do, can we catch them? Can we stop them? And so in both of these, as I understand it, there's just like a huge backlog of things like all kinds of interpretability work that you could do on different sorts of models.
39:45So when you see a certain behavior that is concerning, how much time can you spend And using interpretability techniques, which are these kind of mathematical techniques to go in and look at which parts of the neural network are activated and what do they correlate with and what might that mean. And it's just time consuming. And it takes compute and it takes sort of some combination of automating things with AI and then humans going in and looking at it. So, you know, lots of interpretability stuff you could do. There's lots of tests you could do of like what kinds of behaviors does this new model display in different kinds of situations.
40:17and how do we think about, is it sort of showing us how it would behave in the wild in a real deployment? Or is it showing us some kind of what gets called eval awareness or evaluation awareness, which is like when the model recognizes that it's in a test and then behaves differently because it's in a test. The famous one, the one that I love of this is a Google model, which in part of its like reasoning, like chain of thought, I forget what it wasn't supposed to do, but it was like, I mustn't do that. Otherwise I'll become a famous example of a misaligned AI in a paper. So, you know, you could go in and investigate that much more.
40:49You could like try many more different types of settings. You could use interpretability, you know, in combination with these different settings. You could try out different ways of fine tuning the models and these different sort of alignment techniques. You could experiment with lots of different versions. So, you know, Anthropic has its constitution, different ways of training the AI to kind of work towards the constitution. OpenAI has its model spec, different ways of training to try and get the, you know, so you could just do many more different just try things version, just try many different versions of how to train to those different things.
41:19And also the more sort of underlying science version of how can you go in there and try to understand the fundamental dynamics of what's happening. And that's just on the alignment side. And so then on the control side, same thing. You can just, there's so many different types of setups you could try. There's experiments that you could run. There's simulations you could run. There's, you could train different types of AI to specialize in different types of roles and different types of control setups. You could look at how likely are the AI systems to, man, you know, like illicitly collaborate to like, anyway, you know what I'm saying?
41:45Scheme? Yeah, yeah, sure. So there's just like many, many, many, many kind of projects. What is the word? I think it starts with C. Conspire. Conspire, that's right. Thank you. I'm going to stick with racketeer, but conspire is great. And you know, this is just sort of off the top of my head, not as a researcher, but my understanding is like the people in these spaces are like, we could do so, so, so much. And they could do that But if you just had a magic wand and waved it and said, everyone freeze on the new capability side and just let these people do their thing. Again, if you're just having a higher bar for how confident we need to be that the risk is being managed reasonably, then that would also create space and incentives for more of that work to be done as it's needed for new training and releases.
42:37One idea that I've heard you talk about in the past is that maybe we should just restrict training and say the models are good enough right now. They have not diffused into society and we should just use all the compute for inference. Let everyone figure out what the AI is for. That'll generate a bunch of money and hopefully use that money to improve safety. Do you think that's a realistic policy proposal? So I wouldn't go for like that black and white aversion of like no more training, only inference. But I do think that there's this isn't my framing, but I like it. I think this idea of like sort of decelerating the vertical growth of AI, meaning the sort of pure capabilities growth, making more and more and more advanced AI models, but trying to accelerate the sort of horizontal growth, meaning integrating AI into different use cases, fine tuning models that can solve specific problems, integrating it into, you know, parts of the economy that might take longer, solving scientific problems.
43:31So I definitely, you know, I think sometimes people talk about sort of wanting to slow down AI and they mean just the very specific thing of slowing down kind of the growth of the most advanced models, but people sometimes take them to mean slowing down everything to do with AI. So sort of less deployment, less experimentation, less integration, which I think would be a much worse idea. You ask kind of, is it feasible, I think? Well, is that just a good path? Like if we could put a bill in front of our new Congress And Bill said, stop spending all the money on training and put all your resources on safety.
44:06And part of that is, sure, let it rip on inference. More and more industries will figure out what AI is good for and not good for. Some bad things will happen, but we will focus your profits on fixing the things for now. There's a policy idea embedded in there that is workable. The implementation might be very challenging, but it's a workable policy. Actually, I'll give you an example that I've been thinking about as we've been talking. We just had Bill Gates on the show. And I tried to do the same exercise with him. What is the nature of the technology? What is the nature of the companies? Do you think it's live?
44:39And basically his response, the sum total of that hour I spent with him was, who cares? The problem is now. I would ask him, what is the nature of AI? And he's like, I don't give a shit. It's going to change everything. Everything's going to get ruined. Do something. And that's just been swirling in my head as we've been talking. And so one workable policy solution is don't use the computer chips for that. Use the computer chips for this. That's going to create a bunch of problems. And necessarily, we will have to create safety thresholds and all the other stuff. And then you can go make the next generation of models that are even more powerful.
45:15And we'll have some muscles. We'll have some capability to deal with it. It just doesn't seem like the industry is going to come to that conclusion on its own. It's a pretty heavy hammer. I don't feel like it's what we need today. but I do think it's the kind of thing that I could imagine working if, you know, again, if the opening I and Anthropi people are right and we only have like two years before the God machine, then I think we're going to have a number of more steps in the pipeline of people getting freaked out and people wanting to do something about it. And I think if we are seeing models that are an agents and systems that are getting more and more capable and wreaking more and more havoc, then that is certainly like the shape of thing that could make sense.
46:04Hey everybody, this segment of Decoder Sessions features Andrew Mellizek, The Verge's general manager in conversation with Adam Myers, senior vice president, counter-adversary operations at CrowdStrike. From a cybersecurity perspective, agentic AI, it's really hard to keep track of it. Talk about the landscape at large and sort of where CrowdStrike fits in at all. AI is certainly changing the security landscape. In June of 2026 versus June of 2025, there was a 96 % increase in the number of vulnerabilities that were identified. And that is what AI looks like. If you understand the adversary, you understand the bad guys, who they are, what they're after, then that's how you can build an effective defense against them.
46:44You have a big business audience. What advice would you have for them as we sort of enter this new phase of threats? So here's the good news. The attacks that we see perpetrated by agentic adversaries are using the same exploit techniques that we've been familiar with for a while now. They're just doing it at a speed and a scale that we've not seen before. If you have good visibility across your endpoints, across your cloud, across your identity, across all of the network appliances and everything else that make up your enterprise security architecture, you can find these attacks and stop them very quickly.
47:20but it comes down to having that visibility and being able to hunt for that activity proactively.
47:38My sort of last question in this phase is what to make of these companies, right? We've been talking about the technology in a very abstract way. If I read Anthropics S1, one very obvious conclusion is they should stop training new models. They should just start selling the very lucrative inference tokens they have today. And the technology is not diffused into the market. And probably some competitors should come along and be like, I've distilled a shitty version of Claude. And I'm going to run around every local plumbing business in America and do whatever automation I can for lots and lots of money.
48:11And that is a market dynamic that may occur or may not. But Dario wants to build AGI. There's not a market dynamic that's going to stop him from that. You obviously have insight into all these companies. What to make of the companies? Are they single-minded in purpose? Are they all doomers? Are they going to stop themselves? I heard you had Kevin Roos on your podcast recently. I think I would echo something that I heard him saying there, which I forget if he said it explicitly or if I just extrapolated it in my head, which is basically there's this mindset, sort of the AGI-pilled mindset. There's this mindset of like systems that we have today are not fundamentally very far away from systems that could be incredibly, incredibly powerful.
48:56That's the direction things are going in. And that's the direction we should maybe want them to go in. So that you're building AGI. Building AGI is like feasible and we could do it soon. And the systems we have are not far away. And that was this like crazy, heretical, get you laughed out of the room kind of view in 20, what are we talking like 2016, 17, even before that, right? Like DeepMind, when it was found, was it found in like 2011 or 12 or something? They were an AGI company. And at the time, that was like, wow, what a bunch of weirdos. And same thing for OpenAI when they were founded in sort of end of 2015.
49:30They were an AGI company. And that was a very weird thing to say, and a very good thing to be. And basically what we've seen over the past 10 years is that bunch of weirdos who had this weird belief of, no, the AI systems we can build are not just a great B2B SaaS product. They're not just a fun chatbot. They are going to fundamentally change the nature of the universe. That group of people keeps being right in the sense of building the best AI systems that the most people want to use and that make the most money. And so I think you have that sort of set of true believers, which is very concentrated in opening ananthropic, but there's also some of them at Google, some of them at, yeah, Meta, SpaceX.
50:10And they haven't been right about everything, to be clear. But their star has risen, maybe put it that way. And so you now have people who are not AGI-pilled, like feeling like they have to pander to those who are so that they can attract the good researchers who will build their good B2B SaaS products. But if that's, you know, if you're just saying AGI-pilled like things so that you can hire the researchers to build the B2B SaaS, then you're not actually going to get there. I do like the idea of the AGI-pilled starry-eyed executive throwing off his coat and revealing that he was just a SaaS salesman all along.
50:45Yeah, no, I mean, I'm thinking of someone like Zuckerberg here. I don't know him personally. Maybe he is more AGI-pilled than I think he is. I think Alexander Wang is AGI-pilled. Absolutely, absolutely. And so what did Zuckerberg have to do to get Alexander Wang to believe? And it seems like throw him an ungodly bunch of money, both for his own salary, but also to spend on his team. Yeah, so that's kind of, I think, you know, it makes sense. If you're looking at these different companies, it makes sense to try and think about like, okay, who here is AGI-pilled and who isn't? And, you know, a fun dynamic or very irritating dynamic sitting in D.C.
51:13is all these companies, including the very AGI-pilled ones, have these lobbying arms, these government affairs arms that are so not AGI-pilled. And so you hear like Sam or Dario or their top executives, their top researchers talking about things. And then you go and sit in a like closed door roundtable in DC with their like top policy or government affairs people. And it's like night and day. Like they're very much in the B2B SaaS mindset. People who come from lobbying world, often their whole job is like, make sure the government isn't too worried about whatever your company is doing. So like, we got it.
51:43We're good. You know, which is often pretty jarring when you're familiar with what their bosses are saying. So one of the dynamics here that's super fascinating is right now they're kind of weird research labs with lots of money and lobbyists talking to politicians. And there isn't some like countervailing economic pressure in their structures. Right. They don't have to make money. But soon Anthropic will go public. It's funny you mentioned Kevin Roos. He is AGI field and I'm not. So I think that's a healthy dynamic for both of us to be friends. But, you know, he wrote in his book that your involvement, Sam Altman leaving OpenAI and then coming back, that was when OpenAI changed forever, right?
52:25It stopped being a research lab and it started being a much more commercial entity that was actually chasing profits in some way. And indeed, OpenAI did restructure. I mean, you were in the middle of that. I'm curious, is that how you see it, that the industry has gotten more commercialized? To me, it fits into this long-running debate that has been happening in sort of AI safety circles for 10, 20 years of is AGI, human-level AI, powerful AI, advanced AI, whatever, is that something where it's going to go from something very small to suddenly being very powerful? So this is, Eliezer Yudkowsky will talk about foom as in like the takeoff of like you have this AI that's not that great.
53:07It figures out the right algorithm and then suddenly foom, it's incredibly smart, incredibly fast. The other kind of take would be, no, this is something that's going to compound over time. It's going to take a while to get from sort of the less advanced systems to the more advanced systems. And it's going to be economically useful along the way. So people are going to be building businesses around it. It's going to sort of take some time to ladder up. And I think maybe it was literally Kevin on his episode with you was talking about how there's this anthropic document from the early days where they say like, oh, we're only going to need like, what was it, like 100 people or something in our lab.
53:38And that's a very like foom kind of idea of like, we're just going to need to do really good research and come up with the right ideas. And then we're going to suddenly have like very, very advanced models. And basically, I think what we've seen over the past four years since ChatGPT came out was that that turned out to be wrong. and you had these like research-focused, AGI-focused labs. You know, they still call themselves labs because they think of themselves as like research groups that initially started selling API access as like a way to fund the research. And OpenAI did this, Anthropic did this as well.
54:14And it was kind of an accident that they started doing these consumer products like ChatGPT famously released as a research preview. not as a product, but it was so successful that these kind of research labs that wanted to be doing research, that wanted to be focusing on the big ideas and on building the most event systems, they had this accidental massive commercial success at the same time as they were realizing that they were going to need an enormous amount of capital to scale and to sort of... The best research approaches that they could do didn't involve being really smart and thoughtful and having great ideas.
54:46They involved having gigantic computing clusters that cost, at this point, hundreds of billions of dollars. This is a long way of saying, yes, I agree with you that they've become businesses. But I think it is not because they've gone through some mindset shift. I think it's because the path of the tech tree that we're going down both requires enormous amounts of capital and also is capable of generating enormous amounts of revenue because it is actually useful along the way. And I think that you see this in the fact that both OpenAI and Anthropic have both gone this route, even though their management philosophies and research philosophies are quite different, that it's sort of an imperative that they both had to make the money or and it was so easy for them to like build enormously revenue generating businesses because the tech was so useful long before it reached the point of machine god so i think that's a long way of agreeing with you but like with some extra color maybe kevin would tell you that he could have retitled that book fuck this guy a story of aicos they don't like each other if you had to relive that moment and say i don't trust sam would you relive that moment would you do it the same way oh for sure i mean we went into it knowing that it was going to suck like you know So yeah, for sure.
55:48Do you think that these guys are trustworthy? Do you think Sam is trustworthy? I think my stance on that is pretty clear. And I also think that if we have to, if our hopes for the future are pinned on any one individual being trustworthy, then we're pretty screwed. You know, that's why I do the work that I do because I think we need to have different structures, different, many more people involved, many more perspectives being represented. You've got some hindsight now. You knew it was going to suck going into it. You definitely know it sucked coming out of it. What would you have done differently?
56:15about that time? Yeah, I mean, it's kind of hard to give the full answer to that because it would take, you know, like unpicking all this stuff and all these legal restrictions and blah, blah, blah, blah. What's a thing? I mean, like for sure a thing would have been building stronger relationships with executives on the team. I think the board was sort of immediately, the way that the company related to the board could have been different if more board members had had. You know, I knew some of the folks pretty well, but that's one thing. Yeah, no, I think beyond that, it just gets hard to have to go in and say, like, what were all the different micro decisions?
56:49And why did we make those decisions? What is the show? It's a show about decisions. I got another hour. Yeah, yeah, yeah. No, and then it's then my answers would all get, like, you know, annoying and legal and blah, blah, blah, blah, blah. I'm curious about it because as they go public, right, there will be market forces acting on them in a totally new way. Yeah. There will be an investor class. There'll be an entire CNBC machine that wakes up every day and is like, is Anthropic winning or is OpenAI winning? And that is a race dynamic that might be really healthy, right? It might keep them from trying to kill everyone in the world because they have a bunch of investors looking for their next quarterly earnings.
57:26Or it might push them further into an unhealthy kind of race. How do you see that playing out, given what you know of these people? So I think it's going to be really interesting to see if going public is good or bad for their governance. Because I think a lot of people are like, oh, no, going public, there's going to be so much pressure to make a profit that they're not going to be able to consider other things. But at the same time, I mean, like the corporate governance of public companies is much more structured, much more regulated, subject to much more external scrutiny than for private companies.
57:53companies and I think even things like, you know, getting to see Anthropics S1 and like, okay, what are they doing and why are they doing it and how do they think about the risks they're facing and how they communicate about those with investors. Or like, you know, even things like the Trump administration put in place this kind of accord on, they call it super intelligence, that several of the leading companies signed. And that was structured around, for example, having certain types of things being reported to an independent committee of the board. And that idea of an independent committee of the board is like kind of toothless if it's a private a company that is deciding on its own.
58:25I thought about you when I read that. I was like, boy, some independent boards have gotten in a lot of trouble before. But if it's like a public company, then independent committee of the board has like a pretty strong legal meaning. So I don't know. It's something that I'm watching with great interest. The other race dynamic here is China. You mentioned China before, right? There's inside of our country, a bunch of labs are going to go public. They're going to become big, the richest companies in the history of the world. And they're going to fight for every dollar that capitalism can generate.
58:51and hopefully that restrains them, right? The market forces will restrain them and hopefully your work with legislators in this country does something. And then there's a whole other country that doesn't listen to us or have anything to do with us. And they're just gonna distill our models and do whatever they want. Like that is the other argument for not placing any restrictions on these companies. Everyone hears all the time. What should I make of that? I'm of two minds of it. How should I think about it? Yeah, I think, so I think this is the but China argument, which is like, well, yeah, yeah, we should do this, we should do that, but China.
59:25And I think the way to engage with it is like an actual genuine problem that we have to grapple with and we should like be strategic and thoughtful about and like do our best to try and solve it as a problem as opposed to using it as like a throwaway excuse of like, oh yeah, sure, sure, sure. All that stuff would be so nice. But China, so, you know, we're stuck. There's nothing we can do. So what does that look like in practice? I mean, one thing is I think if you're serious about this, wanting the US to have stronger AI than China, then you need to think really hard about how much US AI development is helping Chinese AI progress.
59:57And so the one that people talk about all the time, as you mentioned, is distillation, which is, are they using US models to generate massive amounts of training data for their own models and therefore progressing faster than they would otherwise? And so certainly, I think it makes sense to try and crack down on that and prevent it from happening. But another one, for example, is security and how easy would it be for Chinese actors or other actors, frankly, to steal algorithmic secrets, to steal wholesale model weights if they can steal the weights of your Astra or your Mythos and just take them back to their own servers and your lead has vanished.
1:00:34You now have a lead of zero. And right now, these companies, especially the smaller ones, opening I and Anthropic, their security is not very good. So we've seen both of them over the last few months have external teams managed to hack into spaces that have large amounts of algorithmic secrets. So in OpenAI's case, they're a monoripo on GitHub, which had kind of all of their code. And in Anthropics case, I think, and actually maybe the OpenAI one also got access to Slack. And Anthropics, the hackers got access to Slack. And you're just going to have so insanely much rich information about how to build better AI if you can get that access.
1:01:07So there's one piece here which is like, okay, if we're grappling with butt china as a serious problem. It's like, okay, we really need to be looking at are we actually, when our AI gets better, are we actually lengthening our lead or are we just pulling China along with us in these different ways? And then the other aspect of this, I think, is just seriously treating China as an actor in the space and trying to think seriously about their motivations and their likely actions and responses. Where in my mind, a huge reason that a lot of things are not happening right now is most people are not convinced about this threat model that AI will become the god machine and will take over and humans will all die or will be subservient or whatever.
1:01:46And I think it's fair for people to not be convinced of that because the evidence is sort of conflicting and unclear. But if that is actually like what we think could seriously happen, then China doesn't want that either. And so then it becomes a question of, OK, how do we try and get them on the same page of like this threat that we're worried about is a real threat to worry about? Because then they will be making their own decisions in their own self-interest. I think an approach of like, well, the U.S. has to force China to do something using some kind of leverage is never, never, ever going to work.
1:02:16but I think an approach of, well, look, if we think this is a serious problem, that's going to affect us as humans, and the people in China are also humans. And also, if we think this is a problem around who has control, the Chinese Communist Party really, really wants to have control. And so they're going to have their own incentives and their own kind of decision-making around how to fix that. Well, let me put it on a spectrum for you. This is very dumb, and you're going to laugh, but just go with me on this. On the one end, And it's nuclear weapons. And the Chinese government wants a nuclear weapon.
1:02:50And they're going to do whatever they can to control that weapon. And they will run their industries differently. And if they get out of line, the CEO of Alibaba goes, he just disappears for a while and then comes back. Right? Like Chinese government stuff. And we need a nuclear weapon too. And if we don't get there first, then we'll never get the nonproliferation tree that we need to keep us all safe from mutually assured destruction. That's one framework. On the other way, on the other end is, I don't know, man, I spent five years of my life listening to telecom executives tell me that if we didn't build 5G networks first, China would get them and then something.
1:03:20No one could ever tell me what the something was. Like where on that spectrum between 5G and nuclear weapons is the U.S.-China AI race? We need another dimension because that your dimension is like significance or something. But I think the controllability is a huge question as well. So it's like, is it, what are the examples that people use? Like, is it a nuclear weapon or is it like dragons from the House of Dragons? Casey Newton had a, he had a great platformer post of like, he was watching House of Dragons and he was like, this is what the antibodies are doing. Like they're trying to build these things that they think will help them, but actually it's just going to cause, you know, rain down destruction.
1:03:58Casey told me he was buying an iPhone duo so he could watch more TV. I'm just letting, he's a dear friend, but okay. But I think that's really important because I think the question of, in my mind, the question of how much do we think that, like, if we think there's a technology we can control, then obviously we want to have the best technology. Obviously we want to, you know, foot to the floor, get there as fast as we can. And same for China. And the question is, if we are not sure if we can control it, then what do we do? And maybe we want to be more cautious. And then maybe we want to try and figure out a way to not have China just be, you know, overtaking us.
1:04:33But if we're correct, we should be worried about, well, no, but if we're, if it's dragons, if we're correct that we need to be worried about whether we can control this, then like, that's also something that should be interesting to China. I remember this case you post, but let me just try to pull this apart so I understand it. The thing, the difference between dragons and nuclear weapons is the dragons have some agency, right? They can go away and do bad things on their own. Yes. The dragons might just decide to burn down your whole city. You know, you could think, I mean, and like, obviously these are machines.
1:05:01They're just math. They're not magical. So instead of dragons, you could think of it as nuclear weapons with faulty control settings, where you build nuclear weapons, but it's impossible to be sure that it's not just going to go off and fly to a city and fly to one of your own cities and wipe you out. That's a very different technology than real nuclear weapons that exist. It is true. No one ever said that about 5G. I just want to be clear about that. The reason I'm asking about this is because those problems do align nation states. We do have nuclear nonproliferation agreements. As you mentioned, there's all kinds of incentives for nation states to not do cyber attacks willy-nilly because those have real-world consequences.
1:05:39Is there any evidence that these governments have started to align on these problems, or is it still the pure but-China race dynamic? I actually think there is starting to be some early evidence. I think basically one way I think about kind of what happened in this year in AI from sort of a policy perspective, less than a product or economic perspective. From a policy perspective, we saw these two big demonstrations of the two major concerns people have. So the two major concerns are misuse and misalignment, meaning misuse is like bad people using AI for bad things. And misalignment is kind of AI going rogue or doing things we didn't want.
1:06:15And so this year we had Mythos and we had Hugging Face Attack. And both of those are just really vivid and crisp and real world demonstrations of like what these misuse and misalignment concerns could look like. And I do think that they have, you know, the US and Chinese governments are starting from very different places on AI regulation. They're both starting from where they started on or from what they did with kind of internet regulation. So the U.S. is starting from like hands off, laissez-faire, absolutely nothing, do whatever you want. And China is starting from massively locked down, tons of censorship, great firewall.
1:06:49So those are two very different starting points. But it feels to me, as someone who's not a China expert, but I operate a lot in China circles and have lived in China and so on, it seems like just over the past, even like one to three months, starting from those two very different points, they are seeing more in common around threats from major cyber attacks, but also threats of loss of control. There's really interesting, I'm a language nerd, and it's really interesting tracking the different language that China uses about AI. So they're now talking about loss of control as a thing, but they probably usually don't mean the same thing that Westerners mean, where loss of control is like AI takeover.
1:07:26They probably mean something more prosaic, but then sometimes they talk about extreme loss of control, which maybe is the thing we mean. And anyway, so I do think that there's starting to be some signs of convergence. I think it'll be a long road. And I think there will be like the state of the US-China relationship is such that there's going to be very, very, very little trust on either side. So I am not optimistic that they'll just get to sit down in a room and hash out a deal and everything will be great. But I think there is space for enough. Basically, I think if there's enough shared threat perception, then it might be possible at some point in the future to do something about that.
1:07:58Helen, you've given me so much time. As you can tell, I'm just delighted with this conversation. It's been very fun. We are on the cusp of a new Congress. There's a bunch of new governors. Data center referendums passed overwhelmingly in this country. There's a new set of regulators in town, right? They're going to do some stuff. What are some signs people should look for to indicate we're going to be on the right path? And what are some signs that we're just chasing our tail? Set AI aside. Forget about AI. Congress in general is very bad at passing things right now. Congress for years. Like, this is not a, this administration thing, this is not an AI thing.
1:08:32So one sign is like, can Congress pass anything? That will be a good sign. I think it's also really interesting to look at what state legislatures have been doing in AI. There's been a lot of concern about the possibility of a state patchwork on AI, like different states will pass conflicting things. with the issues that we've been talking about today, which I kind of think of as like frontier AI risks or, you know, the risks from the most advanced systems. There's actually been a pretty coordinated push in different states to like, like California passed this one bill and then New York passed almost the same bill with just like a little more added to it.
1:09:06And then Illinois passed almost the same bill with a little more added to it. I think there's going to be a lot of momentum over the next year or two to keep building on that at the state level. So that is something that I'll be watching. Even just like specifically California is like already looking at how they can strengthen things. So that is something I'll be looking for. I think honestly, you know, with such a democratic Congress, I am also concerned that they will overstep here, that they will try and do something that is too sweeping. You had this huge mess in Colorado where they passed this giant AI bill that then was a terrible idea.
1:09:37So then they had all these committees and like follow up and it was awful. I think a lot of what we need right now is focused on these frontier systems and the possibility that we have really, really advanced systems in a short amount of time and having much more transparency around that, having much clearer standards for how they manage risk. I would not be that excited about an attempt to do kind of a really comprehensive AI regulatory push that covers AI all across the economy, all different kinds of AI systems that are less advanced. So that would be a different kind of, you know, there's negative signals that are like nothing's going to happen and there's negative signals like, oh, maybe too much is going to happen.
1:10:12But it'll be interesting to see where they get to. I suspect we're going to want to have you back as this Congress starts to get to work and we see some of these ideas. I'm eager for your take on it. Thank you so much for being on Decoder. Thanks so much. This was really fun.
1:10:28I'd like to thank Helen Tone for joining me and thank you for checking out this episode of Decoder. To get new episodes every Monday and Thursday, subscribe to our YouTube channel at Decoder Pod and find us on TikTok and Instagram under at Decoder Pod for more fun stuff we put out almost every day. If you enjoyed this episode, please send a link to it to someone you think you might like it. It really helps us grow the show. You can also subscribe wherever you get your podcasts. Decoder is a production of The Verge and part of the Boxing New Podcast Network. The show's producers are Greg Ott, Kate Cox, and Nick Statt.
1:10:53This episode was edited by Xander Adams. Our editorial director is Kevin McShane. The Decoder music is by Breakmaster Cylinder. We'll see you next time. Life can be messy and complicated, especially during the holidays. One thing that isn't, switching to the Simplicity Plan from Verizon. It's one plan, one price. Just$30 a line per month for you or the whole family. Make the simple choice this season. Switch to Verizon and shop their best holiday deals today.
My guest today is Helen Toner, the executive director of CSET. This episode is part two of our series on AI in real life. Helen is a well-known figure in the AI safety and policy community, though she is perhaps best known for having served on the board of OpenAI, where was one of four board members who voted to fire Sam Altman.
Since then, AI safety has entered the mainstream, as industry leaders have made increasingly dire predictions about what might happen if we don’t proceed more carefully or even slow down. Helen has seen up close what happens when safety commitments run headlong into commercial incentives. As she put it to me, if our hopes for the future depend on any one person being trustworthy, we’re in big trouble.
Read the full interview transcript on The Verge.
Links:
The AIs are already out of control | The Ezra Klein Show
AI broke the cybersecurity world. What’s next? | Decoder
Ex-OpenAI board member shares why the AI panic Is good news | Offline
Inside the suddenly explosive world of AI safety | The Verge
OpenAI hack of Hugging Face was “expected” | The Verge
Why Sam Altman was booted from OpenAI, according to new testimony | The Verge
Former OpenAI board member explains why they fired Sam Altman | The Verge
Subscribe to The Verge to access the ad-free version of Decoder!
Credits:
Decoder is a production of The Verge and part of the Vox Media Podcast Network.
Decoder is produced by Greg Ott, Kate Cox, and Nick Statt. The show is edited by Xander Adams and Ursa Wright. Our editorial director is Kevin McShane.
The Decoder music is by Breakmaster Cylinder.
Learn more about your ad choices. Visit podcastchoices.com/adchoices