The AI Kept Choosing War

0:00Reid:Reid, it is lovely to see you today. First question, I want to talk about a fascinating academic paper out of King's College London that came out last week, and it's looking at how frontier AI models behave in simulated nuclear crises. We often compare the AI age, the nuclear age, although there are many differences. And in this case, researchers ran 21 war game scenarios involving territorial disputes, regime survival threats, Cold War style standoffs. And they had models like GPT 5.2, Cloud Sonnet 4, Gemini 3. They played against each other as decision makers. And what they saw was across over 300 turns of play and nearly 800 ,000 words of reasoning, The models chose, unfortunately, nuclear escalation almost every time.

0:50Reid:Tactical nuclear weapons were deployed in 95 % of situations. They had full strategic nuclear war, happened 76 % of the time. And what's maybe more surprising is that the models weren't behaving randomly. They generated long chains of reasoning, explaining their choices, invoking classic deterrence logic like rationality of irrationality. I think the most important thing, perhaps, is that none of these models ever chose accommodation or surrender in any scenario. And so recently in the news, we've seen the recent interaction between OpenAI and Anthropic and the DoD. And so clearly using AI in times of war and on battlefields is top of mind.

1:30Reid:What do you think this says about how AI might impact nation-level conflict?

1:36Aria:It's a great question. And the depth of it actually illustrates a bunch of different issues to go into. And I will touch lightly as opposed to deeply on each one of them. But there's a bunch of really important issues. The first is, you know, to some degree, when we do all this kind of academic or other kind of theory and game theorizing world, it's to play to trying to highlight where we may need to add into our human responses. And that's what generates all of the analysis, text, political science, et cetera, which the AI agents essentially train on. You know, to some degree, well played in this very old movie, which I recommend to people, War Games.

2:15Reid:So fantastic. Even I've seen it. Yes. And so it's kind of like, look, it's actually trained on kind of pure rationality of a whole bunch of kind of human text.

2:26Aria:But the problem is, of course, those human texts are trying to balance out against a set of human biases where those human biases actually, in fact, have some very important plays in how actually, in fact, we need to have a compassionate, a humanist, a humanist future. And so, you know, in this classic punchline, spoiler for those people who haven't seen it, is the human character gets the AI to play a bunch of no-win scenario games to realize that there's no-win scenarios. And it isn't actually best played out by nuclear escalation in this particular case. Part of that is like where is the value of the human in the loop in these circumstances?

3:14Aria:And what a lot of people don't realize is we actually had a number of different circumstances where we were very close to nuclear war where a human made a decision of saying, oh, actually, in fact, I don't believe what I'm seeing in my sensors because I don't think that people would be that crazy and stupid. There was a Russian colonel who did that. There's a stack of things that it's actually important to realize in these circumstances where it's a, hey, what's the actual context of the circumstance? What's actually happening? What might be the irrational but, frankly, irrational but smart thing to do in terms of this and is to de-escalate and downgrade?

4:03Aria:Actually, I was just at a kind of a gaming scenario thing when 9-11 was happening. I was listening to Condoleezza Rice talk about this. But when 9-11 was happening, it was like one of the things that she had learned from some earlier war game scenarios that she had gotten prioritized on her calendar. She was like, I don't need to do this kind of war game scenario. But when we were going to higher alert because of 9-11, one of the things she says is we've got to call the Russians and we've got to tell them that we're going to higher alert. And it's not because of you. We don't need to have this escalating thing.

4:40Aria:And the Russians said, yes, we understand this because we've done the same thing too. We understand you're going to higher alert and we are going to a lower alert right now because we understand exactly what's happening here. And we don't want to create World War III, right, just out of this mechanics of higher alert, higher alert, higher alert. So you get to it's like, well, but the, you know, the AI people would say, hey, we can train this to not do this. And it's like, well, you can and you can't in this context alert. You can in some ways for doing this, but it's this kind of contextual awareness of this circumstance.

5:16Aria:And then, of course, the fact that you might say that winning the game in this case is de-escalating because even if we're surrendering or even if we're doing something else, it's what's the game that we're playing? And the game that we're playing is actually, in fact, survival of the human race. And actually, in fact, navigating that is really important. So anyway, so that's my first short answer to your first short point on your game. The other ones I think are people paying attention to the details of actually how Anthropic was responding to the Department of War was not saying we will never help in any of these circumstances.

6:01Aria:It was saying in our judgment about where the capabilities of these models are right now, and we are experts in the development of the capabilities models. There were two different points. There was one surveillance, mass surveillance on U.S. citizens, which is we as a private company are not going to provide capabilities for this. And, you know, just full stop and we're a private company. We're a country of freedom and liberties. And we're saying that's kind of a freedom and liberty we have. And the second one is to say on the making autonomous lethal decisions, we don't think that they're there yet.

6:42Aria:for that technology. We think that provisioning it is actually, in fact, bad, and we are the experts on this stuff. Now, to some degree, Hegseth responded in a way that is validating of the anthropic detailed point. They weren't saying we're not patriotic. They're not saying that we were not supporting our country in wars and in times of war in order to do this. They're saying we as the experts are doing this. And it's like, well, Hegseth is going, no, I'm the big dog here. You do what I tell you to do. And it's like, no, no, that's actually not the name. You're failing the Rorschach test in what the conversation is, which is the conversation is, is the technology actually, in fact, ready yet?

7:27Aria:It's Hegseth as being the stupid war games thing. It's like, no, I'm going to escalate. I'm going to nuke them first. And you're like, oh my god that's super scary because the actually the anthropic people here are actually being highly reasonable in what they're doing and what hegseth at all is trying to position them as being anti-patriotic they're saying we have a better decisioning than the than the department of war etc etc and actually you know i think if you go to rank and file within the department of war, I think you'll find them going, no, no, we understand what's being said here, which is the technology is not ready, right?

8:07Aria:And that that's actually, in fact, the point of what's going on. And it makes Hegs Hess himself that crazy AI computer that escalates into bad scenarios where it makes the decisioning framework within the Department of War much smarter. And so I think it's important for those of us who are smart people to say, no, this is not a politicization of red versus blue, of patriotism versus not, because actually, if you look at the details of what Anthropic is doing, it's actually saying, we're actually, in fact, talking about when the technology is ready. We are, in fact, believers in a democracy, believer in elected office.

8:55Aria:Hegseth wasn't elected, right, in terms of this. And being smart about what this means for impact on, you know, what can be like ending of human lives. And we think that technology is not ready for that in a independent theater point of view as experts in this technology. And so I thought that was an important part of that discussion that was badly covered in the general discourse.

9:30Reid:Well, we're all used to founders and CEOs like boasting about their technology and sort of saying that it can do more than it actually can, you know, saying, oh, no, no, it's ready. It's ready. And so when a founder CEO says, oh, no, my technology is not ready, we should probably trust them. They probably know the best. If they're the ones saying it's not ready yet, we should pause. You also mentioned Condoleezza Rice. And for those of you who hadn't listened, she was on Possible last year, a phenomenal episode. She understands, you know, geopolitics and war perhaps better than most.

10:03Aria:Better than the vast majority, not most, the vast majority.

10:07Reid:Better than everyone but five, or maybe she's the best.

10:10Aria:Yes, she's in the set of the best.

10:12Reid:Yeah. And so one of the things she said, sort of similar to what you were saying, is that the personal relationships mattered. When she got on the phone with someone, it's because she had a relationship with them that sort of this soft diplomacy mattered. It wasn't always, you know, puffing up your chest and saying, I have nuclear weapons. It was, okay, let's talk this through. So thinking about these AI models that continue to escalate or lead to nuclear war, do you think that's a problem in the models themselves? Or are they just reflecting back that perhaps the nuclear deterrence of the last 30 years was actually much more sort of aggressive and bellicose than we might have thought?

10:50Aria:There's two different problems in the current models themselves. One is the models are reflecting only the set of things that we have generated as kind of textual analysis and reasoning that we are using to modify our own behavior. But they're not including us and our behavior in that. It's like sometimes you need to not be blinded by like a human fear response or anger response. And you need to be, you know, only doing this. And that's what this stuff is doing because it's the body, the corpus of the material is meant to be training human beings. And it's one of these places where you go, well, actually, in fact, you know, these models are not human beings.

11:38Aria:The fact that they're embodying this kind of how we train human beings suggests that this is the kind of area where you'd want the humans to still very much be in the loop. And it's a very precise point for doing it. Now the question is to say, well, could you train AIs in ways that are not only kind of human in the loop? And the answer is maybe, but it's the nature of the technology is something you'd have to look at. This is one of the things that I've kind of learned from being in a number of conversations, you know, at the Vatican about, you know, what is AI since the, you know, Pope Francis and I briefed Pope Francis on AI.

12:17Aria:And one of the reasons why it's very useful for us as technologists to be in those conversations was like, well, you know, and we were talking about actually paroling and kind of use of the justice system. But it's like there's a role for mercy in human systems. Are you training your models to have a good sense of mercy in them? And the, you know, I think we normally kind of think of efficiency. We think of truth. We think of, you know, weighing all the evidence and all this kind of thing, which is, you know, extremely important parts of this. But you go, well, actually, in fact, would you want to have, call it criminal judgment, without some notion of mercy in it?

13:05Aria:You'd go, well, shit, not really. And can we train it? That's an interesting question, et cetera. And so I think that's another part that this highlights. It's in is, I think, really important for this stuff because it's not that I believe that there's never a place for war or violence or conflict in it. It's not a complete pacifism. But the notion of the fact that you want to be driven by a North Star of we're trying to reduce human suffering. We're not trying to prove that, you know, I am the most alpha or I am the most biggest man, et cetera, et cetera. And that the notion of paying attention to, you know, reducing human suffering across the board is an important thing.

13:59Aria:And reducing human violence across the board is a really good thing. And that you're making decisions that way, not by being, you know, unwilling to step up in the wrong circumstances, because sometimes you have people, you know, like I think the Ayatollah community has been a source of violence and terrorism, you know, in a very bad way across the world and including in the Middle East, as I think a important thing. but you also need to be paying attention to, you know, like, well, you know, that whenever you engage in this, you're causing a lot of people to die and suffer in, you know, like what's the way that you minimize the amount of deaths, period, not, hey, it's only all of those people who we don't care about deaths.

14:52Aria:And how do you navigate that is, I think, is very important and where that would play in to how you're building AI models, what the notion of compassion is, what the notion of mercy is, what the notion of minimizing deaths and suffering overall is, is important. And, you know, like that's where you'd want all of this dialogue to be, as opposed to like the, you know, I am the big man here. You do what I tell you to do.

15:22Reid:Yeah. I mean, it's just So like whether we are patriotic Americans and we want the best for our country, to your point, this this mercy and compassion, we certainly want it in geopolitics and international relations. But as we are training this A.I., like that has to be included. We can't just build sort of smart machines. It must be something that takes into consideration those things as well. So let's let's head to a slightly lighter note than the nuclear war and bombing. Last week, a company posted a job listing for something called an agentic AI developer advocate. And this quote unquote employee isn't necessarily a person.

16:03Reid:The company is looking for someone to build an AI agent trained on certain workflows. They can create content, run growth experiments, and provide product feedback. The company will pay$10 ,000 a month to the developer who maintains that agent. And so in other words, instead of hiring a person for a role, they're effectively hiring a piece of software that someone built and is maintaining. And there are humans in the loop. They want humans to check all of the content that's going out the door. And so my question for you is, is this just a stunt or are we actually seeing the early version of what a new labor market will look like?

16:42Reid:Because we're all trying to predict that labor market. And is it this where people build agents that work for them and are hired out to companies?

16:49Aria:Well, it's definitely a clever stunt. So credit to the stuntiness. But it partially is a clever stunt, not just because it deals with a bunch of current human fears and concerns and everything else, and so therefore gets a whole bunch of attention because of it, but also because it's clearly versions of, call it, some new labor market. The notion of having kind of AI agents doing work is already present. But I think it's a, like, this is a lens. Part of the reason why the question is good is a lens to how certain parts of work and labor are moving towards. Now, part of it is I think that we as a society and we as industries and we as companies and we as technologists need to be kind of saying, hey, look, it is a natural part of building productivity.

17:48Aria:Building productivity is part of how we create, you know, progress in society and part of how we make that happen. But we do, of course, need to be paying, you know, a lot of good attention to, you know, how do we have human amplification? It's part of the reason why I did super agency. Super agency wasn't just for, hey, look, be a little bit more positively minded, bloomer versus gloomer, bloomer versus doomer, you know, as ways of doing this. But it's also to say to the zoomer technologist is to say, hey, pay some attention to being bloomers as well. and to be thinking about how do you build this stuff in relevant ways.

18:30Aria:Because it's not to say make your technology less useful, less economically impactful, less, you know, like never do replacement. Of course, sometimes you do replacement. We replaced a bunch of groomsmen people who are doing horsework when we started building cars, that kind of thing. But it's like the, but also be paying attention to a lot of amplification. in terms of what we can do. Now, that also gets to, you know, kind of questions around, you know, society and industry. And because, you know, say, well, even if we're building mass amounts of productivity amplification, what are the ways that we can be helping human beings economically, but also in finding meaning in the stuff that they're doing?

19:16Aria:Now, part of the difference that I have from some technologists who are like, oh, my God, This is going to be here in a couple of years and so forth. I actually think that there is actually, in fact, a whole bunch of pacing, human organization things. One, that's not necessarily the universe. And two, it's not like AI robotic factory is going to be building other robots any time really soon. And there's a lot of human in the loop and human combination. And we should be building AI agents to help human beings in those circumstances and help navigate. And it's also one of the reasons why for, you know, gosh, I've now lost track of the number of years that I've been trying to advocate for the creation of key human assistance agents running there.

20:09Aria:So even as we have job transformations coming, like everyone can see some of the benefit of AI. Like if you have a medical assistant, an educational assistant, a legal assistant, and they're all there, and all of a sudden you're like, hey, I'm a working class person, but I get benefit. And my children get benefit from the medical assistant, from the educational assistant, from the legal assistant. And so even though jobs are changing, and this, by the way, a lot of white-collar jobs are going to be changing, I'm even seeing benefit even though most people do not like to see job transitions. They don't like to see – like the people who are doing all the horses did not like to see cars coming.

20:49Aria:Absolutely. The people who are doing all of the hand loom did not like to see the power loom. But by the way, the embracing of those societies is part of what makes you have an economically prosperous society, part of what makes sure that your children are part of a and your grandchildren are part of a economically prosperous society. And so those job transitions are hard and not easy. And you say, well, let's at least slow them down. It's like, well, slowing them down means that other people, this is part of the reason why in super agencies said, look, England did not invent the Industrial Revolution.

21:25Aria:They just embraced it more thoroughly, which is the reason why they had a centuries-long empire, whereas France, which had kind of four times the population, or China, which had 10 times the population, did not do this. And it's one of the reasons why it's important to say, hey, in the economic prosperity that's coming from having AI agents doing work, we want to be embracing that as a society because it's actually, in fact, really important relative to the future prosperity of our society. And he's like, well, but there's all these, you know, kind of wealth division issues, like how does it divide between us and those people, between these companies and those companies, and, you know, not just big tech companies.

22:10Aria:But the short answer is you want to have the companies that are generating that wealth because then you can then address the distribution issues. If you try to mess up the fact that you're leading in it, then you don't have anything to distribute. Right. And so it's important to be saying, hey, be foot forward on the economic things. And then, of course, be also addressing distributional issues.

22:36Reid:Well, when you're talking about this job transition and sort of thinking about units of work, you already have people saying like wage work is for suckers, salary is for suckers. You know, the thing that we want to do is to put our capital to work for us. And in the future, you know, is it going to be that the economic shift is going to be from a unit of a person's time to a person's software? Like already we have someone who is working at CVS or Starbucks, like they put in an hour of work to get their paycheck. There's nothing they can do to get more money except that hour. But we have other people who are sort of augmenting or they're having AI help them out.

23:15Reid:And so in the future, is that just going to get even greater where you have the people, you know, you have the nanny that's coming to watch the kid and she's being paid hourly or he's being paid hourly. But then the person who's creating software or AI agents has sort of a multiplication effect that the other folks don't have.

23:30Aria:Yes, we are on a trend that direction. We've already been on a trend direction. It's actually part of the thing that kind of led me to essentially the path that I was on is both my parents are lawyers, both kind of highly paid and I was like, well, actually, in fact, you want to choose career paths and work that's not, you know, kind of by the hour. Like it's how do you, you could put this in various colorful ways. How do you make money when you're sleeping? Because you, you know, have capital, loan assets, et cetera, and done things. There is a question about there's a limit to how much you can charge per hour when you're kind of growing.