Is AI running modern warfare?

0:00This BBC podcast is supported by ads outside the UK.

0:30they want to do. To see how ServiceNow puts AI to work for people, visit servicenow.com.

1:00The highest B2B return on ad spend of major ad networks. Spend$250 on your first campaign on LinkedIn ads and get$250 credit for the next one. Just go to linkedin.com slash broadcast. That's linkedin.com slash broadcast. Terms and conditions apply. If you're betting on when somebody is going to die, might make you want to kill them. And that's horrifying. By analyzing the data of our behavior, It's almost like you can peer into people's minds. He actually said, I have no problem with fully autonomous weapons. Hello and welcome to The Interface, the show that decodes how tech is rewiring your week and your world.

1:45I'm Nicky Wolfe. I'm Thomas Germain. And I'm Karen Howe. Today on The Interface, we will be looking at... Who sets the rules for AI and warfare? How exactly is AI being used in combat zones? And is online betting predicting the Pentagon's moves before anyone else?

2:06So guys, slow news week this week. Yeah, it's been quite a weekend, hasn't it? I have been battling a sickness, but I pulled myself out of bed to talk to you guys. Because honestly, this story... The story is so, this is the biggest week of news, I think. Yeah, like I could not miss it. We're doing just two stories this time to try and give them equal weight because, you know, things are a little more momentous and grave than they usually are in terms of the stuff that we're talking about. We should note to our listeners and our viewers that we're recording on a Tuesday. So based on how fast the news is moving when this release is on Thursday, we might not have the latest up to date information.

2:49but we are going to walk you through what we think is two of the biggest stories that have happened in tech in a while. The first major story that we want to talk about is the entire, I don't even know what you want to call it, situation between Anthropic, the Department of War and OpenAI. So the headline here, Anthropic basically had a massive spat with the Department of War and was unable to resolve that spat over how the Pentagon should be using Anthropic's technologies. And in the middle of this spat, OpenAI swoops in and gets the contract instead. And we should say Anthropic is an AI company.

3:38It's competitive to OpenAI. It's a smaller, but it's a big player on the tech scene. And we should also say that Department of War is what the Trump administration changed the name of the Department of Defense to. Right. And it's like not completely official. It's still also called the Department of Defense, but also the Department of War. It's a little hard to keep track of. So basically, if I were to recount the timeline, the tough one, it is a tough one. Back last year in July, the Pentagon awarded four different contracts to Anthropic, OpenAI, Google, and XAI to start using their technologies as part of their operations.

4:17Anthropic ended up being the first company to also be used on classified systems. The Department of War essentially said the reason why Anthropic got it first was because it was the best technology when they were testing it. Then in February of this year, we learn, because news breaks that Claude was used as part of the Pentagon's operation to capture Maduro in its raid in January of Venezuela. And Claude is Anthropics AI. Anthropics AI model. And is it worth saying that we don't know exactly what is meant by used? Like it wasn't flying helicopters in, we just know it was involved in that operation.

5:01Exactly. There was not a lot of reporting in the details of how it was used. But this basically sets off a series of escalations, first behind the scenes between Anthropic and the Pentagon, and then in full public view as news starts breaking, where it appears that Anthropic is not happy with the fact that their technology was used, and the Pentagon is not happy with the fact that Anthropic is not happy. And so when the Pentagon and Anthropic started having these escalating conflicts, they were actually already in the middle of trying to renegotiate terms around how the Pentagon is allowed to use Anthropic's technologies.

5:41At issue was that Anthropic had this clause in their contract that the Pentagon originally agreed to, saying that they had two red lines for the use of their technology. One is their technology cannot be used for mass surveillance of Americans. And the second one is it cannot be used for fully autonomous weapons without any human involvement. The Pentagon refused to agree on these terms. And their justification publicly was that they were disagreeing on principle, that they specifically felt that the military's chain of command needs to remain in the military. It is not the business of a private company that contracts with the Department of War to dictate those terms and that all they would agree to is lawfully abiding by the law.

6:42Like they would they would just use Anthropic for lawful use cases. And we know this, right, because they like Pete Hegseth made this big announcement about how the contract negotiations were breaking down and started threatening that they were going to either declare that Anthropic was a supply chain risk, which would kind of like ban it from the United States in particular ways. Or they were going to use this law that would basically conscript Anthropik and force them to do whatever the government wants as part of the war effort, which not normally how these contract deals go down when we're not in World War II.

7:22This is pretty unusual. Yeah, there's never been an American company that has been given this designation of a national security risk. This is meant to be for companies of foreign adversaries. So this is completely like a nuclear option that Hegseth tried to pull in order to force Anthropik's hand. But this came up because Hegseth actually called Dario Amadei to the Pentagon for a meeting. And that meeting did not go well. And in that meeting, that's what he then lays out. Like, you have a deadline by Friday. So this was Friday last week to tell us by 5.01 p.m. Not really sure why. Oh, one. That like whether you're going to cooperate and and if not, like we will do one of these two things.

8:14We'll either like completely eradicate you from our systems or we will strong arm you into being part of our systems. suddenly, like the day before the deadline, Anthropic issues out this statement saying, we cannot in good conscience accede to the terms that the Department of War is giving us. The next day, the deadline passes. There's clearly no deal. Trump is tweeting like really angry things at Anthropic. Hegseth then declares that Anthropic will officially be designated as national security risk. And then OpenAI announces via a tweet from Sam Altman that they have struck a deal in the same night with the Department of War.

8:58That's on Friday. And then on Saturday, the U.S. government attacks Iran. Right. So like crazy timeline of events here. And we find out, I think on Sunday, the Wall Street Journal reported that anthropic AI was in fact used in this attack on Iran. Hours after it was declared as a national security risk. Yes. But also, Karen, like you, you know, literally wrote the book on this. This goes back to the history of these companies, right? Because Anthropic spins off of OpenAI early on. And their whole thing is they're worried that OpenAI isn't worried about safety enough. And this is kind of, at least on paper, this is like the company's thing.

9:46Anthropic is like, we're the safety company. We're not doing evil stuff. They frame themselves as the good guys, essentially, in the AI industry, right? Yes, that is their brand. They position themselves as always holier than thou when it comes to OpenAI. And definitely, like, when Dario Amadei, who was a former executive at OpenAI, splits off to found Anthropic, one of the key reasons was that he fundamentally did not trust Sam Altman's character. Altman essentially, he's been now trying to defend himself and say, we're trying to do the right thing here and da-da-da. But a lot of people have been very critical of him saying, because they have this history of a beef and they are competitors, that Altman somehow leveraged this to then benefit the company in a situation where that is the last thing that we should be thinking about.

10:44right? So OpenAI also had a bunch of defenses for itself, one of which was quite interesting. They said that their deal with the Department of War was in fact stronger, they believe, than any other deal that any company has ever struck before, including Anthropics. And the reason is because the way that the Department of War used Claude was via another company, Palantir, which we've also mentioned before, is a company founded by Peter Thiel that is a defense contractor. And OpenAI's defense is like, they kind of like never fully did a direct negotiation with the Department of War on contract terms.

11:29It was through this other platform. Whereas OpenAI negotiated like legally binding contract with the Department of War directly. And Altman said that they in fact agreed to let OpenAI maintain these two same red lines as Anthropic. And not only did they have it in the contract, they also are trying to implement it into their safety stack within these systems. So OpenAI's models will refuse any types of requests from the Pentagon that are beyond these red lines. Which the public, by the way, is not buying, right? Right. The other thing that happened straight after is we should talk just briefly about the public response, which is that Claude Anthropics AI model has rocketed to the top, overtaking chat GPT in the app stores.

12:23Because people did not love that OpenAI was like, yeah. I mean, and in fairness, okay, OpenAI has said that their contract with the DoD also specifies no murder bots, right? But it's just not as binding as the contract that Anthropic wanted. And so nobody particularly believes it, right? Here's where it gets interesting. So Anthropic has been publicly stating that they were specifically having a hard stance on both these things, the mass surveillance and the fully autonomous weapons, and both things they were unable to agree on. The New York Times reported something different. So the Times reported that at issue was actually just the mass surveillance, and apparently they didn't actually have a disagreement over autonomous weapons, which is interesting.

13:13amade also like he had an interview with cbs after anthropic was designated as a national security risk and he actually said i have no problem with on principle with fully autonomous weapons the problem that he said was that he didn't feel like this technology like as it currently stands is ready for that. But like, so I made like a bunch of calls this weekend to like various different people within OpenAI and also people that are just like really smart on military and AI stuff. And one of the people that I called was Dr. Heidi Klaff, who's a chief scientist at the AI Now Institute, which disclosure, I'm also a board member of.

13:59But she has been researching AI and military for years. And what she said is, in this interview, Amadei is essentially saying, the technology's not ready yet for fully autonomous open, so we are going to allow for just decision support systems. We're going to allow Claude to be all the way up until the final let's release the bomb. Claude can be used, but then a human has to make that final, final decision. And studies have shown that because there's a lot of automation bias, that's not real human involvement. Like people just see what the bot says. They're under time pressure and they just accept whatever it says.

14:44So if Amade said that he doesn't feel that Claude is safe for fully autonomous weapons, he also should not be saying that it's safe for decision support systems. And so to your point, Nikki, like the public has automatically latched on to the narrative that Anthropic has sold for a long time, that it is safer than OpenAI and it is like the better lab. But it's actually more complicated than that. Like, I think in both cases, OpenAI and Anthropic did not act in accordance with, you know, what we would want of companies that are this powerful. I saw someone on the Internet talking about how this is like the perfect PR win for Anthropic for Anthropic to step in and say, no, like we're on the side of good and we won't let you make killer robots.

15:32And it seems to have been a total, you know, PR coup for them outside of Anthropic's offices in San Francisco. People were like drawing in chalk on the sidewalk saying, God bless Anthropic. You give us courage, like stay strong, like keep us safe. When, like you're saying, Karen, like the actual thing that they're arguing about is a lot more slippery. Yeah, yeah. He's saying it's not ready. It's not ready. Yeah. There's something so fantastic about the phrase, it is not safe to be a killing machine yet. That's incredible to me. The best B2B marketing gets wasted on the wrong people. So when you want to reach the right professionals, use LinkedIn ads.

16:20LinkedIn has grown to a network of over 1 billion professionals, including 130 million decision makers. And that's where it stands apart from other ad buys. You can target your buyers by job title, industry, company, role, seniority, skills, company revenue, so you can stop wasting budget on the wrong audience. It's why LinkedIn ads generates the highest B2B return on ad spend of major ad networks. Spend$250 on your first campaign on LinkedIn ads and get$250 credit for the next one. Just go to linkedin.com slash broadcast. That's linkedin.com slash broadcast. Terms and conditions apply. This message comes from Schwab.

16:56At Schwab, how you invest is your choice, not theirs. That's why when it comes to managing your wealth, Schwab gives you more choices. You can invest and trade on your own. Plus, get advice and more comprehensive wealth solutions to help meet your unique needs. With award-winning service, low costs, and transparent advice. You can manage your wealth your way at Schwab. Visit schwab.com to learn more. Momentum doesn't appear overnight. It's built, refined, repeated. Puerto Rico understands momentum. As companies rethink supply chains and reshore operations, they're choosing a location that already delivers life-saving medicines, advanced manufacturing, and global-scale innovation.

17:39As a U.S. jurisdiction operating under federal law with competitive tax incentives designed for long-term investment, companies don't just relocate here. They scale here. Not culture or business. Culture and business. Puerto Rico. It's not what's next. It's where. Visit investpr.org forward slash business. Part of this whole breakdown here is about where the power lies and who has control. How is this power struggle shaking out? Like, has anything really changed? So this is a super interesting question. And it really gets to the heart of why this whole mess started in the first place. Another person that I called this weekend was Alondra Nelson, who was the former acting director of the White House Office of Science and Technology Policy under Biden.

18:36And what she told me is like what they were fighting over was who ultimately gets to make these high stakes decisions in the military, irrespective of what tool is being used. And the government actually reveals how disempowered they are in this instance when it comes to private companies that they've allowed to become essential to the national security stack. And these AI models kind of have an unprecedented characteristic about them, which is that they can be continuously updated by the company that provides them. And so traditionally, like when the Pentagon buys something like Microsoft Office, this is a static piece of software.

19:17But when they're purchasing Claude or when they're purchasing ChatGPT, that is not the case. And we don't know the fine-grained details exactly of whether or not OpenAI and Anthropic would update their models continuously the same way, like for the Pentagon, the same way that they would for consumers. But that is part of the crux of why I think the Pentagon is freaking out, because the military has realized that the control over their technologies is slipping. And ultimately, like from the military's perspective, they need that control in order to protect Americans. Okay, so I reached out to a few national security sources over the weekend to ask a little bit more details on what exactly we know about how AI gets used in war.

20:08Mostly, we are talking about data analysis. This is target triage. Target triage is like, okay, we've got a whole bunch of potential things we could hit. Tell us which one we should point the missile at. Right. There is one historical example of autonomous weapon systems. It's unclear if this uses a kind of modern style AI model. It was in 2020 in the Libyan civil war. There was a system called the STM Kargu 2, which is a Turkish made drone explosive, like a suicide drone system. They don't need to be linked to a commander control human system. They describe them as fire, forget and find capability.

20:53That's based on two paragraphs from a UN report on the Libyan civil war. Almost nothing else is known about how it was deployed and what kind of targets it found. Most of the time, my source said that LLMs are database analysis. It is part of what is called the kill chain, which is the chain of command from high up at the Pentagon to the trigger being fired. Parts of that are about picking targets. Parts of that are about identifying where targets are. It does not seem like in this war in Iran so far that a weapon system that is fully automated has been deployed. Nikki, did your sources say how this is connected to LLMs?

21:43Like, is the idea that the LLM is the one that identifies the target and then in a fully autonomous way, it could be integrated to a killer drone to then like hit the target that the LLM identified? Because these are, you know, like the killer drone and the LLM are actually different types of AI, right? Totally. Yeah. And we should say for my Aunt Eleanor listening in Indiana, who maybe isn't as obsessed with AIs, we are LLM, large language model. It's the technology behind tools like ChatGPT. Yeah. Yes. You would feed the LLM model, say, for example, a lot of high definition satellite data and say, rather than having a team of analysts poring over that, saying that has the markings of a disguised oil fuel dump, you would give it a set of parameters and it would bleep, bleep, bleep, bleep, bleep, bleep.

22:37These on the maps are the targets. And the tech is largely not there yet for it to be replacing soldiers in the field. That's not what we're talking about at the moment. And Karen, this is your particular area of expertise, right? Part of what people are so concerned about here is that a tool like Claude, these like AI chatbot models, they make mistakes all the time, right? Yes, this is such an important point. There's this guy named Paul Sher. He's the executive vice president of the Center for a New American Security, CNAS. And this is he's written extensively about military and AI. And he also served in the military.

23:17He had this really good piece a few years back in Foreign Affairs where he said that there is this narrative that there's an AI arms race. And in this narrative, the military that acquires AI and advances it fastest is going to win. But what that doesn't account for is that they're also actually inheriting an extraordinary amount of risk when they acquire these technologies because of the fact that large language models hallucinate and they're inaccurate and they do all these things. And so he had this quote that really struck out to me. AI technology poses risks not just to those who lose the race, but also to those who win it.

24:03And Karen, on top of the concerns you're raising here about the accuracy of these tools and whether they make mistakes, but I think part of what people are uncomfortable isn't just that are we sure that AI is up to the task? Even if it is, there's this question of accountability, right? If a human being makes a big mistake, we can hold them accountable. They can lose their job. We could put them in front of a military tribunal. We could ask them what went wrong. If an AI makes a mistake, whether it's targeting a missile or like driving an autonomous car, no one really is responsible. It's like, oh, well, the tool messed up.

24:41Blame the tool. The tool can't get in trouble. And we haven't as a society figured out what we are and aren't comfortable with these like thinking machines doing in our world. 100%. And we have to remember, this was actually not the only issue that came up with the contracts, right? Like autonomous weapons was only half of the issue. There was also the problem of mass surveillance. Yeah, the thing much more than murder bots that my sources are worried about is how this advances dragnet surveillance capabilities. What this would allow them to do is sift through what is fundamentally a near infinite amount of data, right?

25:24We know that if the US wants to get into your phone messages, your social media messages, they fundamentally can. They're not the only country who can, but we know that they are using this capability. If you set an LLM on a bulk collection of people's messages, it will be able to find anything it wants much, much faster, much more effectively than a human set of analysts can. And it is able to spot patterns in a way that a human analyst would maybe not know to look for. One of my sources went as far as to say that this kind of ends personal privacy. Yeah, Nikki, this question of patterns you're talking about here, I think this really is the crux.

26:14By analyzing the data of our behavior, you can figure things out that we're not saying, that we're not speaking out loud. stuff that in some cases we've never told another human being human like you feel everybody feels special and different and we are but human beings are also very predictable and when you've got information on hundreds of millions or billions of people you can identify trends and behavior and patterns and figure things out about people that they've never revealed in any open way like It's almost like you can peer into people's minds using the power of mathematics. And we're just at the beginning of what is possible here with this new set of tools.

27:02Right. And the problem with that is humans are not statistics, right? Right. If the statistical model decides that you are something, we're looking towards a future where that defines you. And that's horrifying. We have no idea what the U.S. military or the government writ large or governments all over the world are doing with these tools. And I think that's, in fact, part of this fight that we saw playing out between the Department of Defense and Anthropic is like even they aren't sure what they're going to be able to do with this stuff. But they want to have, you know, open range to take it in whatever direction that they want.

27:49And for the public, probably what's going to happen is once this has already been rolled out, once it's already being used, then post hoc, we're going to be talking about whether we're OK with these things that are already happening. That's such an important point, Tom, because one of the things that Alondra Nelson told me is when it comes to what people can do, like the average person in the public right now can do in this moment, it's to demand that these decisions be made democratic again. It used to be that when the Pentagon acquired a technology, there would be a public comment period. And now we're in a state where this has been completely shuttered off from the public.

28:31And we as the American public, but also the global public at large, should be demanding better. So switching gears slightly, but I think this is all kind of tied together in a bunch of weird, interesting ways. As this was happening last week during Trump's State of the Union address, he raised the possibility that the U.S. government was about to attack Iran. And on Saturday, that's exactly what happened. And on this weird part of the Internet, this became a huge issue in the world of online gambling. So people have probably heard by now about these companies, Polymarket and Kalshi, right? They call them online prediction markets.

29:15And essentially, these are platforms where you can bet on anything you can imagine. and when the u.s attacked iran 529 million dollars were wrapped up in bets on the timing of this strike on a foreign country like this huge military action hundreds of millions of dollars changing hands and i've been dying i know nikki you've been following this i've been dying to ask you yeah the thing that polymarket and calci that differentiates them from quote unquote gambling is that there's no house you pay fee and obviously use the platform but you can lay a bet on anything and if somebody takes the other side of that bet it is considered that that's a private wager between two people and then the way that becomes a prediction market is they're suggesting that the wisdom of crowds means that that will predict say the winners of elections now when you're betting on as people were things like the removal of world leaders you wind up in some pretty uncomfortable territory pretty quickly if you're betting on when somebody is going to die that might make you want to kill them right it doesn't take a huge leap of the imagination and there's also uh betting on almost anything right i mean tom you were the subject of some of these bets recently.