How AI safety took a backseat to military money

0:01Hey there, and welcome to Decoder. I'm Hayden Field, Senior AI Reporter at The Verge, and your Thursday episode guest host. I have another couple of shows for you while Neelai is out on parental leave, and we're going to be spending more time diving into some of the unforeseen consequences of the generative AI boom. Today, I'm talking with Heidi Klaff, who is chief AI scientist at the AI Now Institute and one of the industry's leading experts in the safety of AI within autonomous weapons systems. Heidi has actually worked with OpenAI in the past. From late 2020 to mid-2021, she was a senior system safety engineer for the company during a critical time when it was developing safety and risk assessment frameworks for the company's Codex coding tool.

0:44But now, the same companies that have in the past seemed to champion safety and ethics in their mission statements are now actively selling and developing new technology for military applications. In 2024, OpenAI removed a ban on military and warfare use cases from its terms of service. Since then, the company has signed a deal with autonomous weapons maker Anderil, and this past June, signed a$200 million Department of Defense contract. And OpenAI isn't alone. Anthropic, which has a reputation as one of the most safety-oriented AI labs, has partnered with Palantir to allow its models to be used for U.S.

1:21defense and intelligence purposes. and it also landed its own$200 million DOD contract. And big tech players like Amazon, Google, and Microsoft, who have long worked with the government, are now pushing AI products for defense and intelligence despite growing outcry from critics and employee activist groups. So I wanted to have Heidi on the show to walk me through this major shift in the AI industry, what's motivating it, and why she thinks some of the leading AI companies are being far too cavalier about deploying generative AI in high-risk scenarios. I also wanted to know what this push to deploy military-grade AI means for bad actors who might want to use AI systems to develop chemical, biological, radiological, and nuclear weapons, a risk the AI companies themselves say they're increasingly worried about.

2:08Okay, here's Heidi Klaff on AI in the military. Here we go.

2:27Heidi Klaff, Chief AI Scientist at the AI Now Institute. Welcome to Decoder. Thank you for having me. First up, I wanted to talk about how AI companies have moved their goalposts a lot with regard to what they're okay with and what their mission statements allow regarding work with the U.S. military and other militaries. So do you remember the whole controversy over Google removing the phrase don't be evil from its code of conduct? Yeah, absolutely. It reminds me a bit of something more recent, which is how OpenAI and Anthropic both used to have certain bans on military use of their products, and then they relaxed them.

3:02OpenAI walked its ban back in January 2024 when it began to work with the DoD on AI tools. And the month before that, it partnered with Anduril. And for Anthropic, it partnered with Palantir in 2024 to offer Claude to intelligence and defense agencies with the U.S. government. So I wanted to ask what you were thinking when you saw these announcements made month after month. What did you make of that kind of parade of changes that were happening over a couple of months? Well, to many, including myself, the timing didn't seem like it was a pure coincidence of when OpenAI removed the span in January 2024.

3:37If you consider, for example, that Israel at that time was ramping up its mass targeting campaign in Gaza that we now know is being supported by Microsoft's cloud services that offer AI, and in this case, open AI models, as extensions of their IT and cloud infrastructure. And so this rollback was really a signifier on where AI companies were heading, what they were interested in deploying their technologies, despite OpenAI being well aware of the risks that their models posed in defense and safety critical settings, which is something that I actually worked on with them in one of our papers together when we were looking at evaluation of models like Codex.

4:17And interestingly enough, with the announcement of these collaborations that you mentioned, and that includes Meta, Anthropic, and OpenAI, all announcing U.S. national security work that aligns them with defense contractors like Palantir, Angel, and Lockheed, these AI companies never addressed their previous statements on their stance that LLMs or foundation models are unsafe and insufficient for defense use. So it was almost like a clean slate was being created where they behaved as if this was always aligned with their mission, right? For example, they started making claims that U.S. national security is synonymous with safety under this pretense of an AI arms race with China.

5:02And this push for this AI adoption seems quite convenient when you're considering the current unprofitable reality of AI and how expensive it is. where it seems like they're trying to sort of de-risk their portfolio through government subsidies and military contracts. So now we have this complete pivot from banning military uses, all this talking about their main mission being building systems that benefit all humanity, to now their reliance on this narrative of a U.S.-China AI arms race to drive policy initiatives that not only boost the use of AI, but allows them to sort of avoid safety and security scrutiny within military applications.

5:44That makes total sense. And that reminds me of how, you know, I think last week, Senator Warren sent a letter about XAI's own contract with the DOD and expressing concern that the company hadn't done the same level of safety audits as other companies before receiving that type of contract, that they weren't ready. She was worried about how the company would use data it had access to as part of, you know, its government partnerships. What did you make of that letter and kind of XAI's, I guess, it seemed grandfathered in kind of approach to this DoD contract announcement? I think the way that I see it is it's part of the trend of the U.S.

6:29DoD not recognizing that there's a national security risk with the use of commercial foundation models in the military, because they do significantly expand the attack vectors of military systems and defense infrastructures that they interface with, because commercial models are unvetted. They don't have a supply chain that follows the typical military supply chain, and they can be compromised in a lot of ways. So to me, sort of the contract with XAI is another risk that's added onto that, that sort of stems from the issues that all commercial models have, right? And obviously, depending on sort of the platform that these models are trained on and the personal data that these models are trained on, they come with a lot of risks and capabilities that allows them to promote or even deploy surveillance systems because they're able to use data that other companies may not have.

7:28For example, XAI has a huge amount of data, you know, could be from not just public posts, but also private messages of their users. And what does that mean for that to be used in military applications, right? There's a legitimate concern here, but there's also this larger concern that these systems are also unsafe and train on data that can have been compromised by an adversary, including China, right? And that sways the way that the AI system behaves. So there's like risks on both sides here, right? From both the data that is meant to be protected, right? And also the type of data that could have been compromised and then thus change the behavior of an AI system that's being used in something like very sensitive military operations.

8:18So something that comes up a lot in my reporting and in my conversations with other people, even my pitching my editor, is that a lot of these companies are pre-profit. I mean, maybe all of them. And we're seeing a bunch of them unveil government products, enterprise products, and that seems to be where a lot of the money lies. So, you know, obviously, OpenAI, Anthropic, and XAI have all unveiled government products designed for U.S. defense and intelligence agencies to use. They also all received, you know, those government contracts from the DOD. So with companies that burn through cash at super high rates, do we think that the government play is about seeing cold, hard cash come in finally or staying above regulatory pressure or both?

9:01I'd love to hear your thoughts on that. It's definitely both, right? Because as you mentioned, these companies are pre-profit and there's a really big pot of money in the military-industrial complex. There simply is, and I think that's well known. But also, there's the aspect that these companies would not traditionally pass any of the testing and evaluation required for military procurement. And here's the thing that a lot of people don't know, is that defense and military procurement is actually some of the most strict, you know, prior to this AI era, I mean, to sort of evaluate these systems.

9:40They have some of the most strict standards. And I think a lot of people assume that's not the case, but often our safety critical systems, if you're talking about like energy infrastructure, you know, and so on and so forth, is derived from those defense standards because of how robust they are. The thing with AI systems, and we're talking about generative AI systems because AI systems have been used in the military for decades at this point, but these foundation models or large language models, whatever it is you want to call them, they do not meet the sort of very basic threshold that is typically expected for a military system, right?

10:19And so there is this kind of issue now that they want this pot of money. As I mentioned before, they're trying to de-risk their portfolios through military contracts, but they have this issue where safety, as defined by defense and safety critical system is too stringent for their systems to meet just by their nature, right? They're highly inaccurate systems. And when you're looking at, I can't really get into defense systems, but I'm going to talk a little bit about like safety critical system. If you're looking at like a nuclear power plant, for example, you're looking at safety of like 99%, right?

10:55That's like the minimum. And often the accuracy of AI systems is like 60 % if I'm being optimistic about specific types, right? So there's an enormous gap here to make AI systems as they exist for foundation models be able to satisfy the strict testing and evaluation measures often required by military procurement. For the listeners, let's just define military procurement really quick. What is that? Military procurement is, well, it really depends, right? There's a huge amount of processes that exists for different types. And the process is often strict depending on how critical the technology is going to be used.

11:39Like, for example, if it's going to be used for lethal operation versus bureaucratic operations, very different types of procurement. If we are looking at a sort of more general idea, typically the government puts out a specific ask for the type of systems that they're looking for and people submit to that, you know, procurement ask. Ultimately, these systems often have to go through what we call a testing and evaluation process for them to even be considered, you know, even before they sign the contract, right? This process is quite stringent in that it has their specific thresholds on how accurate these systems need to be and how secure they need to be.

12:18Often the security thresholds, it's extremely, extremely high, right? They have to be air-gapped. The supply chain has to be completely traceable. They have to know who coded the system, who developed the system, if there's any sort of backdoors that can be compromised, so on and so forth. And once sort of a system goes through that procurement, that safety and security procurement process, the DOD often just takes complete control of that technology, right? Like this is now something that they possess and are in complete control of using in whichever way that they see fit. right? And so this is often why procurement for the military takes can even take many years, right?

12:57This is not a process that is meant to take a couple of weeks or even several months. Often this is quite a rigorous process. So this is very different from signing a commercial contract, right? Where you have traditional terms of service, the nation state, not just, the US COD often are the ones that get to define the terms of how they want to use technology. and typically people abide by it because people want that pot of money, right? It's very lucrative to even be considered up for procurement because it means that you're sort of be on call for them for potential other technologies. But even to get your foot on the door can take years.

13:33So it's a very different type of assessments than I think what people expect for commercial contracts. We need to take a quick break. We'll be right back.

13:53Support for this show comes from LinkedIn. Imagine if any of the movies that included the line, I need the right person for the job, settled for, I'll just take about anyone. How many heists would have failed? How many deals would have fallen through? How many secret spy missions would have ended in disaster? So why would you accept just anyone when hiring for your business? When you need the right person for the job, you can turn to LinkedIn Jobs. And now LinkedIn Jobs is stepping things up with their new AI assistant. So you can feel confident you're finding top talent that you can't find anywhere else.

14:27With LinkedIn Jobs AI assistant, you can skip the confusing steps and recruiting jargon. It filters through applicants based on criteria you've set for your role and surfaces only the best matches so you're not stuck sorting through a mountain of resumes. Hire right the first time. Post your job for free at linkedin.com slash partner, then promote it to use LinkedIn Jobs' new AI system, making it easier and faster to find top candidates. That's linkedin.com slash partner to post your job for free. Terms and conditions apply.

15:10We're back with Heidi Klaff of the AI Now Institute. Before the break, Heidi was breaking down the standard military procurement process and why it feels like AI systems don't meet the rigorous standards we might expect when it comes to being used for high-risk operations. Now, I want to ask Heidi about the specific AI products being sold to the U.S. military and whether they're really much more secure than the commercial models on the market today. I also wanted to ask the models these companies use for government products, like their government-designed products, like Claude, Gov, OpenAI's government product, XAI's government product, by design have looser guardrails for government use, and they're trained to better analyze classified information.

15:54And although these types of models allegedly underwent the same type of safety testing as these companies' other models, I'm using Anthropic here as an example, they have certain specifications for national security work, Like they have a greater understanding of intelligence and defense documents, and they refuse less when they are asked to engage with classified information that's being fed into them. So in your eyes, how does the development work? How secure really are they? And what are the implications here? So I wouldn't say they're much more secure. They may be more secure in that they're more air-gapped.

16:31So, for example, you can take a commercial model, right, and you can fine-tune it on sort of sensitive military data. And then that model then becomes accessible to the military. But that still misses out on some of the biggest risks of that commercial model is that it was trained on data sets that were publicly available. And so a lot of research has shown that not only can you poison web data that these models are trained on, but you can implement what's called like a sleeper agent, which is given a specific prompt or a command, it will then behave in a sort of a harmful way that sort of the operator of that system did not intend based on something that was implemented in the training data or something that the model was trained on.

17:18And so we see this all the time with like prompt injections, but this can happen on a sort of deeper level with what we call sort of web poisoning attacks, which can then be used to implement these like sleeper agents, as we call them. And so this is in the commercial supply chain, right? The only way that these models are trained is to be trained on sort of mass amounts of data that are publicly available. So they're already compromised, right? These models are also fine-tuned through methods like reinforcement learning human feedback, which unfortunately uses basically sweatshops of people in developing nations that are paid nothing to then make these models behave in a specific way.

18:00And you can imagine a military operation, right, where a foreign adversary is able to sort of have a covert operation in which they run one of these data labeling and data fine-tuning shops, essentially, and are sort of aware that they might eventually be used to be fine-tuned for military application and implement backdoors or sleeper agents, which trigger a specific behavior based on a specific command. And because of that, that's what makes them so unsafe. So sure, you might be able to fine tune it on specific data that isn't then released publicly, which might remove some vectors of attack.

18:40But ultimately, at the end of the day, commercial models are already compromised. So when you're saying that they are more secure, I mean, they're more secure in the traditional security way in that you air gap the system. So you kind of limit the control of people who have access to it. And so that's people who can probe it and get information out of it. But it doesn't remove the fact that commercial models are already compromised from the day that they're built because they're based on public data. I wanted to see if you agree with this take I'm about to tell you. So I once interviewed Meg Mitchell from Hugging Face, and she said that for these types of military contracts, even if you have in your mission statement, like, you know, Anthropic and OpenAI do, that your tech can't be used to directly harm others.

19:23The problem is that you don't have control in the end over how your tech is actually being used with the military. If you do have any control, you definitely don't have control in the longer term once you already shared that with the military organization, especially without having security clearance and knowing really how it's being used down the line. She also was talking about what's considered direct harm. You know, what if you're summarizing social media posts that then lead to making a list of enemy combatants or potential people of interest that have a certain view on a topic on X, for example.

19:59I wanted to see if you agree with that in terms of, you know, these companies often have in their mission statements, oh, don't worry, even though we're working with the military on this, this and this, we know for a fact that our tech isn't being used to directly harm people. But yeah, how can they really know that? Can they? I completely agree with that statement. And I think something that often people miss out on is that militaries do not follow terms of service. They might do that if they're buying like a Microsoft Office suite, right, for their bureaucratic purposes. But when it comes to military procurement, the companies do not have control and they know that over how these systems are being used.

20:38And they actually have no say in terms of the terms of service as well. These things get often determined by international law and also by the nation state itself, right? They have the power here. And so when people tell me, oh, but wouldn't that break the terms of service? And it's like, this is not how military procurement works, period, right? And we've also seen examples of, as I mentioned before, Microsoft working directly with militaries to implement some of these systems. So I would say that in a lot of cases, they are well aware of how their systems are being used to some extent, right?

21:14We don't know all the details, but governments put out procurement documents of the type of data they want, how they want to use it, and how they want to store it, because then companies can offer services like what their AI can do with that, right? So I do think that it's not the case that they are just selling something commercially like they do to everyone else and then they hope the military abides by it. It's a much more involved process to do military procurement that often requires testing and evaluation. And the companies typically have to be in the know about the technical details to see if they can offer support for that.

21:49Again, as I use a Microsoft case as one of the most recent examples of that being the case with their work with the IDF. So I think it's easy for them to point to terms of service. But as someone who has worked on procurement before, this is not a commercial contract that these companies are signing. Okay, so now I want to get into the CBRN side of things. Although, as we've talked about, you know, it may not be as big of a concern as AI companies are making it out to be. It is a big concern for the public, probably because of that marketing and just because it's a scary idea. So let's be real here.

22:25Obviously, smarter AI that can do anything for you isn't always good, especially when people want to use it to do bad things like creating chemical, biological, radiological and nuclear weapons. So top AI companies say they're increasingly worried about the risk of that. Of course, they're not maybe worried enough to stop building. But I want to get into, again, how big of a risk this is. Let's just go into more detail there. We have not seen any proof of CBRN capabilities right now. But those capabilities could come to fruition if we start training very, very sensitive nuclear data, for example, nuclear technologies on these models.

23:07And I actually believe that the risk that comes with that is very different from what most people are thinking about. Most people are thinking about that the AI is somehow going to develop weapons by itself, right? Or it will give access to adversarial actors to do so. But even if it's just a military who has access to a model that has been trained on CBRN data, what that means is that they are likely going to use it for those purposes within the military. And that's extremely dangerous. Like if you're thinking about nuclear command and control, right? Who gets to essentially make the decisions about nuclear weapons deployment?

23:45And it certainly shouldn't be AI systems because regardless of the data distribution, these systems are highly flawed and they're always going to have inaccuracy. As I mentioned before, often when we're looking at military systems that deploy AI, they can have as low of an accuracy rate as like 20%. And if you're being really optimistic, maybe 60 to 80%, right? These are the levels of accuracy that you're looking at with these types of system. And so then to train a model on CBRN data and then to then attempt to use it for those tasks is extremely dangerous when you're looking at those accuracy rates, right?

24:21For me, the concern that I have is that they will then think that these models are reliable because we train them on that set of data. And thus we can then use them and the military and in defense operations to dictate decisions about those types of systems and where they should be used and when. And I think, you know, that's a very different type of risk than most people are thinking about as before, this idea that like they're somehow going to gain CBRN capabilities by themselves, very hypothetical and not really like tied to the reality that we're in right now with AI systems. But if we're taking AI systems today and we train them on sensitive military data, I have a concern of how those systems are going to be used.

25:04We need to take another quick break. We'll be right back.

25:15Support for the show comes from Rippling. If you're a business owner, here's the truth. SaaS promised to make work easier. But now the average company is buried by hundreds of apps that silo your teams, slow you down, and simply don't work together. That's not SaaS. That's SAD. Software as a disservice. That's why you need Rippling. Rippling is the unified platform for global HR, payroll, IT, and finance. They've helped millions replace their mess of cobbled-together tools with one system designed to give leaders clarity, speed, and control. By uniting your employees, teams, and departments in one system, Rippling removes the bottlenecks, busywork, and silos your software created.