AI hot takes and debates: Autonomy

0:03Welcome to the Practical AI podcast. where we break down the real world applications of artificial intelligence and how it's shaping the way we live, work, and create. Our goal is to help make AI technology practical, productive, and accessible to everyone. Whether you're a developer, business leader, or just curious about the tech behind the buzz, you're in the right place. Be sure to connect with us on LinkedIn, X, or Blue Sky to stay up to date with episode drops, behind the scenes content, and AI insights. You can learn more at practicalai.fm Now, on to the show.

0:48Welcome to another episode of the Practical AI Podcast. This week, it's just Chris and I. We're joining you for what we call a fully connected episode, which is one without a guest where Chris and I just talk about a certain topic or explore something that's of interest to us or that we've seen in the news or help kind of deep dive into a learning related resource. I'm Daniel Whitenack. I'm CEO at Prediction Guard and I'm joined as always by my co-host, Chris Benson, who is a principal AI research engineer at Lockheed Martin. How are you doing, Chris? Hey, Daniel. I'm doing fine. Looking forward to today.

1:31We have something fun unplanned here. This should be fun. Yeah. So for our listeners, we've been in the background of the show really talking about, well, a lot of things about the future of the show. Some things in terms of, you know, things like rebranding and updating our album art, which you'll see soon, new intros, outros, but also new kind of focus areas for content on the show, really being intentional about some of the things that we're exploring. But if you've been listening to us for a while, you also know that Chris and I like to have fun together and explore things in interesting ways, just the two of us.

2:14And some of our team here came up with the idea, well, what if we took some shows and had a kind of AI hot takes and debates type of show. So this is the first iteration of whatever we'll end up calling this AI hot takes and debates. That's a good one. But basically, the idea here is that there's a topic which is part of the wider conversation around AI, which people are divided on. And we will take some of the arguments for one side and the other side. And it really doesn't matter what either Chris or I actually think on the subject. Maybe some of that will come out in the discussions. But really where Chris, you take one side and express some of those arguments for the one side, I take the other side, express some of those arguments, and we discuss it together, hopefully exposing the audience to both sides of a topic.

3:20And also, it sounds like fun. What do you think, Chris? I think it sounds like a lot of fun. And of course, as we were coming up with the first topic, you might say I threw a hand grenade into the mix. Good one. Good one. There you go. And that it's a topic of particular interest to both of us, but especially to me. But it's important that I say we're going to talk a little bit about autonomy in warfare and in some areas outside of warfare. But I really want to point out, I want to emphasize that, you know, Daniel and I don't necessarily take hard side on either topic where we're kind of, it's an assignment to take one side or another.

3:56But I also want to emphasize because I work for Lockheed Martin, who's a defense contractor that I'm definitely not in any way representing a Lockheed Martin perspective. This is just a fun debate issue that we're having today. So I don't often have to do disclaimers, but I felt given the topic that was important today. Yeah, well, and I was going to say autonomy means a lot of things and it can be applied in a lot of industries. And so I think some of the arguments that we're going to talk about here are equally applicable to autonomy in, you know, self-driving cars, you know, airplanes, surveillance systems, like all of all of these sorts of things where, you know, manufacturing, whatever it is, there are similar concerns to those that we're talking about.

4:43Obviously, if you have autonomous weapons systems, there's a particular kind of life and death element to it. And it intersects with conflict around the world. In the YouTube videos I've watched of interesting, interesting debates. I've learned over time that you should frame this sort of thing as a question or frame it as a, not a question, I should say, as a statement, one side taking the affirmative, one side taking the negative side of that. And so the statement, if you will, is autonomy within weapons systems or military applications is an overall, you know, positive thing for the development of, you know, of the world and safety and, you know, resolving conflict and all of those things.

5:41So I'm, because Chris is maybe a little bit closer to this, I'm going to take the affirmative side of that, which is, hey, autonomy could actually provide some benefit and more safety and less loss of human life, more ethical sort of application. And Chris is going to take the opposite side of that, arguing that we shouldn't have autonomy in these sorts of systems. Does that make sense, Chris? Did I explain that in a reasonable way? I'm not much of a debater. No, that sounds fine. We will go with that. I wanted to mention to the audience, it was actually someone who kind of got this into my head a little bit through a post was one of our former guests on the show.

6:29So back in the beginning of 2024, on February 20th, we had an episode called Leading the Charge on AI and National Security. And our guest on that show was retired U.S. Air Force General Jack Shanahan, who was the head of Project Maven and the DOD's Joint AI Center. He actually founded it. And I follow him a lot on LinkedIn. And he had highlighted two papers written. And one was kind of a pro-autonomous weapon systems paper that was written by Professor Kevin John Heller, who is professor of international law and security at the University of Copenhagen Center for Military Studies. And he's also a special advisor on war crimes at the International Criminal Court.

7:16And he wrote a paper called The Concept of the Human and the Critique of Autonomous Weapons. And subsequent to that, after he wrote that, two other individuals, Elke Schwartz, who is professor of political theory at Queen Mary University of London, and Dr. Neil Rennick, who is also an expert in this area. He's a researcher at the Center for Military Studies at the Department of Political Science for the University of Copenhagen, wrote a counter paper on this. And that counter paper is called On the Pitfalls of Technophilic Reason, a commentary on Kevin John Heller's The Concept of the Human and the Critique of Autonomous Weapons.

7:58And that was a very recent paper, May 23rd of this year, 2025. And the original paper written by Heller was December 15th, 2023. And it just seemed like an interesting topical thing to jump into. And like I said, we're not going to we're not going to hold ourselves bound by the strictness of their topics, but mainly just have fun with it. Yeah, yeah. Have fun with it. And we should say at least our plan for these types of conversations between Chris and I are that one of us would express core arguments of one of the sides of these debates, but then maybe open it up and just have some casual conversation about each of those points, the kind of how we think about it, the validity of it, that sort of thing.

8:42Maybe that's just because I selfishly don't want to come off as a bad debater and not have a good rebuttal for my opponent, but also Chris and I are friends. So that makes it fun. It's all for fun in the end. Cool. So again, the kind of hot take or the debate of today is that autonomy is an overall positive in terms of application to autonomous weapons systems or in military applications. But But I think as you'll see, some of these arguments might be able to be extracted to other autonomous systems like self-driving cars or planes or maybe surveillance systems or automation and manufacturing, etc.

9:28So I think we can start with the first of these. And I'm happy to throw out the first kind of claim here. And really, I'm basing this claim on. So again, I'm kind of on this affirmative side and taking part of that article by Heller and really highlighting. So my claim is that autonomy within conflict and weapon systems is or could be positive because real human soldiers or humans that are part of any process are biased, emotional and error prone. And if an autonomous system is able to outperform humans in adhering, for example, to international humanitarian law, then that actually minimizes harm of these systems, as opposed to being solely reliant on humans who are, again, biased, emotional, and error prone.

10:31So part of the article that motivated this talks about how decision making in particularly high impact scenarios like a combat scenario is distorted by these sort of cognitive social biases, negative emotions, actual limitations of humans in terms of how much information they can process it at any given time. And one of the quotes that I saw from there is, you know, very few human soldiers in a firefight actually contemplate the implications of, you know, taking a life or extrapolating that to other things. Maybe humans flying planes or humans flying cars or humans doing manufacturing activities.

11:15They aren't contemplating the implications of every action that they're taking. In some ways, they're just trying to survive in those environments or get through the day or deal with their emotions. So that's kind of the first claim. What's your thought, Chris? Well, I think the opposing viewpoint in that is going to be really that we humans value our ethical and moral judgments that we make. And we put a lot of stock in that and that our ability as humans is really important. So, you know, the argument against the fact that, you know, kind of the fog of war takes over for the individual and they're not thinking about higher thought is the fact that if you – on the other side is we do obviously want humans in combat to have the ability to make moral judgments.

12:08And that does happen, you know, where you have someone thinking, you know, that's a child out there that has the weapon and I'm just not comfortable no matter what happens. I'm not comfortable doing that. And that's the kind of moral judgment that we humans really value. And we don't necessarily, you know, trust autonomy to be able to make such distinguishments in the near future on that. And so, you know, the notion of taking a life and death decision out of a human's hand is something that we really struggle with. And it creates ideas like the accountability gaps that go along with that and ensuring that as horrendous as war is, that there's some sort of ethical core to that that is at least available to the common soldier that's making decisions in the heat of the moment.

13:00Yeah, I this is a really interesting one, Chris, because I I also I mean, I know you and I have talked about this and you're also a pilot and thinking about airplanes, been watching a lot of with my wife. There's this, I think it's been produced for like a very long time, but it's on streaming now. This series of shows, I think it's called Mayday Air Disasters. And it just highlights a different sort of air disaster, like air, commercial airline disaster that happens, you know, has happened historically. And kind of what happened and they go through the investigation. And some of the clips are kind of funny, not in terms of what happened, because obviously they're tragic.

13:43But in terms of how it's produced, it's very, very literal. and some of the acting is maybe not top notch. But what I've learned through that show is I have been surprised at the amount of kind of information that pilots, let's say, have to process and given a certain state or fatigue or even just unclear leadership, like who's in charge in a cockpit, can lead to just very irrational decisions. And so I do understand the argument, whether it's in terms of, you know, weapon systems or flying airplanes or driving cars. Certainly people are not always, aren't always making those rational decisions in terms of how they process information.

14:27I totally agree. The, there is, you know, but there's also, it's kind of funny as we talk about that and talk about the, you know, the flaws that, that humans have in, in these processes. And, And we certainly do in that there's also kind of the notion of when you're processing that much information and knowing you have a lot of information you have to put in contextually in the emergency of the moment. And pilots do that quite a lot as things happen. That, A, there's that uniqueness of the human brain being applied to that, that we don't yet feel that autonomy can take that all the way. And certainly if you pull the flying public right now, you know, in terms of airliners and stuff and other places where they would put their own lives into the hands of autonomy, most people still, and I don't have a particular poll in mind, but having seen a bunch of them over the last couple of years would argue that they, that they are not comfortable not having a human in that.

15:30And once again, that is the notion of having someone that cares for them, that is in control, that you have trust with a tremendous amount of expertise, being able to ensure that a good outcome occurs. And I think that that's an important thing to accept and recognize that people are not there yet. They're not to the point in general. Now, I will note as someone who is in the military industrial complex world that there's a lot more autonomy on the military side than the civilian side. But I think we need to get to a point where our autonomy can match the expectations that we already have in our human operators.

16:34Okay, friends, build the future of multi-agent software with Agency, A-G-N-T-C-Y. The Agency is an open source collective building the internet of agents. It is a collaboration layer where AI agents can discover, connect, and work across frameworks. For developers, this means standardized agent discovery tools, seamless protocols for interagent communication, and modular components to compose and scale multi-agent workflows. Join Crew AI, LangChain, Lambda Index, BrowserBase, Cisco, and dozens more. The agency is dropping code, specs, and services. No strings attached. You can now build with other engineers who care about high quality multi-agent software.

17:25Visit agency.org and add your support. That's A-G-N-T-C-Y dot org.

17:36Okay, Chris, continuing on in relation to the arguments for autonomy, meaning AI or computer systems totally in control of things like weapon systems or other things in a military context or other autonomy. So the next argument that I'll put forth in terms of pro-autonomy or the affirmative here is that we've been talking a lot about kind of human morality or their intentions infused in that. And the claim here would be that responsibility essentially remains, like there's no unclear responsibility here. So some people might say, oh, if you have an airline crash and it was the computer's fault, right?

18:28Then how do you, where do you put the blame, right? And here, I would say, or in expressing this kind of part of the argument, the argument would be, while responsibility still is clear, it just remains with the designers or the commanders of these systems, just as with other kind of automated systems. And if you think of like automated weapon systems, they don't disrupt these kind of legal and moral frameworks because it's the designers of these systems that actually impose those frameworks or should impose them within the systems. So there's no, in this case, the argument would be there's no significant difference between kind of human soldiers or autonomous weapons in terms of criminal responsibility, because responsibility lies with those who design, program, or authorize the deployment of these machines, not with the machine itself.

19:29These are just instruments of the will of their developers and those responsible for employing them. So that's the argument. What's your thoughts? And I think the counter argument on that is going to be that, you know, reducing civilian casualties alone through automation is not enough. And recognizing that automation can follow rule sets like the like the laws of war and other other applicable laws that we don't want to lose the notion of of our moral of moral decision making and the emotions that go with that. We have empathy. We have the recognition of moments where under a particular rule or law, an action might be authorized, but that there might be a kind of a human notion of a moment of restraint because you recognize the distinction that is complex.

20:27The world is not simple. It's not black or white. And that these kind of qualities of recognizing the situation, that there are lots of ancillary concerns that aren't necessarily covered under strict rules of engagement, that those matter and that we care about those. And that the, you know, if you put in a model that's trained on reacting to specific, you know, tactical situations and is following the rules, so it's entirely legal, that they still aren't able to recognize those complexities. and they they're not able to have that empathy and that you can't replicate that with present technology or near future technology to to take the human out of the equation.

21:15So it's one of those. It kind of depends on how you're how you're seeing that in terms of do you want to take that those human qualities entirely out of the equation and rely on autonomy. And that's I think that's one of the great questions of our time. Yeah. I, I have a followup question here, Chris, or maybe it's a proposition. I don't know, but you mentioned how many rules or, or things that a human intervening in a situation, like they, they make kind of decisions in gray areas, things are maybe fuzzy. I'm wondering if like here we're framing this very binary, right automation or or not automation one of the things i found just practically outside of this realm and in like putting in automation with customers of ours that are in manufacturing or healthcare or wherever wherever is there could be an automation and you can analyze the inputs to that automation to understand if things are similar to things you've seen in the past or different from those you've seen in the past.

22:30So, for example, maybe you're processing clinical documents for patients, right? And there's a set of scenarios that you've seen quite often in the past and you've tested for and you have a certain level of confidence in how your AI system will process these clinical documents, maybe make some decisions and output a notification or an alert or a draft of email or whatever those automations are, then you could receive a set of things on the input that is very dissimilar to what you've seen in the past. Oh, this is a unique situation for this patient in healthcare or something. And you kind of don't know what the AI system is going to do, right?

23:18And often we would actually counsel our customers and our partners. Well, that's an area where maybe you want to flag that and just not have the AI system automated, but those are the things maybe that should go to the human. So I just wanted to call out that kind of scenario where we're talking in a very binary way, like, is it automated or is it not? There's kind of this area where it's like kind of automated and kind of not. And you're looking at the situational awareness of what's coming into the system in light of the distribution of data that you've seen in the past. There is, you're posing a really interesting question.

24:04And I guess I would frame that as you're looking at, you know, these different, these different types of outcomes and, and, and based on the data that you're doing that, that within these structures, there are, there's this kind of notion about where the human fits in real, in relationship to that. I mean, you kind of raise that up about, you know, the autonomy, raising those issues up to the human and stuff. And that's, I think that's a big part of figuring out in the path forward, how, what that relationship looks like because I think I think it varies quite widely across use cases there in industries what you're looking at I know in in my industry there's the notion of kind of a human in the loop versus a human on the loop and so there is there is autonomy where humans and and the autonomous systems are working as partners in a very direct way and some people might say the human perspective on that is to the autonomy kind of makes the human who's in the loop kind of superhuman by giving them a lot more ability to process the right information at the right time to make the right decisions.

25:13And then contrasting that against the notion of a human on the loop, which kind of is kind of what you mentioned where it raises something up that might be an exception or a special case or a specific thing in a larger quantity of data that you're trying to bring to attention as a key finding. And they suit different use cases, but it's not always very clear what those are when you're trying to design a system that does that. So I think there's a lot of room for both. I think that right now we tend to look at human in the loop more often. But I will say as someone just as making an observation about the industry that I'm in, that in this particular case, the nature of warfare is changing rapidly and the speed of war is increasing rapidly.

26:05And we're challenged with what can a human do if things are speeding up all the time. So, yeah, you've raised some really great questions there in terms of how to solve for some of these, especially considering what we've talked about, about, you know, the ethics versus the deferring to the autonomy. Yeah, I'm glad that you brought up like speed and efficiency here. I guess that's one while you were talking. That's one thing I was considering. And maybe I'll give just a couple examples. It could be that in, let's say, an aircraft situation, I'll go back to that. Maybe it's just because I've been watching that show with with my wife.

26:48But in that situation, maybe a situation comes up and you realize the automated system should alert the human operators of something. Well, if the human operators haven't been tuned in to what's going on with the system, right, in this case, the aircraft, that alert could happen and they could have to gain so much context and so much information to understand why the alert is happening, what systems are malfunctioning, etc. that by the time they catch up, right, then the planes crashed or something like that. Right. And so in that sense, like the speed efficiency piece is really relevant. I was also thinking in a maybe less life or death type of situation.

27:39If you're using, you know, these these vibe coding tools and you're creating X, you know, web app or something like that, you're creating this new system, this new project that you're working on, you could have those automated systems working for, you know, lots of time creating lots of things. And then, oh, all of a sudden, like, there's this error that the Vibe coding tools can't resolve, right? Well, then you step into that. And you're like, I, I don't know, anything of what's going on here, what files have been created? How is this configured? What is really the context of this error, right?

28:23And it could be more inefficient for you to actually, you know, debug and deduce that than it would have been, like the overall efficiency kind of would be less than if you had just like written all that code. Now that may not always be the case and different scenarios happen, but that element of speed and efficiency really strikes me as a key piece of this element of what we're talking about. Yeah. It's funny. Your example really resonates with me because I think that's, as I've tried vibe coding myself and stuff compared to more conventional programming. And I think I've run into that and it kind of depends.

29:08There's a lot of context. If you're not only running into problem, but you have to decide to some degree, there are, there may be extraneous reasons to structure, to pick certain tools and certain things. And if you're vibe coding and essentially the model may have a lot of freedom to choose how it's structuring, you know, what you're building and the tools that it's using and things like that, that, that may have an impact. And I think that's where I run into vibe coding challenges myself is giving the model the autonomy to make choices that I really think I should, whether I'm right or not, whether I think I should be making myself that may have to do with performance or constraints or future team skills or whatever else we may have.

29:52So a great point you're making there. Okay, Chris, the third affirmative claim that I'll express from the side of pro-autonomy in things like weapon systems, conflict, and maybe other areas of autonomy, is that autonomy could actually reduce harm, casualties, suffering, because autonomy actually results in greater precision or consistency. So banning these sorts of systems on principle may cause more harm than good, essentially. And, you know, you could kind of play this out. A couple of things that are said in the paper, again, that we're kind of inspiring this conversation around by Heller is, you know, it's only a matter of time before these systems maybe comply with international humanitarian law better than human soldiers or, you know, these systems, I think it would be true, Maybe at least in certain cases now, these systems have more or have the potential to be more precise than even the most kind of most precise non-autonomous systems in terms of targeting or performance or operation in some sort of way.

31:22So the argument here is that a kind of outright ban would actually result in more suffering rather than less suffering. So how does that how does that strike you? Well, before before I make a counter argument against it, it's interesting that it's something that you noted there. I'll just throw out a two second example. You know, the the the Russian invasion of Ukraine has has been going on now for for several years. and the one of the you know there there is a as reported in the news there's quite a bit of small autonomy drone warfare going on and that and one of the things that we're that we've seen there a combination of both human i think this will kind of lead into my side a little bit is that you may have drone operators flying but often when they make the strike the final bit of strike has been made autonomous uh where the actual hit itself in the last uh last couple of seconds will be autonomy driven to your point about, you know, precision and such there.

32:23That's a little bit, you definitely have a human in the loop through most of that operation. And then that human has to decide to go fully autonomous for the precision thing. But, you know, on the, on the counter side of that, it's important to note that as you are bringing more autonomous capabilities, especially when you're moving the human out of a direct loop and to be just kind of human on the loop where they're kind of overseeing the autonomy versus directly, you know, directing the weapon itself. There's a dehumanizing aspect to that. There is this notion about I have a tool and I'm as a human, I don't even need to drive my tool to the bitter end and to strike my objective.

33:05I can just let the machine take it home. And I think there is something that is potentially revolting to a lot of people about the notion of kind of the sense of devaluing a human life, that even though that may be your adversary, that your mission is to go eliminate that adversary, that it's still a human being, and that you're still just outsourcing to your machine to go take care of your current problem, which is your target. And that is something that is a big struggle in this ethical conversation that we're having is, is even if it is your enemy and your adversary that you're trying to to address um in the in the manner that you have to do it does you know what does that mean to outsource it does that make it you know even even more uh dehumanizing and inhumane to do it that way so i think both sides have really interesting arguments and i hope for listeners who've been listening so far you know seeing kind of that uh there's not a right or wrong answer to either side on this, on these issues.