Leading the charge on AI in National Security

24:59Electronic warfare is really, really important right now because it's killing Ukrainian drones. The Russians are very good electronic warfare. So now the Ukrainians are trying to adapt based on what's going on. So we have to listen to those lessons and apply them to the U.S. Department of Defense. And I, by the way, and I'll stop here, is I think what you're seeing is the replicator initiative, which is buying thousands of drones of various sizes and capabilities, is in a large part, I think, based on the lessons that they're absorbing from what we see in Ukraine. I wanted to go back for a moment to software-defined warfare.

25:32There is the paper that you just referenced that you wrote with Mr. Molchandani, if I'm getting his last name pronounced correctly, who is your CTO at the Jake, and I believe he's the CIA CTO at this point. That was quite a landmark paper. Ironically, for those of us who are in industry and maybe not military related at all in the audience, you're used to Daniel and I always talking about, you know, AI is still part of the software. It's all bound together. You can't do AI without the hardware and the software and the systems written together to make it practical AI that's usable. And you really went there in software-defined warfare.

26:13It's called Software-Defined Warfare, architecting the DoD's transition to the digital age. And it was quite a landmark paper for those of us in that industry because it really laid out the future of how software needs to be integrated. Do you have anything that you wanted to comment on about that, just given I thought it was very important to anybody concerned with AI in the DoD? And it was really written by Nan Mulchandani after his experiences in Department of Defense with the JAIC. Then he left. And before he took the CIA, he had had all these ideas germinating. And he said, you know what, I'm going to write about this.

26:50And we're co-authors, but he's the author. I'm the editor. And I placed the operational imprimatur on that report. And it's so important because it represents all the commercial software industry best practices that need to be brought into the Department of Defense. And what's sad about this, Chris, is to your point, is if you're in commercial tech industry and you were to read this report, you would be flabbergasted that the department's not already doing all these commercial best practices, which are just the way of doing business. You know, microservices, platform as a service, software as a service.

27:22It's just so foreign to the Department of Defense. Getting there, it's come a long way, but that's why he wrote that paper. It is a blueprint and say, we can talk about, this is what non-denied is to have these conversations toward the end of my time there is that we can talk about AI till we're blue in the face and we will, it's a wonderful conversation, but unless you make the Department of Defense modern and digitally modernized, which includes data best practices and do all the things that's in that software defined warfare report, AI will be meaningless. You'll never get there. You'll at least not at scale.

27:55And by the way, I am now a member of the Atlantic Council's Commission on Software Defined Warfare. So they've taken that mantle and are making it the centerpiece of all these recommendations they're going to have. So it's a really important piece. And I thank Nan because, as I'll say, for as long as I talk about this, Nan changed that organization the day he showed up because he understood there was a different way that we had to be doing business than the standard traditional Department of Defense way. One of the interesting things that I think I haven't seen the specific paper you're referencing, Chris, but I like the ideas that are being discussed here because one of the things that I've seen, and I'm kind of trying to parse through and figure out how to put words to is when I'm on customer sites or interacting with people, it's something about this like AI technology that seems to disconnect people's mind from the fact that there's still a need to like have error checking or like, it's not just that you have this AI and you send something over and it kind of automates things and then you move on.

28:59There's still the idea of a software application and there's still a call at the minimum to an API that maybe you need like retries around or some sort of health checks or backups or this sort of thing. Does that perception hold water in terms of what you've seen, Jack, as well? And you were talking a lot about culture. Any recommendations for those kind of dealing with this fact of like swapping software for AI versus embedding AI in software. Yeah, completely resonates. It absolutely resonates because this is, I even say, maybe this is going a little bit too far, but I don't think it is. And Ukraine's validating this, by the way.

29:41The next conflict is going to be an API-driven conflict. If you get that piece right and you can update faster than your adversary on this idea of software-driven warfare, you will have a competitive advantage. Because the battle speed, let's face it, you put out an AI model and you never update it, then you might as well have never done it in the first place. The idea of the battlefield is going to change, the battle space will change, but also models will drift and all the other things that happen. They get exposed to different data, new data, whatever. You're going to have to do a continuous integration, continuous delivery or deployment.

30:13That's part of this. But you can't do that in the traditional way of doing it, which is you have an entire weapon system that you have to pull apart, completely pull apart, rebuild and put back out to get. That doesn't work. You're going to have to break all this apart and just focus on those little bits and pieces that have to be updated. You mentioned API calls. There is so much more that can and should be done in the department of getting those status updates and two-way fees. One, feeding those updates of higher headquarters down, but two, putting all the things that the end user is seeing, feeding that back uphill.

30:46That's what we have to be thinking a lot more about to be able to handle what I think is going to be a very, very chaotic environment. Now, This AI, however great it may eventually become, does not change the fact, as I said in my testimony, that warfare is a very chaotic, nonlinear, dirty, ugly place. And it's horrible that we have to go to war. But this technology could provide that competitive advantage that makes a difference in a future fight.

31:30you know when we started podcasting back in 2009 and online stores just the furthest thing from our minds now we have merch.changelog.com and you can go there right now and order some t-shirts and that's all powered by shopify it's so easy all because shopify is amazing shopify is the global commerce platform that helps you sell at every stage of your business. From the launch your online shop stage to the first real life store stage, all the way to the, did we just hit a million dollar stage? Shopify is there to help you grow. Whether you're selling security systems or marketing memory modules, Shopify helps you sell everywhere from their all-in-one e-commerce platform to their in-person POS system, wherever and whatever you're selling, Shopify has got you covered.

32:17Shopify helps you turn browsers into buyers with the internet's best converting checkout up to 36 % better compared to other leading commerce platforms and sell more with less effort. Thanks to Shopify magic, your AI powered all-star. You know, nothing gets me and Jared more excited than when our guests get that coupon code in their email when their show ships or to everyone out there who loves change law podcasts and can go to merch.changelaw.com and get your favorite threads to support our podcasts. It is just the best thing ever from stickers to threads. All that is at merch.changelog.com. And did you know that Shopify powers 10 % of all e-commerce in the U.S.

32:57and Shopify is the global force behind Allbirds, Rothy's and Brooklinen and millions of other entrepreneurs of every size across 175 countries. Plus Shopify's extensive help resources are there to support you and your success every step of the way. Because businesses that grow, grow with Shopify. Sign up for a$1 per month trial period at shopify.com slash practical AI, all lowercase. Go to shopify.com slash practical AI now to grow your business no matter what stage you're in. Again, shopify.com slash practical AI.

33:54to extend what we were just talking about uh right before the break you kind of finished talking about the study of warfare being asymmetric technology advantage you talk about that in your paper and just the challenges and you've talked about integrating in these latest greatest technologies. One of the things that you address a fair amount is you talk about teaming between AI and a human being in the context of national security and adjacent with that, which I know Daniel asked you a little bit of a question about that earlier, kind of having to do with safety and the security of the AIs and stuff.

34:28Can you talk to us a little bit about, you know, that's obviously a question that many people have in their minds that have nothing to do with this space, you know, the military or the industry around it. Just concerned about what does teaming mean? How far in your thinking does autonomy go? Is there a point, just to infer something, that as the pace of warfare speeds up, you know, we have drones and we have different types of autonomous technologies and stuff, but as the pace of warfare speeds up and you look into the future versus kind of what we're looking at in the near term, how does that change the human AI teaming equation as you're trying to keep up, you know, and maybe having humans in the loop can be come a problem or a serious logistical issue.

35:13How are you looking at the future in terms of where we are today and where that's going? And what are some of the challenges that we need to overcome to get there? I think so strongly about this idea of human machine teaming or human systems integration that if we don't get this right, we're in trouble. And what I mean by that is there are people who have been writing about human machine teaming in some form for 50 years, no different in terms of the concept of how do you get the boast out of human machines. But to me, we're at a period where it is changing and it's changing because the machines are going to be so much smarter than we're used to talking about that it will be a different view of human machine teaming than we're used to.

35:53So I think there's a whole new research field. There are already people working on this and I'm amazed at some of the work that's being done, but we just don't know how good some of these technologies are going to be. Let's just take the example, very briefly example of sort of chat GPT or the equivalent. There's this idea of prompt engineering. I look at that as almost like the elevator operator in the 30s or 40s, whenever it was, and it goes away eventually because people get comfortable with it, but they're not as comfortable with it today. You need help figuring out what that interface looks like.

36:21But the other thing I say, and this is where I come back to, I really do believe there will be situations. I call this the bell curve of military operations. Some cases, you really do want that human making that final decision. Maybe to launch nuclear weapons is the president of the United States and nobody but the president of the United States. On the other end, you need the machine to do what the machine does best. There is no time to get in the way. Everything else in between, I don't know what that is, 80%, 90 % is human machine optimized for both, that centaur idea. And that, we need a lot more work working through.

36:56And here's the example I would give from my time in the flying world. There were a lot of times the machine did not operate as intended. The human was there to make sure that everything went fine with the mission. Oh, that broke again. We're going to have to pull that circuit breaker. I started in the F4. Circuit breakers popped all the time. You'd piss out hydraulic fluid out the back end or whatever. So the human would have to make up for a lot of the machine's mistakes. I'm not sure that's going to be a luxury in the future, that you're going to have to let the machine do what the machine does and the human do what the machine does very well.

37:27What do the humans do well? They do reasoning. They do inductive reasoning. They do deductive reasoning, do abductive reasoning. They put things in context. They deal with other human emotions. Machines don't deal with emotions. They don't understand emotions. And I'm not sure, despite what some people claim, they ever will. So what does that look like? I think there's a lot that has to be done in this area in experimentation. Play around with it. See what works because if you don't get it right, and there's a couple of examples where it's went dramatically badly. And I think the 737 MAX is one of those examples.

37:59Well, we're going to put this software in. Trust us. You don't need to be retrained as a pilot on this. Just listen to what it says in the cockpit. That was wrong. And it's proven that it was dramatically the wrong thing to do. So I think this is an area that needs a lot more explanation. There are people writing about this and then really good writing in various places. But we have to figure out how to get this part right because the machines are going to be smart enough that they have to be allowed to sort of run when they should run. We're seeing both modern models are getting so impressive in certain areas, in certain capabilities.

38:34There's a lot of hype around them. They can't do everything. But what they do, they tend to do quite well, recognizing that there are hallucinations and other technical issues to be worked through over time. But we're seeing kind of a rapid increase in capability in a lot of these areas. So one of the things that I get asked all the time, just kind of on the side with people, is as those capabilities go to some level of increase in the future, whatever that is, and whatever the tasking is, and you've kind of alluded to, it kind of changes the balance in human teaming. Is there a set of metrics or some guidelines that you have in your mind on as you see the technology progress, maybe not today, maybe not this year, maybe not next year, but maybe five, 10 years out.

39:19And you see these capabilities in specific areas increasing far beyond what a human can do for a given task. How do you rebalance? How do you assess that? You talk about assessments in your paper as well, so that you say, this is one of those moments where you let the model do what the model does really well because it does it faster, it does orders of magnitude better than a human could do in the same amount of time. How do you make those assessments? Because I think that for us humans with emotions around technology and warfare, I think that causes a lot of concern. It seems to be the foundation of many questions that I get asked.

39:58How do you make that metric judgment on those adjustments to the culture of how we interact in that way, you know, kind of the way we think about it. Yeah, a few thoughts on that. And while I remain an AGI skeptic, I do believe these machines, these AI-enabled or smart machines will get so good, the human interfering with the machine could be a worse outcome than human-machine together. And the example that I've heard, I think it was Gilman-Louis raised this about AlphaGo and the Move 37. If a human was there to override the machine, it would have lost the game. Machines said, no, you really do want to make this move.

40:35Leave it alone. And it won't. So that's an example of humans are going to get to the point where they've got to be more comfortable with this technology. And they're not necessarily today. Why? Because just like when I use ChatGPT, whatever version of a large language model, it does get some things still very wrong. Maybe the ratio is only 10 % versus 20 % just a year ago. But if that 10 % is dangerously wrong, then do you really want that in military operations? Not yet, you don't. So how do you do that? This is the core of your question. To me, it is test and evaluation. It's experiment. It's putting it into the hands of users as an MVP, a minimal viable product, which again is a little bit different or a lot different than the way the Department of Defense has fielded systems in the past, where you field any system when it was as close to perfect as you're going to get, which may have not been perfect, but close to nothing.

41:25for government work, as we say. But in this case, you need to put things in the hands of users sooner rather than later. This is why I'm a big fan. Despite my reluctance to say we should not be using large language models for putting items in the presidential daily intelligence briefing, you should have all sorts of experimentation being allowed in all these federal government organizations and agencies. In fact, the White House executive order on AI says, go try this out. It basically says, go do it. So I'm a fan of the experimentation, what we'd call sandbox experiments, put it in users' hands because we don't know yet.

41:59But part of that is core test and evaluation. We do not want a short circuit test and evaluation. And why is this so important? Because I think that's where we're going to see the biggest risk, at least initially. Some risks are going to be hard to figure out until you use them in an operational world. We just know that, just like in commercial business. Some things are going to surprise you, but that's why you then update the models because you learn. But in that early stages of design and development, before you get to the fielding part, there is this thing that we have done really well in the government for many, many years, especially on the hardware side, it's called test and evaluation.

42:33That does not go away when we talk about AI. In fact, AI is still novel enough. I think it's more important than ever to spend a lot of time on the T &E. It doesn't mean you're going to go slowly relative to sort of putting out other things, but I am a little bit cautious about moving too quickly on some of this because we just don't know what those risks are yet. So there's a lot of work going on, including some things I've been working on, on a risk management framework for AI systems in the military. Because this risk, it is a hierarchy, right? There's one end, there's AI-enabled nuclear weapons, really, really bad.

43:06On the other end, there's process automation for a finance system that touches nothing having to do with warfighting. negligible risk, move really fast on that one. And then a lot of things in the middle, you've got to do test evaluation, determine how many risks there are, and then come up with risk mitigation strategies. I'd love to weave a couple things together from what you mentioned in your testimony. One of those having to do with a concept which I thought was really interesting around techno-economic net assessments. And I think there's a lot of people, so I'll ask you to maybe explain what you mean by that.

43:40There's probably a lot of people that see news articles about, oh, this country is ahead of this country and this type of AI, whatever. And of course, you see articles about, oh, China, whatever company in China bought up this many GPUs or like other things like that. And depending on where you're at, you get concerned. But it's all sort of very amorphous and it's hard to really grasp where countries are in terms of the AI stack and their capabilities. And the other thing that you mentioned in the testimony is encouraging the DOD to take bigger bets. And I imagine that that's also connected with sort of making sure that our techno economic net assessment is sort of on the upward trend.

44:27Could you talk a little bit about those ideas and maybe how they're connected and how you would love to see those kind of big bets going forward? Yeah. If anybody was not in the government and skimmed through my testimony, they probably would have went past that paragraph and not paid any attention to it. There's a reason I put it first. It's that important. It really is the center of attention. And that is, let me put it in terms of say one commercial company competing against another commercial company in the same general business space. CEO of one is always looking at CEO of two saying how much faster are they moving?

45:00What special sauce are they bringing into their product that threatens our market dominance? This is going on every day in commercial industry. Well, it's also going on between states, in this case in AI. So the problem is with states is you can get a lot with industrial espionage, but it's a little harder to do when you have a nation state like China and both sides, the United States and China, are talking about how much they're each doing in AI, but how much is reality and how much, again, is the hype? Well, you need a lot of intelligence assessments. And I found this out from my earliest days in Project Maven, where I would ask these questions about, okay, and I'll stay very unclassified here, the People's Liberation Army, what does their AI stack look like?

45:41The Intel community is not spending any time collecting, or at least they weren't at the time because nobody told them to. What is a GPU? Okay, well, we know where we have to start this conversation then is, what does their compute look like? What do their models look like? Are they using open source or who's building them for China? What does their talent base look like? So this idea of not just technology, but then equally important on both sides, US-China or US-Russia, whatever you want to say, is what are they doing with the technology? What are they building new operational concepts? Are they actually reorganizing?

46:14When you start seeing bureaucratic reorganization, they're far along. We don't see that yet. We don't see it in the Department of Defense because we haven't figured out exactly what this new technology is going to do for us yet. So that's what I mean by this. And by the way, the U.S. government used to have this thing called the Office of Technology Assessment. I think Newt Gingrich managed to kill that as part of the revolution in government, whatever. Now there's some serious work to bring something like that back. Now I'm just talking about within the Department of Defense and the intelligence community on the classified side, how do I bring in all this information, both unclassified and classified information to give us a relative net assessment, a net assessment, us versus them, where do we stand?

46:54And it turns out that's pretty difficult to do because technology is not so easy to take a picture from a satellite of a GPU and decide how much farther they are ahead. So to me, we've got to do better at understanding vis-a-vis, you know, the United States, vis-a-vis China or Russia or anywhere else around the world, what they're doing in this area. And so it's such a big concept to get right. And it's hard because it's just not like collecting against tanks or nuclear weapons or something. I can't see it anymore. And as I say, there's no fluid coming out of a building somewhere to tell me that they're working on a particular project.

47:27Well, Jack, we have covered so much material here and time that we've been talking has flown by. I think we're going to, if you're willing, we're going to have to have you back on the show because there's a lot that we haven't been able to address yet. As we finish up, I'm kind of thinking in the background about culture and what you were talking about and just the massive organizational and even national change that we're doing here in the United States in government and military and intelligence. It's changing the way that we are looking at the future of warfighting. You mentioned in your paper, the joint warfighting concept, which I'm very familiar with.

48:03And I believe there is an unclassified version out there that we'll add into the show notes, but it's changing the way we think about all that we do with the military and intelligence. And I think AI is a big part of that. As we wind up, do you have any, we often ask guests to kind of take the last question to be whatever you want it to be, paint a picture of the future with whatever you want to be. Can you give us a sense of kind of what you're excited about going forward, how you think some of this may evolve and what that means to the United States military and intelligence communities so that the listeners to this, mostly not involved in that directly, have a sense of where things are going.

48:45Any thoughts you want to finish with today? Yeah. Thanks, Chris. And thanks, Daniel, for allowing me this time with both of you today. And it's a big thought and it's something I've been thinking a lot about. And when I retired, I went back and got another master's degree, thanks to the GI Bill. And I did get a chance to think big thoughts about technology over the course of human history. I mean, really, back to the very beginning. I took some courses about looking at what we would call, maybe not technology today, but it was back then, say sugar or tea and how it sort of diffused globally. What I say in my testimony, I do believe in my core that we are going to be in the middle of, at some point, the third revolution, sort of agrarian revolution, industrial revolution.

49:24This is different. It's not the fourth industrial revolution. It's some kind of digital revolution. We don't know what it's going to look like because we just haven't been there yet. I say in my testimony, the future is, to a large extent, both unknowable and unpredictable. Why? Because it's not determinism. It is not technological determinism. It will be dependent upon the decisions by many, many, many, many thousands of people, from leaders to citizens of countries deciding they like or don't like the technology. It's going to take maybe 50 years to 100 years to play out. And historians will be the ones that look back and define when this revolution began.

50:02But to me, it is fundamentally different, which means warfare will be different. The character of warfare is going to change. I will not say the nature of war is changing, because the nature of war is a human-centered decision, like why we fight and why one country fights another country and so on. So I don't believe the nature of war, but the character of warfare is going to change dramatically as we're seeing in Ukraine. And we have a chance to be on the right side of that, as I say in my testimony, asymmetry equation. If we're on the wrong side of it, we risk losing and we're not used to losing.

50:32And this is a very serious risk. So this idea of it's playing out as we speak, get involved in it. Don't wait for it to catch up. You've just sort of got to dive in and start working these big projects in the government, wherever you are or anywhere else in the industry. Well, that's a great call to action to finish up with. Jack Shanahan, thank you so much for joining us on the Practical AI podcast. Really, really interesting conversation. Thank you for your insights. And hopefully we can get you back on the show to cover things going forward. Really appreciate your time. Thanks, Chris. Thanks, Daniel.

51:06And of course, I'll be glad to come back.

51:15All right, that is Practical AI for this week. Subscribe now. If you haven't already, head to practicalai.fm for all the ways and join our free slack team where you can hang out with daniel chris and the entire changelog community sign up today at practicalai.fm slash community thanks again to our partners at fly.io to our beat freaking residents breakmaster cylinder and to you for listening we appreciate you spending time with us that's all for now we'll talk to you again next time

52:00Game on!