#135: Thales' Chief AI Officer Ajay Chakravarthy, AI on the Battlefield: Where Should We Draw the Line?

28:53So the human operator is still in charge, but they're able to deploy five, six different agents, which are able to look at different parts of the requirements and then code up the entire end-to-end system for you, whilst the human is still being in control. So that's one example. In terms of autonomy, if you're talking about drone management and potential mission planning with respect to a swarm of drones yeah yeah if ai spots a pattern and then ai acts upon that as opposed to waiting for a human to press the sort of approval button do you see what i mean because ultimately if the ai can take action and resolve the problem in real time then that's more efficient than waiting for someone who may be on his lunch break to go to approve it in a critical environment you know what i mean so example but you know what i mean yes i i understand exactly what you mean so within In Thales, we have a philosophy called True AI, which stands for transparent, understandable, and ethical.

29:51So Thales has always followed these sorts of philosophies when it comes to developing any technology component. But particularly true with AI is, yes, you can allow a degree of autonomy based on who the end user is, what level of risk are they prepared to accept, and how do you constrain your agents to perform within those within those bounds of risk level and then are you able to design your ai keeping in mind the true ai principles from the ground up which is are you able to explain your decisions on why an ai has taken a particular decision all the way through to the long tail of information so where exactly has the data come from why did make a decision and what was the result of that what level of reliability is it performing with So is it able to give you confidence scores on the objective it has achieved?

30:40And the other thing is objective matter, but also very important is ethical principles. So within the European Union, we got something called the EU AI Act, which all AI systems now need to implement by default. So it's making sure that all the ethics are embedded within your systems right from the beginning, rather than a tick box exercise as an afterthought. so once you're able to follow these kind of these kind of principles from the ground up just like it was security by design you need to be true by design you're then able to provide more and more level of autonomy with within a level of acceptable risk and then ultimately the human is able to make the decision depending upon the criticality of the situation so if you're talking about kinetic effects then i would say the human will always be involved in those kind of things human in the loop Yeah, particularly when it comes to defense and national security.

31:31Oh, yeah. Yeah, I suppose it's even more critical there, isn't it? And that's something I want to dip into a little bit more. But I just want to get your thoughts on the recent Jacob Coxon situation, as I'm sure you heard about, who left his employer because he wasn't particularly one of the big frontier model companies because he wasn't comfortable with the level of safety measures that were in place and the way AI is just continuing to just progress. and safety, not being able to keep up with it, perhaps. Where do you stand on that, AJ? I know this is a bit of a tangent question, but where do you stand in terms of, should we restrict progress?

32:07Should government step in and sort of say, hang on, we need to slow down with this, or do you just think we should let it take its course? So there's a few points on this particular topic. I won't go into any particular companies as such, but the frontier model development is going at a pace which is quite rapid. So I read some stats recently, which says that the capability of the models is doubling every seven months or so. And then the power consumption requirements are growing to a level where in about five years, we won't be able to use all the energy on Earth to train a front-term model, which is still not fully trained.

32:43So although the energy requirements are growing, the capability is doubling. So the first thing to consider is, is it sustainable in terms of the energy required to train these models? and secondly when it comes to this particular comment which you are referring to stopping the growth of ai there's there's two aspects to it one is the prevalence of ai agents so we've seen a recent attack on the huggy face website through many different agents yeah that was crazy by the way that was crazy having done a deep dive into that it wasn't it wasn't some sentient being you creating sky it's not skynet so what actually happened was many different agents collaborating together to exploit a cyber security vulnerability in order to achieve the goal sure but having said that's not that's not an excuse to say we shouldn't be concerned about the rate of development of these agents so this was a this was a case where there was a thousand agents working with each other what if there were a million agents deployed out there in the world and what level of collaboration could they achieve, for example.

33:46So the lesson to take away from that is we actually need to use more AI in order to create those defensive barriers, those defensive agents in a situation where this kind of attack may happen, which is likely to. So even though the big companies, big tech companies may not do it, they could be state-sponsored actors, they could be lone wolf individuals who may want to spawn this kind of agent capability in order to hack through systems. So which is why you need to use more AI. And the second one, when it comes to the frontier models itself, the thing we are hearing about this particular individual is a company called RSI, which is recursive self-improvement, which means the models have gotten to a stage where they don't necessarily need humans anymore in order to improve their capability.

34:32So the frontier models can work within themselves in order to improve the capability. So what are you going to do in this particular thing? So in this particular scenario, which is if an AI has reached the point where it's recursively able to self-improve itself and make decisions on top of that for its energy needs, for its objectives, etc. Then that's the point where we need to be extremely cautious, I would say, on the development of this and say, okay, just because it can be done, should we be doing it? So one example, which Demis Hassabis has a really good talk about AG. Who was that, sorry?

35:06Demis Hassabis. So he's the founder of DeepMind. Oh, yes. Okay. Yeah, we'll pull that in the show notes so people can check it out. Yeah. Okay. He's really a pioneer in this field. And his push towards AGI in an ethical and reliable manner are quite impressive. So he was the developer of AlphaZero. So this DeepMind. So his company developed AlphaZero, which was able to win the game of Go, for example, beating human components. but the main thing he describes is how AI could be the descriptive language for biology and they designed something called Alpha Fold which was able to predict the folding structure of proteins so on average it took about a human would take about a PhD of six years in order to predict the folding structure of a single protein and Alpha Fold was able to predict a thousand protein structures within a matter of three to months.

36:02So that's the scale we're talking about. The implications of that. You won the Nobel laureate for that, I think, didn't you? I think you won the Nobel laureate prize. He did win it. So this is the next version. So they've got a next version called 4.3, which is able to model the behavior of every protein known to man. So it's able to do that at scale. So he speaks about something very interesting in this moving at pace matter when it comes to frontier models. Yes, it could be the descriptive language for biology and we could design you know better vaccines we could design a you know potentially cure for cancer and many other diseases in the future but you could also use it for you know nefarious purposes you could define design you could design you know super viruses you could design mutant genes you could design all these kind of things so whilst it's good to see the opportunity you always need to look at the risk these technologies can create because they're all of dual use.

36:58So my overall conclusion in a long-winded way is I think fear-mongering is a big thing, which is lots of people are spreading misinformation about AI proliferation, largely because they don't understand how the technology works. But that being said, you always need to take a cautious approach and ask the moral and ethical questions of just because we can proceed at pace, is that the right thing to do? But the crazy thing is just how fast the picture is developing you know from not just the safety conversation the technology is growing moving fast of course but all everything the whole picture is moving so much faster than any other innovation in history so it's it's a it's a hard one to stay on top of but okay let's get back to the defense sector i'm really keen to understand that that sort of where robotics meets meets ai i think that's where there's some terrifying technology going to be developed for battlefield tactical purposes do you see what i mean so are you guys seeing are you personally seeing ai being combined with robotics to you know develop battlefield type technology at the moment is is that going on yet in the defense sector with one of you any of your competitors or yourselves or what is that happening is all is all you know is sort of like i'm i'm talking about robots fighting on the battlefield is terrifying stuff but surely that is something that that's going to become realistic in the future aj isn't it so the most rapid innovation we are seeing in in in the robotics and autonomy sector when it comes to battlefield technology is in the drone market and one of the one of the front lines for this is is ukraine of course and that's where we are seeing innovation not just start in cottage industries but in people's personal kitchens and garages and everywhere else because the the threat is real and if if they are being attacked on a on a day-to-day basis with the new and novel drone technology then they need to create the countermeasures and they need to create the defensive position in order to counter these kind of threats so that's where we see autonomy ai based autonomy implemented more and more in a real-life environment out in the front lines.

39:16So I think Ukraine is really the innovation hub and the testing ground for this kind of technology where it's being done in anger, per se. When it comes to wider development of this robotic battle technology, as you're talking about, in Thales, of course, we always follow these true AI principles. So again, it depends on the ethical constraints and the policy within the state itself where you are being deployed. So Thales, as you know, is a global technology leader and it's based in more than 60 countries. The ethical policies and principles largely depends on the geographic region where it is at.

39:55But the overall perspective on the group is we always follow true AI principles. With respect to the wider world on where this technology is being produced, I think the greatest advantage we are going to see in terms of physical AI is what I call it. which is robotics, etc., is going to be more in the industrial and home sector. So Elon Musk has released something called Optimus 3, which is the next generation robotic technology. Oh, I've heard of that. Human-level dexterity is what they're promising at the moment. So imagine the kind of positive effect that could have if designed in the right way, with the right ethical principles, to taking care of the elderly or doing house chores, etc.

40:41Or in an industrial setting, helping with manufacture, reducing the safety risks, particularly where it's dangerous environments. So I think that really is where the physical AI is going to flourish rather than any kind of military technology. Having said that, there are military-based equipment which are under production, as we see, primarily US-based companies like Boston Dynamics. So they've got these robot dogs, which you have seen. So they're making leaps and bounds towards the development of that technology. But I think we are still a while away from, if you are imagining a Star Wars clone-based war, then I think we...

41:18Two minutes. Yes. So I think that is a bit unrealistic at the moment. Because often what we see is the dynamics in the battlegrounds are completely different when compared to idealistic lab-based situations. so yeah yeah but do you think that ai and and that leaps because you know it has leaped forward in the last say 5 10 you know you've been in this space for a long time but do you think that the the recent innovation around ai is a genuine genuine inflection point in warfare or is it just simply a bit of an acceleration of what already existed so when it comes to warfare i think a lot of the strategies and tactics and everything else which has been developed over many, many, you know, many, many hundreds of years, I would say.

42:04So all that still holds true. I think where AI will come into play is to firstly help on the back office function front, because defensive prices are usually so huge that knowledge management becomes a very big issue very quickly. So rather than thinking about dystopian futures where, you know, AI takes over and Skynet, I would say the most productive environments are firstly in the back office functions, which is aligning your data, achieving better acknowledgement, and making better decisions. So ultimately, it is leading to, it is indirectly leading to a very direct impact, which is you're able to make better decisions in order to defend your own nation.

42:48Yeah, of course. And operating more efficiently at an operational level, I suppose, as well. Where is AI having the biggest impact at Talus, AJ? So, Athalis, the biggest impact I would say is in augmenting the human decision-making process. So, we use these terms called sense-decide effect. So, particularly in the defense sector, these are the three main components you need to have when you're deploying technology, which is you're able to sense the particular environment you're in. So this is collecting data from all the sensors, which sensors you would deploy in operational theater. You're then able to fuse that data.

43:27Often these data sources are quite heterogeneous. So you need to be able to fuse the data in the right way so that it makes sense. And then you're able to reach a decision. And that's where the human in the loop comes in. And that's where a majority of the time is wasted if you have extremely large amounts of data or inconsistent data. so that's where that's where the that's where ai can make a maximum maximum impact within defense and it is making that for thalus and then when it comes to effect again it's helping you go through that effect chain in order to augment the humans so it's always in terms of think of it like a companion for for a human operator or teams of human operators yeah yeah absolutely i gotta be honest i know it's going back to what you said earlier on though but i think having that that sort of AI assistant is very attractive, isn't it?

44:20You know what I mean? It just allows you to cover so much more ground, make better decisions, be more productive. It's an exciting prospect. And obviously Optimus is exciting as well because you never have to do the dishes again, which is fantastic. I'm sure they'll be very expensive, those robots, though, when they come out. But we will see. But I just wanted to talk about the internal returns from AI then. So obviously I know you were probably involved in product development and the outward facing elements of AI innovation, but I'm sure you're involved in internal endeavors as well. Are you seeing sort of good internal returns in things like IT spend within the organization, HR endeavors?

44:59Are there use cases for artificial intelligence, technology, large language models being rolled out across the organization for inward looking endeavors and operational facing endeavors as well, AJ? Yes. So we have an enterprise AI function, which is solely dedicated on doing exactly that, which is how do you deploy AI technology in order to make our lives easier internally. So we've done, Thales has done a number of things to support this initiative. We have authentic AI solutions for engineering functions, because Thales, as you know, is heavily based on engineering principles. The introduction of AI companions in order to help you through your engineering process, making sure that all the checks and balances, the compliance and the technical rigor is in place through the use of these companions makes a huge difference.

45:49And he's already showing really good signs of improvement. The other thing is day-to-day processes like checking your emails and summarizing your meeting minutes, making sure what your next actions are, checking for the amount of leave you've got left and notifying the users. So all these can be done through co-pilots. So there's a lot of co-pilot technology available already out there. So this is not even a case of reinventing the wheel or designing something from scratch. So your existing platform providers are able to provide many of these co-pilot kind of capabilities. And that's where you can really gain, really see the low hanging fruit, as I call it, to see that ROI impact.

46:29But ROI impact is again a very interesting topic because often people define AI success metrics, which is that's almost like patting your own back so people say okay i'm going to create a new metric for called an ai success metric and show that the token cost has gone down that's fine but to a board level that doesn't mean anything so yes the token costs are gone down but why were the tokens in the first place so i think the real roi comes in from when you are you're able to define your business kpis quite clearly and then relate to your ai investment to how the business kpis are being addressed and improved in a significant way.

47:05So if your business as a whole is improving after AI implementation, then that's the automatic way in which you can measure return on investment rather than defining spurious AI metrics. Yeah, no, for sure. I mean, they call it tokenomics, don't they, in terms of the consumption versus cost versus productivity for the business. I mean, I think this is obviously very new territory for a lot of people. what what what general do you have like a framework in place for that then in terms of measuring roi from the areas where you know our ai technology has been rolled out how do you generally aim to get that right aj so as you say this is this is a this is a new kind of venture for many organizations who are on the journey so my technical answer to that would be yes so every every AI technology we implement, it could be a co-pilot, it could be a chat interface, it could be some kind of processing thing.

48:09We always make sure that there's observability on the process. And what that means is you're able to monitor all the metrics associated with that right from you use a prompt, token costs, through to, you know, reliability of the system itself, latency, availability time, etc. So you're able to monitor all these different things. But the thing which fascinates me most is I was lucky enough to be in a talk where Fei-Fei Li was speaking. So she's known as the godmother of AI. So I was in Vegas recently and she was speaking live. So she said something very, very interesting, which is lots of companies are moving towards AI for productivity.

48:47But are we really thinking about AI for shared prosperity? This is something I think every company needs to consider, which is you're in this rat race of productivity, ROI gains, etc, etc. And in some cases as we have seen in some large software companies redundancies because you're letting go of many software software design teams for example i think the the thing companies need to really think about is is is the ai implementation you're doing internally resulting in that shared prosperity which is easy making people's lives better so again like it like i spoke about the true ai principles in the in the defense world you kind of need to apply it internally as well so it's a big cost for a lot of organizations who don't have a handle on managing this stuff isn't it isn't it so but that's a really interesting interesting topic that we could dive into more but i know i'm just mindful of time so what i want to ask you is is about you did touch upon it there it's just in terms of you just job cuts coming as a result of the proliferation of ai displacement what is your forecast for the next five years is this going to go is this going to escalate you do you see a significant you know job cutting spree before maybe new careers are created?

50:00What was your forecast for the next five years, generally, for businesses and, you know, people losing their jobs because of AI? That's a very interesting question. So firstly, I would say, AI is a civilizational technology, in the sense that, like, there were, you know, people talk about the factory floor being created for the first time, and then people thought all their jobs would be lost, but it only created new jobs. So whilst that is the case, the impact of AI and the pace at which it's growing is civilizational in nature in the sense that, yes, there will be some job losses moving forward, particularly where the tasks are repetitive and can be automated with a high degree of accuracy.

50:47So we will reach that inflection point where those kind of job losses are inevitable. But That being said, it comes back to the shared prosperity point I was making, which is how are employers aligning their workforce so that they are re-skilled and they're refocused on the jobs which will make a difference to the company as a whole rather than doing repetitive jobs. So I think that's the key. There's a really good example from IKEA, I think recently, where they deployed authentic technology in order to do most of their customer experience management. So that left many, many thousands of people potentially without a job.

51:22But that's not what IKEA did. So what IKEA did was anticipated this already and looked at what is the market where we are making most amount of money. And that was in the interior design sector. So they re-skilled all those employees into the interior design kind of space and made them interior design engineers and made a billion pound profit on the back of that. So that's a really good example of how if management takes the right moves into getting this technology implementation right, firstly, and then planning in advance the effect it's going to have on your local population. In this case, it's the staff.

52:02And putting in the measures to re-skill and re-deploy them to where it's actually required for the business, then it can actually be a really good force multiplier for any organization. yeah i think it's going to elevate the quality of jobs isn't it you know what i mean but as exactly as you said aj in the industrial revolution it was the same thing in the digital revolution around the turn of the century it was a similar pattern is that there was an initially there was a you know a lot of jobs were destroyed by this technology innovation but ultimately eventually it's like a j curve it kind of goes jobs sort of get the they we lose jobs but we create new careers new vocations which ultimately leads to a net gain in jobs you know this could be different it does seem to be a more dramatic impactful force year with ai but you know there's so many i think we all need to maybe not focus on the doom stuff so much and look and think about the the incredible benefits we could get from this technology if we play our cards right if we if we regulate we've regulate it and yeah you know we have to use it to you know for the for the greater good of the collective of of the human race of the planet as as the godmother of ai was was pointing out so two last questions aj what sorry to ask you a negative question again but what worries you most what are you concerned about i think what what i'm most concerned about is as you said in the last thing, there's a huge amount of opportunity and we need to grab that opportunity, taking into account all the risks and going ahead with that.

53:46Whilst some nation states and some societies may adopt this approach, the world is a very big place and there's lots of people and there's lots of nation states with different intentions, different approach to ethics, etc. So whilst some nation states may take this approach, this ethical approach, other nation states may not take that approach which will force force us to you know keep up with the keep up with the game or you know you're left behind so what is me most is we get into this cat and mouse game and we are forced to proceed with the rapid development of this technology in a completely unbounded way without really understanding what civilizational and societal impact is going to have on us as a whole and the earth because AI revolution needs an energy revolution and I'm not seeing an energy revolution at the moment whilst there's some efforts to create you know contain fusion reactions it's not nowhere near the pace which we require in order to fund these AI ventures so it's that it's getting the balance which worries me most.

54:51Yeah that's one of the biggest limiting factors I think isn't especially in the UK where we've got high energy prices which So that's a really good point, actually. What are you most excited about, AJ? What I'm most excited about is the combination of really powerful AI models, like the latest frontier models, which are able to, you know, design better protein-holding structures, design better vaccines, design better mathematical equations. But when you combine that with another technology, which is, I think, going to be a complete game changer for the entire world, which is quantum because quantum is able to deal with very large spaces and solve extremely complex problems so if you are able to if you are able to combine the power of ai with the power of quantum technology when once it becomes stable then the the opportunities are limitless you could you could understand the universe as a whole i mean some of the biggest questions ever asked the universe you know why was the how did the big bang form or what was what was before the big bang what is time so you could answer you could simulate entire universes within within a quantum environment and do what if scenarios yeah that really excites me so i know that's it may sound like it's way out there but i don't i don't think we are very far away from a scenario where we are able to answer some of the most fundamental fundamental questions in the universe and the combination of those two things will will provide that yeah i think that confluence of quantum coupled with you know the increase you know increasing sophistication of of the models that are getting built and just ai and large language models and agendic ai but when those two meet i mean that's going to be very exciting but also terrifying as well listen you know how far do you reckon quantum is away in terms of being accessible to normal people and businesses how far away are we from i think the biggest biggest problem in quantum at the moment is to be able to stabilize those quantum bits to work at room temperature because you have to get it really cold or they don't perform as well.

56:56Yeah, it comes down to energy again, isn't it? Yeah, so there are some technologies coming up. Microsoft has released something called the Majorana chip where it's designed a new kind of architecture where quantum computer performs at room temperature. But I would say the next five years is something to keep an eye on for this technology to become widely available. Yeah, for sure. AJ, it's been an absolute pleasure. I could speak to you all day about this stuff. It's very interesting stuff. But look, fascinating conversation. Where can people reach out to you, find out about what you guys are doing at Thales and keep track of what you're posting and stuff?

57:33Where's the best place to find you? The best place to find me personally would be to follow me on LinkedIn. Everything I see which is interesting and home projects I've done or what Thales is doing, I make sure to post it on there so you can keep up to date with that. Apart from that, Thales has various media channels all across social media, right from LinkedIn to some of the most prominent social media outlets. So be sure to subscribe to Thales to get to know more about what it is we do. Absolutely. Fantastic. Well, thank you so much for coming on the Tech Leaders Podcast. Well, thank you. It was a pleasure.

58:16that was a fascinating conversation i have to say it was also slightly terrifying at times as well loads of takeaways i mean the biggest thing that stands out for me is just how quickly ai is moving beyond software into the physical world i think you know in terms of robotics and you know when you start combining ai with drones and robotics and autonomous systems the possibility are incredible, really, but also so are the consequences of getting it wrong, which I thought was a really important point. But what I did find very reassuring was AJ's clear emphasis on keeping humans involved when those consequences really matter.

58:55As these systems become more capable, deciding where we draw the line is going to become increasingly important. Okay, as has recently been shown with the hugging face incident, Yeah, and there's some of that incidents as well, actually, where AI agents have gone out of their sandbox and caused mayhem. But yeah, the emphasis on safety was very reassuring, is what I'm saying. So perhaps we're not quite heading for Skynet just yet. But, you know, there certainly are some big decisions and some big questions ahead. But look, fascinating conversation. Huge thanks to AJ for joining me. Thank you so much for listening.

59:31And please don't forget to give us a subscribe, a review. It really helps us bring you more amazing guests in the future. But thank you so much for listening.

1:00:07AI technology capability. And on the last point, Be Digital have developed a cutting-edge AI readiness assessment, which provides tech leaders with a platform they need to make well-informed decisions about AI adoption strategy. Go to Be Digital UK to find out more and get in touch.