Anduril with Gokul Subramanian

0:00Anduril is a technology defense company with a focus on drones, computer vision, and other problems related to national security. It's a full-stack company that builds its own hardware and software, which leads to a great many interesting questions about cloud services, engineering workflows, and management. Gokul Sabramaniam is Senior Vice President of Engineering for Software Programs at Anderil Industries. He joins the show to share his knowledge of the national security problem set, how Anderil operates, and what the company has built. This episode is hosted by Sean Falconer. Check the show notes for more information on Sean's work and where to find him.

0:51Gokul, welcome to the show. Hey, Sean. Thanks for having me. Great to be here. Yeah, awesome. Actually, I probably should have said welcome back to the show because you were actually here like four years ago. That's right. In the middle of COVID, actually, I recorded it from my apartment. so it's great to be here in a proper studio with the right recording setup. Yeah, for sure. I think people's home equipment setups have really evolved in the last couple years because so many people working for remote. That's exactly right. I remember slowly upgrading my home office over that year. Yeah, exactly.

1:19Went from like a laptop on your kid's bed to like actual like functioning office. You start thinking about your background, what are people going to be staring at? I got to like, you know, put some posters back there. You got to get the lights, you know, beaming at your face. So yeah. Yeah. I remember I was at Google at the time of the start of the pandemic and like the senior director of engineering, like his first meetings were legitimate. He was sitting on like a child's bed, a child's bedroom. And I think I'm sure it evolved significantly since then. Yeah. And for us, you know, we're such an in-office culture and now we completely are back to that in-office mode that, you know, we were wholly unprepared.

1:53And I remember during that COVID period when the rules hadn't been established completely yet, you know, we had people taking tables from the office and driving them to their houses because they didn't have a home office set up and taking their monitor from the office back to their home. It was crazy. Yeah, I went through similar things. How does that transition to bring everybody back to the office? I think it's been good. I mean, we're such a tactile organization in the sense of the things we build, you can feel them, you can touch them. Even as a software organization, we fly our stuff at the range regularly.

2:21So I think that it's been much needed for us. We're so collaborative across hardware and software. I will say, though, that COVID period really made us invest in remote operation, remote telemetry, remote management, which enables us to now run test sites out of state, right? So when we were recording during COVID, our test site was 45 minutes from the office, but that's just not possible anymore. You know, our new test site is, we have two, one in Nevada, one in Texas. The one in Texas has one of the largest runways in the United States, It's the entire United States, and we're going to be flying our airplane next year.

2:57So you just can't ship all your engineers out to the test site. And so we've had to really double down on that remote testing capability. I would think that it's probably forced a lot of organizations to sort of like upgrade, I guess, both like their asynchronous communication and how they think about like remote work, remote test sites, like even organizations where maybe historically you had to have some sort of field engineer essentially located at a customer site to get them set up, you're going to have to change things significantly in terms of not only how you work, but also probably the support systems that need to be in place to deploy software remotely and monitor it essentially.

3:36100 % true. And yeah, it's kind of a platitude now, but necessity is the mother of all invention. And so when you're basically told... And during COVID, it wasn't like work stopped entirely, but we had a safety protocol. So it's like, one engineer can go to the site, and there will be four test engineers, and that's it. And they have to be separated by this much. And it couldn't have prepared us better for hitting the real world and scaling to these, you know, more realistic, more safety conscious government systems. So given that you're on the, you know, show four years ago, and maybe not everyone listening to this had checked out that episode, I do recommend And if you are listening, you should go back and listen.

4:15But can you give, before we go too deep into things, can you give a little background on Andrel and what exactly does the company do? Yeah, happy to. So Andrel, we are a defense company. We're a defense technology company. And our thesis is we want to do for defense what SpaceX effectively did for spaceflight, which is if you think about the 60s and 70s and 80s, defense was the place to be. I mean, there was a real national need. We were in the thick of the Cold War and all the brightest minds were going into defense. What we have happened over the last 30 years is many of those minds shifted to working in commercial, whether they went to the big bang companies or otherwise.

4:53And what Andrew wants to be is a beacon to all of those people to say there is a place for you to come back and work on these really critical national level problems, problems that only a nation state can tackle. and you can do it in a culture that matches the modern world. And you can do it with access to technology and you can really push the needle. And we hope to lift the boat across the industry. You know, I think that the existing defense primes are feeling this as well. And I wish for them to be able to access that talent as well. And there's a whole set of companies that are coming in behind us.

5:26And so we really, I mean, the broader thesis of Andrel is reigniting that arsenal of democracy because we feel that we're moving into a world of great power competition. That's a bunch of jargon, but if you just look at what's happening in Ukraine, if you look at what's happening in Israel, if you look at what's happening in the Middle East, it's very clear that America needs to continue to hold its role in the world. How do you go about appealing to that talent pool? You want to recruit the best engineers, product people, best people across essentially any functional area to be able to have, you know, sort of, I guess, like fan quality talent pool, but at a non-fan company?

6:04Yeah. And actually, you know, it's very interesting because we don't want, quote unquote, fan quality talent across the whole company. Actually, one of our biggest recruiting draws is the diversity of talent, the diversity of opinions across Andrel. So I'll give you a statistic, 20 % of our staff, maybe 25 % are veterans. So they formerly served in the military. You know, I would say another 30 % come from bang companies. I would say 30 % come from traditional defense companies. And I think it's that intersection that really excites people. And it's feeling like you get to work across talent and people that you wouldn't meet otherwise.

6:39You get to work across, you know, for a lot of people that I talk to that work in pure SaaS companies, you know, getting to touch hardware is a huge draw for them. And getting to work with the hardware side of the company. for the hardware side, getting to work with fan quality engineers is a huge draw for them. So I think that's been the key to our success is that intersectionality of disciplines. Okay. And then I want to talk about, you know, Lattice and Lattice SDK and some of the things that you're doing there. So just to start, like, what exactly is Lattice? How do you typically like describe that?

7:10Yeah, you know, there's a philosophical definition. And then there's like a very specific technical definition that we can go into, you know, at the highest level, if I just take a step back and think about history, the defense world has been very good at building hardware. You know, I call these appliances. So like, if you think about what the defense community has done since the Cold War, they build airplanes, they build submarines, they build, you know, battleships, and they think from a very hardware centric viewpoint. And the net result of that is that these systems are designed in silos, and they don't talk to each other in any way.

7:40They were never designed to talk to each other. It's nobody's fault. Everyone did great work. But from the beginning, they were never designed to talk to each other. And what it means is that if you think about how a modern conflict will evolve, this is a huge Achilles heel for the United States. And you can see that the military has been talking about this for literally decades. So if you think about what Andrel set out to do is inverting that paradigm. So you build the software as the core from the get-go, thinking about every different domain that you'll want to work across and build a common core.

8:11And then you design the hardware to take advantage of that software from the beginning. And the net result of that is you get huge economies of scale. So it means that I don't have to reinvent the same code. You get higher quality. You get a better cost structure, which is really important to our customers. And so fundamentally, that's what Lattice is. It's a common way we think about how we build systems, whether they're command and control systems, laptops and servers we're putting all around the world, or they're robots, whether it's our school bus size submarine that's coming online and going through sea trials right now.

8:44It's our spacecraft that'll be launching next year. They're all sharing a common DNA and a common code base. And that means they work together. And that means that we get higher quality. And then to get to your point about the release that we're super thrilled to do today, what we realize is that that shouldn't be locked up to just Andrel. And so we are releasing that Lattice SDK, that capability for anyone to take advantage of. And we have a thoughtful onboarding experience. And we're releasing with a number of partners from the get-go who are all taking advantage of Lattice to be able to do those sorts of missions.

9:19And Lattice itself, so if we get into the very technical definition, breaks into a microservice architecture, and I can go into the layers of that architecture. And what we're offering is for people to take that a la carte. So they can take different parts of that architecture, whether it's our networking capability, our command and control capability, our autonomy capability, and they can start integrating that into their systems. And if I'm someone who has some of this, like, essentially legacy hardware, I already have, you know, airplanes and all these other things that military organizations would build up.

9:48Can I use Lattice or do I need to essentially be building with Lattice and then building with essentially the hardware instantiation of that from the ground? Yeah, no, I mean, we do this all day long. We integrate with legacy systems because the reality is, and we knew this from day one, you know, we're super thrilled to see the government adoption of our hardware. But the reality is the vast majority of the kit that the average soldier has access to is not handle stuff, at least not yet. And so we need a way to onboard other third-party systems. And so the short answer to your question is, yes, it's totally possible.

10:22And there are layers of integration with Lattice that folks need to take advantage of. So if you want to be able to send data into our mesh network so that that data can route seamlessly wherever it needs to go, taking advantage of Starlink and mesh networking technology, you can do that. And you can buy a compute module from us, or you could actually use your own compute module and drop our software on there, and you can be onboarded into the network. And you can go all the way up to if you build, and this is one of our partners, build unmanned surface vehicle, and they want to integrate more deeply into that stack, they can go do that and they can work with us.

10:58We can give them recommendations on hardware if there are any changes necessary in order to get that deeper integration. What's the actual architecture of Lattice look like? And how do I go about using it to essentially start a project with it? Yeah, that's a great question. So at the base layer, you can think of it as we have a set of OS-level primitives that we recommend. You don't have to adopt them, but we have thoroughly tested against them. So Red Hat operating system, for example, or Prometheus for how we do telemetry. So those are a base layer of Lattice. Then if you get into the domain-specific layer, you've got three basic components.

11:38The first is a networking capability. The second is what we call command and control capability. How do I know where you are and send you basic commands? And then the third layer is the autonomy capability. So then how do I go from sending you commands to letting you move autonomously within the boundaries of a plan or a set of behaviors? And you kind of think of that as like Maslow's hierarchy. It's like, I got to be able to talk to you. Then I got to be able to send you like basic commands, like move here, fire this missile or whatever. And then I got to give you high level intent. And that's kind of the direction we see technology going.

12:11So in terms of like the commands, am I essentially, is it like sort of a client server model where I'm sending commands from the server and the client essentially is the piece of hardware that's going to react to the commands? And then is the client sending information back as well? Yeah, so we think about the Lattice Stack as two different deployment targets. So the first target is what we call the node. And the second target is what we call the robot. The thing I want you to take away, that the main point here, the node has the human involved. So it is where the human, you or I would interface into a set of systems, whether they're manned or unmanned.

12:44And once you realize that there's a human involved, it creates a huge set of follow-on, knock-on technical decisions. For example, you've got to have a user interface because a human cannot talk via APIs. And if you have a user interface, you've got to have a database that's going to store a ton of data so that a human can rewind and go forward and all those sorts of things. So that's the node. The other thing we have is called the robot, and that's a different deployment target, kind of different stack that gets composed. Now, the thing that connects that is what we call the mesh. And that's the announcement that we made today.

13:14Once you are on our mesh, and the mesh is what's called a pub sub network, it's a publish subscribe network, anybody can publish topics on that mesh, anybody can subscribe to topics on that mesh. And so if you are a node, which is where the humans operate and interact, they can subscribe to a set of topics, I want to know where the submarine is, I want to know where the airplane is, I want to subscribe to the radar data coming off the airplane, and that data will get seamlessly routed back to you. You know, we do things like, you think about a typical cellular network thing your phone uses. There's a concept of quality of service.

13:46So we will ensure the most important data comes to you first, and then the second most important data, so on and so forth. And that's how an end user or even a partner gets onboarded into the stack. And that mesh takes care of things like security, it takes care of things like quality of service, et cetera. So that's really the fundamental piece of the Lattice stack that then lets you access all the higher order things. In terms of that PubSub, what are you running underneath the hood? Is that something that you rolled your own or are you using an existing PubSub provider? Yeah, this is a core piece of IP for us.

14:19We spent a lot of time studying existing providers and we ended up rolling our own. It's one of the few areas we've patented as well. And so it's kind of custom implementation. we do use protocol buffers as the mechanization, as a serialization mechanization. And then we use things like, we work with all sorts of open source authentication systems. So if you've got PKI or if you've got OAuth or any sort of those, we can integrate with them. But the actual networking layer of the networking protocol, that's all custom built by us. And since you're using protocol buffers, are you using things like drpc for communication?

14:55Yeah, precisely. So you can send data over protocol buffers over the mesh. You can publish data, but then you can also call commands. So if you want to task something, you want a direct ability to expose what I can task and then be able to leverage that task and get an act back. And that we would use something like Google, a gRPC. Okay. So you have essentially this data flowing back and forth between these nodes in the server. Where does this actually run? Are you running these within your own cloud deployments? How does that sort of stuff work? Right. So one of the fundamental premises that we had to make and design for is that we cannot assume availability of any cloud stack.

15:39And so the entire system is designed to be agnostic to both any individual cloud vendor. If we have them, we'd love to use them. And also the ability to run on-premises. The node itself can go all the way down to the form factor of a laptop. and all the way up to the form factor of a data center, an on-prem data center, or in the cloud. The robots tend to use NVIDIA form factor, the NVIDIA Jetson family of chips, but we're compatible with any sort of embedded form factor that we can run on. And that's why I call out that distinction, because you've got very different compute available to you, depending on if you're a robot versus a node.

16:18And within this kind of like mesh infrastructure, where this information is flowing between all these different nodes. Is there essentially a component of AI-based decision-making that's also part of it for supporting things like defense operations? Yeah. And this is an area that we're all kind of leaning into, that DOD is leaning into this as well, but we're all trying to figure out where do we draw the line and how do you do this in a safety conscious way? So there's this concept that is basically traditional systems have been human in the loop. So the human is directly inside the decision-making loop.

16:53The idea with AI and a lot of the latest generative AI technologies is how do we move the human from being a blocker inside of the loop to becoming sitting on the loop? So they are observing the loop entirely. They can stop at any time, but they are not directly an actor within the loop that they have to do something. So, you know, the closest example I would describe is, you know, if you've got a Tesla, for example, you can put your car in self-driving mode. You are on the loop. The car is going to make a set of decisions, but you need to be watching it extremely closely, and you should be ready to take over at any time.

17:25And so as it comes to AI, that's the way we're starting to think about it. And we're starting to lean into a lot of new use cases. Your viewers may have seen our announcement with OpenAI, our partnership there, and we also have a partnership with Palantir. And so we're really excited to lean into that. And then the second thing I could say on AI is, you know, what I really think about, this is my personal opinion. If you think about what's happened in the last five to 10 years, Since the advent of transformers, I don't know that the innovation has really been on the model architecture side. And there's certainly been new model architectures and things like that.

17:57But the real breakthrough has been on the data side. And it's been moving from increasing the volume of data that's been available to train these models, multiple orders of magnitude, hundreds, if not thousands, X more than we had before. And how did we do that? We got the labelers out of the loop. So what we basically did is we created a situation where the data could be self-labeled. To give you an example, right? I'm going to look at a Wikipedia article. I'm going to take the first paragraph, feed it to the model, ask it to predict the second paragraph. I don't need a human labeler in the loop because I can just check, did you guess the second paragraph correctly?

18:29And by doing that, you can explode the amount of data you have available to train. That's our goal, which we are sitting on top of, we hope, a hugely valuable data set for defense purposes that no one else has access to. Only the US military has access to this data. And we want to really treat this data with respect with the right authorities, but start to use this data to start being able to generate the next generation of AI models for our use cases. Yeah, I think the primary bottlenecks today with both on the model side, evolution of the models, and then also with building AI applications are really like data challenges.

19:06That's right. Essentially, if you're hitting the limits of what you can scrape publicly, then you need to go find data somewhere else. And then there's challenges around generating purely synthetic data for training in terms of degradation of model performance and integrity. And then even on the other side of essentially building AI applications during prompt assembly, we now have the advantage of really large context windows. where, but then it's like, how do you provide the right context, you know, with the right information in real time to generate the most reliable response. And both of those things are really around essentially data problems.

19:41Yep. Everything has followed from how do we get access to high quality, huge volumes of data. And then you figure out like, okay, well, I need to train against this huge volumes of data. So I need these huge server centers. I need to figure out how to network the server centers together. And I need to be in a, there's this concept of coherence when you're training, but all All of that is second order that follows from, can you get access to these huge volumes of automatically labeled data effectively? So the big idea that we're after is how do we do that for our use cases, this national security problem set, where we think we are closest to those problems.

20:15And as we talked about with our mesh networking architecture, we're just constantly sucking up that sort of data. Yeah. And even outside of AI, like what are some of the engineering challenges you face with trying to scale Lattics to handle like the complexity of these like dynamic environments where you got like, you know, classified servers, battlefields, you know, you probably know all the terms better than me. But I mean, these are like not sort of the typical places that most engineering organizations are deploying things. Yeah, I'm so glad you asked this question. I think this is what gets people fired up when they come work here.

20:49And they see the scale of our problems very differently than what you'll see in big tech. So I think one of the fundamental building blocks for us is we build for a decentralized world. So we have no concept of all the data. There's not gonna be one server to rule them off where all the data will come back. You know, if I go back to my Tesla analogy, every Tesla car can communicate back to Tesla headquarters and they can kind of pull all that data together. We live in a world where our systems have to be designed to work against degradation of communication and reforming these meshes wherever they are.

21:23Maybe they can connect back to Washington, D.C. Chances are they can't in the real world. And they'll have to work in a dynamically evolving kind of decentralized paradigm. And what that means is the way you design applications has to be designed with that in mind. Let me give you an example. Right. If someone sends you a text message on your phone. and your phone is off, you have an expectation that when you turn your phone back on, that message will be received, and you'll be able to see it. Many DoD systems are not architected with that in mind, you cannot make that fundamental assumption. Furthermore, you can't make the assumption that that DoD system can ever see Washington, DC, it may be able to see another DoD system if you know, the bad guys aren't jamming it.

22:06And so how do you architect and build applications for that world, that kind of becomes our seminal challenge. And if you think about the Lattice SDK that we're releasing, it contains those lessons learned that we've spent seven years learning those lessons about how to build for the decentralized world. And we're trying to share those lessons. And it's a different paradigm fundamentally. How do you test for those types of environments? Yeah. So, I mean, this is where the real world test range is incredibly critical. We have a really robust simulation architecture that we're continuing to lean into.

22:36And one of these ideas is that we can't have one central team do it all. So we have people building the architecture. So if you're interested in that kind of work, you can come here and work on that. But then we actually have people embedding close to these problems, really getting in the weeds with the end users, often traveling to the end users, working with them when they do their testing. Remember, I mean, we've got to build our stuff so that we're not there when it's actually being used. So we train with our end users and learning from them what those edge cases are. And then we bake it back into our integration tests in our simulation environments.