NVIDIA’s Marco Pavone on AI Simulation, Safety, and the Road to Autonomous Vehicles - Ep. 260

0:10Hello, and welcome to the NVIDIA AI Podcast. I'm your host, Noah Kravitz. If you've been in San Francisco, Phoenix, or a number of other cities recently, no doubt you've seen cars navigating the streets without anyone in the driver's seat. Robo-taxis and other autonomous vehicles are becoming a part of our lives, whether through prototype testing on public roads or driverless cars ferrying passengers across city streets. Autonomous vehicles hold a lot of promise, but first and foremost, they have to be safe. This is where advanced simulations powered by AI can help. Here to explain the ins and outs of autonomous vehicle safety and simulation, and why it's so important to all of us going forward, is Marco Pavone, Senior Director of Autonomous Vehicle Research at NVIDIA and Professor at Stanford University.

0:55Marco, welcome, and thank you so much for taking the time to join the NVIDIA AI Podcast. Thank you for having me. So, Marco, if you don't mind, could we start by having you tell us a little bit about what your role is at NVIDIA and Stanford as well, if applicable, when it comes to autonomous vehicle research, safety, simulation, and everything else we're going to talk about? Sure. At NVIDIA, I lead autonomous vehicle research, and our research spans a number of different research tasks. First of all, we work on developing new architectures for autonomous systems. We work on developing AB Foundation models that can empower the full development lifecycle, the full development program from the onboard stack all the way to simulation, data curation, and even safety evaluation.

1:49We work on enabling high fidelity behavior and sensor simulation. And we also work on developing tools to ensure the safety of AI-based stacks. At NVIDIA, I focus primarily on ground robotic systems, namely self-driving cars, while at Stanford, I focus primarily on autonomous aerospace systems. Oh, fascinating. Well, we'll have to have you back to do an episode on the potential future of flying cars, putting those two things together. So in the introduction, I mentioned robotaxis. It was something that came to mind for me as an easy way to kind of visualize what we're talking about. But as I understand it, your purview and what NVIDIA is working on more broadly isn't just fully self-driving vehicles.

2:35There's more to it than that. Yes, the field of autonomous vehicle is very diverse, both in terms of the technology from semi-automated systems all the way to fully automated systems, like for example, robot axes, and also in terms of application domains, personal mobility, namely self-driving cars for people moving in cities, all the way to freight transportation, autonomous vehicles for agriculture, for construction, and more. Right. Excellent. So let's start maybe by talking about Halos. NVIDIA recently announced, well, I'm going to leave it to you, something called Halos. Can you tell us about it and how it relates to AV safety?

3:20Yes, of course. So safety obviously is paramount for autonomous systems. They are safety-critical systems. And HALOS plays a very important role in this regard. So specifically, HALOS is a full-stack, comprehensive safety system for autonomous vehicles that unifies safety elements from a vehicle architecture all the way to AI models. and in particular halos comprises a hardware and software elements tools models as well as design principle for combining them to safeguard ai-based end-to-end av stacks the hardware and software is this in vehicle in the cloud does it span both as i mentioned before it spans everything from the vehicle architecture so basically the hardware and software goes into the car all the way to the development processes that happen in the cloud to build a safe AI system.

4:19For example, Halos comprises the world-first safety-assessed platform for AI-based AV stacks. So this goes into the vehicle, and it also comprises dedicated MLOps workflows for safety data curation. So as you can tell, it's a very broad program. Very comprehensive. Yeah, great. Okay, so let's talk about AV safety. This is one of those questions that I have a million answers to in my head as I ask it, but I'm sure none of them hit the mark. But why is safety important for autonomous vehicles? Well, autonomous vehicles are safety critical. So the consequences of a mistake, of course, can be catastrophic, including a loss of life.

5:01So obviously, you have to make sure that these systems do not pose an unreasonable risk to society. One, because this is the right thing to do. And two, because otherwise they will not be accepted by society. So this is the reason why NVIDIA, who is developing its own AV program, but is also working on developing an ecosystem where multiple stakeholders can develop their own AV programs, is so invested in pushing safety from vehicle on the way to the cloud. So when we dig into this a little bit, what are some of the key considerations when it comes to ensuring AV safety? Yeah, so I like to say that there is no silver bullet to ensure AV safety.

5:48There is no single technology or single process that is going to make your system really safe. Safety really has to characterize the full development program from design time, making sure that, for example, we train the AI systems on safe driving behaviors, all the way to runtime and deployment time. So once the system is deployed on the road, building monitors that, for example, keep in check the AI system or building guardrails, all the way to the iterative development of the system. So basically how you learn from your deployments to increase the safety of the system throughout its lifetime, potentially over the years when the system is deployed.

6:38As I understand it, and mine is very much a layperson's kind of working understanding, but one of the things or perhaps the overarching thing that makes developing autonomous vehicles so challenging is that the conditions that they're dealing with are literally unpredictable. They, you know, things, weather, driving conditions, the conditions of the roads, a kid on a bike darting out across the street, all kinds of factors that, you know, you have to account for even if you can't actually imagine them ahead of time. But can you talk about some of the common challenges that developers face at different stages of the life cycle when developing AVs and how they make sure that safety, drivers, passengers, pedestrians, everybody is prioritized?

7:24I mean, you're exactly right. So by definition, an autonomous system is a robotic system that operates in scenarios that were not foreseen at the design time and requires some level of reasoning. So how do we ensure the safety with respect to conditions that potentially might be quite different from those seen at design time when the system was developed? And as I said before, there is no silver bullet. It goes down to fundamental principles of safety engineering. So making sure that we build a stack that is diverse, meaning that comprises a component that are different and with overlapping responsibility.

8:07It goes down to the principle of monitoring, meaning that you have to constantly monitor, if you will, the health of your system to make sure that, for example, is not operating in situations whereby you cannot assure anymore the trustworthiness operation. So basically, the system needs to understand when it is outside its domain of expertise. And of course, it boils down to the principle of testing and validation. So how do we robustly test the system in a way that covers as much as possible all the most important situations in my face at the moment time? Right. And are there accepted standards or benchmarks?

8:54Or how do you determine when a system is deemed safe? So the level of safety really depends on the product we're talking about. Okay. So obviously a semi-automated system whereby the human is in charge of monitoring the system itself has lower safety requirements than a robot access. Right. The human can intervene if need be. Yeah, exactly. So there is a continuum spectrum of safety requirements depending on the level of automation. And there are well understood processes to go from an analysis of the potential risks and consequences to requirements on the stack, for example, in terms of mean times between failures.

9:42Then how basically you achieve those. Again, there is no silver bullet. It goes back to making sure that you have a safe design, for example, reflecting the principle of diversity, that you have a robust dataset curation pipeline to make sure that you train your system on safe behaviors, to make sure that you have a robust runtime monitoring pipeline that keeps the system in check, as well as a very strict test regimen to test your system using a combination of both real-world data, you still need that, even though that is expensive, and also simulation data. And this is something that is becoming increasingly available to developers, which I think is very exciting.

10:22So to talk a little more about simulation, I mentioned in the introduction prototype vehicles testing on public roads, city streets and all of that. But it sounds like that's not the only way to test and validate the safety of these systems. How do simulations work in this context? I mean, simulation has always been the holy grail of robotics more broadly. If we had a perfect simulator, robotics development would be dramatically accelerated. And of course, perfect simulator doesn't exist, at least yet. But in the past two, three years, the robotics community broadly has made a number of breakthroughs in the field of simulation, both in terms of how we simulate the environment from a sense of realism standpoint.

11:12So basically rendering images and environments in a way that is ultra realistic. It will be very hard for you to tell the difference between a computer generated image and actually a real image. And also in terms of technologies to replicate how other agents, for example, bikers or pedestrians would behave on the road. So all these technologies together have basically enabled simulation capabilities that even two, three years ago were not even imaginable. There's been quite a few years in this industry. Yes. And now simulation is more than ever part of the development process. For example, for testing, if you have a new design, you want to make sure that you improve with respect to number of metrics in a number of different operational design domains, simulation is a great tool for that.

12:09Using simulation for validation is still something that is a bit creaky because it means that you're not just interested in directional information, that is whether something is better than something else, but you're also very interested in the absolute numbers of your metrics. And so the question is, how do you trust your absolute numbers? But for example, we've been doing research on this topic on developing statistical methodologies to provide confidence bonds on the metrics that are generated by the simulator to help developing a validation case that comprises both tests on real data. As I said before, you need that.

12:50I don't see there is a way around it. but also simulation data, thereby dramatically decreasing the cost of validation. Are there specific, I guess I'm thinking of both points in the development process, but then also situations that a vehicle, I'm thinking about it as a driver, but a vehicle might encounter that simulation has proved particularly adept at? So simulation helps both with respect to testing against nominal conditions, like for example whether you have good performance when negotiating an intersection and so on and so forth but also with respect to anomalous conditions that might be very hard to see in real life and one recent work that we had that i think is very much related to your question was to use large language models to recreate crash scenarios that have happened in real life.

13:53So basically mining the police reports that have been written over the years because of crashes, and then using a large language model to interpret them, adds a little bit of generative AI, and then recreating scenarios that will be very hard to recreate just by asking designers and artists to do so. Right, right, right. So to recreate those scenarios, but in a way that they're still plausible. It's very easy to simulate very challenging scenes. The question is, are those things that can really happen, are they plausible? By basically grounding the generation in crash reports, you have a direct grounding on those scenes.

14:35Oh, that's fascinating. And we've seen how actually these scenes can improve in both testing and validation. So this was a long answer. The short answer to your question is yes. We've seen situations where we were able to generate scenarios that would have been very hard to generate otherwise without our AI-based workflows. And they have helped with improving the performance of the system. Right. That's fascinating. It's, you know, one of those things that when you look at it, you're like, oh, that's such a simple idea. It makes so much sense, but very clever. It's interesting. How does simulation, AV simulation for autonomous vehicles, how does that differ from traditional automotive simulation or even other methods of simulation?

15:12Yes. So a V-simulation has some features that make it particularly challenging and some other features that actually make it simpler than other domains, like, for example, robotics. One feature that makes a V-simulation, particularly in the context of a high level of automation, so semi-autopilots or robotaxis, is that you have to reason about the interactions with other agents on the road. Human drivers and pedestrians, bikers, animals possibly. And those interactions are very difficult to model because essentially you're modeling a decision-making agent. And you want to do that in a way that is highly dynamic.

15:58So basically, it reflects the physics of an interaction, but also the behavioral dynamics. So how, for example, a biker might respond to what an autonomous car is doing. It's sort of a ballet between the two agents. So that part is very hard. But it's very important if you really want to push the utility of a simulation, particularly for higher levels of automation. For more basic levels of automation, like for example, collision avoidance or lane keep assist functions, you don't have such a requirement on modeling interruptions, for example, which make it simpler. There are other parts, though, that make AV simulation easier.

16:45Like, for example, the dynamics of a vehicle, they are relatively simple and well understood, at least as compared to the dynamics of a humanoid robotic system. That is, for example, grasping an object and then you have to reason about friction. Maybe the object is a software to reason about deformation. So they're very hard and it is not something that you have to deal with in AV simulation. So I think that our future, since I said, they make it more difficult, rather than make it a little bit simpler, but definitely it's very hard. Yeah. How do developers navigate testing, using simulations to test in controlled environments versus uncontrolled environments?

17:29And, you know, you were talking about, I almost was going to ask you if you have to kind of weave some unpredictability into these other agents, you know, imagining, well, is it a competent, somebody who's used to being on a bicycle or is it a kid just learning for the first time and will react differently? But obviously, controlled environments, uncontrolled environments are both important parts of the processes. But how do they relate and how does the use of simulation kind of differ in the two? Yeah, so as I mentioned before, simulation helps both in terms of simulating nominal behaviors, which is still important, for example, to test whether a new autonomy stack is potentially regressing on some metrics, like, for example, how well and how comfortably you're driving on a highway.

18:15But at the same time, you're also interested in more rare cases, corner cases. And here is where you leverage different technologies. Like, for example, for sensor simulation, there have been significant advances in the past two, three years on the topic of neural reconstruction. So basically reconstructing a scene potentially from a different point of view to allow for new trajectories that a vehicle can execute in a simulated environment. To back up just a step, Marco, I apologize, but when you say sensor simulation, does that refer to simulating based on inputs gathered by sensors, or is it something else?

18:57Oh, I mean, basically recreating a scene as it will be perceived by a camera or a ladder and so on. Okay, thank you. Basically generating, for example, images that are synthetic, but they're very close to the ones that your cameras would see at the moment time. Okay. So what I was saying is that there are technologies that allow you to reconstruct very faithfully a scene. But of course, reconstruction technology reconstructs a scene that was observed in the real world. So basically, it's bound to reconstruct whatever has been seen. So that's where other new technologies related, for example, to video generation, like for example, the Cosmos model that NVIDIA has produced, come to the rescue or help because it provides the capability of generating new scenes, something that is completely different from what has been seen in the real world, maybe in a conditional fashion.

20:02So these models can be conditioned, for example, in text prompts to create the scenarios that are deemed of particular interest. Like, for example, I want to simulate a kid that is chasing a ball on a crosswalk. And now we have the technology to do that. Right. And I would imagine the models and these systems have guardrails that, to speak to what you mentioned earlier, that kind of make sure that the scene being simulated is something that could actually happen in the physical world. and not out of those mountains. Or at least plausible. Or at least plausible. Right, plausible is a better word, yeah.

20:37One of the reasons why there is substantial effort at NVIDIA and in the community in general to make sure that the simulation is so-called controllable. Right. Or in other words, you can add cabinets with. Right. We've talked on the podcast recently about digital twins and the role of digital twins in industry and industrial AI and being able to simulate, well, all kinds of things happening in a factory environment, say, before actually trying to construct or deploy them physically. How do digital twins play a role in AV simulation? Well, in a way, an AV simulator is a digital twin of a city. An interesting digital twin, because as I said before, differently from other domains, for example, the domain of industrial robotics, or the aerospace domain, in the urban digital twin, you also have to model humans.

21:33But in general, yes, digital twins are revolutionizing a number of fields, including aerospace, which is my other passion, by providing unprecedented tools for designing aircraft systems completely or almost completely in simulation, along with the control laws to control them. And actually, I do have some projects along those lines that are also quite exciting. Not to take us off course, but I don't know if you can speak to this briefly. It might be an unfair question. But how different is it simulating, running simulations, working on AVs, cars, vehicles on the ground, as opposed to working on, I don't know if you call them AVs, but in the aerospace industry?

22:16Well, in aerospace, depending on what you want to do, So depending on whether you want to test a new design or maybe you want to test the new control laws or maybe you want to test operations, of course, you have different type of simulators. But in general, one important aspect of the simulator is that of accurately modeling the aerodynamics of the system, which particularly in the case of aircraft design has to be extremely accurate. And today, thanks to advances in AI, we can run simulations where we have, you know, very detailed simulation of the airflow around the aircraft in a fraction of a time that was even possible a couple of years ago.

23:06And interestingly, I do have a project whereby we try to develop a sort of mini digital twin that can even be run online and used for the purposes of control to control highly flexible aircraft. So these are basically unmanned aircraft that tend to tend to be much more lightweight than other aircraft. And so they have deformable modes, meaning that the aircraft can actually, the wings can actually deform a little bit. And so you want to have a digital twin that allows you to predict how your body might deform as a consequence of your control actions. Well, now we have the technology to do that.

23:44Yeah, amazing. I'm speaking with Marco Pavone. Marco is the Senior Director of Autonomous Vehicle Research at NVIDIA and also a professor at Stanford University. And in addition to, well, as we've been talking about, in addition to his work on autonomous vehicles, cars and the like, also working in the aerospace realm at the same time. And Marco, you alluded to this, well, throughout the conversation, but just now again, the past few years have just been breakthrough after breakthrough in AI, in generative AI in particular. But how are these breakthroughs in generative and also physical AI influencing AV simulation now?

24:19In multiple ways. So breakthroughs in terms of reconstructing environments, for example, visually, in a way that is very difficult to distinguish from the real world, at least to a human eye. breakthroughs in terms of generating completely new scenes and scenarios by harnessing for example the power of video generation models breakthroughs in terms of using large language models or other foundation models to be able to parse human knowledge in terms of potential crash scenarios to recreate those in simulation. Right, right. And then also breakthroughs in terms of modeling the behaviors of humans in simulation.

25:11So it's not just one single breakthrough, it's the convergence of multiple breakthroughs that interesting have all happened in the past really few years. It's interesting listening to you talking and thinking like, wow, you see so much of the world when you take a drive in a car somewhere, you know, and all these different elements you're talking about needing to simulate to really reconstruct a situation and test and validate for safety. You know, you see it, the natural world, the changing road conditions, the kids and the animals and all of that. So it's fascinating to hear you talk about the ways those come together by virtue of these AI breakthroughs that allow us to model these different things.

25:51Yeah, and interesting what you're saying, because one of the things that these breakthroughs allow you to do is to tap into highly heterogeneous data. Meaning that traditionally, people were using data collected by a fleet of vehicles owned by a company developing the system. But now with these new foundation models like video generation models, vision language models, blood language models, and so on, we can actually tap into internet scale knowledge. For example, we can use videos that have been recording through dash cams, maybe by taxis. And all of a sudden, instead of relying only on the data gathered by 1 ,000 vehicles that are part of your fleet, then you can collect an enormous amount of driving hours.

26:42Not only incredible, not only in terms of the magnitude, but also in terms of the diversity, because of course, humans across the world are driving. So you're able to collect data across the world simultaneously. So this is yet another opportunity that has been unlocked by the most recent AI breakthroughs. So along those lines, what are some of the other potential opportunities you see that World Foundation models are opening up for AV simulation? So, so far we have discussed the use of World Foundation models mostly for the purposes of simulation. And that's absolutely a very important use case.

27:22But then you can think a bit more. So if you have a model that can really predict what will happen in the future, or at least in the next few seconds, with a high level of fidelity, both in terms of how the environment will look like and in terms of what the other agents will do, it means that your model has implicitly learned a very powerful model of how the world works. And so you can use that understanding as the basis for policy construction for basically building the AI that goes into the vehicle. Right. Okay. So I think that another opportunity that is right now still under development is to distill down this extremely powerful knowledge in a way that can be used also as part of the onboard stack.

28:14Vida is working heavily on it. And of course, other people and other bodies can be also working heavily on it. This is going, I think, to be a lead potentially to another substantial breakthrough. Right. Amazing. So you've touched on this a little bit in the conversation here and there, but are there any research projects that you've worked on recently or still working on now related to safety and SIEM that might be helpful for the conversation, for the audience to hear about? So, yeah, multiple projects. So one that I'm particularly excited about and that I alluded to at the beginning is how do we best leverage simulated data and real-world data to provide metrics with rigorous confidence bounds so that we can really trust them and use for validation purposes.

29:08Then we are doing research on how to curate data to ensure that the data is safe, so that you only learn on safe behaviors. And you might think that defining safety is easy. You don't want to collide. But actually, safety is much more nuanced than that. So think about how you may want to teach a kid how to drive. You start saying, well, you shouldn't have put yourself in that situation because if the other driver were to be inattentive, then you might have collided. So how do we capture all those potentially dangerous situations and distill them down into a data set that we can use for data curation and testing?

29:53So this is another area that is very interesting. And the third one is how do we automate safety evaluation? As I said, the safety assessment, understanding what the reasonable risk is, is complicated. Of course, we can rely on human judgment, but that doesn't scale. So how we can actually train the separate AI models that can help with the safety evaluation. So that's another interesting project we have currently ongoing that we hope will bring real value soon. And so along those lines, looking ahead in the next couple of years, five years, maybe it's more like 10 years, where do you see autonomous vehicles and specifically talking about safety and simulation?

30:35But you can go more broad than that. Where do you see the technology, the industry, society's relationship with AVs headed? So I think that in terms of semi-automated vehicles and largely automated vehicles with a high confidence, this technology will be pervasive in our society within the next five years. We already have companies that are productizing these systems, multiple companies that are doing so, and it's delivering real value. So I'm quite confident in this statement. In the case of robotaxis, the estimate gets a little bit more difficult in the sense that is in a situation whereby we do have a technology proof.