Driving Safer AVs Faster with Smart Simulation, Neural Reconstruction, and Data-Centric Tools - Ep. 289

0:10Welcome to the NVIDIA AI Podcast. I'm Noah Kravitz. NVIDIA GTC is this March 16th to 19th, online and in person. Visit nvidia.com slash GTC to learn more about the premier global AI conference and to register now. Today we're talking about autonomous vehicles and AV simulation, using AI-powered systems to make driving safer and more efficient. With me are Rohan Basen and Dan Goral. Rohan is Senior Solutions Engineer for Sensor Simulation at Fortelix, and Dan is head of technical partnerships and a machine learning evangelist at Voxel 51. Gentlemen, thank you so much for taking the time to join NVIDIA's AI podcast.

0:51Welcome. Thanks for having us here. Thank you for having us. So before we get into talking about all things AV simulation and driving, maybe we could start with kind of self-intros. Dan, maybe you can start. Just kind of tell us briefly what your company does, what your role is there. Yeah. Like you said, my name's Dan. I'm from Voxel 51. we were one of the premier data platforms for physical AI and AV, allowing you to help find all the kind of problems and curate your data set to make sure you're always focusing on the most important data. One of the things we always love to say around here is garbage in, garbage out, right?

1:27So if you're not training on the right amount of data or the right data, you're not going to be able to get that model that you're hoping for. Personally, my background is in all things physical AI. Before we called it physical AI, I spent a lot of my time working on edge devices such as drones, cars, robotics, and many such. I have a very strong background in hardware and things like that as well. But now, of course, as we evolved more towards this physical AI world, I can now kind of slap that label on it and everyone should know what I work with. Awesome. Rohan? Hi, I'm Rohan. I'm a solutions engineer at Fortelix, and I help our customers work with sensor simulation tools from a variety of providers to meet their synthetic data generation needs.

2:09My background is in sensor simulation as well. I was part of developing Ford's internal simulation toolchain for their Level 2 and Level 3 driver assistance stack. Very cool. Maybe we can start with a little bit about what AV simulation is. We've talked about it on the show before. A lot of listeners will be familiar. But maybe one of you guys could give just kind of a brief overview of what AV simulation is, why it matters in your work and in the broader mission of making autonomous vehicles real and safe and worth using. And then you can get into talking about what the ecosystem is like today.

2:47I'm sure Rohan can give a better answer for this one. So I'm going to pass this one to him. Okay. Yeah, definitely. I think the AV simulation ecosystem has definitely changed a lot since I started working on this six, seven, eight years ago. A lot of focus then was on camera-based solutions and using bespoke neural networks for individual perception tasks. So think tasks like object detection, lane detection, semantic segmentation, and fusing those together in the vehicle. Now we see a drastic shift away from that into kind of more end-to-end solutions where we have one stack that does all the perception controls and outputs steering throttle and brake commands to generate a driving trajectory that's safe.

3:27I think a big reason for that has been developments in generative AI, right? We talk about 3D Gaussian splatting or diffusion-based models, and both of those have contributed significantly to the shift in sensor simulation data and the fidelity of sensor simulation data that we're able to get and use for downstream training or validation of our AV stacks. I'm sure there's not a one-size-fits-all answer to this. But when we're thinking about, for the listener who's thinking about, you know, sensors on an autonomous vehicle and gathering all this data and what does this all mean, on a typical, you know, car or truck, whatever works, how many sensors are we talking about?

4:05Roughly, how many sensors does each vehicle have? And, you know, what are the kinds of data that you're talking about when we're thinking about all these different sensors pulling data? Yeah, that's a great question. and it's always fun to get that question because it really depends on who you ask, right? If you go walk into Waymo, they'll tell you one answer where we have 25 different sensors and this thing and that thing. If you go ask Tesla, they're going to tell you a completely different answer, right? Okay. Just cameras. In reality, what we typically see is you're going to have a mixture of almost always camera data.

4:38That's things like videos and images. And then we call LiDAR data, which is, if you're familiar with things like radar and such, it's laser-based, pointing to understand how far away things are. Basically, your car shoots a laser at the car in front of it. That laser comes back. I know the car is 10 feet in front of me now, right? Famously, Tesla does not have LiDAR, just calling that out. But for the majority of everyone else, they do have either LiDAR or radar on their cars. Now, something that Rohan touched on as well is we're seeing this new adoption of even additional sensors on top of this.

5:09Traditionally, when we were doing lane line detection or actor prediction, things like that, We only really relied on the video and the LiDAR in order to get these predictions. But now we also have a lot of additional sensors to measure a lot of the physical components of the car that is being fed into our models as well. Things like very things you'd be common to find on every car, like a speedometer, of course, to understand how fast we're going, but also very complex physics-based sensors to help understand exactly how the car is kind of positioned, where it's moving, which way it's turning, which way it might be getting pulled and such as well.

5:41And so when we're kind of moving from physical world, real world, I hate saying the real world, but the physical world and then simulations, where are the gaps between what's available in the tools and the workflows that devs work with in these simulations and what AV devs actually want or even need for the work that they're doing? Is it a matter of getting more and more accurate simulated data? Is it something entirely different? Where do the gaps exist that you're trying to address? Yeah, this is a great one. And I'm sure Rohan and I, we can bounce off of this one because he's really good at the second half of this answer I'm going to give.

6:22And then I'll cover the first half here, right? So the first thing I want to touch on is, is it just more data? The answer is absolutely not, right? If it was just more data, we would have self-driving cars, right? If you're not familiar with this, most very serious self-driving car companies, your Wamos, your Waves, your Zeekses, they're on the 100-a-petabyte scale, typically, or somewhere close to that. If you don't know what a petabyte is, 1 ,000 terabytes is one petabyte. So we're talking about... It's a lot of data. A lot of data. Exactly, right? So where I see a lot of the issues today with a lot of users that I interact with, a lot of problems, is something that you touched on there.

6:59Is the actual translation from the physical to the digital. Did you do everything correctly? Is everything calibrated correctly? because it's all well and good if you've got exactly that scene or that drive that you were looking for in the real world. But if you can't properly communicate that back into your model, you're going to have a lot of issues, of course, right? If it doesn't see it the same way it actually happened, your model is going to degrade because of that performance. And that's a lot of what Voxel 51 is supplying to our AV users and such. It's ensuring that digital to, or physical digital translation is as clean as possible.

7:33Now, there is a lot of other things that people are going to be expecting whenever you do these translations and what happens after the translation, right? Things such as, well, I don't want to be doing this just for pulling out of the garage, for instance, right? There's scenes that matter more than other scenes, right? And this is where Fortelix really, really shines in their tool on how they can help identify these cases. Thanks for that. Yeah, exactly right. I think for us, as Dan said, there's a lot of data already out there and Fortelix's goal is to help AV developers make sense of their data, right?

8:03Identify what data they have and not necessarily generate or collect more data, but use that data smarter and fill in the gaps with exactly what they need, right? Not scenarios of you pulling out of the garage, but niche edge cases that are safety critical. You know, we have tools that allow you to generate smart replays and create unique and interesting scenarios which are going to stress your stack and test whether it's actually up to the mark. And that's the kind of, you know, smart and I guess niche data that you need to generate that little bit of additional model performance rather than throwing hundreds more petabytes than what you already have.

8:41Right. Can you guys talk a little bit about foundation models and neural reconstruction and how these two technologies in particular have changed expectations around simulations, AV simulations? Yeah, neural reconstruction definitely is something that's created like a step change in what AV developers look for in simulation. The level of fidelity that we can get from neural reconstruction is unmatched by any kind of physics-based rendering engine. It's just not even a close competition. You know, this technology is still developing. It's only a few years old, and there's still improvements to be made.

9:14But the goal is much more approachable now than it ever was before. And with Fortelux's Smart Replay technology, you can take a relatively uninteresting snippet and generate variations on that. We can have pedestrians crossing in front of you, other vehicles driving in behaviors that you don't expect, breaking the rules and creating interesting scenarios with a level of realism that was unimaginable even just five or seven years ago. Yeah, I mean, I think to add to that, I think the most important thing to understand here, I like to make this point very often, is that the most valuable resource when you're developing an AV system is not GPUs, it's not data, it's not people, it's time, right?

9:56How can I train as fast as possible? How can I test as many cases as possible? Right. And so every time you train a scene that it's not going to make your model better, you just waste it a lot of time. Right. And so with neural reconstruction and foundation models, we don't have to go drive and find the snowy days. I'm not checking the weather to figure out when it's going to rain next time. Right. I can use models like NVIDIA Cosmos Transfer is a really great example to just generate, say, hey, this is the scene I'm interested in. But how do I do when it's raining? How do I do if it's snow? oh, you can use a tool like Fortelex to change the actors.

10:31What if I didn't turn left? What if I turned right, right? And the main goal here is to save time, right? Both because these neural reconstructions render faster, it's easier to move things around. I mean, just to show you where we were five years ago, this is not a joke. This is not an exaggeration. We literally had people playing GTA V and crashing into other people to capture data, right? This was not a joke. This was state-of-the-art even at the time, right? And so think about how much time we're saving and how far we've come in only a few years where now we can just imagine these things with foundation models and neural reconstructions, Gaussian splatting, where we can skip not just driving in the real world, not playing GTA, but actually just doing this in a matter of a couple prompts or clicks in order to figure out where we need to go next.

11:16I don't know if I'm more amazed by the fact that that was only five years ago or that like we've come that far in five years ago you know what I mean I'm kind of like absolutely oh man that was a long time now it's only five years you know but but I mean things have advanced so much is it I shouldn't say I don't know if is it still is the right way to ask this but Rohan you mentioned this the edge cases of, you know, somebody, a kid running out onto the road, you know, what have you, sudden weather change or an object in the road, that kind of thing. Are these the kinds of things that are still amongst the most, I don't know, sort of time and bandwidth intensive problems to solve when it comes to, you know, building a safe AV stack?

12:03Is it still kind of solving for these unknown edge cases? or what are, I see you guys nodding, people can't, you know, listening. Of course, yeah. Yeah, yeah. What are these things? Because, you know, I was going to ask you about this process of, and again, you talked about it a little bit, of, you know, creating this high-fidelity 3D reconstruction and then you can kind of turn it into all these different edge cases to test for and to kind of stress test on your stack. Is it these, you know, again, like the unexpected person or obstacle in the road? Is it weather conditions? What are some of these, you know, either most important, most difficult edge cases to test?

12:40Yeah, I think these are definitely amongst the most safety critical cases. Obviously, the tails of the distribution are very long and it's impossible to capture all of them. But I think the hope, you know, for us, for Voxel 51 is that we can capture enough of these events in your model that your model understands what to do, even if it's not seen the exact scenario before. Right. As humans, you know, we learn that whether it's a dog running in front of us or a cat or a human, we know that we should stop and kind of create that same learning in your models. Again, with like the high volume of data that's being collected by a lot of different companies, right, they're collecting a lot of nominal cases of you driving smoothly along the highway where nothing's happening.

13:19And that data is not necessarily incrementally useful to your model. And so at Fortelix, definitely one of our key offerings is being able to create a huge variety of edge cases just from one relatively nominal log and allowing you to understand how your stack reacts to each different event. Right. And just to add to that, I'll say it in a different way, right? Where if I gave you, Noah, a billion dollars in a year and a half and said, build me a self-driving car, right? I am highly confident that a person of, you know, even minor technical, not to undermine the great work that people are doing, but even someone with minor technical ability can build a self-driving car that drives correctly 90 % of the time.

14:01There's just no way in the world you would ever release a self-driving car that only drives safely 90 % of the time. Of course, this is how we think about it. Anyone that works with machine learning models know how difficult this is. Closing the gap from 95 % to then 97 % to 99%. And now we're in this 99.9 % of the time kind of region, right? And still not being able, it's not safe enough, right? We need to make sure it's safe enough. That's where the real problem lies, right? Yeah. Each of those little jumps is like an order of magnitude, more effort, more time, more data. You're the last mile problem.

14:37No, for sure. When you're working so deeply in simulated environments, how do you measure, and maybe there's a better word than this, but how do you measure the realism of scenarios, of objects of behaviors within these simulations? And how can you come up with benchmarks, other ways to measure things that will translate into safer outcomes, you know, out on the road? I want you to go because I have a really big devil's advocate answer to this question. Okay. Oh, it's a teaser. I like it. I can go first. Yeah, for us, realism falls broadly into two categories, right? And it depends on what you're trying to do.

15:12If you're trying to train your stack or if you're trying to test your stack's performance. In each case, you know, obviously better realism leads to better AV stacks and safer driving behavior. For training, we want the synthetic data that we are generating, right? Whether it's through neural reconstruction or diffusion-based, you know, generation like Cosmos, we want that data to increase the model performance of your end-to-end driver stack, right? And there's, you know, whether that's making sure that you're following the rules, minimizing disengagements or collisions, essentially. We want training data to improve the model performance.

15:48And we can kind of use intermediate like task-based perception models as proxies to make sure that this is something that's happening. That's typically what you would do with like data generated from Cosmos for Spark. For testing, we want to see the same performance in the reconstruction that we saw in the real world. That means we want our stack to fail in the same way, right? Whether that's predicting false positives or false negatives to understand or ensure that our simulation is behaving exactly like the real world. It's not exactly applicable to end-to-end models, but again, using task-based perception models can be a good proxy here.

16:25It gives us confidence that adding reconstructed snippets back into the training set with labels will improve the driving behavior. When you're able to see new scenarios, add those to your training data set, and hopefully learn not just, again, like I said, that a person is running in front of a car, but have the same reaction if it's a dog or a cat or something else unexpected. Yeah. To add to that, right, I'll give the marketing answer first, right, which is very important first, which is we do the same thing very similar, right? And Rohan and I, our companies serve different purposes inside of these orgs at work, right?

16:56So Rohan's very focused on safety and make sure it fails in the same way. These are all absolutely crucial. It's not to say that I don't plus one everything you just said. But in BoxL2D1, we focus a lot on how the machine learning model interprets the context of this, right? Does the model think it looks like the real one? And there's many different metrics and stuff you can do, such as embedding models. You can extract embeddings from your model that is driving your car. So whenever we do a scene and then we reconstruct it synthetically or I make it rainy or if I make it snowy or something like that, does the model still think it's driving in the same neighborhood?

17:29Because in the past, we see this with not the case. It looks like a butterfly to me. The model does not think it looks like it. It looks like it's completely different. That's exactly what we're trying to avoid. There's also a lot of classical kind of photogametry, videogametry methods you can apply here, which is like, hey, this doesn't really look like a normal video to us, right? Like the pixels are in the wrong place. It's not really moving in the right directions. It's a lot of generative AI noise, right? That we don't want to have. That's throwing our car off and such. Now, the devil's advocate answer here is I don't think it actually matters as much as you think it does.

18:00How so? The goal of training a self-driving car is to make sure it doesn't crash, right? Right? So I actually don't care if it doesn't look exactly like the real world, as long as my car gets better at driving. Right? And this goes back to my point earlier about time and things like that, right? Where if you can make me a synthetic scene that looks 90 % of the way like the real world, and my car can drive it in, it learns from it, it learns from its mistakes. And that we see it's reproduced in the testing scene that Rohan is talking about. And that's great. Right? Now, if you spend 10 times as long to get to 100%, and now I have all my RTX shaders on, I have all my ray casting, my shadows are perfect, I look at the puddle, it reflects my reflection, all these kind of things.

18:48Studies have shown that it doesn't matter. That the car doesn't care if it sees its shadow or if it sees a puddle and things like that. The important part here is that it's learning how to drive better. And so there's always a trade-off here in terms of we always want to be as real as possible, of course. We're getting closer and closer to that goal every day. But there is this acceptable little zone here where as long as you're being efficient in this process, your car is driving safer, and you're always maintaining that real-world data inside of your testing and validation loops, then some level of not-realness is acceptable.

19:22Right. That makes sense. Yeah, I completely understand that perspective as well. A little bit different of machine learning engineers, the data side of things. But yeah, there have been studies and papers published that show mixing real data and synthetic data, even if that synthetic data isn't 100 % realistic, still leads to improved model performance, at least on traditional task-based perception models. So it's very reasonable to think that that should apply to end-to-end models as well. And to reemphasize, the digital twins, Gaussian splatting, the on-verse new rack, is still way better than what we were a year ago.

19:55It's way more realistic and everything like that. That's not to say that we're personally thinking about it or anything like that, right? Of course, we're getting so much better at it every day. I'm speaking with Rohan Bossin and Dan Gorel. Rohan is Senior Solutions Engineer for Sensor Simulation at Fortelix, and Dan is the Head of Technical Partnerships at Voxel51. Rohan, I want to ask you about a Fortelix technology or an early step in the pipeline that you guys do called Scenario-Driven Data Curation. Can you unpack that, talk a little bit about what it is and how it differs from traditional ways that AV teams pick drive logs for simulation and for retraining?

20:37Yeah, absolutely. Scenario-driven data curation is what we call our automatic scenario labeling tool. It allows us to automatically label temporal events like approaching a stop sign or turning right on a junction when pedestrians are crossing in logs that we ingest and we add them to our database. It allows us to search for individual events or a combination of overlapping events to identify scenarios of interest. We also have dashboards and metrics to see what data you've collected for your stack and what data is missing and how, you know, my role is to help our customers fill those gaps. Right.

21:12And then I understand you guys also have a product called Fortify that helps when customers are scaling up their programs to big fleets and using larger models. Talk a little bit about what Fortify does. Yeah. Fortify is, I guess, like our overall tool chain. Okay. And this is a step within that. I think in the way that it's different from, as you were asking before, like the way it's different from kind of traditional ways, AV teams pick drive logs, that this is like an automatic process that we have when we ingest logs, and you don't have to go searching within your logs for certain events. Oh, great.

21:47Okay. As Dan said, you know, you've got hundreds of petabytes of data. Digging through that is difficult. Yeah. process. And our scenario matching tool helps kind of label those events and let you search within each category to understand what data you have in your ODD and what's missing. Fantastic. Dan, from Voxel's side of the house, how do your neural reconstruction and data curation tools change the way that AV teams think and go about operating? Yeah, right. And if I haven't heard it or said it already, it's all about efficiency here, We're trying to be as efficient as possible, as well as holding as much context as possible for the right job.

22:27So Fortelix, for instance, as we just heard, takes a really interesting approach, a really great approach about scenarios and scenes. And can you label and find the temporal aspects such as left turns and U-turns and pedestrian crossings and things like that? For us, it starts earlier in the pipeline from data curation based off the videos or the 3D assets there to help understand one of two things. One, is this the right thing I'm looking for? I'm really looking to see where is the model uncomfortable. And I link that kind of ambiguous because it is an ambiguous thing where we don't always want to make and project what we think the model needs to learn.

23:04We need to hear and listen to our model to understand where we're going to learn. right so using different methods of embeddings and curation understanding model performance and evaluation help us understand exactly kind of where the model feels uncomfortable so that way we can make sure that we're only reconstructing the training on the seams that are going to give us the most value right that we can then pass with tool like foretellics and say hey this is the subset of that petabytes of data that we think are the most interesting let's label these first and then we'll move back down the chain and we'll you know gradually grow our data from where we go from there.

23:37Additionally, it's all about considering how accurate your translation is from physical to physical, right? We can always get better, right? There's always going to be microsecond differences, centimeters off. There's always going to be additional machine learning context you can add to it. Similar to how Rohan talked about those perception models in the middle to evaluate kind of the performance of your autonomous driving stack, you can actually do this before training as well to understand things like if I'm really interested in traffic lights that are turning yellow, for instance, right? Well, I can use a machine learning model that finds specifically yellow traffic lights and then only grab those seats to see, hey, did my cargo through the intersection?

24:16Should it have stopped? How close was I to the intersection, right? This is a very interesting scene from an AV scenario perspective and something that we can do inside of the box of the tool to make sure that you find the best suits what you're looking for. Right, right. Amazing. From the engineer's point of view, how much do things like interactive exploration and visualization of data sets, how much do those help in helping the engineers trust these reconstructed 3D worlds as they use them to simulate things? Yeah, it's a great question, right? I think visualizations are always going to be important, right?