NVIDIA's Auto Play and the Future of Autonomous Driving — With Danny Shapiro

0:00NVIDIA's Automotive VP is here to speak with us about the state of autonomous driving and how the latest AI innovations translate to the car. That's coming up right after this. The truth is AI security is identity security. An AI agent isn't just a piece of code. It's a first-class citizen in your digital ecosystem, and it needs to be treated like one. That's why Okta is taking the lead to secure these AI agents. The key to unlocking this new layer of protection? An identity security fabric. Organizations need a unified, comprehensive approach that protects every identity, human or machine, with consistent policies and oversight.

0:36Don't wait for a security incident to realize your AI agents are a massive blind spot. Learn how Okta's identity security fabric can help you secure the next generation of identities, including your AI agents. Visit Okta.com. That's O-K-T-A dot com. Capital One's tech team isn't just talking about multi-agentic AI. They already deployed one. It's called Chat Concierge, and it's simplifying car shopping. Using self-reflection and layered reasoning with live API checks, it doesn't just help buyers find a car they love. It helps schedule a test drive, get pre-approved for financing, and estimate trade in value.

1:16Advanced, intuitive, and deployed. That's how they stack. That's technology at Capital One. Welcome to Big Technology Podcast, a show for cool-headed, nuanced conversation of the tech world and beyond. Well, look, the car is a computer. And so, of course, we got to speak with NVIDIA. We have a perfect person to do it with us here today. Danny Shapiro is here. He's the vice president of automotive at NVIDIA, someone I've spoken with before privately. But we're so excited to bring him here and have him share everything that NVIDIA is working on with you. And then also give a real important picture into the state of autonomous driving, which I think is one of the most interesting tech innovations that we're in the midst of currently.

1:56So, Danny, great to have you here. Welcome to the show. Hey, Alex, thanks so much for inviting me. Really excited. So let's talk at the beginning, just looking at the big question. How far are we from fully autonomous driving? You know, they're operating on our roads today. There are vehicles, robo-taxis, even some trucks that are truly driverless. They've gotten to the point where the safety driver can be removed. And they're operating some as part of a revenue program as well for those companies. It's not widespread in the Bay Area where I am. I see them on the roads all the time. I've ridden in them.

2:34It's an amazing experience. It really is transformative. But I think we're seeing a lot of the technology come down into consumer vehicles. So you have driver assistance systems, things that are originally based on the technology for self-driving. But we're bringing it into vehicles like Mercedes, like Jaguar, Land Rover, vehicles like Volvos and a number of other brands all over the world. are integrating NVIDIA technology to enable much safer driving on the streets. Yeah, and I had experience in San Francisco last summer where I spent a bunch of time riding in Waymos and then riding in Cruises.

3:10And we had the Cruze CEO at the time, Kyle Vogt, come on the show and talk about how his plan was to 10X Cruze rides every year going forward. And of course, we know it didn't happen. They ran into some safety concerns and he's out. and it looks like their ambitions have scaled back pretty dramatically, although maybe they'll ramp back up again. But I'm curious to hear your perspective of what is the difference between a Waymo and a cruise? I mean, not specifically going into the technical details, but you mentioned that we have autonomy already. So why is it taking us so long to, and I know so long is relative, but so long to have that spread from one company that does it really well to every car through the economy?

3:59Is it a safety thing? Is it a cost thing? Like what is the roadblock now? Well, for us that I believe a lot of our partners, safety is the primary concern. I think if you look back to 2016, when a lot of predictions were made, everyone was talking about 2020 was the year. And so from a compute standpoint, from a software development standpoint, that really looked realistic. I think everyone underestimated the true complexity of being able to make sure you get it right virtually all the time. And that's what's really challenging. So the basics are easy when you can drive down the freeway. Cars are all going in the same direction.

4:37There's no pedestrians. There's good lane markings. That's really a solved problem. But as soon as you really introduce the complexity and the anomalies that come about from human behavior, people either falling asleep on the roads or driving recklessly or being impatient or road rage or whatever it is, it's really hard to predict that and creates hazards for the self-driving car. So I think what we're doing is we're seeing a whole new wave of innovation now. Part of it's based on the same fundamental technology from ChatGPT, these large language models, more of an end-to-end system that's able to look holistically at the whole environment around the car and be able to anticipate and predict what other drivers will do and understand how to react.

5:23Just like ChatGPT, you can say anything now and it knows how to respond. It's kind of, it's quite amazing. So while ChatGPT is a large language model and using a form of generative AI, that's text in and text out. What we're doing is applying that same type of algorithmic approach and training of the neural networks, really. The technology innovation from that, we're doing with training video in and imagery from cameras or sensor data to be able to understand that environment and then determine what is the best course of action for that vehicle to safely navigate. Yeah, I was going to ask you what the path forward is, given that we know that, okay, this is something that these models are struggling with.

6:05So you think that this is the path forward, the way that you just described? You look at the innovation that's taking place from players out there. Companies like Wave in the UK have put together a really amazing underlying technology that lets the front-facing camera be interpreted in a way that the system can communicate to the occupants of the vehicle what's going on outside the car and also use that then as a way to determine what is the car going to do. How is it going to steer, accelerate, or brake? So it can interpret that video feed and explain that there's somebody jaywalking or somebody ran a red light or there's a child waiting to cross or whatever it may be.

6:43So this generative AI approach is really, I think, going to accelerate the adoption. Very recently at a conference called CVPR, so that's Computer Vision and Pattern Recognition, it takes place annually. Last month it was in Seattle, Washington. And there was a competition that they held. So it's a research conference, over 400 entries into this autonomous driving challenge. And it was basically looking at sensor data and trying to predict the best trajectory for the vehicle moving into the future. NVIDIA submitted and won the challenge. Our research team had developed a new large language model, basically end-to-end training of that sensor data system for then controlling the vehicle.

7:28And so over 400 entries, NVIDIA came out on top with this new large language model type of approach. And I think really that's where we're seeing a lot of innovation now. And instead of having a lot of individual neural nets that are trained on lanes and on signs and on pedestrians and all these individual things, a more end-to-end approach looks at the entire environment and can then understand what to do in the case that maybe there are no lane markings. So again, I think as you look at how dynamic the environments are, how there's no standard of streetlights all look the same or signs are all the same or lane markings are all the same, this end-to-end approach is really going to help get us to that point where we don't worry like we used to worry about those end cases that haven't been explicitly trained.

8:16I think this is important, what you're saying. So basically, if I have it right, the current set of autonomous driving vehicles, at least the base layer of technology they use, they have a bunch of different basically artificial intelligence systems picking out each different feature in the road. and they combine that to eventually make their predictions of where to go. But you're saying that the cutting edge today is not these individual systems, but one system that looks at everything and then is able to predict? I think what's really key about being safe is the combination of diversity and redundancy.

8:53So you want backup systems, but also you want a variety of different algorithms. So I think we're seeing a layering of different technologies and that these things that you're looking for lane markings, but if you don't find the lane markings, there's an end-to-end network also that's there to guide the vehicle and determine what to do. And we have neural networks now that are looking at signs and can interpret complex signs. So if you're trying to figure out, can I park here or not? The sign net will actually be able to read that sign and understand, is it a Saturday? Is there street sweeping?

9:24Or whatever it is, it has the context. And so the complexity of these networks is quite elaborate as well. Trust me, I could have been saved a lot of tickets if I had something in that. So what you're saying is basically if people have cars with this technology embedded, they'll be able to pull up to the curb and it'll tell you, no, you can't park there? That's right. So it's interesting to me that NVIDIA is developing a lot of this technology. Like I went to your automotive section on the website and was like, wow, there's a tremendous amount of models coming out of NVIDIA. I thought it was largely the car makers, like the Waymos or the Teslas that are developing the autonomous technology.

10:05So how involved is NVIDIA in developing these models itself? And then who's the customers? We work with hundreds of automakers, truck makers, robo-taxi companies, software startups, the sensor companies, the mapping companies. It really is quite an ecosystem that we've built. We're not creating the vehicles, but we work with those manufacturers. And so we offer the compute hardware. That's our drive platform. So that's the brain that goes inside the car. Our drive OS is the safe operating system that's part of that package. We have a lot of different middleware and libraries that they can use to develop their applications, algorithms, the neural networks.

10:44That application layer, though, is generally built by our customers. So Mercedes-Benz or a Jaguar, Land Rover, Volvo, Neo in China. And so they can pick whatever parts of the software stack they want. And in many cases, our customers are taking the whole stack and they're developing some of their own algorithms as well. So there might be a pedestrian detection algorithm from Mercedes running along a pedestrian detection algorithm from NVIDIA. And we collaborate on that. So starting through the end of this year, the introduction of the new CLA, it's already been announced from Mercedes. That's the new Mercedes model, the C-Class.

11:27And so every Mercedes will be built on NVIDIA Drive with the software that we've developed and rolled out by NVIDIA. So it starts with that C-Class vehicle and then will go through their entire line over time. Do you sense a time where eventually NVIDIA will be able to build, like using some of the technology that you've discussed already, build the software suite that empowers autonomous driving and then just its plug and play for these auto manufacturers? Is that the goal? Well, that's essentially what we're doing. We're making the software available. So we're developing the whole stack. And really, it's a three-computer problem, we call it.

12:03So you have the computer we just talked about inside the car. So that's the drive platform. That's a very high performance, energy efficient, automotive grade supercomputer. You plug all the sensors into that and it's purpose built for a vehicle. It's going to operate in the heat of the desert sun. It's going to operate in very cold temperatures in Alaska, unlike your phone, which if it gets too hot or cold, it'll shut off. So we have to make sure that the temperature range will work, that the shock and vibration, the dust environment. So all of that goes into making this computer automotive grade.

12:39But then in addition to that, we make the computer that's used to train the artificial intelligence. That's our DGX. That's the supercomputer. And so we have a huge business in terms of the automakers building out their own data centers or using cloud providers like Azure, AWS, and Oracle to train. And then we also have our OVX. That's our omniverse computer for simulation. So again, that's another data center solution for first developing and then testing and validating and simulation before the software even goes into the car. So NVIDIA is the only company that has these three computers. And it's really this whole life cycle of developing, testing, and deploying the software.

13:23And really, it's a continuous flywheel. Just like your phone gets software updates, all these cars are designed to get software updates and get smarter and smarter over their life. Yeah, that's wild. You know, Danny, a lot of folks think of NVIDIA as just like the chip company for AI training. And I'm always like, it's a little bit more than that. And it's just wild that like just in this one discipline, it's deeply involved in pushing the cutting edge forward with autonomous. Absolutely. I think you're right. People tend to focus on what happens in the car, not realizing there's so much work before you can get to that point and so much development.

14:04So what's great about the customers that work with us, it's a single architecture. It's the same chip technology that's in their data center that they're training on, that we do testing. It's called hardware-in-the-loop testing. So we actually test the whole software and hardware that goes in the car. We can test that in the data center first in these virtual environments called a digital twin. So we create a model of a city and we simulate the camera, the radar, the LIDAR signals that are detecting everything happening around the car. Motorcycles cutting off vehicles or pedestrians, jaywalking.

14:39And so all of that can be tested before we actually even put it on the road. So it's very efficient. It's very safe to test it that way. And it really helps create a much better product in the end. Yeah, I'm going to talk to you a little bit more about that in a moment. But first, I want to ask you another technology question, because it sort of seems like there's two different schools of thought when it comes to building autonomous cars. One is, I'm just going to call it shorthand, the Waymo school of thought, which is that you need like a gazillion sensors and cameras. And your car is going to look like a submarine and it's going to cost hundreds of thousands of dollars, but it's going to work pretty freaking well.

15:18And another school of thought, I'm just going to call it the Tesla school of thought, is that you just need a few cameras. And then eventually you'll be able to train machine learning models to the point where you can run your Tesla autonomously without a LIDAR. Who do you think is right? I think they both have merits. The reality is, as I mentioned before, this diversity and redundancy is really how you get higher levels of safety. So cameras are great, but they don't work in all conditions. And so when you combine radar, when you combine LIDAR, you have the strengths of many different types of sensors and they complement each other.

15:59So I think we see in the case of Waymo is really a higher level of security and confidence and that redundancy that comes from the system. And so they're operating fully autonomously with no drivers, whereas I do have a Tesla, and it's quite remarkable being camera-based. But every once in a while, I still need to jump in and grab the wheel. So it's not there yet. Can it get there? I think it probably can eventually, but it's not there today. How's the latest full self-driving update? Like I said, it's pretty good. I rely on it every day. It is still considered beta, and so I'm watching it. I'm in the industry, so I'm curious about each software rev and what it can do and what it can't do.

16:43But it's quite remarkable. And there was a report in the Wall Street Journal this week that talked a little bit about some of the deficiencies of the way that Tesla operates. And points exactly at this issue, right? That it's decided to not use LIDAR, for instance. Some of them have radars. But Elon basically wants to do everything cost as cheap as possible, right? Seen it in SpaceX. You're seeing it in Tesla. and it's because these systems are just good enough that people trust them that we've seen some of the tragedies happen there was a video that they put out the journal put out that showed effectively a car driving at night and there was an overturned tractor trailer blocking the road and because the car's computer vision models hadn't seen effectively like black you know dark underbelly of the of the truck dark evening couldn't pick it up because it hadn't been trained on enough examples of this and the car went right into the tractor trailer.

17:43But thinking about your answer, you think that this is just a temporary thing that eventually they'll be able to figure out these sort of outlier scenarios over time? I think if the models can be able to detect that there's a physical obstacle, even if it doesn't know what it is, then it will be able to take the right action. But again, this is where having the diversity of sensor data becomes a really big differentiator. Right. Now, talking again about what NVIDIA is doing internally, it's pretty wild. So you're actually simulating collisions. Is this something that you do in that sort of world that you talked about where you go through these?

18:26Absolutely. So talk What we try to do is create the scenarios. It's less about creating accidents, but creating the scenarios to ensure that the systems will have a safe outcome. There's no way to ensure that there's zero accidents in the world, right? There always can be crazy stuff that's going to happen on the road, and no human driver could avoid something falling right in front of the car or somebody getting pushed in front of a vehicle or something like that. But what we want to do is be able to anticipate all that and be able to avoid it or mitigate what would happen in one of these hazard scenarios.

19:04One of the things we're able to do actually is record drives that we're taking and then use that as input to create a huge range of different scenarios, permutations on that and test the software. And so we can actually capture cars in a scene and make any one of those cars in the scene the autonomous car and see how it would behave. So we're building a massive database of scenarios and ways to test and validate that the technology is good. The other thing that we can do is we can take accident reports. And now using these large language models, we can input these accident reports and be able to create scenarios from a text input explaining what happened or if there's a map or something like that.

19:51So then how important is simulation and training? Anything having to do with autonomous driving? Simulation plays a really, really big role in ensuring the safety of the system. them. There's really no way that, first of all, driving around collecting data, you rarely are going to see the dangerous scenarios, the hazards, the things that very rarely occur, you're not going to capture them on your data collection. So we need to use simulation and really what we call synthetic data generation to create those kinds of scenarios. So we can create fake potential hazards, things falling off of trucks, people running across the street at night, whatever it may be, somebody running a red light.

20:33And so we can create that data to augment the real data for training the AI. And then we can actually simulate all these dangerous scenarios to ensure that the system will do the right thing. And the benefit of using simulation too, it's repeatable. So we can adjust the software and test something that maybe didn't pass a month ago, but we We can run it through the same scenario and see, oh yeah, we fixed that. There may be situations, it often happens that the sensors are blinded by the sun right as it set in, right? The sun is coming right into the eyes of the car, into the driver's eyes, into the camera's eyes.

21:12You only have a few minutes a day where you can actually capture that data as well as test. In simulation, it can be sunset 24 hours a day. We really can control that environment. and we can create rain, snow, fog. It's really remarkable. And it gives us the ability to test those things that again, may never be seen in the real world. So if you're driving a real vehicle, trying to test it in autonomous mode, you may never know. In simulation, we can be sure it's gonna work before we release it. And you also have a assistant that you're working on that sort of appears within the car that helps you like navigate tough things.

21:48Like for instance, like it will look around, It's called LADA, I believe. And if you're in New York and you want to make a ride on red, it will be like, hey, listen, like can't ride on red is illegal here. Or if you're in Mexico and like you're taking a hairpin turn, you know, there's like some traffic patterns that allow you to kind of go outside the lane to take a peek and then come back in. And it will help you figure out what's going on with that. I mean, I'm about to go to Ireland and I definitely need something like that to tell me what side of the road to be driving on. So talk a little bit about the progress there.

22:20So I think you're referring to a video series we put out called Drive Labs. Right. And so what we do is each episode, we sort of pick one little piece of technology that's part of this huge software suite that we're building. And so some of these things come out of our research team. So it might not be in our software today, but it's sort of a preview of what's coming out soon. And so what we're able to do is train these systems and create these different modules, essentially, that become part of the software stack that our customers can use. So how they decide to bring it to a customer sort of is the decision of a Mercedes or a Volvo or others.

22:56But the technology is there so that we can train it on what are the laws of a particular region. I mean, the signs, the light signals, the lane markings in different regions of the country or around the world differ quite a bit. And so basically creating these large language models for each region that understands what the car is allowed to do and not to. And so it could give you an alert. If it's been implemented in the car, it could give you an alert if you try to turn right on red in New York, for example. Perfectly fine to do in California. Okay, we're going to go to break. But before we do, I've got to ask you a stock question.

23:34The stock is down like 7 % today and 16 % on the month. Do you notice that within the company? You know, things move around a lot. The market just shifts a lot. If you look at the page often, everything's green one day, and then everything's red another day. And I think there's really no way to predict. We'll put out great news and we never know what's going to happen. I think what we do know is NVIDIA is just creating amazing products. We have really exciting technology we're bringing to markets, not just in automotive, it's in healthcare, it's in energy, It's in finance. Gaming is still a big part of the company.

24:12People don't necessarily talk about it as much, but it's a huge, huge part of our company. It's super exciting. This week going on is SIGGRAPH, which is the big graphics conference annually. And that's something that NVIDIA has always been a part of. And that's kind of our core accelerated computing for graphics. And now that conference is turning into an AI conference and it's all about robotics. And so NVIDIA is still at the center of all these different industries, providing technology, platforms, ecosystems for other companies to build. So I think things are feeling good. Yeah, I have been tuning into SIGGRAPH this week, and it is funny, right?

24:45At a gaming conference, and I think Jensen is just so happy to be back in a gaming environment because at the start of every session that he's had, he just shouts out, this is my home turf, which is, you know. Yeah, I mean, it's so true. SIGGRAPH is where all the graphics researchers kind of unveiled the new technology and graphics is so core to nvidia but so many other things whether it's filmmaking whether it's manufacturing whether it's now visualization for um you know engineering with fluid dynamic simulations or or weather um forecasting we have a system we're working on called earth 2 which is going to just do amazing things for that um for the meteorologists and be able to predict weather with incredible detail beyond what has ever been possible and look way in advance.

25:34So that has huge implications for just a lot of predicting natural disasters, for example. Right. All right. Let's take a break. Come back. You mentioned robotics. We'll talk about robotics, including how this technology will help translate the state-of-the-art advances to generative AI as well. We've learned what generative AI can do to autonomous cars. What can autonomous cars teach gendered AI? All right, that's coming up right after this. Capital One's tech team isn't just talking about multi-agentic AI. They already deployed one. It's called Chat Concierge, and it's simplifying car shopping.

26:11Using self-reflection and layered reasoning with live API checks, it doesn't just help buyers find a car they love. It helps schedule a test drive, get pre-approved for financing, and estimate trade in value. advanced intuitive and deployed that's how they step that's technology at capital one this holiday season train smarter not longer the hydro rowing machine delivers the best results in just 20 minutes a day it works 86 of your muscles in one seamless motion twice as effective as running or cycling hydro is your go-to for the ultimate full body workout how ultimate it works 86 % of your muscles, arms, legs, and core, and it's twice as efficient as cycling or running.