Building the World's Most Trusted Driver

0:01Hello everyone, welcome back to the A16Z Podcast. This is Stuff. Now, one of my favorite podcasts we've recorded since I joined the team was just about this time last year. That episode was on autonomous vehicles, but it was actually also in an autonomous vehicle. That was my first ride in a self -driving car, and over the last year I've seen so many others have their first as Waymo has expanded to the public in Phoenix and San Francisco, while also placing its roots in Austin and LA. In 2015, Wainlow tested its first fully driverless ride on public roads and then open to the public in Phoenix in 2020, but it wasn't until 2022 that autonomous drives were offered in San Francisco.

0:43And by the end of 2023, it clocked in over 7 million driverless miles. Slowly, then all at once. So with this space moving so quickly, we want to give you an update on where this industry is today. Passing the baton to properly introduce this episode, here is our very own AI Revolution host and A16Z General Partner, Sarah Wang. As a reminder, the content here is for informational purposes only. Should not be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security and is not directed at any investors or potential investors in any A16Z fund. Please note that A16Z and its affiliates may also maintain investments in the company's discussed in this podcast.

1:27For more details including a link to our investments, please see A16Z .com slash Disclosures.

1:37Hey guys, I'm Sarah Wang, General Partner on the A16Z Growth Team. Welcome back to our AI Revolution Series. In this series, we talk to the Gen AI builders who are transforming our world to understand one where we are, two where we're going, and three, the big open questions in the field. Our guest this episode is Dimitri Dolgov, the co -CEO of Waymo. Dimitri has led Waymo to solve some of the biggest challenges in bringing AI to the real world. And after tens of millions of miles of testing, Waymo's vehicles have shown themselves to be safer and more reliable than human drivers. Myself included.

2:16Dimitri has a unique perspective, given that his work has banned multiple AI ML development cycles across decades. He was an early pioneer in self -driving cars, working with Toyota and Stanford on DARPA's Grand Challenge before joining Google's self -driving car project, which then evolved into Waymo. In this conversation, from a closed door event with A16Z General Partner, David George, Dimitri talks about the potential of embodied AI, the value of simulations and building training data, and his approach to leading a company focused on solving some of the world's hardest problems. Without further ado, here's Dimitri in conversation with David.

3:01Maybe to start, take us back to Stanford, if you will, and that was when you first started working on the DARPA project, And maybe give us a little bit of your history of how you ended up from there to here. My introduction to autonomous vehicles was when I was doing a post -doc as Stanford. You just mentioned David. This was during, I got to be pretty lucky with the timing of it. This was when the DARPA Grand Challenges were happening. DARPA is the defense advanced research project agency that started these competitions with the goal of boosting this field of autonomous vehicles. And the one that I got involved in was in 2007.

3:44That was called the DARPA Urban Challenge. So the setup there was, it's going to look a toy version of what we've been working on since then. That was supposed to mimic the driving in urban environments. So they created a fake city on an abandoned air base and they populated it with a bunch of autonomous vehicles and a bunch of human drivers. and they had them do various tasks. So that was my introduction to this whole field. And it was a bit of a, I think, a dark, these challenges are often by people in the industry, considered a foundational pivotal moment for where this whole field. And it was definitely that for me.

4:23It was like a light bulb, light switch moment that really got me hooked. What was the hardware and software that you guys had at that point? This is 2007. Yeah, I know it's a very high level, not unlike what we talk about today. A car that has some instrumentation, so you can tell what to do and you get some feedback back. Then you have what's called a post system, a bunch of inertial measurement system accelerometers, gyroscopes that kind of tell you in GPS, tells you how you're moving through space. And it has sensors, radars, lighters, and cameras, and those same stuff we still use today. and then there's a computer that gets the sensor data in and then tells the car what to do and then bunch of software.

5:09And software had perception components and decision -making planning components and some AI. But of course, everything that we had, I can't one of those things over that how long has it been? 18 years more than that. It's changed drastically. So when we talk about AI today versus AI, we had back in 2007, 2009, nothing in common. and similarly everything else has changed. The sensors are not the same, computers are not the same. Yeah, of course. So then, okay, so at that point, that was the pivotal, that was like the light bulb moment. And then at that point, you said, okay, I'm at Stanford, I wanna make this my career, right?

5:45Is that, and then was Toyota, and then where did it go from there? I don't know if I thought about it in those terms, I was like, this is the future. I wanna make it happen, I wanna be building this thing, career, okay, you know, they can wait. But it was, that was the next step. That was a nice big step is a number of us from the DARPA challenge competitions started the Google self -driving project. It was about a dozen of us. Then in 2009, I came together at Google with support and excited from Larry and Sergey to see if we can take it to the next step. And that then, we worked on it for a few years.

6:24And that project then became Waymo in 2016. and we've been on this path since then. Okay, so we have this new big breakthrough in generative AI. Some would say it's new, some would say it's 70 years in the making. How do you think about layering advances that have come from generative AI to what many would describe as more traditional AI or machine learning techniques that were kind of the building blocks for self -driving technology up to that point? Yeah, great question. So maybe a generative AI is a broad term. Sure. So maybe you can maybe take a little bit of a step back and talk about the role that AI plays in autonomous vehicles and how we saw the various breakthroughs in AI map to the space of our task.

7:09So, like I mentioned, AI has been part of self -driving autonomous vehicles from the the early days back when we started, it was very different kind of AI, ML, classical maintenance, decision trees, classical computer visions with kind of hand engineered features, you know, kernels and so forth. And then one of the first really important breakthroughs that happened in AI and computer vision, but really was important for our task was the advancement in convolutional neural networks right around 2012. Many of you are probably familiar with the AlexNet, and the ImageNet competition, this is where AlexNet blew away out of the water all other approaches.

8:01So that obviously has had very strong implications for our domain, like how you do computer vision and not just on cameras, right? How you run, you can use CONVNET to interpret what's around you and do object detection and classification from camera data, from wider data from your imaging radars. So that was kind of a big boost around that 2012, 2013 timeframe. And then we played with those approaches and tried to extend the use of confidence to other domains, just be able to be a little beyond perception, with some interesting but limited success. Then another big, very important breakthrough happened around 2017 when transformers came around.

8:40had a really huge impact on language. Like I mentioned, understanding language models, you know, machine translation, so forth. And for us, it was a really important breakthrough that really allowed us to take a Mell and AI to new areas well beyond perception. And so if you think about, you know, transformers and the impact that they had on language competition is that they're good at, you know, understanding and predicting and generating sequences of words, right? And in our case, we think about in our domain, about the tasks of understanding and predicting what people will do, like other actors in the scene, or the task of decision making and planning your own trajectories, or in simulation, generating, generative AI, our version of generative AI, and generating behaviors of how the world will evolve.

9:33These sequences are not unlike sentences, You're kind of operating the state of objects. There's kind of local continuity, but then the global context of the scene really matters. So this is where we saw some really exciting breakthroughs in behavior prediction and decision making and simulation. And then since then, we've been on this trend of models getting bigger. People started building foundation models for multi tasks. And most recently, all of the last couple of years, all the breakthroughs in large language models, and also in a modern state, modern -name, generative AI, visual language models, where you kind of align image understanding and language understanding.

10:13And there's been most recently one thing I'm pretty excited about is kind of the intersection or combination of the two. So that's what we've been very focused on. At Waymo most recently is taking kind of the AI backbone and all of the waymo AI that is over the years with build up that is really proficient at this task of autonomous driving and combining it with kind of the general world knowledge and understanding of these, you know, VLMs. One of the things that you just mentioned is the role of simulation and how that has been, you guys have had major breakthroughs in the use of simulation. And this idea in the recent breakthroughs in generative AI around synthetic data and its usefulness is somewhat in question.

11:04I would say in your field, this idea of synthetic data and simulation is extremely useful and you've proven that. So maybe you could just talk about the simulation technology you guys have built, how it's allowed you to scale, build that real world understanding, and maybe how it's changed in the last few years. Yeah, definitely. It is super important in our field. And the largely, if you think about this question of evaluating the driver, is it good enough? It's how do you answer that? There's a lot of metrics and a lot of data sets you have to build up. And then how do you evaluate the latest version of your system?

11:48You can't just throw it on the physical world and then see what happens. You have to do an simulation. But of course, the new system behaves differently from what might have happened in the world otherwise. So you have to have a realistic close loop simulation to give you confidence in the other. So that is one of the most important needs for the simulation. You also mentioned synthetic data. Yes, that's another area where simulation allows you to have very high leverage. I just got to explore the long tail of events. Maybe there's something interesting that you have seen in the physical world, but you want to modify a scenario.

12:25You want to kind of turn one event into thousands, or tens of thousands of variations of that scenario. Now, how do you do that? This is where the simulation comes in. And then lastly, if you sometimes want to evaluate and train on things that you've never seen, even our very vast experience. So this is where purely synthetic simulations come in that are not based on anything that you have seen in the physical world So in terms of technologies that go into play and It's a lot and that that is like a huge generative AI problem They do what but what's really important is that that Simulator is Realistic, right?

13:07It has to be realistic in terms of your you know sensor or perception realism I guess you it has to be realistic in terms of the behaviors that you see from other dynamic actors. If other actors are not behaving in a realistic way, like if pedestrians are not walking the way they do in the real world, you need to be able to quantify the scenarios that you create in simulation to the realism and the rate of cover currents in the physical world. It's very crazy to sample something very easy to sample something totally crazy and simulator, but then what do you do with it? So I think that that breaks me to the third point of realism, is that it has to be realistic and quantifiable at the macro level, at the statistical level.

13:55So there's any connection, there's a lot of work that goes into building the simulator that is large scale and has that level of realism across those categories. And they've intuitively think about it. To build a good driver, you need to have a very good simulator, but to have a good simulator, You actually have to build models of like realistic pedestrians and cyclists and drivers, right? So it's good, you know, it kind of do that iteratively. Yeah, of course. And then by having this simulation software that is very good at mimicking real world, and very usable in the sense that you can create variables in the scenes, you can actually give the driver multiples of the amount of experience that they have on the road.

14:33That's exactly real in real miles. That's exactly right. This is exactly right. I have driven tens of millions of miles in the physical world. At this point, we've driven more than 15 million miles in full autonomy. We call it a writer -only mode, but we've driven tens of billions of miles of simulation so you get orders of magnitude of an amplifier. Speaking of multiples of miles driven, one of the hotly debated topics in the AI world today is this concept of scaling loss. So how do you think about scaling laws as it relates to autonomous driving? Is it miles driven? Is it certain experience ad?

15:12Is it compute? Like what are what are the ways that you think about that? So model size matters. So we're we're seeing you know scaling laws apply to a lot of typical, you know, old -school models are you severely under trained and And so if you have a bigger model, you have data that actually does help you. You just have more capacity that generalize better. So we are seeing scaling laws apply there. Data, of course, usually matters. And but it's not just counting the miles, right? For hours, it has to be the right kind of data that teaches the models or trains the models to be good at the rare cases that you care about.

15:56And then there is a bit of a wrinkle because then you have to, you can build those very large models, but in our space it has to run on board the car, right? So you are someone who can be a trained, so you have to distill it into your onboard system. But we do see trend, we do just come in trend, and we see that play out in our space, where you're much better off training a huge model, and then distilling it into a small model, than just training small models. Yeah, I'm going to shift gears a little bit, and I'm going to do a sort of simplifying statement, which is probably going to drive you crazy.

16:27But the DARPA school of thought is, you know, there's sort of a rules -based approach, right? A more traditional kind of AI -based approach with a massive amount of volume, and you document edge cases, and then the model then learns how to react to those. The more recent approaches from some other large players and startups would say, hey, we just have AI from the start, make all the decisions, and you don't need to have all that pattern recognition and learning, like the end in driving, that is kind of a tagline out there. What is your interpretation of that approach, and what elements of that approach have you taken and applied inside of Waymo?

17:14Yeah, I think it's kind of, sometimes it's a, the way people talk about it is kind of Well, this weird dichotomy is this or that? Yeah, that. But it's not. It's that end and some, right? So it is, big models. It is end to end models. It is a generative AI and combining these models with VLMs, right? But the problem is it's not enough, right? So I mean, like I only know the limitations of those models, right? And that's when we've seen, you know, through the years a lot of these breakthroughs in AI, right? like condom, transformers, big end -to -end foundation models. They're huge boosts to us.

17:54And what we've been doing at Wayman through the history of our project is constantly applying, pushing forward these state of our techniques ourselves in some cases, but then applying them to our domain. And what we've been learning is that they really give you a huge boost, but they're just not enough. So the theme has always been that you can take you're kind of latest and greatest technology of the day. And it's fairly easy to get started. Like the curves always look like that. And they've been kind of the curves in their shaping, but the really hard problems in that remaining 0 .001 percent.

18:30And there it's not enough. So then you have to do stuff on top of that. So yes, you can take nowadays, you can take an end -to -end model, go from sensor to unit trajectories or actriation. You typically don't build them in one stage, you build them in stages, but you can do like backprop through the whole thing. So the concept is very, very valid. You can combine it and whether VLM, and then you add close simulations, some sort. And you're off to the races. You can have a great demo like almost out of the box. You can have an ADES or a driver's system. But that's not enough to go all the way to full autonomy.

19:06So that's where really a lot of the hard work happens. So I guess the question is not is it this or that? is this and then what else do you need to take it all the way to have the confidence in, you know, so that you can actually remove the driver and go for a full autonomy. And that's a ton of work. That's a ton of work through the entire kind of lifecycle of these models and the entire system, right? So it starts with training, like how do you train, how do you architect these models? How do you, you know, evaluate them? Then, you know, if you put in a bigger system, the models themselves are not enough.

19:35So you have to do things around them. You have to, you know, they have modern, gender, of AI is great, but there are some issues with, you know, hallucinations. hallucinations. You do it like, it's pointability. Exactly. Exactly. So, you know, they have some weaknesses and kind of goal -oriented planning and policymaking and kind of understanding this, you know, 3D -spatial world, right? So, you have to add something on top of that. We talked a little bit about the simulator. That's a really hard problem, you know, of itself. And then, you know, once you have something, you know, wants you to deploy it and you'll learn how do you feed that back.

20:04It's like this is where all of the really, really hard work happens. So it's not like end to end versus something else. It is end to end, and big foundation models, and then the hard work. And then all the hard work, yeah, it totally makes sense. That is a great segue into all of the progress that you guys have made, right? Writing in the Waymo for those who have done it is an extraordinary experience. It's not to say that you have solved all of these complex tasks, but you've solved a lot of them. What are some of the biggest AI or data problems that you still feel like you're facing today? The short answer is going to be taking it to the next order of magnitude of scale.

20:46Multiple orders of magnitude of scale. With that, come additional improvements that we need to make a great service. But just to level certain in terms of where we are today, You know, we are driving in all kinds of conditions. Yeah. Driving the 24 -7 in San Francisco, in Phoenix, a little bit, those are the most much more markets, but also in LA and in Austin. And all of the complexity that you see, go drive around the city, right? All kinds of weather conditions, whether it's fog or storms or dust storms or rain storms down here, like all of those are conditions that we do operate in. So then I think about what makes it a great customer experience.

21:34What does it take if you grow by orders of magnitude? There's a lot of improvements that we want to make so that it becomes a better service for you to get from point A to point B. We ask feedback from our writers. A lot of feedback we get is, it has to do with the quality of your pickup and drop of locations. So we're learning from users like we want to make it a a magical seamless delightful experience from the time you start to app on your phone to when you get a decision. So that's a lot of the work that we're doing right now. Yeah, pick up and drop off for what it's worth is an extraordinarily hard problem, right?

22:09Like, do you block a little bit of a driveway if you're in an urban location and then have a sensor that says, oh, actually, I just saw somebody opening a garage door I need to get out of the way. how far down the street is acceptable to go pull, or if you're in a parking lot, we're in the parking lot, do you go? Like this is an extraordinarily hard problem, but to your point, it's huge for user experience. That's a great, right? And it's just, I think that's a good example of, like just hey, just one thing, one of the many things that we have to build in order for this to be an awesome product, right?

22:41Not just like a technology demonstrator. And I think you just like, you hit exactly on a few things that make, you know, something that kind of at the face of it might seem fairly straightforward, right? Okay, you know, I know there's a place in the map and it pulled over, so like how hard can it be, right? But really, if it's a complicated, you know, a dense urban environment, there's a lot of these factors, right? Is there like, you know, another vehicle that you're going to be blocking? Is there a garage door that's opening, right? Like, you know, what is the most convenient place for the user to pick up?

23:11What is, you know, so it really gets into this, yeah, the depth and the subtlety of understanding the semantics and the dynamic nature of this driving task and doing things that are safe, comfortable, unpredictable, and elite to a nice, seamless, pleasant, delightful customer experience. Of course. Okay, so you've mentioned this stat, but 15 million miles, I know the number is probably a little bit bigger than that, but do you really say Tuesday? Yeah, it's growing by the day. 15 million autonomous miles driven. That's incredible. Even more impressive and you didn't share this stat yet, it results in 3 .5 times fewer accidents than human drivers.

23:55Is that right? And I think 3 .5 acts as the reduction in injury, and that it's about 2x reduction in the police reportable can lower severity incidence. This sort of comes to a question of both kind of regulatory and business or ethical judgment. What is the right level that you want to get to? Obviously, you want to constantly get better, but is there a level at which you say, okay, we're good enough? And that's acceptable to regulators. Yeah, so there's no, you know, simple answer, super simple, short answer. Right, I think it starts with that. It starts with those statistics that you just mentioned.

24:31Yeah, I can then have the day what you care about is that roads are safer. So then you look at those numbers, yeah, where we operate today, and we have strong empirical evidence that our cars are in those areas safer than human drivers. So on balance, that means a reduction in collisions and harm.

24:52Then, actually on top of the numbers, we've publicly been publishing this, you're quoting the latest numbers that we shared, and consistently sharing numbers as our service scales up and grows. If you can also bring in an additional lens of, What, how much did you contribute to a collision? We actually published, I think it was based on about 4 million miles, 3 .8 million miles. We published a joint study with Swiss RE, which is, I think, the largest global reader in the world. And the way they look at it is, you know, who contributed to an event. And there, we saw, like, the same theme, but the numbers were very strong that, you know, It feels 76 % reduction in property damage collisions.

25:38And it was in 100 % reduction in claims around bodily injury. So if you bring in that lens, I think the story becomes even more compelling. But there are some collisions where we'd be, and that's the bulk of the events that we see. We'd be stopped at a red light, and then somebody just plows into you. Sure. But then we do know it's a new technology. new product, so it is held to a higher standard. So when we think about our safety and our readiness, you know, framing methodology, we don't stop at just the race, right? We build over the years as, you know, one of the huge areas of investment and experience over the years, like, how, you know, what else do you need?