Waymo's Dmitri Dolgov: 20 Million Rides and the Road to Full Autonomy

0:01We have an unbelievable treat next. A founder who's touching a ton of lives and who's been at it for a very long time. How many people here have been in a Waymo? Okay, well, that's a relief. The chef eats his own food. I do, I do a lot of it. All right, and how many people love the Waymo experience? Rock on. I'm a daily active now. It's incredible. Excellent. Thank you. We have here the creator, a man who has been at this mission for Get This founders who've been in AI since 2022, almost 20 years building in the autonomous vehicle challenge. And he has not only been at this, he's been at it in the great times, at the tough times.

0:52He's been persistent, and he has created something that is unlike anything else on earth. Truly exceptional. Please join me in welcoming Dimitri Dolgov.

1:04Thanks. Great to be here. All right, Dimitri. So we've got about 25 minutes together. The goal is to understand a little bit about you, what makes you tick, what has made you persist since the early days of the DARPA challenge 21 years ago, all the way through Waymo, early days to today and the future. Let's start on you, and then we'll get into technology very quickly. Sound good? Sounds good. Okay. So, Dmitry, you are known by your team as technically brilliant, incredibly intense, but also very kind and humble. You were born in the Soviet Union, raised in the States, and then chose to go back to one of the most prestigious, intense physics programs on the planet in Moscow.

1:45How did those first few years of your life shape you, and how did it shape your character? So my parents went to the same school so that to a large degree drove my decision to go back and went to high school. I actually traveled around quite a bit. I spent a year in Japan, then went to high school in the States and came back to college to do math and physics in Russia. So that was the same school that my parents went to. And I kind of really grew up hearing stories about what it is like to be at that place. I really wanted to go back. And I think in terms of how it shaped me, I think it really set the foundation, technical foundation.

2:21And in those early days in college, one of the most important things is acquiring the ability to learn and independently explore. So I think that really helped me in my future career. Now, you did this very intense program at Moscow Institute of Physics and Technology. And then you decided to keep going on the AI path. You earned your PhD also in AI. And then you pretty quickly were attracted to autonomous vehicles. In 2005, you were part of the DARPA challenge. Can you tell us about those early days? What drew you to autonomy? That was kind of a light switch moment for me. In the early days when I went to college and in grad school, it was more about just the learning of the fundamentals.

3:07and I didn't have at all a clear picture, an idea of what I wanted to do after. And then I think the timing was just incredibly lucky that this was when I was finishing up grad school, the urban challenges, the grand challenge, and then the urban challenge, the one that I took place in, were happening and it clicked. It was just, the technology is incredibly interesting. The mission is so powerful that nothing else come close, and it's a real product there. You can be hands-on, experience it yourself. So it really checked all the boxes for me. And as you said, that's been 20-some years ago. You know, who's counting?

3:48And then I've never looked back, and that's what I've been doing since. Amazing. So Waymo started out of a project at Stanford Automotive Lab. There were two sides of this building. There was the autonomy side, and then there was a solar car side. Fun fact, I was an idealist. I worked on the solar car. I got that bet very wrong. You bet on autonomy. Tell us about the first few years of Waymo from 2009 to the formative years. So we started in 2009. That was at the time the Google self-driving car project. and the first couple of years it was all about learning the problem space, understanding what it means to try to put an autonomous vehicle on public roads.

4:37So when we started in service of those goals of learning, understanding the problem space, we created a couple of goals for ourselves. One was to drive 100 ,000 miles total in full autonomy, which at the time was not heard of. And the second one was to drive 10 routes. Each one was 100 miles long. They were all over the Bay Area, chosen to be very difficult. And we had to do each one from beginning to end in full autonomy. I was still with a person behind the wheel that can take control. But the challenge was to complete each one without an intervention. So it was a small team of us. It was about a dozen people.

5:15It was the early crazy startup days. Everybody working 24-7. writing code and building hardware during the day, doing some testing at night. And it took us about 18 months to complete both of those challenges. Incredible. It seemed impossible at the time. Now you guys are on hundreds of millions of miles. Absolutely. Okay, so early Waymo days, extreme challenge, starting to achieve there. Next few years, you have a reputation in your team for grinding really hard. You were sleeping at the office. Tell us about Dimitri in the first few years of Waymo and how you formed your leadership style. I gotta say, those early days was probably the most fun I've ever had in my professional life.

6:05And it is that momentum and that pace of early startup days. when you are making so much progress every hour of every day, and you're doing everything. You are working on setting up the hardware in the cars and then configuring and calibrating the sensors and your pose estimation system. And then you're writing software during the day, and it's everything. It's the core of the software, the algorithms that drive the car. It is all of the tools and UIs and the user experience in the car. So you're doing everything. You're learning at an insane rate, and you're making progress at an insane rate.

6:47So those were the early days of Project Chauffeur. And then in those couple of years, we've convinced ourselves that, yes, this is worth pursuing. So we doubled down and started actually building towards the future of a fully autonomous product. Okay, so exciting first few years, intense, fast-paced, technically really difficult. Now take us to 2016-17 for a moment. This was a period where we actually had a hype cycle in AI. Turns out there's been a few of them. And AV, autonomous vehicle, was at the center of that hype cycle. I mean, I remember just so many companies going after this. And then there was a massive slump.

7:30And when most people gave up or failed or fell apart, you guys persisted. And you were a leader in that persistence. For all the builders in this room, how did you navigate through the hard times? So first a comment on what these cycles kind of look like to me and how I've seen them. And you said there's been many, some in an AV, but more generally. And what often leads to a cycle like this is some rapid, some breakthrough that leads to very rapid progress in the early parts of the problem and very rapid advancement in the early part of the problem. And in the AVs, the problem has always had this property that it's very easy to get started, but it's very difficult to take all the way to a real product, full autonomy, and superhuman performance.

8:25So it's somewhat natural, given those ingredients, that whenever there's been a big breakthrough in technology, whether it's convolutional nets or transformers or large language models, it's led to this cycle. well, okay, now the problem is going to be, and it kind of reshapes the early part of the curve, but it doesn't change the long tail of it. So for us, I think it was understanding that it's not going to be an easy problem, but it's a very important one, and believing in the mission. I think today, worldwide, somebody loses their life to a crash on our roads every 26 seconds. So I guess it's the combination of knowing that the mission is really, really important and then understanding what you're up against and not looking for kind of easy wins or quick solutions or silver bullets that help the team have the right stamina to go the distance.

9:30Brilliant. So you guys were in this moment where it was really easy to get started. A lot of people got there, but you guys actually persisted and got through to the other side with a truly magical experience. Pretty much every hand in this room went up, a truly magical experience because of that persistence. Let's talk about technology today. A lot of people are talking about world models. You have had all the components of world models for many years. How do you think about a world model and what is Waymo's version of a world model? Yeah there are a few things that a few terms that people use nowadays people talk about world models world action models omni models you know visual language action models and at the core of each there's an ingredient that is relevant and really important for Waymo and for what we've been building in our AI ecosystem.

10:24So at the core of our AI ecosystem is what we call the Waymo Foundation Model, and it powers three main pillars of our AI and our tech. It's the driver, the simulator, and the critic. And those are very related but distinct tasks. So at the core of what our foundation model needs to be capable of are things like it needs to understand how the world works, the physics, the dynamics of the physical world. And it needs to understand what it is to be a good driver and how the effects of the actions of that driver or our agent on other agents in the world. And then we need to instantiate those in the physical agent that we're putting on the roads.

11:15So in a way, that foundational model that we've been building over the years is a multimodal world action language model. It's multimodal in that it needs to be able to reason about not just images or video, but also other sensors like lidars and radars. It is a world action model in that it really has to have a deep, precise understanding of the 3D spatial properties of the world, the dynamics, the physics, the behavioral aspects of other agents like cars, pedestrians, cyclists, and so forth. and we are not just passively modeling those worlds, we're an active participant in it. So we not only have to, you know, the world model has to be controllable, but also we need to have a deep understanding of what it means to be a good agent in that world.

12:04And finally, it's aligned with language, and that allows us to kind of pull in the general world knowledge of a VLM into our model that is very, very useful and giving us a boost in understanding the semantics and the deep social aspects of driving. And we've been working on productionizing that model for years, and really it requires an extremely high degree of performance and accuracy and realism in every aspect of what we just talked about. Brilliant. So with this driver simulator critic architecture, there's also been a lot of conversation about end-to-end architectures. Is that the appropriate dichotomy?

12:51How do we think about the approach to getting us to extreme performance efficiency, autonomous vehicles that are totally generalizable? That's very clear. The world model that is described, the Waymo Foundation model, is an end-to-end model. So when we talk about an end-to-end model, typically mean that it's one model that goes from sensors to decisions or actions. And there's some very nice properties of such a model. One of the most important ones is that it learns the right rich representations between different components of the system, like the encoder and the decoder or the perception and the planning part of your system, as opposed to something where that interface is engineered, which is not sufficient for a task like driving.

13:40Now, I do think there's a false dichotomy there. There's end-to-end or something else. Really, in my mind, it's always been the question of, it's end-to-end, and then what else? And what else do you need to build if you want to have a product that is fully autonomous, has superhuman level of safety, and you want to deploy at a scale and drive hundreds of millions of miles. And there it turns out that kind of the basic vanilla end-to-end system is insufficient. So there's a massive difference between using end-to-end versus purely relying on it. So at Waymo, we've really gone beyond that kind of basic vanilla end-to-end approach, and we've augmented the learned representation with structured, materialized, intermediate representation.

14:27And what that allows us to do are a few very important things that you might not actually need if you are, you know, building a different product, if you are building a driver assist system or, you know, a prototype, a demo or a small scale deployment. But again, those things are absolutely critical if you want to go all the way to a fully autonomous safe system with superhuman performance. And those are things like having extra validation at runtime of the agent that's running on the car in the physical world. It's things like richer training and evaluation recipes that are very difficult or impractical to do in a pure kind of basic ATN system, where this structured materialized representation gives you a boost in things like closed-loop evaluation, closed-loop training, rich reward functions for reinforcement learning.

15:15So that's been our approach. And all the human feedback that you get from support and drivers dropping in and all of that, it's essential to have this type of architecture to do that. Exactly, exactly. Makes perfect sense. So not only have you innovated on the software stack, but also the hardware stack. There's a sixth generation now of Waymo driver, and you guys have always focused on being the driver. Tell us about the new sixth generation, and what was it like the first time you've interfaced with it? So the sixth generation is our most advanced hardware suite and sensor suite yet. the focus there has been on performance but also on simplification, drastic cost reduction and high scale volume production.

16:01This is the driver that's powering our latest vehicle platform, that's the Ojai. We earlier this year started fully autonomous operations. It's currently only open to employees but coming to all of the riders later this year. And I had a chance to take a ride in one as soon as we started running fully autonomous operations. And I spent a lot of my life in various generations of our cars. Every once in a while, there's kind of a new first moment, and that was definitely it. It's just the cool cars designed around the rider experience. It is, even though the external footprint of the car is about the same as the eyepiece, but inside you get in, it feels like it's a living room.

16:51There's so much space in the back. We have new screens. We have these doors that slide open and will open automatically when you approach the car. So I had a blast, and I can't wait to have this car in our fleet open to everyone. So you guys are going through a period of incredible scaling. For many years, you were in the lab, R &D purely. It took 16 years-ish to get to 100 million miles, six months-ish to get to 200. Things continue to scale really rapidly. 11 cities now, many, many more on the horizon. Tell us, what is it like to scale a new city? And then tell us about your daily life with Awaymo.

17:28How do you use it as a creator? Well, there's a lot. Okay, so exponential scaling. First of all, absolutely. It's been a phase transition for us in how we're scaling. So to give you a couple of additional data points, it took us eight years from the day when we started our fully autonomous operations to the day when we had our service, our driver, providing rides to the public in four cities. Earlier this year, just a few weeks ago, we launched four cities in one day. we've given over 20 million fully autonomous rides. Ten of those million happened in the last seven months. So that's what exponential scaling looks like.

18:15Amazing. Launching new cities, there's operational components show up. You have to collect the data, characterize the environment, validate the driver. A significant part of it is starting the conversation with the local communities because it's a new thing. It's a new product. So it's on us to earn the trust of the people there. And then more often than not today, we're seeing that the driver is generalizing incredibly well. And it's just a matter of high fidelity, rigorous evaluation and validation before we deploy the fully autonomous product. And then we grow from there. And then the last, what was the last part of the question?

18:57Oh, what is it? My daily life. It was a multi-part question, yeah. I mean, Waymo is how I get around nowadays. That's how I got here today. It was a great ride from Palo Alto up to San Francisco on freeways. My family use it. I have three kids. They love Waymo. I think nowadays they get annoyed if on a rare occasion we have to be in a car that's driven by myself or my wife or another human being. They're like, okay, what's going on there? I feel the same way at this point. They love it. It's been part of their lives. For the entirety of their lives, when we are driving around, there's two things.

19:41There's only two things that get cold outs from my kids nowadays. It's doggies and it's Waymos. Nice. Okay. Probably similar amounts of cognition between those two. Okay, so let's talk about safety. You know, one of the most meaningful, exciting parts of partnering with Waymo has been the fact that there's 1.19 million people a year on Earth that die in road accidents. It is this is life or death. Not only does it touch everyone in this room, but everybody has some connection who's been impacted by this. You have been about safety from the very beginning. And it's actually pretty hard. In a Silicon Valley where it's move fast and break things and see what happens, you guys have been incredibly patient with safety.

20:31Can you tell us about a story that made it very real to you and how you keep that safety culture at Waymo?

20:40So the numbers you mentioned, that's what drives all of us at Waymo. And the status quo is not okay. We've kind of grown this over time. but challenging the status quo is really important to everyone at our company. You're absolutely right that how you go about building a system like this is different from you might do in other areas, in other fields, in other industries, where safety has to be the non-negotiable foundation. And you have to build that into everything that you do from day one. your model architecture, your training and evaluation recipes, the mindset of the team, it can be very tempting to focus on capability first and get to the 90 % very quickly.

21:28But how you go about the first 90 % is a totally different problem for how you go about getting to your next N9s. So keeping that in mind and focusing on safety as the non-negotiable fundamental layer from day one is super important. And then, you know, today we're driving more than 4 million miles in full autonomy per week. And you see a lot of events from the field. And today we have the data over 170 million miles, fully autonomous miles, where we see that the Waymo driver is more than 13 times safer than a human driver when it comes to serious injury causing collisions in the cities where we operate.

22:16And you see that sort of superhuman safety behavior manifesting itself on the roads daily. I see examples of, recently it was a little while ago, there's an example that I saw of a person, I think it was a young woman, on an electric scooter on a road, and then she lost control and tripped and fell right in front of the Waymo. And the Waymo driver showed superhuman accuracy and reaction time, I was able to, you know, suburban brake and everybody walked away. So it's things like this that I myself personally and the team, the whole team, find very rewarding in terms of actually having a real impact on the safety of our roads.

23:01And the scale that we're operating, that 13x reduction means that we are preventing a serious injury every eight days. And that impact will just grow as we scale up. Wow. We're going to open the room to audience questions a couple in just a moment. But before we jump in, I heard a story about the LIDAR detecting or the radar detecting the footsteps of somebody behind a bus. Did that happen? And how does that work? Yeah, this was one of those moments where I was positively surprised by the emerging capability of our system. And the situation was, this was I think in San Francisco, the Waymo driver was at an intersection.

23:45There was a bus that crossed and we're sitting there waiting at a red light So the bus crossed and stopped partially blocking the intersection. So our light intersection So then our light turned green the Waymo driver started to proceed and as it's proceeding it detects a pedestrian on the other side of the bus and You can't see through the bus. It's you know not through lighters not radars not cameras you know the the windows are reflecting the people inside the bus and then you know starts to react defensively and sure enough a pedestrian emerges from behind the bus and then we're able to nudge around them and everybody you know goes on their way so when I saw that I was have it blew my mind I'm sure what's going on I guess capable as superhuman as the way my driver is it doesn't cease through solid objects so actually what turned out was happening is that our lighter had was balancing the signal you know under the bus and got a little bit of a sparse return from the movement of the person's feet under the bus and that was enough for the Waymo AI to not only detect that there's a pedestrian there but also make a prediction about what's going to happen in the future and keep everyone safe.

25:02Mind-blowing pretty unbelievable we've got time for one question from the group if anybody has a key Jim sorry no free codes not at this one no yes Jim please thank you this work yes I was just saying congratulations on all you've achieved it's really mind-blowing if you think about the next five to ten years really focus on the business model what are the milestones what happens in major cities what's gonna be different than where we are today just kind of walk us through your vision of the future. So we're heads down in execution mode. We've transitioned from intentional, sequential de-risking of the driver and key parts of the business to rapid parallel global commercialization.

25:51That means deploying the Waymo driver in more places across the United States. And today we're in 11 cities operating fully autonomously and serving our riders. That we're going to expand in those existing places. We're going to add new geographies, new cities. We're also expanding internationally. Announced that this year we'll plan to offer a service in London and in Tokyo. So you will see us just accelerating that deployment all in service of our mission. Good news to our team in London. Well, we covered a lot, Dimitri, from the very early days where you could get a lot of distance. with not a lot of technology to then persisting through extremely hard times in autonomous vehicles and getting that extra mile.

26:35We talked about world models, driver simulator critic architecture. We got into the hardware, the sixth generation hardware, safety and scaling. But most of all, I hope that we learned a little bit more about Dimitri, the man who's brought the magic that is Waymo to so many of us. and as I've gotten to know you more and more, I'm constantly struck not only by your brilliance and your persistence and performance, but also your humility. It says a lot for accomplishing this much. Thank you, Dimitri. Please join me in thanking Dimitri for all he does. Thank you. And the many lives saved. Thank you.