Waymo’s Journey to Full Autonomy: AI Breakthroughs, Safety, and Scaling

0:05Hi, listeners. Welcome to KnowPriors. Today, we're hanging out with Dmitry Dolgov, Co-CEO of Waymo. Waymo started as the Chauffeur Project within Google back in 2009 and eventually spun off as its own company. Now it provides over 100 ,000 paid rides each week across San Francisco, L.A., Austin, and Phoenix. I love taking Waymos and I'm regularly campaigning for better South Bay coverage. We're excited to dig into all things robo-taxis, self-driving, what it takes to deploy this technology on a mass scale, and what's next for Waymo. Well, Dimitri, thank you so much for joining us today. Thank you for having me.

0:38Yeah. Maybe we can start off with just a little bit of a history of self-driving at Google. How you got involved and how things have evolved over time? GUSTAG WONEVITSENGERPENHAIER - I've been doing this for quite a few years. I think about 18 now. I got started in around 2006. This was the time of the DARPA Grand Challenges. This is when DARPA organized a few competitions. that they call the Grand Challenge in robotics for the purpose of advancing research in autonomous vehicles. And so the first competition they had was the first Grand Challenge. The challenge there was to create a car that could drive autonomously in a desert.

1:22I just completed a stack in Weimar, drive for about 100 miles. Nobody succeeded, but there's a lot of great progress that was made. And then they repeated the challenge and a few teams succeeded. So on the heels of that, they created another challenge called the DARPA Urban Challenge, where the setup was kind of a mock city that was supposed to imitate what driving on public roads is like. And that's the one that I was involved in. I was on Stanford's team. This was kind of my, you know, moment where it clicked for me. I saw the future and the benefits, and I've never looked back. That's what I've been doing ever since.

1:57And then we started this project at Google in 2009. It was a small group of us. And then that grew into what now is Waymo when we started the company in the very beginning of 2017. It seems like a lot of the lineage or history of this field all traces back to a handful of labs. It's like Sebastian Threm's lab at Stanford and a few others. And it seems like the founders of a lot of the companies that ended up eventually existing in this ecosystem all came out of the same sort of cohort of people, which I always think is fascinating to think about in terms of lineages. Yeah, definitely a small world.

2:29the CMU team and the Stanford team. There's a few people who came and started this project at Google in 2009 came from those teams. I think when you started working on this, this was considered a little bit of a crazy thing to do, right? It was early on. A lot of parts of the waves of deep learning hadn't really quite happened yet in terms of applications across all sorts of areas. Like AlexNet hadn't existed yet, like all these other sorts of things. Nothing existed, right? And you're absolutely right. people, you know, we heard a lot about us being crazy and never going to work in whole honesty with ourselves.

3:04We're not exactly sure if, you know, we are just a little bit crazy or, you know, completely insane when we're trying to go after this problem. Did you treat it as an open-ended research project or you treat it as like, hey, you had an idea in your mind of like an endpoint date where this would be viable on public roads? More of the former, but it was not, it was not research, right? So, you know, and then kind of the University of the Upper Challenge Day, it was a research project. Then when we started Google, it, you know, was under the belief that, you know, we can make it work. And if we can, then the impact, the positive impact of this technology on the world and the mission is worth it.

3:44But it was early days. So we actually, you know, had very little data to go by in terms of thinking about how long it's going to take and how hard the problem is. How long did you think it was going to take? Like at the time that you started? Well, I don't know if we had a specific date, but that was actually the first question that we posed. We said, you know, let's not build a product, right? In the first couple of years or so, we didn't have like a product in mind or a target data in mind. The first order of business was to explore the space. So towards that end, we created some milestones for ourselves.

4:17with the goal of prototyping and learning and just understanding. So after those two years, we said, ah, okay, you know, there's something there. Let's start talking about, you know, what the product could be. And actually our first product that we thought was gonna be viable was, you know, what nowadays you would call kind of an advanced driver system, right? And we had some expectations of, you know, a small number of years that it would take for us to get there when, you know, we, after working on it for a while and making more progress on kind of the core of the technology, we decided that was not the right path for us, that we want to go after full autonomy.

4:47That was, you know, that pivot around 2013. You're now doing something like 100 ,000 rides a week. So 5 million rides a year sort of annualized out, which is incredible. What was the inflection point or what suddenly caused that sort of volume to happen or all these things to come together? Because it feels like a reasonably recent phenomenon in some sense. Yeah, you're right. But there was a discrete, you know, some discrete jumps. So, you know, if we rewind the clock a little bit, I think in my mind, there were a few kind of generational discontinuous steps on kind of that progression from that point in 2013 when we said, let's go for it, to where we are today.

5:19That's exactly right. 100 ,000 trips per week, more than a million miles per week, and growing exponentially. So some interesting ones were in 2015, that was kind of our zero to one moment. This is when for the first time, we put a car in the road. That was what we call the third generation of our system. This was the third generation of our self-driving hardware suite, sensor of the computer, and we put it on a custom-designed vehicle that we call the Firefly. We took a few rides, but nobody behind the wheel, zero to one moment. Then the next evolution, that kind of generational skip, was our fourth generation of our driver.

5:55Those are the Pacific minivans with the fourth generation of the Waymo hardware suite. And we deployed those in a full autonomous mode in Arizona, in Chandler, and we actually opened it up to the public in 2020. But at that point, the focus was on doing it repeatedly. And the focus was on maturing the technology, the building of the driver, the evaluation of the driver, and of doing releases in a regular cadence, getting it out to real customers, hearing from the customers, understanding the feedback, and iterating. So that was the focus of that fourth generation was not to grow and scale and capture the market.

6:36right and then at that point we made the decision to jump to what you know we now call the fifth generation of the waymo driver it's on the jlr uh i basis this is what you see in you know in the fleet today in in those four cities uh phoenix san francisco la and austin that was very smart actually started in arizona versus in california and i think you know for example um i think crews ran into some issues in san francisco where there was activists like putting cones on the cars and trying to stop them and doing other things. And so it seems smart to start in Arizona. I was just sort of curious, what are the criteria that led you to start that as a sort of a test bed or a place to?

7:12So I guess, yeah, it depends on the different time horizon. So on the fourth generation, we picked a deployment area that was kind of medium complexity. And the goal there was, as I mentioned, is to kind of go end to end. So we picked an environment where we thought it was, we check enough of the boxes to help us learn the most important things that we wanted to learn and de-risk, right? And that was the deployment. And then there's the development of the system. So for the development of the system, you wanna go after the hardest problems possible, right? You wanna go after the densest environments, you wanna go after the harshest weather.

7:42So we've kind of in parallel been doing that. So we've made a decision to deploy, you know, in Chandler in 2020, to learn from the end to end system while pushing on developing the system. Then when we've learned enough and we made that discontinuous jump to the fifth generation of our driver, and then we said, okay, like that's the platform we believe that we wanna take to scale. What's the hardest environment for a self-driving car? So density matters, speed matters, weather matters. Okay. So where those come together is where the most complexity is. So New York in the winter is really bad. And then, you know.

8:17That's right. Yeah. That's right. But then this is why we picked San Francisco. It's good for learning and advancing the driver. It is a very interesting commercial market. We also, at the same time, went after downtown Phoenix. It has different makes more of the higher speed roads. And that gives us kind of the way we think about it in terms of the development and evaluation of the driver is kind of in the space of the operating domain, not necessarily areas or zip codes. And then you kind of take cities and you map that to the operating domain and you deploy it. And then looking forward is the kind of in terms of how we think about future cities.

8:50It is a few lenses. Is market? Is there a good market from the commercial perspective? What is the technical complexity? You know, what is the regulatory environment? And that's how kind of the lens that we apply. What were the big technology breakthroughs that got you to this fifth generation driver that is really the one that you think you can scale now? So the biggest one's an AI. Probably not surprising. Yeah, there was generational breakthroughs. And with every generation of the hardware, of the driver, there's new hardware. So it's getting more capable. It's getting simpler. It's getting more inexpensive.

9:24And there's a lot of simplification. But that's a boost, right? It really is all about AI, as you mentioned. And what specifically in AI was the shift or change that was important? Was it moving to sort of end-to-end DL for everything? Was it the transformer backbone? I'm just sort of curious, like, was there a specific thing there? So for that last jump, the models, you know, transformers, as you mentioned, played a huge role. Before that, you know, we had the big breakthrough. Before that, you know, it was ComvNess. You mentioned AlexNet. That was around 2013. So that gave us a big boost, but it was still kind of plateaued at kind of the wrong place.

9:53And then it was a few of those things coming together, right? It is transformers, it has bigger models, more compute, coupled with kind of the whole evaluation. We often talk about the architectures and what's really more important the kind of the machines surrounding the architecture. You need the architecture as an enabler, but really to make it work at the level that we care about, you need like everything around it. The data engine, the flywheel of training the system, evaluating it, and you kind of have to think about the problem of evaluating the driver in tandem of building it, right? And you need the simulator, you need the data.

10:25So all of those coming together, I think, is what leads to the breakthrough and discontinuity that you're seeing with where we are today. Can you explain how you guys think about evaluation internally and then also how that might differ from how regulators evaluate this from the safety case perspective? So that's a big question. But I think I'm glad you're asking that because it is a super important and insanely difficult problem. We often talk, again, about the building of the driver, but there's two problems. the building of the driver and the evaluation of it, and they go hand in hand. So internally, it starts with figuring out what metrics you care about.

11:04Then bringing the data to support the evaluation of those metrics, which we have hundreds. Then you need all the infrastructure, including things like the simulation. Some things you can evaluate in open loop, and some things you need closed loop simulation for. So you need to build a realistic, scalable simulator to support all of that. There are all of these metrics that guide our development of the system that help us improve and kind of train the Waymo AI, the Waymo driver. And then it funnels into what we call validation and evaluation. The aggregate of the evaluation and validation methodology is what we call the readiness and safety framework.

11:41You know, if you look at miles driven in an urban setting or, you know, some comparison to human drivers in a given city that you operate in, what is a relative safety level of what Waymo is doing versus a human driver at this point? We are pretty proud of our record. I think we have now, now that we've driven tens of millions of miles in fully autonomous, we call writer-only mode, and driving today more than a million miles per week. I think we can, with pretty good confidence in empirical data, say that we are better than human benchmarks. So we published some of that very recently on what we call the Safety Hub.

12:14The latest data point that we shared was based on 22 million fully autonomous writer-only miles. and we compared our performance versus human benchmarks by different severity levels of context of collisions, different severity levels. So we see, depending on the severity level, we see that for the lower severity outcomes, we're about a factor of two better than the human benchmark course. And as you look at more severe outcomes, the gap increases. So for the most severe, that's exactly what we want to see, right? So for airbag deployment type collisions, we're about a factor of six better than human drivers.

12:52And that's without any notion or attribution of causality or fault. So if you bring that into the picture, most of those are unavoidable. There's very little you can do. We have done a study with Swiss RE, the biggest global ratio in the world. And we partnered with them, we shared the data, they've done the analysis, and they found that for damage claims, we had about a 4x reduction versus the human baseline. And for bodily injury claims, we had a 100 % reduction. It was a smaller data set. This was earlier, but that was based on just a little bit less than 4 million miles. But also, you know, it was, from their point of view, statistically significant.

13:31And that, you know, again, we feel pretty proud of that. Yeah, it's amazing. Given that, what do you think the regulatory stance should be? I think we want to make sure that, you know, we enable this technology and service of the mission of making roads safer. And we've been engaged with regulators for many years and have that dialogue. And so far we've had good success getting all the necessary permits and all of that to a lot of scale. The way we think about it internally, and that's how we have the dialogues with the regulators, with communities, with writers, it needs to be based on transparency and it needs to be a responsible, iterative, gradual process, right?

14:11Because this thing is very new. The technology is very new. The product system is very different. If you're doing a million miles a week now, what prevents faster rollout? In other words, it seems like you've proven out that this is a safe solution. It's working very well at scale. You have your Generation 5 driver that seems to be quite performant. Why not go bigger, faster? We are scaling exponentially. It took us about three months to get from 50 ,000 miles to 100 ,000 miles. So we're moving at a good rate. But the main thing is... What is the total number of miles that are driven in the US per year?

14:48Old vehicles, I think it's in different modalities. Large vehicles, small vehicles, I think it's just over 3 trillion miles. So quite a bit. We have more something. But the way we think about it is that it's important for this to be an iterative process where we earn trust. It is not a thing where you build it and then you just turn on the switch. It needs to be, it's new, it's different, it needs to be a dialogue. It needs to be like, we talked about establishing the safety record, right? And we needed to build up to that, right? We collected in tons of millions of miles. Now we feel pretty good about that.

15:24So then we, you know, that gives us confidence. That earns us, you know, trust. So you have to be transparent about where we are. And then we, you know, grow a scale. What do you think is the biggest shift that's coming from a technology basis going forward? We had Andrej Alparati on our podcast a couple of weeks ago, and he sort of contrasted what he viewed as the Tesla approach, which is more, in his view, sort of software driven, to the Waymo approach, which was more sort of hardware centric. Do you think that's a correct characterization? And also, how do you view things sort of changing over the next, you know, a couple of years or year or two in terms of your roadmap?

15:57Yeah, I would think, yeah, I mean, it's all about AI. Full stop. We talked about, you know, a few big breakthroughs that allowed us to get to where we are today. You know, Continent, Transformers, Big Nest now, you know, most recently kind of combining the Waymo. AI with the congenital knowledge of, you know, VLMs. This is at the core of it. And, you know, hardware matters, right? We all have to operate in the physical world. So the hardware that we have on our cars gives us, I tend to think of it as an advantage, right? Like you have to see well, right? So, you know, as a human driver, right?

16:31Like if you can't see, you know, if you close one eye, you don't lose depth. If you, like, if you're, you know, your vision is not 20, 20, you don't have your glasses, like you'll still drive, right? with driver k but you know you're not going to be a scuba driver but the point i guess the the way i see it is it's all about ai it's all about building the system building the ai building the software and being able to evaluate and that's what we have today right so far i mean we have you know achieved good quality of the driver we have built all the machinery to evaluate it and know what it takes so now for us it's an optimization right it's a it's an optimization simplification and if you have that you know that the thing that works and you have a good mechanism to value it, then it really is drastically different in terms of how fast it can move.

17:15I guess I've been in this other mode for years where we haven't cracked the nut. And you're kind of hypothesizing what it would take and what is the yield of this technical breakthrough? And you're kind of climbing uphill, right? And this was old analogy of going to a mountain, you see the peak, then you get there and you thought it was the summit, but no, now you see the landscape. And that's what it felt like for us for many, many years. That's kind of the trend that the industry usually falls. And I just find that it's a qualitatively different place to be in when you've cracked the nut and you have the valuation and now you can optimize and scale.

17:45As part of your optimization, do you think of reducing or simplifying the sensor suite as an important priority? Every generation of the hardware would increase capability, but also simplify drastically. And then the cost comes down every generation. So that was true for the previous generations. We made a big jump from the previous generation to today's, and then from the fourth to the fifth, and then now going to the sixth generation. This was the primary focus simplification and bringing down the cost, as well as with a vehicle, making it about the user experience. So absolutely. And then, of course, the economies of scale, right?

18:21There's nothing like fundamentally, look at the components that we use in our hardware. There's just, all of them are fairly commoditized and with scale. You get to ride the typical curves of the hardware. Radars used to be expensive in the past, right? When you put them all in the cars, you bring them down the cars. Same with computers, right? I guess qualitatively, we know that people can drive well with a very simple vision system, right? And obviously, in the self-driving world, we have dramatically more and different types of sensors. Is there any sort of analytical framework that you all use in terms of the amount of data slash types of data you need to collect in order to have a performance system relative to the curve on the AI side?

19:00In other words, can you follow some sort of scaling curve on AI that you can pre-predict the set of sensors that you can do without? What we have now is, now that, to jump to the core of the answer to your question, now that we have cracked the nut of the driver and the evaluation, we can answer that question with data. So we have, we talked about different scenarios. Start with humans, we have just two cameras, but like on a pivot, right? And we can drive kind of okayish, right? And then if your vision is not very good, or if your windshield, like you maybe not drive as well as computers can, right?

19:36So that's why we have vision, we have cameras, we have lighters, we have radars, right? And that will give us some benefits. And we can talk about what the pros of that they all and kind of how they bring to the table, the data they bring to the table and how they're nicely complementary. And what you get from the different sensing modalities just by the kind of the physics of it. But again, in terms of how do we answer that question, for years it was more theoretical and you hypothesized how much of a certain thing you need. Now that we have a thing that works and we can evaluate it, we can bring data to the table, we can empirically, more empirically answer that question.

20:11Say, what happens to the way in the driver if you take away something? Maybe you add noise to the system, maybe you degrade, maybe you take away a full sensor modality. Maybe you take away lighter and you take radar away completely and you just drive with camera. You know, will this drive? Yeah. Can you use a human drive with one eye or, you know, with blurry vision? Yeah. Is that acceptable performance? Will you get a license? No. Same for us. We can, you know, take something away and we can answer that question. Like, is the performance, can you still drive? Of course. Is the performance good enough for full autonomy?

20:39And is it good enough for our bar of readiness and safety? And the answer is no. Right? And, but again, this is in the context of full autonomy. It is in a context of scale. And it is in the context of the responsible deployment and the high bar for safety and readiness that we set for ourselves. If you change some of those inputs, let's say you talk about small scale, or you talk about not full autonomy, you talk about a driver's system, then the answer changes. Like you might have a different, if you still have a human in the loop and they're responsible for safety, a different configuration, whether it's just you get rid of sensor modalities altogether or in, you know, maybe use all three modalities, but you pick a different operating point because, you know, it's cameras, it's cameras, right?

21:23Resolution, dynamic range, cleaning, you know, same for lighters. But, you know, the answer of your operating point would change based on kind of where you set the bar and what the product actually is. If you look forward a year or two, like now, you know, crack the nut, Waymo is looking at scale and probably more about the business. Like, do you think of robo-taxis at scale as the near-term business plan? Are there other modalities or like deployment? Like if I think about this as just like a CapEx problem, like other avenues that are important for you guys to explore. That's the main one. Right-hailing is the main one that we're focusing.

21:53So we're focused on technology. We're focused on the product. We are learning from our users. We're every day, we're earning trust. And we're setting up the ecosystem of partnerships in that space. So that's our primary focus, right? We're very excited about the commercial opportunity there. It's not the only one. I always think of Waymo as a technology company. we're building a generalizable Waymo driver, with the mission to build the world's most trusted driver. And we want to deploy that driver, not just in ride hailing. There's more than 3 trillion miles in the US. There's more than 10 trillion miles worldwide.

22:29So the vision and the mission is to deploy the Waymo driver in different commercial products and different applications and maybe different modalities across all of that spectrum. So that includes things like deliveries, There's things like, you know, long haul trucking. It includes, you know, things like personally owned vehicles. But right now we're very focused on ride hailing. Do you think you'd license that out or you think you'd actually build the vehicles for these different use cases? We're not, you know, we've never been in the business of building vehicles. We partner, right? And I guess this is how, like, this is, the mission is so important.

23:04The opportunity is so massive that we don't want to go it alone. So we think about partnering and weaving together the ecosystem to pursue all of those different commercial applications and all of those products. And you see us doing that in the right-hailing business. We don't build our own vehicles. We partner with Tier 1s. We partner with OEMs. We partner with other companies who help us on the operational side. We partner with companies who help us on the network side, so forth and so on. How do you think a car ownership will change over the next decade or two? So it's very exciting to see this ramp in terms of driverless rides that are happening.