Autonomous Vehicle Research at Waymo

0:03Welcome to the Practical AI Podcast, where we break down the real world applications of artificial intelligence and how it's shaping the way we live, work, and create. Our goal is to help make AI technology practical, productive, and accessible to everyone. Whether you're a developer, business leader, or just curious about the tech behind the buzz, you're in the right place. Be sure to connect with us on LinkedIn, X, or Blue Sky to stay up to date with episode drops, behind-the-scenes content, and AI insights. You can learn more at practicalai.fm. Now, on to the show. Well, friends, when you're building and shipping AI products at scale, there's one constant complexity.

0:45Yes, your wrangling models, data pipelines, deployment infrastructure. And then someone says, let's turn this into a business. Cue the chaos. That's where Shopify steps in, whether you're spinning up a storefront for your AI powered app or launching a brand around the tools you built. Shopify is the commerce platform trusted by millions of businesses and 10 % of all U.S. e-commerce, from names like Mattel, Gymshark, to founders just like you. With literally hundreds of ready-to-use templates, powerful built-in marketing tools, and AI that writes product descriptions for you, headlines, even polishes your product photography.

1:24Shopify doesn't just get you selling, it makes you look good doing it. And we love it. We use it here at Changelog. Check us out, merch.changeall.com. That's our storefront. And it handles the heavy lifting too. Payments, inventory, returns, shipping, even global logistics. It's like having an ops team built into your stack to help you sell. So if you're ready to sell, you are ready for Shopify. Sign up now for your$1 per month trial and start selling today at shopify.com slash practical AI. Again, that is Shopify.com slash practical AI.

2:16Welcome to another episode of the Practical AI podcast. This is Daniel Whitenack. I am CEO at Prediction Guard, and I'm joined as always by my co-host, Chris Benson, who is a principal AI research engineer at Lockheed Martin. How are you doing, Chris? Doing great today, Daniel. Lots of, as always, lots of AI and autonomy to talk about. And you know what? We have Waymo to talk about as well. We have Waymo to talk about. Yeah. Speaking of Waymo, we're very excited to welcome back Drago Engelhoff, who is the vice president and head of the AI Foundations team at Waymo. Welcome, Drago. Thank you, guys.

2:57It's great to be back after five years or so, right? After five years. Yeah, we were commenting before we started the recording that the last episode with Drago was on September 1st of 2020. So that was episode 103. So a few things have changed in the world generally, but certainly in relation to AI. I'm wondering if you could maybe just catch us up at a high level, Drago, in terms of driverless cars, autonomous vehicles. How do you see the world differently now than you did in 2020? So one thing I would say is in October of 2020, we opened our Waymo One service in Phoenix East Valley to everybody.

3:46So, you know, just one month after we talked. But since then, we have launched and scaled quite dramatically in now five major metros. And this is San Francisco, Los Angeles, Phoenix, Atlanta, and Austin. And we are also serving hundreds of thousands of rides a week to paying customers. We are expanding. We announced expansion to at least half a dozen or no more cities that will be going on through next year. And we may announce yet more. In the cities we were at, we continue reporting the safety performance of our autonomous driver. And we are over 100 million autonomous miles driven on the road at this point.

4:41So it's fairly statistically significant. and in those miles our safety study at close to 100 million miles showed that we are five times less likely to get into accidents with critical injuries and over 10 times I think 12 potentially less likely to get into collisions or injured pedestrians so that has been happening and And we are on to doing more and more right now. I think we work on improving the driver further. We have a sixth generation vehicle coming up. We have started partnering with different companies. For example, we're partnering with Uber in Austin, Atlanta. So our vehicles show up on their app in those cities.

5:34We have partnered actually with Lyft in Nashville, if I believe. and we partnered with DoorDash to explore delivery. So we're exploring and expanding the scope and the partnerships that we are doing as well. But I think in 25, I would say a lot more people have had and continue having the opportunity to try Waymo. I'm quite a convert myself. To me, probably the aha moment, the big aha moment was in 22 when I got riding in San Francisco by myself fully autonomously. And so since then, it took some time for more people to get exposed. But now I think the phenomenon is out there. And I think also the autonomous vehicle industry went through cycles.

6:23There was certainly around 22, 23 time of pessimism in autonomous vehicles. But I think through our success, through generative AI, And I think there's other companies now. It's, again, a very lively space. There is others that are also trying to push what's possible with autonomous driving and robotics. So it's, again, very, very happening place. And, yeah, we are contributing probably, I would like to think, the most advanced version of an embodied physical AI today that you can do without. That's fantastic. I got to say, as a native Atlantan, I'm so happy that you guys are in my city. and we're a very, very car centric city as well.

7:08You know, you really have to have a vehicle to get around. And I noticed, you know, as you were naming the cities that you guys are in, that tended to be the case in terms of very auto. Does that play into any of the way that you guys think about testing in terms of being, you know, like Atlanta traffic for its size is notoriously bad. And I would love to see ever more Waymos and other autonomous vehicles here because I am terrified of all the drivers around me with our daily collage of traffic accidents and stuff like that. So I keep telling everyone, just wait, autonomous vehicles are coming.

7:44I'm kind of curious how you pick these different testing cities that you guys engage in, and what are some of the things that you're testing for that maybe those locations are particularly apt for helping out on? So, I mean, it's a bit of a combination of both technical and business reasons. I think we are trying to do large metros where autonomous vehicles can be a big market and help a lot of people. So that's one. And also we've intentionally been growing our ODD, so to speak, operational design domain. Our first service, Waymo One in Phoenix East Valley, Chandler, that's maybe a bit suburban with up to 45-hour arterial roads.

8:28And we learned to master it and then went to San Francisco. that is dense urban with fog and some rain and hills and windy roads and narrow roads and tons of pedestrians downtown. So we dealt with that. And then we started expanding. I think some of this is Atlanta is a big city, also different state. There's some differences across the various states in both how people instrument the roadway and how people drive. So we're spreading geographically more and more. I think also we're spreading to other domains. A few that are really top of mind is highways. We have been working on highways for a long time.

9:10We've gotten to a certain point with highways. Generally, to have a good taxi service, you need highways, right? And it turns out that's a very fascinating, interesting problem. They're difficult because Because whenever you move at high enough speeds, like 65 miles an hour or so, the consequences of any mistake are really high. And many things can happen. And so it pushes your robustness and safety capability there. So we've been doing highways. But one thing I did do is I rode. Now we can give highway rides to employees. And I rode one to Millbrae Station to get to the airport. and it's fantastic.

9:52So hope to be able to bring it in the future to more and more people. I think that will make the service a lot more useful. Also, we announced that we will drive in other cities like that have snow. So potentially even in 26, right? So our sixth generation platform is designed after the Jaguar, right? It's a Zikr vehicle and Gili Zikr. It's a Zikr, and that Zikr is designed with our hardware suite to be able to handle snow. And we are also heading out to other countries. So we announced that we intend to launch driverless capabilities in London next year. And London is left-side driving city. And so it's Tokyo, where we currently have vehicles and we're testing, right?

10:44So you can see we're trying to cover little by little the operational design domain of most large metros with all of their properties. We're, of course, also in Texas. That's its own unique state. But we started with most southern states, large metros, so you don't have to worry about snow at least. You want to tackle these challenges in some order, not just try to do everything at the same time. it's very difficult to validate your ability to do well in everything all at the same time, right? So we're just taking, we're kind of mixing what makes business sense with actually expanding the capabilities to become truly global driverless.

11:23And you started, you mentioned the driver, the car. I'm wondering if for those out there, those listening, which this is kind of maybe hard to do just from an audio standpoint, But if you kind of imagine the driverless car as a system in 2025, how would you kind of describe that architecture, that system? What are the kind of main components? Like I imagine, you know, the sensors, the actual car, the compute, like what does that system look like in 2025? Just at a high level. And then, of course, I'm sure we'll get into some of the modeling things and foundation models and all of those things.

12:03I mean, the car is, you know, ultimately it's a robot on wheels, right? The main distinguishing capabilities are that it has a set of sensors. In our case, camera, lighter, radar, and microphones. Our microphones are quite helpful for many things, including listening to sirens, right? And occasional instructions. Then you have compute on the car. It's a non-trivial amount of compute. It's more than you can put on a phone, right? and all our vehicles are electric. That was an explicit choice of the company. I'm personally quite proud of this choice. I think that's good for the environment to actually have such cars and I think can accelerate, I think transition to more electric vehicles, which I think is good personally.

12:51And so they have this robot on wheels with computer and sensors and then you have actuators, right? And then there is a lot of system design engineering to make sure, you know, steering and brakes and all these things, they need redundancy and robustness to make sure that if any system goes wrong or we need to think also if compute parts of it can go down, that you have contingencies, so it needs to be designed with redundancy. You need to think of what if, you know, steering wheel column, like there can be also issues with steering, what is the redundancy. So for autonomous vehicle, you need to think additionally and build these things into the hardware.

13:30So it's a robot designed for safe transportation from the ground up, even though we're using, we're just extending existing platform. And we work with the various automakers to do this. As you're doing this, and you guys have progressed over these five years, you know, since we last talked, you know, one of the challenges is not, probably not every person out there is a Chris or a Daniel who's very invested in this kind of technology going forward. You have a lot of people out there. Here in the South, we joke that every other driver thinks they're a NASCAR driver and stuff. And that notion of control and safety and the general population may not have as much confidence in some of these technologies because they're not following it closely and living it the way you do all the time.

14:21How do you approach that? And how has that changed over these last five years since we talked to you last in terms of getting buy-in from the public and getting them feeling, you know, like it's – as you talk about the safety statistics, which are amazing, but getting them to really feel that deep down, you know, inside that they can – that they know they can trust and believe in this mode and that it is, in fact, much, much safer than what they're typically doing on a day-to-day basis. So there is – you know, people do not feel statistics. It's hard, right? Because they're a product of many, many rides.

14:56You're doing 10 or even 100 safely is not enough. I think what people feel is when they get into the vehicles, and this worked for me in my high moment, even though even before, and also my wife and friends of mine, people get comfortable really, really fast. You need to pass a certain bar where they feel, okay, this thing actually is a really, really good driver. My mother-in-law sat in it just a few weeks ago for the first time. And she rolled around. She's like, this car drives much better than me, right? And once she thinks this way, she's immediately at ease. And I think people relax after the first several minutes are very exciting.

15:37And then they relax and enjoy the experience and mind whatever they like to mind, either the environment or their phone or other things. People get really used to it if you cross this threshold of can I trust you? and I think your driving immediately shows this. Now, us in the industry also understand that, you know, coming back to statistics, you need to back it up. And so with regards to backing up, Waymo, right, we believe in transparency and we're quite open with the incidents that happen. We file the details and we also track the statistics and do our best estimate. We have a great safety team.

16:19They publish these reports. In them, we evaluate and try to estimate how are we doing compared to a fleet of human taxi drivers or human drivers driving in our area that we are handling. And this is both by us, but also like their studies done by insurance companies who, of course, want to quantify this very well. And so there is a Swiss Rez study also proving our numbers. They also believe we significantly can decrease claims of different kinds for injuries, for accidents, and so on as well. So that's another external validation for the kind of thing we provide. So that's what I would say to people.

16:58Now, you know, it's a process. You need to work with the local communities. You need to work with police. You need to work with, you know, the various city stewards, officers. We train a lot of people. We engage with them. We work over time. I think you can see that in the cities we have been over time, I believe generally the trust in us increases. And I think that the satisfaction of Waymos by the users, if you look at the apps like in the stores, so I think on the App Store, we had a five-star rating, right? So there is a lot of, a bit of almost like people that would just use Waymos now, if they could.

17:40And that's a testament to the value that people see in the rides. But it drives, of course, to safety and ultimately engaging these people, getting them comfortable. Often when people experience this, many of them become converts. So I encourage people, try it. you may be the next convert if you have not yet I personally love it I take it as much as I can and it's always a pleasure working on a product you enjoy yourself so I feel blessed that way

18:28well friends it is time to let go of the old way of exploring your data it's holding you back but what exactly is the old way well i'm here with mark de poie co-founder and ceo of fabi a collaborative analytics platform designed to help big explorers like yourself. So Mark, tell me about this old way. So the old way, Adam, if you're a product manager or a founder and you're trying to get insights from your data, you're wrestling with your Postgres instance or Snowflake or your spreadsheets. Or if you are and you don't maybe even have the support of a data analyst or data scientist to help you with that word.

19:02Or if you are, for example, a data scientist or engineer or analyst, you're wrestling with a bunch of different tools, local Jupyter notebooks, Google CoLab, or even your legacy BI to try to build these dashboards that someone may or may not go and look at. And in this new way that we're building at Babby, we are creating this all-in-one environment where product managers and founders can very quickly go and explore data regardless of where it is, right? So it can be in a spreadsheet, it can be an Airtable, It can be Postgres, Snowflake, really easy to do everything from an ad hoc analysis to much more advanced analysis if, again, you're more experienced.

19:39So with Python built in right there and our AI assistant, you can move very quickly through advanced analysis. And a really cool part is that you can go from ad hoc analysis and data science to publishing these as interactive data apps and dashboards, or better yet at delivering insights as automated workflows to meet your stakeholders, whether they are in, say, Slack or email or spreadsheet. So, you know, if this is something that you're experiencing, if you're a founder or product manager trying to get more from your data or for your data team today, you're just underwater and feel like you're wrestling with your legacy, you know, BI tools and notebooks, come check out the new way and come try out FAPI.

20:18There you go. Well, friends, if you're trying to get more insights from your data, stop wrestling with it. Start exploring it the new way. Learn more and get started for free at fabi.ai. That's F-A-B-I dot A-I. Again, fabi.ai.

20:39Well, Drago, I understand that every driverless car company is going to have a different approach to modeling and all of those sorts of things. You've talked a little bit about the hardware and the car, but I think it would be good for people to understand. We talk about this driver, or you mentioned the driver. People might have in their mind, because we do talk a lot about models now after the generative AI boom, that there's this model that can reason and blah, blah, blah. And so people might have this view of there is a model that drives the car. Could you help us really break down, like in 2025, is this a system of models, models that do different things, a kind of combination of different types of models and even non-AI pieces?

21:27Could you just help us kind of generally understand how that works? So when you think of the stack, right, let's talk first about what it needs to do. It needs to perceive the environment using the sensors. It needs to build some representation of this environment. It needs to use this representation of the environment to make a set of decisions. And so traditionally, I mean, autonomous vehicles are around a long time. We have around over 15 years already, right? So it's a rapidly developing technology space, but traditionally you can think of, there's this, historically people thought, okay, there are these models.

22:09There's a perception model that builds a representation of the world that can be useful for certain things. And then there is some kind of behavior prediction and planning module that reasons what we could do. And potentially some people like to also reason what others could do to cross-reference our behavior with the other folks. And then based on all this information, eventually select promising decisions. So that's what the stack normally does. Now, there's different ways to implement it. Generally, the trend has been to have few, and in some cases, people claim they have one, large models, AI models on the car.

22:52And you can say ML or AI. For a while, it was called ML. When the models became big enough, people called it AI, right? So you have these large AI models on the car, a few or one, depending on the various companies. and they're connected in certain ways. You can train them end-to-end or not. That's also an option. Different companies can choose. The two are orthogonal concepts. Whether you have modules and whether you can train them end-to-end is different concepts, right? So it can be structured end-to-end. So essentially you have modules end-to-end, end-to-end. These are two. And so different companies on this very coarse taxonomy fall somewhere in this bucket, right?

23:32And I think Waymo always has used AI or ML since I've been there and it's been the backbone of our tech. I think over time, our models have streamlined and become fewer and fewer. I can say that. I think off-board, what my team does is build these large foundation models for Waymo that are not limited by how much computer latency constraints you have. and they can be quite helpful to essentially curate data or teach the models that actually run on the car in the simulator. We can get to simulators later. So we have experience with most aspects of these options, whether it's end-to-end and whether they're structured or not.

24:15I think off-board, I can definitely tell you we've explored a lot with large vision language models. That's one of the latest technologies that's relevant to us. I think in the field of robotics, people talk also about vision, language, action models, because you can tie in one model, you know, both understanding vision and language inputs and potentially ask for certain actions as outputs, right, which is ultimately what the robot needs to generate. So that's an exciting area that has developed in 2025. I think in our foundation model, Waymo Foundation model, we combine benefits of these vision language models, but also combine it with some bespoke Waymo architecture innovations.

25:02I think in areas such as fusing these new modalities that vision language models typically are not trained on, like LIDAR and RADAR is one. Another one is modeling the evolution, future potential evolutions of the world. There is some interesting Waymo technology on how to do this well that we also use. But we've used all of this and VLM technology, world knowledge from also other bases, whether it's a world model or visual language model into something that then is able to do well on autonomous driving tasks. So that's off board. On board, we don't typically talk exactly what is there, but I think we're trying to get state-of-the-art the best architectures that we believe solve the problem and put them together on the car.

25:54I think it's a really, really high bar to have a model perform in all the conditions and all the situations we need it to, right? And so we also have some notion of, as you know, VLMs also have this weakness of hallucination. So we have the safety harness around them to prevent hallucination, to double check what they're predicting, right? So we also have that aspect in our style as well, which we have worked on historically. So that's what I can say on a high level. I hope that's not too scattered. Maybe you guys, if you want anything specific, we can discuss that in a little bit more detail.

26:35So I do have a follow-up to that. And recognizing that you're not able to get into the specifics of how, of the architectural decisions and model decisions that Waymo is engaged in, if you could abstract it a little bit and maybe just talk about the space a little bit. I'm curious, as you talk about world models and having representation of the environment, that brings in not only AI, but the notion of simulation as one of the tools in the tool chest, if you will. I suspect we have a lot of listeners that are hearing lots of different AI use cases in general, but may not have as much expertise in autonomy.

27:18And so as you talk about that notion of representation of that environment, could you talk a little bit about what that problem looks like and what are different things that you might think of to solve it without having to get into how you guys have done it but just kind of like what is that what is that juxtaposition of simulation ai and representation of the world in the environment around you look like so maybe i mean simulation if we're gonna go there maybe i can just juxtapose two things there i like saying this historically i've been doing this for a while there are two main problems in autonomy one is to build this onboard driver.

27:59And another one is to test and validate this onboard driver. And both are really, really hard problems. And people usually talk about the first one. But I think, imagine there is some collection of models and you need to prove that it's safe enough to put them out in the real world. That's in itself a really challenging problem, arguably no simpler than putting the model, the first model together. And that one ultimately, because you need to be a bit more exhaustive, potentially takes even longer time to build the full recipe to validate things properly, right? So these are the two problems. Now, in autonomy, what is different maybe than the standard AI models is there's a few things.