Ep 119: Former Waymo Leader Boris Sofman on Autonomous Construction & Bringing AI into the Physical World

0:00Digitizing the physical world is probably one of the most exciting things about the physical world. 85 % of our GDP is physical industry. At the end of the day, we have to build things. You became a key leader at Waymo. What was that like? The fundamental challenge of autonomous driving, you can't ship until you're really almost perfect. Statistically, it's five times safer than a human. You decided to move into construction. Why did you find bedrock? There's something like$120 billion of excavation contracts per year in the U.S. You can capture incredible signals and actually kind of do pretty magical things.

0:27The machine becomes autonomously capable. It's just a physical manifestation of the AI wave. A lot of people aren't going to understand that. They assume you're destroying jobs. How do you think about it? You can't hold this back at the end of the day. There's an inevitability to it. And the entire country will benefit from it.

0:50Boris Softman lives at the cutting edge of AI and the real world. He was a PhD in robotics at Carnegie Mellon with some of the top roboticists in our country. He built a robotics consumer company, a lot of really cool popular toys. and it was bought by Google. At Waymo, he helped develop some of the most important systems for self-driving, for trucks and otherwise. After Waymo, he's left and he's now revolutionizing another industry, which is construction. Construction is one of the largest industries in our country. It affects the cost of pretty much everything. Boris is taking some key lessons from Waymo and applying it to the construction industry.

1:21Most people have no idea how much there is to gain in construction. There's so many projects that are not quite affordable that we wish we could do. There's so many things that are broken with infrastructure and with our buildings. There's so many things that cost so much more than they should. So there's going to be this huge argument in our country from the Luddites against the AI. But talking to Boris, it's fascinating what's coming and how positive it is for our civilization. Excited for you to meet him. Boris, thank you for joining us. Yeah, thank you, Joe. My pleasure. I'm really excited to be here.

1:48Bedrock, of course, is a very exciting company we're going to talk about that's just doing new things with autonomy and construction. It's going to change a lot of possibilities in America. It's a big week for you. You guys have a big launch going on as we're releasing this. But let's talk about your background a little bit more before we go into real world and AI. Tell us about you. You're a PhD in robotics at Carnegie Mellon. Is that right? That's right. Yep. I was in Carnegie Mellon, both foreign grads, stayed for a PhD, focused on autonomous driving, where that was kind of an epicenter of a lot of that early research.

2:17And that was the type of work that really kind of got me into robotics. It was one of the holy grails of the entire space. What was going on in Carnegie Mellon when you were there? What kind of robotics work? So what's interesting is that at that point, nobody could imagine how aggressively we would get pushed to commercialization. At that point, it was, you know, the Robotics Institute existed since I think the early 90s was the only university that really invested in that degree. It was a lot of work funded by NASA, by DARPA, Caterpillar and Deere were actually starting to fund a little bit of kind of more kind of industrial applications.

2:52But it was a lot of like applied research for autonomous driving. So navigating on-road, off-road, starting to develop the beginnings of perception systems, path planning systems, being able to ship this onto large-scale mobile robots and actually try to drive intelligently. And a lot of the people around that time ended up being the seeds that then propagated to all the Grand Challenge, Urban Challenge, and then Waymo, Aurora, Uber, ATG. It was this really fascinating pocket of talent, actually. Yeah, a lot of great talent came out of there. And going back further, your Your dad's a scientist, right?

3:24You come from a little bit of that background yourself? Yeah. So I was actually born in the Soviet Union and immigrated when I was pretty young. I was about six years old when we left. Came into the U.S. through New York, ended up in Texas. My dad was a mathematician and worked in the telecommunications industry for a long time on kind of optimization of cell phone networks and data flow. And I've always been excited about engineering, math, science, and then really, really got into AI just being around it so deeply at Carnegie Mellon. And did coming from the Soviet Union to America give you an appreciation for business and freedom and building things at all?

4:01Yeah, it's interesting. Not even the probably like talking to my parents, the things that they went through. It's hard to even imagine. I try to grasp it. But they've been trying to leave for about 15 years before they were finally able to. and were really deeply involved in even kind of helping organize the first kind of batch of immigrants that were able to leave legally in a deal between the Reagan administration and Gorbachev administration. And then we immigrated and it was, you know, it was like a very, you leave everything and start from scratch. And actually, it was kind of interesting because like last year I hit the age that my dad was when he left the country, barely spoke like a tiny bit of English, zero assets left.

4:43Every asset he had, had a six-year-old me and my sister was one month old and it was like a fresh start. And so it kind of hit me hard. I'm trying to imagine what that would feel like. And it's really hard to put yourself in those shoes. Just to totally start over. Wow. Amazing what he's been able to do for his kids and what you guys have been able to do here. You've built multiple companies, of course, after your PhD, you started something called Anki. Tell us about that. That's right. So Anki was a consumer robotics company. So we're using really some of the most kind of novel and innovative technologies in robotics, AI, computer vision, and trying to create very interactive and entertaining consumer products, so toys and games.

5:20And we were able to use both really novel kind of technologies on the software side and leverage the smartphone industry actually on all the components that were dropping in price and pushing a lot of complexity to software. And so we made really interactive high-end games and toys that were super popular. They were like number one selling products for certain years and or something cool is for like almost like a Pixar character coming to life in the real world. And so you had these really popular consumer robots. It was working pretty well, though I remember I guess it's a tough business to sell almost all of your stuff in the Q4.

5:53Oh, it was horrible. The toy didn't appreciate how bad that would be. It's like it's you almost have to have a little bit of ignorance going into some of these industries to, you know, and then you kind of tackle the problems. But yeah, we were like it was the most seasonal business I've ever heard of still is 85 % Q4. And we were like, kind of close to a hundred million in revenue. And so your cash flows are spiking up and down like crazy. And you're locking in forecasts and end of the summer for what you think is going to be the demand. If a toy doesn't go quite as well, you're in big trouble basically every year.

6:22Totally. And every year you're over under, you never get it right. And so you're like, either, like I remember one year we were the number one selling product on eBay and the, the$150 product was going for a thousand on eBay. And then another year you're like massively overbuilt and then you're like figuring out how to do promotions and, and get it out. So it's, that's about as hard as it gets, uh, in terms of a kind of a business flow. That's a tough one. You sold on key to Google and you went there and you became, you became a key leader at Waymo, which everyone knows now we see him driving all around our cities.

6:53What was that like? Yeah, that's right. So they pulled us in where like a lot of the team, um, uh, the, The core of our company was basically these types of backgrounds that were from perception, planning, robotics, autonomy, electronics. And so we basically became the trucking team. And so I was leading autonomous trucking there and bringing up a second application. At the same time, I led a handful of the technical verticals, like the overall perception team, where we helped launch the Jaguars into San Francisco. and then we also eventually led reliability and freeway driving. We launched Ravos on freeways and it was pretty amazing.

7:34It was like this, you know, autonomous driving went for probably like 12 years where it was hype, but then it wasn't quite there and it was just like endless R &D where it felt like it was getting closer, but not quite there yet. Everyone thinks of Google as being like too cautious, but I guess like you have to be cautious on this thing. You're not allowed to release it. It doesn't work, I guess. No, so this is, yeah, exactly. This is one of those where oftentimes you think of like AI or products where it's like, yeah, get it out there and iterate. This is one of those where you're optimizing for the worst case, not the average case.

8:03This isn't search where you give some bogus result. Like you could kill somebody. Elon maybe iterates fast in public with XAI. We had to do some interesting things recently. But at least that's something. It's an eventful few days. It's very eventful. But it's not. I guess Waymo is like much more. And what years were you there? I remember I remember I was like drive by Google and Mountain View on the one on one, maybe a decade ago. And there'd be these cars with these things on them. The lighters. And I'm like just super obnoxious, you know, young guy at that younger guy at that time. So I'd like to test it and see like if I can get it to like break quickly.

8:37So like if no one else is around, I wasn't trying to cause any damage. It's kind of funny. Yeah. I'm sure you've seen much worse than me, but you're curious as a driver, like how will this react? And it was 10 years ago. When was it? Yeah. So it's 2019, summer through spring of 2024, so about five years. And it was like a really cool phase because it went from that kind of really aggressive R &D point to where that launch in San Francisco was the beginning of this incredible hockey stick that is like actually the beginnings of like real scaled commercialization for autonomous driving. And there's a couple of things that were amazing about it, where one, there was a real aggressive transformation of the technology stack to kind of embrace these very machine learning, kind of new age of machine learning and kind of data driven approaches, particularly on the behavioral side where you're learning from large scale data instead of engineering a system with heuristics and search and so forth.

9:30Yeah, I remember I was talking to Elon about this. It was maybe late last year when he said that like everything had been redone by the AI. Like it just got rid of all the heuristics because it was safer and better just itself. Yeah, and there's a couple of properties there where, you know, for Waymo, it's an interesting kind of a hybrid because at the end of the day, some layers of structured heuristics are actually helpful because of long tail safety challenges where you have some guarantees that no matter what happens, you're kind of like, you have some guardrails. But the way to solve the mass of the problem has to be ML.

9:59It just doesn't work any other way because without it, you have such a brittle system that is engineered and you run into a problem. You can go and solve it, but you'll create five other ones in this infinite space. and the further you go, the more complex the system becomes. ML is the inverse where the further you go, the better competency you've gathered and you end up using data to explain whatever gaps you have and those gaps are less and less the further you go. And so the system from San Francisco generalized really well to Los Angeles and Phoenix and Austin and even from car to truck where we unify the tech stacks and that jumped to 80 ,000 pound big rig.

10:35you're basically, again, using data to explain how the dynamics are different. And so you 10 % to 15 % more data. Oftentimes, we get the models to driverless level quality. And you end up getting this kind of like flywheel in that subsidy. When you say that quality, does that quality mean you have to be like 10 times safer than the human driver? Like, how do you quantify what's safe enough to put on the road? Yeah. So Waymo still has an incredible safety culture where, you know, today, like, there's over 70 million miles and statistically it's five times safer than a human it's like genuinely uh like it is saving lives from a on a from a statistical basis so if there'd be 20 deaths on average in those miles now there's four deaths yeah and i mean and there's been zero like really significant accidents almost every accident that actually has happened has been um you can kind of they've released safety reports where it's actually human caused by you know it's probably some jerk like me experiment oh it's not the word i mean there's like people that were jumping on the front hoods of the car.

11:35We actually had a rate on it.

11:39It would happen in different rates of different cities. And so you can estimate how civil a city is by how often people jump on the front hood of your car. Oh, that's funny. You can quantify it. It could be a very dangerous, politically incorrect issue if you start to quantify it too much. Let's be careful. It's a rating. Going back to the AI here. No, but the system, it actually then works And you start to really think about it, not in terms of geographies, not in terms of anything else, but what are the gaps or companies that you need to add on? And when you expand the new cities, the burden kind of shifts more towards the operations and qualifying the system.

12:18And there's always little things that you learn, but that flywheel is just working incredibly well where you've seen that exponential growth. So I want to push back on one thing you said on this before we move on, because I think it relates to some of your new challenges. You said you just have to add a little bit more data and adjust it for a truck. data sometimes. Well, sorry, but a lot more data for a truck. But like for me, just intuitively, like the way you safely drive, like a really large truck, it's going to have different braking ability. It's going to have to stay farther away. It's just like much scarier to me as a human, like thinking about trying to maneuver a giant truck, but you're saying the data still like really applies.

12:51So highways in general. So this was a big aspect of our focus over the last few years, just kind of unifying the car and truck stack, But highways in general have a pretty gigantic extra complexity where you can never stop. Like you just can't stop in the middle of the road like you can, you know, on a 20 mile an hour street. And so you have extreme reliability challenges and you have a really difficult sparsity of the events that you see on a highway. And so the foundations of how you drive that that holds like you can carry over and really learn. And you start thinking about how you, you know, how you interpret the environment, how you detect vehicles.

13:28how you think about lane changes and so forth. But the long tail becomes really, really challenging. And it's actually both cars and trucks where you're dealing with, you know, events that happen once every many millions of miles. And so the fundamental challenge of autonomous driving, which is why it took so long and so many billions of dollars, is that you can't ship until you're really almost perfect, like not perfect, but you're really beyond a human relative safety. and you have to solve every one of those types of flavors of challenges. And so there is no traditional kind of startup release process where you get something out, it's imperfect but pretty decent, and then you kind of keep improving.

14:05You really have to be able to handle everything that San Francisco can throw at you or a highway can throw at you before you can launch. And so even though the ML systems fundamentally do carry that property that things carry over, the cost of actually shipping a system is astronomical, including redesigning the platform itself. Yep, that makes sense. Well, let's shift. So you guys, a lot of top talent in the world, obviously crushed these problems at Waymo. Everyone's seeing what it's starting to do now. It's amazing. And you decided to move into construction and starting with excavators for the real world.

14:37So how do you make that shift? Why did you find Bedrock? Yeah. So part of it was really appreciating how incredible those scalability capabilities we just talked about were, where you start with a beachhead, in Waymo's case was San Francisco. So and now you have this fly where you're adding new capabilities, new geographies, new platforms. And setting aside the complexity of the public road driving domain, the generalization actually like really, really works well. We saw an opportunity to apply that to a space that never had that type of approach in it, which is automation of specialized heavy machinery.

15:14And construction was a especially exciting place to start where you have large numbers of machine types. that are all in these slow-moving, semi-controlled environments. You have astronomical amounts of work that needs to be done. You have an industry that fundamentally has these astronomical tailwinds where manufacturing has to be built at massive scale. You have onshoring of manufacturing, reindustrialization. Infrastructure is going through a refresh cycle where you have to repave giant amounts of roads. You have housing shortages. You have energy. So you have these huge demands. At the same time, labor is going the opposite direction where you're seeing the average age go above 50.

16:00The up to 40 percent of construction workers are retiring in the next 10 years. And so you have this like crisis level pain in this industry that's like about double digit percentages of our GDP. And it's a perfect problem for autonomy to help, you know, really diffuse some of these kind of constraints that are holding it back. And it's genuinely supply constrained. And it has very similar properties where you have a lot of different types of machines doing very diverse types of work, but you have pockets of incredible volume of work that needs to be done that is incredibly well suited for learning and automation.

16:35A lot of it's very repeatable, I guess. It seems like in some places it's easier and in some places it's harder, right? So the thing that seems maybe easier to me is you're not driving with like tons of other cars around. Safety is still a problem, but it seems like maybe not as… Way more controllable. You can iterate a lot more of something by itself practicing. So it's like probably easier and cheaper to iterate and learn. One thing that seems a lot harder to me is you're like moving on very uneven ground and weird hills and things which you don't encounter. And then you also have like an excavator where like maybe in some cases it hits dirt.

17:04Some cases it hits rocks. Some cases it hits something that slips. Everything moves. So you have to like model all sorts of different things here. Yeah, there's some gives and takes. So on the easier side, these are actually very, very significant where you can't launch a robo taxi as a legitimate product until you solve 100 % of San Francisco. We can actually be very surgical on the product kind of categories and mixes of machines and capabilities and become incredibly competent in a certain type of work. For example, heavy earthwork for an excavator. And that has astronomical amounts of value.

17:32It's one of the most capable machines, one of the highest volume machines and highest utilized. And it's the hardest to learn. and there's jobs where that's all they do for a year, moving gigantic amounts of - For a year, just moving things. Setting a foundation of a factory, right? So construction is like$2 trillion industry. Excavators might be some like, it's like 100 billion. Yeah, so the way I think about it is that there's one and a quarter billion hours of operator time operating these heavy machines just in the US. So one and a quarter billion hours a year. Wow. That's massively constrained because there's giant amounts of jobs that just don't pencil out and everybody has shortages.

18:04And every general contractor we talk to is turning down work that they can't handle. And so and that's just one foundation. There was a study we saw that there's something like one hundred twenty billion dollars of excavation contracts per year in the US. Now this is everything. Right. Like but and excavators we're seeing are oftentimes like twenty five, thirty percent of fleets. And so the fact that you can isolate the product side is a huge value add safety. You have a very slow moving, semi controlled environment. So the interactions that are at the core of the complexity for qualifying a vehicle on a public road, you now can actually have a much more controlled problem that for a variety of reasons, we actually can chop off a giant percentage of the complexity of safety.

18:45And the hundreds of millions of dollars it would take to reinvent a vehicle platform for public road safe driving, these machines are actually beautifully designed to where we can upfit, retrofit existing machines and go to market without a vehicle. So you could take something like a Caterpillar machine or whatever and just put stuff on it and go use it. Sensors, compute, and now the machine becomes autonomously capable for the situations it's cleared for. That surface area increases through software updates over time, and it can still be manually operated, and the product becomes the digital operator and the driver of these machines and all the services that you can kind of get beyond it.

19:19And so there you can actually start to gradually get that beachhead, And then you use this flywheel to add new capabilities, demolition, material handling, other kind of more nuanced tasks, new machines like wheel loaders, motor graders, compactors, new types of work. And you start that expansion, but you can start getting a really genuine business from the very beginning. Tell us about some of the problems with excavating. I'm just curious, what's the tech stack look like for when you're digging and it's going to have different types of ground? Does it learn over time about the ground from looking at it, from digging it?

19:52Can you tell what's going to happen? And like, does an experienced operator know, watch out, this is going to slip. And you have to learn that too. Like, how does it work? So there's maybe like two really big buckets of things that are super different and challenging about this. So one is the fact that you're like manipulating the world and you have these like interesting weird dynamics. The second is the interface on how you actually define it. So on the first one, you're totally right. There's this physics where you're cutting through earth. Clay is different than different types of soil. You have rocks, you have like pipes, you have all these different things.

20:20And if you screw up, you're just going to fall into it probably. So you could, there's actually some pretty bad things that can happen. And so that's actually like pretty modelable. You can like, you see the terrain model of everything. You can model the physics, what safe orientations you can be on. The harder part is probably what is inside the earth and how do you actually kind of The pipe being there that you destroy. The pipe, the rock, how do you not fight against it? How do you not cut fiber to DFW, right? Like, so apparently that's like somebody told us it's like a million dollars every few minutes.

20:51You cut that line or something. Don't cut that line. Don't cut that line. So there's like the lava guy to just make sure you don't cut that line. So anyways, so there's stuff like that. But when you think about the interaction of the soil modeling problem, what's interesting here is that you can treat like the broader problem is like a giant scale imitation learning problem. where you're learning from expert kind of demonstration and you're learning the nuances of how you interpret this kitchen sink of everything that's happening through massive scale data and picking up these nuances of how does this actually, how would you actually navigate these challenges?

21:25Very similar to how, in a way we can learn from human demonstration, how you navigate the kitchen sink of stuff that's on a public road. And so in some ways, the fight against the earth and everything, there's an element of that that you just directly learn because you see how people react to it. And the sensors data you're collecting is not just the data from the top of the machine. We're actually seeing all the signals inside the machine that give you kind of the pressure gauges, the like IMUs, the resistance. You feel that resistance and the fact that you're not moving as much as you could.

21:54And so you can tackle it through directly learning. There's also a lot of things that we're thinking about in simulation where we actually have simulations that model kind of like different Earth densities. and over time you can actually even think about modeling kind of huge diversities of properties of earth and just making sure that you're sufficiently able to navigate yeah i mean could you could say there's resistance to resistance here oh we found a large rock that's underground here that you actually have to change your strategy to remove this that's right you change your strategy you change your pressure you do all these things and um uh and so you can treat it as a reinforcement learning problem to some degrees and so there's a lot all sorts of interesting ways to kind of like isolate this out where um the interesting things about these physics problems is you don't even have to like perfectly model the earth.