#152 Alex Kendall: How Close Is AI to Taking the Wheel?

0:00The world model for me is a model that can understand the state of the world and predict how it's going to change given an action you put into it. So mathematically speaking, it's a function that takes your current state, your current action and predicts the next state. Essentially a simulator. It's a model that can allow you to understand how the world will evolve given different things you might want to do or interact with that world. The result of this is a world model that can simulate the future. And if you want to, you can take that state and decode it back to video so you can produce the video output of what's actually going to happen.

0:36But you might not want to if you want to keep this real time and efficient. You can just stay in your embedding space and use it to drive your car. Hi, I wanted to jump in and give a shout out to our sponsor NetSuite by Oracle. I'm a journalist and getting a single source of truth is nearly impossible. If you're a business owner, having a single source of truth is critical to running your operations. If this is you, you should know these three numbers. 36 ,000, 25, 1. 36 ,000 because that's the number of businesses that have upgraded to NetSuite by Oracle. NetSuite is the number one cloud financial system, streamlining accounting, financial management, inventory, HR, and more.

1:2825 because NetSuite turns 25 this year. That's 25 years of helping businesses do more with less, close their books in days, not weeks, and drive down costs. One, because your business is one of a kind. So you get a customized solution for all of your KPIs in one efficient system with one source of truth. Manage risk, get reliable forecasts, and improve margins. Everything you need, all in one place. As I said, I'm not the most organized person in the world, and there's real power to having all of the information in one place to make better decisions. This is an unprecedented offer by NetSuite to make that possible.

2:17Right now, download NetSuite's popular KPI checklist designed to give you consistently excellent performance, absolutely free at netsuite.com slash ionai. That's I-ON-A-I-E-Y-E-O-N-A-I, all run together. Go to netsuite.com slash ionai to get your own KPI checklist. They support us, so let's support them. I'm Craig Smith, and this is Eye on AI. This week, I speak with Alex Kendall, CEO of Wave AI, to understand Wave's innovative approach to autonomous vehicles using a world model called Gaia One. Alex explains the advantages of world models, which we've explored before on this podcast, with Jan LeCun and how they can be used in AI agents.

3:21The discussion offers a unique view on the progress, promise, and obstacles in developing AI to act in the physical world. I hope you find the conversation as fascinating as I did. I'm an engineer at heart. I've loved building things ever since I was growing up and did so in the back garden. But I got the chance to work on a bunch of robotics in my childhood in New Zealand, whether it's building drones to chase some sheep around the field that I grew up in or other projects I did at university. But one way or another, they ended up taking me to the University of Cambridge. I spent many years there doing a PhD in research fellowship in computer vision.

4:06I'm fortunate enough to publish some of the first work that applied deep learning to scene understanding algorithms like semantic segmentation, depth motion, other forms of scene understanding. And yeah, I think that work really inspired some of the ideas that I've had to be able to build machines that can make decisions for themselves. Interestingly enough, a lot of my PhD work stopped short of understanding the future or doing future prediction, which is, I think, one of the big topics we've been able to address with world models and Gaia, which I'm looking forward to talking about. Yeah, well, that's fascinating.

4:48And as I said, I just had Jan LeCun on the podcast talking about world models. He mentioned Gaia. and so where do I start I'm interested in in world models as an alternative to large language models and Gaia won your model uh Jan says it's a little different uh than his JEPA architecture that he's using to research world models. So can you tell us a little bit about Gaia One? And then I have a lot of questions. I'm interested in marrying this tech with robotics, because the big challenge in robotics beyond the hardware is building an AI brain that can plan and make decisions and that sort of thing.

5:53And there's a lot of talk right now about LLMs being able to play that role, but I also think there are a lot of problems because of the hallucinations or the fact that large language models don't have a very concrete underlying model of the world. So why don't you talk about Gaia, how that came about, what it is,

6:27both generally and then how it's built, and we'll go from there. Well, taking a step back, maybe some background. I lead Wave, an autonomous driving company, and how we've set off on a different path to build autonomous driving systems that have the onboard intelligence to drive different vehicles in new places, including places they haven't been to before, and understand the complexity and the long tail of situations that you see on our roads. And taking an AI approach to autonomous is driving is quite contrarian and different to how people usually look at this problem. When we started six years ago in 2017, we set off to build an end-to-end neural net that could learn to drive, take the data as input and output a motion plan to control a vehicle.

7:21And this end-to-end AI approach, I guess, why are world models interesting here? So a world model for me is a model that can understand the state of the world and predict how it's going to change given an action that you you know you put into it so mathematically speaking it's a function that takes your current state your current action and predicts the next state so it's a it's a essentially a simulator it's a it's a model that can allow you to understand how the world will evolve given different things you might want to do or interact with that world why is this important for self-driving? Well, the first thing you might look at when you're building an end-to-end neural network to drive a car is to build something that's autoregressive.

8:03Build something that creates a function that takes your input state and produces the motion plan that you should drive with. And that's kind of what people did in large language models. And the problem with self-driving is it's a safety critical application. If you make the wrong decision, you're not just going to put out some hallucinated text, but it's life and death decisions of driving on our roads. So for that reason, it's really important that you are aware of the implication of your decision and you can understand the dynamics of the world. So that's really what motivated us to start off with world models as a concept.

8:35And in 2018, we actually published a blog with one of the first examples of doing this. On an autonomous vehicle, we published a model-based reinforcement learning system where, actually only on a quiet country road, but we learned to drive a car with a world model. So this system had never driven an actual car, it only learned in its imagination in a world model, but it used this model that it trained of the dynamics of the world to be able to learn to operate this car and drive it down a quiet country road. And I guess over the last six years, I can talk more about Gaia, but over the last six years we have scaled that approach to the point it is today where with the latest and greatest in generative AI, we can now understand the full, diverse, rich, dynamic urban scenes that we operate in today, like central London.

9:22And building the world model, I mean, the autonomous driving systems to date are, help me out here, but they're primarily reinforcement learning systems that are taking in data from various sensors and have trained on what the best approach is in any particular situation. Is that right? Or what is the current state of autonomous driving systems? Well, we're having an AI conversation and fundamentally autonomous driving is an AI problem. It's a problem of complex, high-dimensional decision-making. And so So you'd assume that you're going to use a data-driven method like reinforcement to do it. But actually, that's not the case.

10:15If you look at all of the large autonomous driving, if it's out there today outside of WAVE, you know, primarily the approach is a traditional robotics one. It's one of, yes, you use deep learning for perception, but once you have the state of the world, it's very much a, you know, hand coded optimization approach to produce a motion plan that's aided by a set of infrastructure like HD map, high definition maps that tell the car where and how to behave. So it's actually not an AI approach. And what we've done is, you know, I think the first time an AI system has actually driven on the roads at this level of scale.

10:51So that's not how the industry has traditionally approached things. When you bring in an AI approach, of course, you know, the challenges there are how do you understand what it's doing? How do you make sure it's safe and making the right decisions? And so that's brought up some of these challenges that led us down the road of world models. So, well, that's interesting. I didn't know that about autonomous driving systems. So they have all these sensors that data is coming into a central decision maker. And you're saying that decision maker is a traditional control system and not probabilistic AI system?

11:33Broadly speaking, yes. I mean, a lot of the systems running around San Francisco today, for example, are of that approach. Now, more and more machine learning is being used throughout over year on year in these systems. But it's not an end-to-end neural net. It's not a large transformer that decides the whole decision making. And that's the step that we've taken to replace that entire stack with one big neural network that learns how to drive end-to-end. Right. And I've seen a lot written recently about using the reasoning power of large language models to play that role, to decide on actions.

12:17And then, you know, with some other piece of software to translate that action, to execute on that plan. And can you talk about, well, first of all, for Gaia, so there's a world model that's building a state of the world in its, I guess, its weights. and then a reinforcement learning model that learns to act on that state of the world. Is that right? Maybe you can describe the architecture a little bit. Yeah. Let me jump into some of those details. We've got a research paper online that talks about them in great depth. But one of the interesting things for me is that, you know, if you look at the three big major trends that we've seen in large language models this year, I mean, at the start of the year, it was all about scale.

13:23Everyone was talking about how many parameters, how much data, how much compute are these models trained on. In the middle of the year, it became about multimodality. We pushed scale to some degree, and now it's about how do we understand across different modes. and a lot of image text systems came out, for example. And then more recently, it's about synthetic data. The benefits of synthetic data are clear. You can control the bias in your training data. You can ensure that the training data is equally sampled across the things you care about. You can control the distribution of your training data.

14:01And you can often get information that is harder to understand and from noisy real data alone. And so I think those three trends that have really driven the state of the art and say large language models, the interesting thing is we've seen the exact same thing play out in robotics. So for us at WAVE, we've been pushing the scale of our neural network that drives the car. And in the next year, our roadmap is going to be pushing this in terms of parameters, data and compute by 100 X further, two orders of magnitude further. And so the results, the emergent behavior we're seeing come out is just remarkable.

14:38The ability for the car to nudge its way through crowds of pedestrians, to do complicated unprotected turns, to predict the behavior of other agents cutting in or moving around our vehicle, all of this kind of thing emerges at that level of scale. The second trend on multi-modality, you know, that's where I think it's really important to be able to learn to understand between different modes because ultimately if you're training a self-driving car just off the video data it has, it's going to be intelligent but you know perhaps it'd be more intelligent if it doesn't only have that video data but also you know text and other information sources it has when you and i learned to drive um i learned to drive when i was 16 uh and you know i had what maybe my my mom and dad probably had had 20 or 30 hours in the car with me uh maybe not that long maybe i maybe i learned something like five or ten hours i think i hope i was a fast learner but you know that kind of length and time to learn how to drive um but But it wasn't just that that allowed me to drive.

15:36It was probably the 15 or 16 years of experience I'd had of learning how the world works, learning what objects are, how things might behave on the roads. And it was that observation that gave me the intelligence to drive. I think that's seen the same as true in robotics. We can train our autonomous driving system now, not just on the video data of it driving, but also internet, video and scale video and text. You know, we can literally feed it the PDF document of the highway road code that the government writes and give it that as, you know, further context to understand. And so I think multimodality is becoming really important to bring together different sources of information and improve the intelligence of your system.

16:14But then secondly, with tech specifically, you know, I think the future of how we interact with robots is going to be through language. We are going to be talking to our robots, interacting with them. you know there's a reason why you and i have evolved language as a way we communicate is because it's the most efficient way to get information across that that you know that we understand uh um and uh and so for that reason or maybe not most efficient but most natural way and so for that reason i think the accessibility of robotics will be greatly improved by us being able to just literally converse with it you should be able to be in your self-driving car and say take the next left take the next right drop me off here or i'm worried about this why are you doing that and you should be able to you know build a sense of trust through it and we've done exactly that at wave we've produced a system called lingo which is a first a vision language action foundation model that combines those modalities of video of action and robotics and language that allows us to talk to our autonomous vehicle and ask it why what it's doing and then finally uh the third trend on synthetic data and this is where gaia comes in uh gaia our world model um not only as a system that allows us AI to understand the implications of decisions it's making, but also produce synthetic data.

17:28Generative AI is very good at recombining data in new ways. And so we have lots of experiences of foggy scenes on the car. We have lots of experiences of jaywalking scenarios, but we have very few foggy jaywalking scenarios. And Gaia can not only allow us to understand how the world's going to evolve, but it can create new examples we haven't seen before. And again, we can do that by connecting vision, language and action. We can prompt it and say, literally give it a prompt and say, I want an example of a jaywalking pedestrian in the fog. Or we can take a scene that exists in the real world and change it and ask it to recreate it with new features and things like that.

18:07I didn't really get into the architecture guy and I'm happy to, but I just thought those three trends have been really powerful for us in ai and uh and you know uh exactly applicable to robotics as well yeah uh so you can tell from my questioning i'm a journalist uh not a practitioner uh so my knowledge is fairly surface but i understand large language models uh to a degree i understand the transformer algorithm and and And the large language models are predicting the next token in a sequence. And because of the volume of training data, it does a very credible job most of the time. A world model, I understand Yen's JEPA architecture, the joint embedding predictive architecture, to a degree.

19:08he says your model is is something different so can you kind of walk us through at a very high level of uh how uh the the model is trained and and what it's doing it's predicting the the the next state of the world whether it's video or text or whatever is it doing that with uh a transformer algorithm uh how is it doing it just i'm sure the the audience would also like to hear absolutely uh so i mean yan and i share a lot of common belief around these systems we think that um to go beyond the autoregressive nature of language models are just predicting the next word and getting to systems that can understand and be safety critical we need to have world models um we share the vision these should be unsupervised they should be able to be trained through self-supervision uh through you know whether it's signals like contrastive learning or building energy spaces or things like this um you know i think we share a lot in common there uh um i mean jepper is is a great architectural approach uh i you know i think um uh i think there's a lot in common these systems in terms of there's some representation space um you want to train it with with unsupervised learning and and so for our approach in particular with with Gaia what we first do is we take um you know we tokenize up the different inputs whether it's images action or language and uh essentially it's a well today it's a large transformer but you could use whatever your favorite flavor of neural network or let's say machine learning system that you want to use.

20:58But essentially you take those inputs and you learn this dynamics model, this ability to take current state, current action and predict next state. And then the result of this is a world model that can simulate the future. And if you want to, you can take that state and decode it back to video so you can produce the video output of what's actually going to happen. But you might not want to if you want to keep this real time and efficient. You can just stay in your embedding space and use it to drive your car. And, you know, right now we're working on Gaia 2 and Gaia Drive. And these are systems that very much are going to see Gaia embedded in the vehicle, able to actually increase the intelligence, understanding, and improve the safety criticality of our system in a production setting.

21:45So that's really the guts of the architecture. Today, it's a transformer that's able to predict future states. And again, forgive me. I'm sure you'll cringe at my repeating this back to you in sort of super layman's language. But you're encoding the data coming in or tokenizing it and turning it into embeddings in, I guess you call it a feature space or embedding dimension. And you're making predictions based on that at that level. So they're, and then to see the video, you decode it into pixels. But in that space, you can predict the, is it, how specific are those representations? When I said embedding space, I guess I meant representation space, yeah.

23:01One of the other big challenges of self-driving compared to large language models is the amount of data you have as input is enormous. I mean, take our latest vehicle, for example, it's six or seven cameras. They have eight megapixels each, and then you care about multiple frames over time, and not to mention if you consider an imaging radar as well, or choose the sensors you want to use. Just in cameras alone, the data, eight megapixels, your RGB value is there, so So, you know, 24 bytes, 24 million bytes alone from those, multiply that by the six cameras, there you've got about 120 million bytes, and then multiply that by multiple timeframes.

23:44I mean, you're talking about gigabytes of data there alone. And that, you know, that data, what it does is it makes it, you can't have an embedding space where you're dealing with gigabytes of data. It's just too much to process. And so to make these models practical, you need to be able to take that extraordinary amount of input data, all those videos, and compress them into an embedding space that you can reason about. So I guess the question is, how many factors do you think you care about for a driving scene? You know, you can list them out. You care about the positions of the cars in front of you, the pedestrians, the cyclists, the direction they're facing, the way they're going to move, the weather conditions, the traffic light, all these kind of factors.

24:27Now, interestingly, if you go down the path of trying to list them out by hand, you end up in the AV 1.0 or the classical robotics approach to autonomy. And that doesn't scale because it's very hard to enumerate all those factors and to reason about them a priori. So we can do it as a thought exercise, but I wouldn't advocate for that approach. But the point is, is that it's not, you know, hundreds of millions of numbers. There's probably a much smaller set of things that you care about there. And so we want to learn that. you know you don't care about the pixels in the sky except you want to know the weather but you don't really care about all those things it's there's a lot of redundant information in in the signal compare that to large language models you have sentences of text that are really high signal to noise ratio you know text is precise at saying this means that and impacts this right it's a direct description of what you care about compared to videos where most of the pixels in an image you don't care about clouds you know second story of a building you don't care about when you're driving So the point is that you want to be able to take this data and embed it in a very efficient space.

25:31And the way we do that is through end-to-end learning about, you know, what do we care about for driving? What actually is going to impact how the world's going to evolve? And that's what we look to learn. So we look to build a transformer and a self-supervised learning approach that learns an embedding that is really efficient, is as small and compressed as possible, but has the information that we need to understand the safety critical natures of the scenarios that we're driving through. So that's the primary task of that embedding model and that learning model of the scene representation. Yeah.

26:07And then right now that, you know, I've seen some remarkable videos that you've done and maybe I can, you know, splice one into this podcast. But they're awesome when you go drive in the car. I go out most weeks and when you see it, learn new behaviors week on week. Like just yesterday, I went for a drive down in part of London in Notting Hill and we went through Portobello Road Market. It's this crazy area with loads of pedestrians all on the road. the fact I've never seen our car sort of nudge its way through a crowd of pedestrians before, but it did so really safely in a way that, you know, if you just stopped and waited for the pedestrians to clear, you'd be stuck there for an hour.

26:53So these kind of new behaviors over time, there's tons of videos that we have online of this stuff, but it's pretty amazing seeing AI operate in the physical world like this. Yeah. The videos, so you're then decoding the representation space into pixels in a video, that's useful, I presume, for creating training data. But when you're actually driving in real time, how is that state of the world being translated into action and that's where i think you said there's a an rl engine or agent that's that's learning over time how to act on that can you talk about that part yeah absolutely um i mean one of the amazing things about gaia properties are that they can create photorealistic and diverse scenes that controllable by text you can modify these environments and you can also create multimodal futures i think the really powerful thing is when you only observe the past you know you can't predict how the future is going to unfold in a driving scene and the fact that gaia can generate diverse multimodal plausible futures is a really important a really important factor but uh yeah in order to actually control the car so this is what we call Gaia Drive.

28:35It's not just decoding to images, but also, well, there's many ways that you can go about thinking about incorporating Gaia into a driving system. For example, you could generate future data and use that synthetic data to actually just train a system. You could use it to predict the future and use the information it learns about predicting the future to improve the driving representation. Or you could actually bring it into a full on, you know, model based reinforcement learning, a model predictive control or some kind of learn simulator. What that means is that let's say you're in a driving, you're at a green light and you want to decide whether you drive through the intersection.

29:16What the system can do is it can perceive, run its world model, run Gaia for a few seconds ahead and see what might happen. Maybe it runs it a few times and sees how various different things happen. and then it can make a decision based on how it thinks the future is going to play out we do that in in our brains and in our hippocampus we have mechanisms that are you know most famously referred to as sort of thinking fast and thinking slow thinking fast the reactive decision making you don't really plan ahead you just you just do whereas thinking slow you take a step back and use a reason what might happen should i do this and you sort of play chess a few a few steps forward and we can do the same thing in robotics robotics has typically had you know two levels of control there's like a low level control that runs it over 100 times a second that controls parts of a robot and you have a high level control that operates typically around 10 times a second that does the high level reasoning but actually what these kind of models that you do is maybe move one more level up abstract and have a three-tiered system you can have a thinking slow thing which can involve a large language model to interact and to reason and to plan it could involve a world model to actually understand the implications of these decisions and that might happen at anything from one hertz you know one time a second to maybe even one time every 10 seconds it's quite an infrequent high level if you think about when you're driving that sort of high level kind of topological task planning you do can happen at the highest level then that middle tier you know you're deciding you're designing the motion plan that you want follow to ensure you don't hit things and that you follow road rules that's more reactive and it runs it at that kind of 10 times a second player and then 100 times a second is sort of the minute changes and breaks and and steering to make sure that you actually achieve that plan that you've set out and that three-tier uh system i think is uh uh so i think world models can play a really great role in that top one a new level of higher level reasoning into into robotics and uh the advantages of that over uh agents built purely off of large language models is that they're they're they require a less compute i would imagine but also um their their their decisions or their predictions are grounded more firmly in reality where as a large language model even if it's tokenizing pixels in a video it's it's only predicting one token ahead as it goes along is that right yeah look i think large language models and world models can be complementary right like large language models give you the ability to under well give you a text interface they give you a ability to interact through language but they give you the ability to learn a really incredible understanding of the world through internet scale text.

32:14And world models, the advantages of world models is they give you the ability to understand the implication of your decision, whether you are making a driving task decision or whether you are outputting sentences and text, you know, whatever it may be, it allows you to understand, okay, what is the implication of that decision in the environment I'm operating in? Is that a good thing or not and that understanding can help you do things in a safety critical environment so i think world models become really important in an application like self-driving maybe not so much in like an internet search problem but when you have safety critical applications whether that's in you know medical correspondence of language models or self-driving for for our application that's when uh world models give the ability to do that the other advantage i'll describe those is not just at runtime, but also at training time.

33:03World models give you ability to learn more efficiently. We do this in our brains as well. As people, we daydream or nightdream. When we dream, we actually go through a process that solidifies our actions and lets us replay experiences to learn to do them better next time. Yeah, like if you're learning to play tennis and you you hit a ball once you know you're not going to hit a ball with every single permutation of the angle that your racket might be to learn how it might go you might only hit it a few times but from that you need to learn the general way of how to hit a ball to get it in the right part of the court to be able to play tennis and what we do is from the ways that we hit the ball we actually um you know replay this in our internal world model many times when we daydream to be able to learn how to do that more effectively and the same is true with machine learning when you have a world model you can get more out of your training data you can replay recombine reconfigure and use that to understand your training data and and learn a lot more performant effective or safe policy or decision making system from your training data because of the fact that you can learn and recombine those experiences in new ways.

34:18So world models are really powerful, both training and inference or testing time. So, and you have this system not only creating training data, but acting as an agent to drive a car. How developed is that system? how far is it from commercialization for example how many cars do you have it in and how many road miles have you logged and that sort of thing yeah we've been we've been spending the last uh six years building a different approach to autonomy and where we're at today is we've been able to demonstrate that it can do a lot of things that have been blocking the industry for many years. It can drive on the kind of equipment that's on modern vehicles today, a single GPU computer, surround cameras, maybe a forward-facing radar.

35:14It can drive in different places it's never been to before, and it can drive different vehicle types. So we are very excited to be in the process of commercializing this technology now, to seeing it deployed across the world's most innovative to fleets and vehicle manufacturers, and to see this deployed in a way that can realize value quickly and accelerate the growth from assisted autonomy through to full autonomy. And so we are in that process right now. So far, we've been excited to partner with some of the UK's largest fleets, fleets like DPD, Asda, and Ocado Group. And these are large fleets that do things like grocery delivery here in the UK.

36:01And those partnerships have been wonderful. We've been delivering groceries throughout this year with our partners as, for example, in London, it's showing some of the value that this autonomy technology can bring to society. Yeah, just on that, I imagine you still have a safety driver in the car. How is the regulation? I don't want to get lost in that regulation discussion, but how are you managing to operate? As you know, crews in California has run into all kinds of trouble, but how is the regulatory environment allowing you to operate fleets, for example? Yeah, that's a really important question.