Distilling Transformers and Diffusion Models for Robust Edge Use Cases with Fatih Porikli - #738

0:00The world knowledge representation capability of LLM's act as a regularizer or conditioner to generalize the solution. So we don't need to really go every time for each long tail specific, very rare scenario and try to learn them, model them. We want to harness the LLM's world knowledge with the efficiency of this vision-based low-level perception stack.

0:44All right, everyone, welcome to another episode of the Twimble AI podcast. I am, of course, your host, Sam Charrington. Today, I'm joined by Fatih Pridikli. Fatih is Senior Director of Technology at Qualcomm. Before we get going, be sure to take a moment to hit that subscribe button wherever you're listening to today's show. Fatih, welcome back to the podcast. You're somewhat of a veteran with us. Thank you so much, Sam. I really love this podcast and I'm very happy to be back. I'm looking forward to our chat. It's been just about a year since we last spoke and we are back once again to dig into Qualcomm's papers from this year's CVPR conference.

1:29And in particular, we'll be digging into a couple of them. the DEMA paper, which looks at distilling multimodal LLMs in an autonomous driving context, as well as the sharp depth paper, which looks at diffusion distillation for computing absolute depth maps and some of the new capabilities that that unlocks, which kind of speaks to multimodal foundation models. Of course, another unifying thread that is often present in my conversations with you and your colleagues at Qualcomm AI Research is the idea of efficiency and distillation. And that's a key theme in these papers. But I guess let's jump in and talk through the DEMA paper.

2:15The full title of that paper is Distilling Multimodal Large Language Models for Autonomous Driving. Maybe let's start with kind of your broad take on the state-of-the-art autonomous driving and in particular the shift towards end-to-end autonomous vehicular systems? DEMA is about autonomous driving, as you just mentioned, and specifically it aims to provide safe motion planning in long-tail scenarios, rare events, and those rare events are very critical events because those represent accidents scenarios as well. So we don't want them to help, right? Previous trends, it was mostly modular systems, they sequentially and independently trained models, components.

3:06And the reason for that, there are kind of smaller data sets and, for instance, for object detection, vehicle detection, lane detection, lane marking detection, road detection, road segmentation. So there are small data sets and it's easy to maybe train such component than, you know, kind of build a larger system. But when we do that, it is, you know, kind of easy to train such things quickly. But since we are now focusing at each module, their own performance goals for better segmentation, better detection, better tracking. it doesn't mean that at the end, you know, altogether they will work in harmony.

3:50And our goal is to drive safe, right? Drive our destination. I mean, when we are driving in our brain, we don't really segment the things, detect the things or track them explicitly. But so this is the shortcoming of such approaches. And end-to-end now is a breakthrough in a way that training objective applies to all components at the same time. So we are not really focusing on test-specifically trained modules on limited dataset. But now, you know, kind of we are trying to optimize everything, all the processing with the end goal in mind. That's why it is called end-to-end. Frequent listeners to the podcast will know that this idea of physics-based versus model-based or modular versus end-to-end is kind of a battle that's been raging in the machine learning community and these various communities like autonomous driving and robotics for quite some time.

4:54We've been talking about related themes for many years here on the podcast. There is a big reason for that because end-to-end systems doesn't only offer better, more robust solutions for a wide variety of long-term scenarios, but also they are better in terms of interpretability and their KPIs are much better. So DEMA is an end-to-end solution. It establishes the new state of the art, literally 25 states of the art for autonomous driving and AI. Can I pause you there? Because you said something that when I think about modular versus end-to-end or when I reflect on the conversations that I've had with folks in which this comes up, it's usually positioned as given enough data, we think it will perform better because we're optimizing against the thing that we really care about.

5:57But modular is still useful because of explainability and interpretability, because you can look at, you know, the output of each of these modules and use that to get some intuition about why decisions are happening at the planning layer. But you just said that end-to-end system has some interpretability advantages. Can you explain that? That interpretability is more the one that I implied is semantic interpretability. For instance, we have AI planner. There is a planner at the end, right? We do all such low-level perception things that I mentioned to make a decision to whether accelerate, slow down, change lanes, you know, or turn, those type of things.

6:41And when you are sitting in a vehicle, you know, kind of, yes, like maybe modular system can tell you how many people are in the scene or, you know, kind of what are the locations to the positions of the kind of other vehicles. But kind of when it is, it decides to change the lane, you know, you don't know why it's happening. It is a very stressful experience if you do not know why vehicle is making such decisions. So Entrance Systems and DEMA in particular, the way that we kind of like incorporated the language model into overall system provides explanation. Look, I'm slowing down because there's congestion ahead or we are approaching a zebra cross.

7:30So analogous to explaining thought traces in a reasoning model, the model can to some degree explain what it's doing. Yeah, high level explanation, useful explanations. But you are right also, you know, kind of like the modular system, we know kind of their detection results for instance, the thin advanced system. And sometimes they do not need to reveal that because maybe that is not critical at that point. And that's why I described it as a bit of a battle between these. Okay. I'm trying to be like a judge, but honestly, I'm more kind of supporting end to end at this point. Yeah. All right. So, sorry, I interrupted you.

8:20You were talking about... I was talking about DEMA's kind of position advantages. So for DEMA, I mentioned it is the new state of the art. It can reduce the collusion rate, which is a very important KPI for autonomous driving system, 80%. It's respect to the 2024 or latest 2025 baseline SOTA solution. So it is a big improvement and also 40 % reduction in, for instance, waypoint trajectory estimation and other KPI. Let's say I know how vehicles should drive, how close is planner making better decisions. So that is also kind of a 40 % reduction in the error rate. So more than 40 % improvement in accuracy for long tail.

9:15So in the paper we show, it's a long paper, and we have many results, comparisons, and, you know, kind of I invite everyone to take a look at that. So it says the new SOTA, we compared with the previous SOTA, including VAT, Vectorized Autonomous Driving, which is an amazing paper also. And this NEMA can also do two things that they are not in the training data. For instance, there was a zero-shot scenario about three-ponged turn. We didn't have any such example for planning, but then when we gave it for testing such examples, it was capable of, you know, kind of doing the right planning. And then I mentioned that it can answer questions or the system can proactively, you know, kind of system prompted and then provide explanations while it is driving itself.

10:11You know, it can tell you why it is making such decisions. So overall, it is, you know, there's a low-level perception state. There's an end-to-end planner, AI planner state. They work together. It integrates all the perception coming from the cameras and other sensors, the vision-based front-end low-level perception into this language model, LLM-based, you know, kind of AI planner. This improves generalizability. It improves the robustness. It is now the explanations are semantically grounded. And also, we have two versions. You know, we can run this AI planner, a transformer-based model, still leveraging the language model or language model itself, as the overall AI planner.

11:03So both of them are in the paper. Both of them are better than the existing SOTA. Is the transformer-based model the distilled model? That is the distilled model, right. We are using the model to distill it, yeah. When I think about the idea of using LLMs in an autonomous driving context, the elephant in the room, if you will, is that these things are slow, right? inference for LLMs is very expensive. And I'm presuming that's where the distillation in DEMA comes in. You know, kind of. Yes, absolutely. You have a very good point, LLMs. At this point, of course, we are talking about the systems that we can run on the vehicle, not like on a very expensive H100, you know, GPU.

11:56that would be more expensive than the vehicle itself. So using this reasonable price accelerators, the token rates are not that high. Maybe I can say that depending on the model, size of the model, anywhere from 30 token per second to maybe 1 ,000 token per second. But running everything end-to-end using an LLM would be much more compute intensive than the transformer model. That's why we have the transformer models. We are saying, well, if you are interested in efficiency, here's the transformer model. Still leverage this still using the language model, but there is also a language model version of it, a multi-model version of it.

12:40You are right. Let me take a step back and kind of recap where we are. So we've got this autonomous driving problem. We're moving quickly towards autonomous driving solutions, a promising direction for autonomous driving is to introduce LLMs into the autonomous driving loop. Why even LLMs? Given we kind of jumped ahead and talked a little bit about the challenges or the cost of LLMs, like what is the role of an LLM in this loop and why do we even want it? Like when I think about what I'm driving. I'm not verbalizing questions to myself about what I'm seeing or what I'm doing. Why do we think an LLM should even be part of an AV system?

13:29Yeah, that's a good question. We were also asking the question, this question to ourselves. And the reason we want to use LLM is that they represent world knowledge, many things. So they capture all the knowledge, or most of the knowledge, yes, they hallucinate at the end when they are generating this muscle, but they are much better than the things that we manually crafted. So there is something there that would make it richer. The word knowledge representation capability of LLMs act as a regularizer or conditioner to generalize the solution. So we don't need to really go every time for each long tail, specific, very rare scenario and try to learn them, model them.

14:27We want to take advantage of this world knowledge of the LLMs to do it automatically. So that's why we want to harness the LLM's world knowledge with the efficiency of this, you know, vision-based low-level perception stack. And so is that world knowledge access via traditional prompting, or are we talking about doing something like taking the embeddings of an LLM and somehow directly accessing them to unlock this world knowledge? By the way, DIMA is not the first end-to-end solution. There is UNID and VAD. I mean, it's, you know, let's say big brothers, but we do, I think, better than them by incorporating LLMs there.

15:16So those models, they try to represent the scene in terms of vectors and replacing these, you know, kind of dense representations to reduce the cost and also, you know, kind of like provide better improved capability. So there is this thing called scene representation and I will kind of, we can also consider it as tokenizing. Tokens are very common term for language models, LLMs. So what goes into those models separately, let's kind of think them as tokens and visual data. And this visual data is coming from multiple cameras, six cameras, eight cameras, and for each camera or better we are taking them and low-level perception projecting them into a bird's-eye view map and then we have things that ego vehicle does and then we have things for other agents other vehicles and the things moving in the same pedestrians and also there is map we have a map so altogether so let me hit pause there so the ego view is kind of this bird's eye view and that is familiar.

16:31Yeah, I've got that in my car. Like it's got cameras on the mirrors and in the back and in the front and it kind of projects into this top down view. Top down view. That's kind of the... Exactly. So top down view vehicle, ego vehicle is in the center and everything is around. Looks like 2D from top down. And so the other, did you say the view from other vehicles? No. the tokens or their kind of not necessarily the views, but the useful information that they kind of provide into. So maybe localizing other things in the scene or something. Yes. And also their previous trajectory, their headings, that type of information, not necessarily available in one bird's eye view picture, but because they are about other vehicles, like maybe also how they intend to drive, which is not available in Berzavu.

17:36So this system is not only, you know, looking at a frozen Berzavu, but, you know, thinking on behalf of the other vehicles, agents, how they may move. Are they accelerating? Slow down, you know, do something unexpected. So we are trying to anticipate what the other person is going to do, other vehicle is going to do, like a real person, right? We are on a traffic stop, and then I'm looking at the other vehicle, I'm thinking it's going to go ahead, or it's going to wait for me to go ahead. So in other words, the LLM input is kind of a string of tokens that represent this top-down view from the perspective of the vehicle, but also almost like other features, you know, representing the vehicle's trajectory, what it thinks other agents in the scene are doing, whether those are vehicles or pedestrians, and like somehow mapping out the scene.

18:36So it's like a featurization of the space that the vehicle is operating in. Exactly. That is what we call as scene representation and tokens. Everything now kind of standardized in terms of tokens. We have this ego vehicle tokens, you know, coming from its own Qformer and then other vehicles tokens coming from their Qformer and Intense. And there is this bird's eye view scene at that. We are tokenizing it also. And there is MAP also. There's tokens for that. But they are not all the tokens. When we are training with LLM, these are going in, but then there is also a question, right? Or a prompt, text prompt going into there.

19:22And then this language model, what it does, provides an answer. And then we are, you know, giving in training time, we have the answers. We have this, you know, scene representation tokens and we have the question. So we are updating this language model. At the same time, we are also looking at these tokens are also updated. You know, this ego or agent or bird's eye view or map tokens, they are also updated. These tokens goes into language model. And then within this language model, when we are running layers, each layer has cross-attention and also self-attentional layers. We self-advice those tokens.

20:11So in a way that you can think that these tokens are now changing. In a way that they are more consistent and they are more useful and they are answering, for instance, this input prompt better. So at the end, we will have updated representations of those tokens. So some raw tokens goes into and then better tokens comes out. So those better tokens are used to update two versions again. Transformer versions, we can distill the planning transformer to capture, you know, how planning transformer will have some additional compute. So raw tokens will become better tokens. And then the other tokens will allow us to make better decisions.

21:02So that's why we are improving to performance these planning constraints. And I should note that there's a really good image or figure two in the paper that is kind of a block diagram of all the things that we're talking about. And hopefully if you're watching the video, we will have dropped that in so that you can follow along. Certainly if not, or if you're watching the audio, then we'll be including a link to the paper in the show notes and you can pull that up. part of what I'm trying to do here is reconcile what you're saying with the image which is you know it's obviously a static thing and so I'm trying to understand how the things that you're saying happen in time like so you've got you know I think your data set consists of at a given point in time, six images from the cameras on a vehicle, right?

22:04And you just have many of those. Is there additional data associated with that or is it just those images? We represent all of these current six images and let's say T minus one, six images. It could be T minus N also, but it looks like T minus one is sufficient. In terms of their tokens, there is this scene encoder And then... That scene in Kodo is kind of like the feature extractor that we talked a little bit about, right? It generates tokens, you know, at the end there are these tokens for the six images, current images and then previous images. We have those tokens already in the LLM. So again, two versions to kind of distinguish.

22:48If LLM is making the AI planner decisions, Once those are in the KV cache long context of the model, so model retains those. And then when it is going to make, for instance, a waypoint estimation or a driving decision, a planning decision, it will leverage those previous tokens up to the buffer size of KV cache. And then the transformer version, we use previous token instance and current tokens and update this planning transformer. It's a transformer, but it's not like language model, but transformers are the building blocks of language models, by the way. It's not explicitly asking a question, you know, it doesn't have that capacity, but it's with tokens.

23:38So those tokens, what I mentioned, are now in training time, we learn better tokens. And now we take it and we learn an adapter, you know, kind of for this planning transformer. So next time in the transfer... Meaning so you can get the tokens that come out of the scene encoder and give them to the planning transformer directly and get a prediction as opposed to going through all of the machinery of the LLM. Exactly. So that version is transformer version is more efficient. Now, C9 encoder, again, generates the raw tokens. It goes through this adapter and then it goes to the planning transformer.

24:17And that is, there's no LLM anymore in that version, but, you know, kind of a better transformer for planning. And this is the core idea behind distillation in general. It's like you've got this, you know, surrogate model in this case, a transformer or the student model that I'm referring to in this case that is, you know, it's a function approximator. And so the distillation process is kind of teaching it how to approximate, you know, the function. In this case here, the teacher model, it's output tokens given the beam tokens from the scene encoder or the function that we're trying to approximate.

25:02Absolutely. Of course, the devil is in the details, how you do the selection, how you do surrogate tasks, how you do VQA, is it part of it? What are those data sets and what are those functions? We tried many things and more details are in the paper. You mentioned surrogate tasks and VQA. DQA. What are surrogate tasks in this context? For instance, they are mostly around trajectory prediction. Some of them are about also mass token prediction, future BV token prediction to encourage the LLM to learn spatiotemporal cues, useful for planning and scene editing. Scene editing is a part of it because we have to hypothesize where the other things, agents, are going to be.

26:01So, I mean, this is in training time. How the surrounding agents, other vehicles will impact the vehicle's future path, you know, kind of. So this is what I meant by scene editing tests. But that's a part of the surrogate test or trajectory prediction. and there are a couple of more also. And so they're not necessarily tasks that you want to use, but rather they're exposed as tasks so that you have a separate loss function that you can incorporate so that the model attends to those things. Is that a fair way to think about it? Exactly. We are not explicitly saying here's the trajectory, here's the plot form equation.

26:49We are using that to make sure that the temporal aspects of drawing is captured through those surrogate tests, like prediction. I said prediction, right? The scene itself, the other agents, and these are future predicting. And also leveraging on what kind of it observed before. So the model would learn a better representation itself. But it is not like we are telling a model to go optimize for these future instances, something like that, or change it. It is through the training, it learns itself, changes its coefficients to better understand the dynamics of the traffic scenarios. And what's the role of incorporating a VQA component, visual question answering?

27:40Yeah, we want this capability, that's an important part also, this work knowledge to be a part of overall kind of solution. We do not want to just drive, steer NLM towards surrogate tests, you know, the feature prediction or, you know, generating better tokens for the planner. But we want it to be grounded also. So that is keeping it real, connected to the semantic information. For instance, those questions could be, okay, in this scenario, what are the safe actions to take for ego vehicle? That is literally the question and the answer. It could be the action is to brake gently to a stop. So for this input, we want model to be still able to generate such answers.

28:38given the prompt and all the visual data. So that is going to allow us to retain the semantic word information of this LLM without destroying the LLM. And also the second version, we can ask questions and then it can answer, you know, kind of, again, two versions. Transformer version doesn't answer the questions, but the LLM versions in inference time can answer the question. So it has two purposes, keeping it grounded, semantically grounded, still not destroying the world knowledge, but then also having capability to answer questions if you are running LLM as the planner. uh and so you've talked about the kind of the the outcome here so the state-of-the-art performance on some of these long tail on both trajectory uh error and kind of long tail collision detection or avoidance.

29:44What you didn't mention yet is any kind of SLAs or performance around the distilled model. Is it fast enough yet to be able to use in a real-time AV loop, or does more work need to be done to make the model more efficient? Distilled model we can run as of now on Qualcomm accelerator faster than real-time. It's a new solution. I mean, this paper is not for publication research sake, but it has a very practical purpose also. We hope to have a vehicle running around soon and showcasing this technology. Most likely it's going to be at CES, But, you know, kind of, I don't know the future, you know, kind of, but it's there.

30:45Given what you've done with this paper, where do you see this kind of direction of research going, this particular line of research? What's next? So we see that, you know, such models are useful. And other companies are also looking into that. And I see one trend of running both transformer and language model together. Language model is, as you know, kind of as we talk about, is slower, compute intensive. So maybe not that fast at this moment. So it is on a longer horizon, you know, in a lower frame rate. It is estimating, doing its own thing, you know, estimating waypoints and, you know, some useful AI planning decisions that the current AI planner using Transformer is also running on 3D or 2D perception stack.

31:35And then these are combined. So there is redundancy in the system. If it is something unexpected for the low-level perception stack, at least we have high-level understanding that would provide some safety mechanism for the long tail. So this is like a trend. But many companies, research labs are looking into that. So I think it will become more popular. But as important as this one, using language model in the vehicle to interact with the vehicle, decision mechanism, change its behavior, customize for a geospatial location, like for a city or for a country, because rules are different and driving dynamics are different and preferences are different.

32:23Maybe you want more aggressive driving or more conservative, you know, kind of. Yeah, they are also happening. I think kind of, yeah, I can say that we actually, we are in the process of having an even more capable version of DEMA, which is domain adapting to some of the scenarios that I mentioned. Very cool. Very cool. All right, so shifting gears to the second paper that we want to dig deep into. Oh, deep. No pun intended, is sharp depth. And the full title of this paper is Sharp Depth, Sharpening Metric Depth Predictions Using Diffusion Distillation. So distillation, again, a theme here. My impression is the general space that this paper is exploring is monocular depth estimation.

33:20So you've got a 2D image, a photo, and you want to essentially reconstruct it as a 3D, in a 3D space. Is that the right way to think about this? Very good, yeah. You said it very well. And so talk a little bit about, you know, the background here and state-of-the-art prior to this work, and then we'll dig into what this work continues. Absolutely. So this is monocular depth estimation, but more specifically, this paper chart is about metric depth estimation. You don't know whether it's a tiny chair or a huge chair or, you know, some reasonable chair. So there are two ways, two separate, you know, type of algorithms.

34:05One are maybe, let's call them as generative models. And those are fine-grained. they provide very sharp molecular depth estimation because they can use synthetic data. There's a lot of such data like Marigold and Lotus and many molecular depth estimation algorithms are like that. So they have also dense ground truth because it's synthetic, it's there. But their skills, it is point. It is not like inches or millimeter or anything like that. You don't know what that point each pixel, how big it is. It could be one meter or one millimeter, you know, very different. So there are... It's relative within the reconstructed image, but it's not absolute.