Gen AI at the Edge: Qualcomm AI Research at CVPR 2024 with Fatih Porikli - #688

32:56It can be constantly learning or it could be distributedly learning across multiple people. But I think this is a very smart way of getting additional information to further fine-tune, tailor, customize such models, personalize such models. The next paper is called Math Search, a benchmark for multi-hop step-by-step visual reasoning over plots. What's the challenge there? Models, multimodal models, like the previous one was a video multimodal model, and this one is, let's say, image multimodal model. There's an image as an input, and we have this language model, and we are trying to do things around math now.

33:45So these models, they use, they take these input images and there is something called this image encoder, tokenizer, which could be a clip, VIT, or kind of a dyno-way to any type of image, respectable image encoder network, which takes this input image and generates tokens. So it is many of those solutions, they need to scale down the image and they are looking at, you know, patchwise information, texture information in that type of input. For instance, I have a picture of a plot, a plot, literally, maybe it's coming from MATLAB, you know, or some kind of Excel graph, you know. there isn't much texture and there are maybe small but important details like there is a curve, it is crossing some axis, you know, something like that.

34:47And the questions that we want to answer, like how many, what are the, how many zeros does a function t have? Like where does it crosses the, you know, kind of, let's say in this case, vertical axis, or list all the discontinuities of this function. So it has to understand this continuity and everything. Such information is difficult to learn. So this is a dataset paper. It introduced this benchmark dataset. In the benchmark, We provide more than 200 ,000 example plots, and we have 2 ,000 also questions, more than 2 ,000 questions in the test validation part. So we have single function plots. We have multifunction plots.

35:40We have plus within plots. You know, kind of, we have multiple plots. So this is a very challenging data set created, put together to allow research better solutions, better encoders, better fine-grained visual information learning for such multimodal models. And I got the impression with this one that the sole purpose wasn't necessarily to create models that are better at math or interpreting charts, but also that asking the model to focus on some of the details in these math pictures might help models that aren't specifically, you know, purpose to do math better. attend to details in other types of images.

36:39Because of the current image encoders, visual data encoders, they focus on the global information. And with this, we are motivating, giving momentum to further research that will leverage on the local information. And was the evaluation and the development, What models did it use? We have results in the paper on this data set, like how human would perform, how, for instance, there is this lava, which is a model which can take image and text and then do reasoning about that. There is QN, another multimodal model, LMM, large multimodal, some people call it like this. There is also chart lama. There is GPT-4V, the visual version.

37:34So we provide some baseline results for this, including human performance. We show that there is a gap between human performance reasoning on the mathematical math plots with the gap between human and the existing genetic AI models. and the gap is actually quite large, you know, quite big. On average, let's say human does 70%, human accuracy, let's say around more than 70%. But these models, even the best one, which is GPT-4V, is around 25-ish, you know. It shows that there is significant kind of progress to be made, promise, you know, we are motivating people to look into this gap and provide, think about better solutions.

38:27The next paper on speculative decoding for multimodal language models. What's this paper trying to do? There's something called speculative decoding for language models, LLMs. What it does, there's this, let's say, big model, or, you know, kind of, and there's this drift model, tiny model, small model. So big model, we need to go to parameters from memory and to one iteration, then we do another iteration. So it's a big model, it takes time. And the token rate is something, how many outputs, let's say, words, you can think that tokens are similar to words, how many words it can generate per second is an important metric.

39:15We want it to be high, generate many, many results very quickly, not slowly. But there is associated cost of compute with the big model. So what speculative decoding does, it uses a tiny model approximating the performance of the big model. It computes multiple tokens, let's say three, five tokens at a time, then we run the batch big model just once as a batch and we look at the results and if If results are same, then we kind of accept it. We don't need to step by step run the large model. Large model only runs on the batch, which allows us to accelerate significantly. If we look at the results coming from the draft and the kind of batch results from the big model, then we can very easily skip, let's say, two tokens, four tokens.

40:17if you are using three, five, you know, it is a significant speedup. No one so far tested whether, okay, such speculative decoding idea would extend to multimodal. In this case, we have lava. Lava has, we just mentioned, there is like this LLM part, which could be, let's say, Lama 27B or Vikuna, any model, any large language model. And there is this image, and both of them goes into the model. image goes through its own token, so you can ask a question and then it will answer. So the test goal, can we make it faster by adapting speculative decoding like ideas here? And that's what we investigated.

41:01We use a tiny model of 150 million parameters and we observe that we can significantly accelerate up to 2.3 times, you know, using the speculative decoding for a multi-model, not only language model, but multi-model model as well. This graph model could be any model, you know, kind of. It's not this kind of observation. we show that it goes beyond a specific model you know big models draft model it is kind of a useful information it shows that spd speculative coding very easily extends to you know multi-model and just to understand is the is the draft model is it part of a larger model for inference?

41:56Is it like a subsystem or is it a replacement model that's been, you know, like it sounds like distillation in a sense? Draft model is learned to do a good job in approximating, estimating, or predicting, you know, the output of the big model. So we run draft model and we run also the big model. both of them coexist in the inference time. I mean, it's not like we only run draft model and then go with its own decision because draft models sometimes make mistakes. It is, we intend, we want it to approximate the big model, but it is not the big model. It's going to make a mistake. That's why I keep saying on average, sometimes it fails.

42:47So if it fails, then you go around the big model at every, let's say, three, five tokens, you don't accept wherever it is not aligned with the big models, batch processing results. We go back to that earlier token and run the big model. So, of course, if the draft model makes a lot of mistakes, running the draft model and then running big model, like every token is more computationally intensive. It takes more computation. But on average, if draft model does good job, you know, it can allow skipping three token, five token, more tokens, you know. So, yeah, we need to run both of them. In the inference time, we have both draft model and the big model.

43:33Is the draft model used in inference in training or just for like plain inference? Meaning if you're just doing inference, how are you evaluating the output of the draft model to determine whether you need to invoke the real model? So it generates tokens, draft model. We run it iteratively, let's say, five times. We generate five tokens autoregressively, not as a batch. So we have this token, another token, another token, and sequentially. Yeah. And then we have the results. One time runoff to big model comes as a batch. We generate multiple tokens. We do not do it autoregressively. So there's a danger, of course, you know, if they are aligned, if they are same, then looks like we are good.

44:29But if they are not the same, then we need to go to the big model, then auto-aggressively, sequentially, start from that false generation token, update it, and then generate the rest. Because it took a while for me to understand and appreciate how speculative decoding is working. I was thinking, yeah, you do more computation though, right? kind of then what is important in some cases we can run this drift model and the large model in parallel on different platforms that one could be running on nsp or mpu you know kind of another one is running on cpu or gpu so we actually do more compute but then we can very fast generate multiple tokens you know let's say token per second there is a metric called this tps uh if ordinary big model uh token tps could be let's say eight you know on a mobile device with speculative decoding it can go up to 20 40 you know kind of that you want to generate results outputs very fast you know but but you do some additional computation or you need to take advantage of parallel processing.

45:46That is also a fact. By the way, I'm not one of the authors of this paper, but that is how I understand, you know, I will truly invite everyone. And one of the, a couple of authors will be at the conference, you know, as questions, you know, if they are interested to the, our authors, you know. No, it's a really interesting idea. And in terms of the training of the model, Is it analogous to like a distillation type of training or? Yeah, draft model. You mean the draft model? The draft model, yes. Yeah, exactly. It is, there's the large model, big model, and we are trying to, we are distilling to learn this smaller model that generates similar outputs.

46:29Yeah. And distillation is something we do not only for, you know, speculative decoding, but also So for the large vision models, LVMs or BLMs, you know, we take larger models, bigger models, and we distill them to make them faster. You know, in this case, that is also the reason. It seems like this idea of running a distilled model in parallel with a full resolution model and then checkpointing frequently that that's a broader idea than, you know, this particular application of, you know, speculative decoding for language models. Is there something that exists like that in vision or in other domains?

47:10That's a good question. Yes. for instance, this is maybe older, more conventional computer vision networks. There is something called for video activity recognition slow and fast. You look at a bigger model with using much lower frame rate, the original video or smaller resolution frames but more capable model, big model. you run it from time to time and you run the other one, you know, smaller model, more frequently, and you look at how consistent their results, whether to regularize the, you know, kind of the small model outputs or combine them, fuse them. There is that. There is also hierarchical resolution trees, you know, kind of at these trees, you have the small version and you accept small version based on some confidence criteria.

48:12If you are not happy, then you will switch to higher resolution. It could be object detector, for instance, in this case, or a segmentation solution. These are more conventional things, but I see the similar analogies like having a draft model and large model or having a tiny model, but applying many, many times or a big model applying from time to time, you know, kind of something like that. And this thing is actually extends to end-to-end training for auto. There are now these models, very interesting areas. There is this genetic AI-based big models that are doing scene interpretation. And there is more conventional, let's say, vehicle detection, segmentation, tracking, like more conventional And these two, one is kind of high level, providing high level regularization to the conventional pipeline.

49:12And like both of them could be estimating the trajectories where the vehicle is going to be in the future. But the big one, of course, bigger. It cannot run real time, you know, kind of at this point. So it is not running all the time. And but they are running together, you know, kind of. So there are similar things all around the research landscape, I guess. The next paper on our list here is Segmentation-Free Guidance for Text-to-Image Diffusion Models. Yes. What's this paper attempting to do? So this paper applies to text-to-image generation, like stable diffusion. You give a text prompt and it's going to generate something.

49:55it doesn't require any fine-tuning of the model. But it does. When we run such models, diffusion models, we have the conditioning prompt and also a negative prompt. We do actually initial models. They had two runs of the unit, one with the text conditioning. So it is trying to generate things that you ask it to do. And the other one, without any conditioning, it takes the same feature maps and do another run of it. And that is called as, you know, kind of like a class free or, you know, kind of like a negative prompt. So this information applies to each patch of the image. You know, there's some negative prompting.

50:39But let's say we run the network once a couple of times, unit, And we now know a spatial region of the image is more representing a given object describing in the prompt. For instance, the prompt could be a dog sitting on a couch in the kitchen. I just made it up. And after a couple of first iterations, there are some image regions where you would likely to see a dog and other parts, maybe couch and maybe the background, the kitchen. So when we do negative prompting, we are saying the region also penalized for the dog region, penalized for the couch for the couch region. So this is kind of like, it doesn't sound right, right?

51:35We are trying to generate a dog in an image region, but we are also penalizing for that. So this paper, by just changing the attention map coefficients, these attention space are adjusted, you know, kind of solves this problem of, you know, I'm going to generate a dog, but I'm also penalized for the whole thing, you know, kind of. So it doesn't penalize us only for the dog, only for that region. So that's the beauty of it. And when you do that, Sam, when you look at the output results coming from, let's say, stable diffusion. And by the way, maybe I mentioned it, this thing doesn't change the diffusion model.

52:16It applies to anything. It's model agnostic. It's free. It's cheap. It's an amazing idea. So the output images, the quality look much better. What I'm hearing is that it is using this internal ability of the model to do implied segmentation as a way to allow the text guidance to happen in a more fine-grained way, essentially. So there are different metrics to measure how good it is. Some of it we just talked about FID and CLIP. There's Inception Score IS and there's also Peak Score, a network that approximately human learns how humans are responding. Someone sits down and trains such a neural network because that part is automated.

53:02So we look at all these kind of metrics and also we run our own human studies. But we observed that, so here is the base, original model, which is a very good model also. It could be, like I said, stable diffusion 1.5, 2.1, STX, anything. Here is the model, the attentions are changed. Model architecture, coefficients are everything the same, like we are doing segmentation-free. So we asked people, what would be the better one for you? And 60 % of the people with high preference said that I like segmentation-free. And only 19 % said that I like the original model. So the gap is 19 to 60. There's a big indicator that people really like the results coming from segmentation-free.

53:53And also in this paper, what we did that, okay, you want to do this kind of evaluations. of such networks and the input is just a prompt, right? And usually there are these prompt data sets like the MS COCO 30K, which means that there are 30 ,000 prompts in the camera. But then, you know, taking all of these prompts results, like for this model A, model B, model C, and showing to people, it is a big feat. It is very expensive. So you cannot ask, I mean, a person give me your feedback for 30 ,000 prompts. Then in this paper, we are also talking about how to make it better to keep the number of human evaluations manageable, not like 30 ,000, 100 ,000.

54:43But still ensuring that the selected subset of these prompts are representative in terms of the content and fear in terms of the model performance. So in this paper, we are also talking about this subset, like a benchmark plant, data set. So we just made it through the generative papers. And there are even more papers on the traditional side. I don't think we'll be able to get through them at the same degree of depth or coverage. You know, one of the things I noticed is that your team is continuing to investigate the idea of optical flow quite a bit. That's something that we talked about last time.

55:34And it looks like several of the papers are kind of continuing to push on ways to do optical flow better. Absolutely. You are absolutely right. Last time we had another CVPR conference paper, District Flow was its name. It was optical flow. Yeah, we put a lot of efforts in optical flow because it's very important, necessary to improve the image quality. First, when you take a picture, you actually take multiple snapshots and then you do some kind of alignment to get rid of noise, you know, kind of, or improve the high dynamic range. motion information is essential in compression in anything you do in video, you know, understanding how everything is moving.

56:17We are trying to, in these two papers, the first one is the kind of occlusion and consistency area interpolation. In that paper, we are trying to squeeze all the information we can get from the available limited amount of optical flow data sets. Optical flow is very difficult to annotate, you know, let's say I have video, you know, I need to go at every pixel, find which pixel it went to, right? This is, it cannot be done manually. That's why there are lots of synthetic data sets and approximating, you know, computed optical flow manually. This is very, very challenging. So amount of data is limited, Sam, for that reason.

57:03So we are trying to do the best. And in this case, that's the difference from what we published before. We are looking at now, I mean, we do all type of augmentations. We are trying to develop the best models in terms of accuracy and all the leaderboards, you know, Qualcomm is there. But also in this case, we are now looking at, okay, I have two frames. What about if I generate intermediate frames? This is video frame interpolation, VFI, and then, you know, generate the corresponding optical flow also, and then let's use it in addition to the original frames, you know, T0 and T1. Now I have T0.68, you know, something like that.

57:49And this is now going to give me more information, more frames, when I want to go train the optical flow models. Two things in the paper, how you would get the best, these interpolated frames between two frames, and then how you would take it and use it to come up with a better optical flow solution, which is model agnostic actually. And then you've got companion paper in the same, well, not necessarily a companion paper, but it's another optical flow paper called Self-Cleaning Inversion. The first paper was about how to make it, augment data sets even with using intermediate frames. And also, you know, we also talk about a better way of self-supervised training.

58:40That's separate. Here in this paper, CyFlow, which stands for self-cleaning iterations or, you know, inversion. So we are aiming a very efficient optical flow model, which is also accurate. For instance, recent optical flow solutions, most of them are using based on draft-like idea. It starts with initial optical flow or noise and then iteratively refines it, you know, using the features coming from different networks. It makes it better and better. But if there's a problem, there's a drift, they cannot correct it. So in this paper, we are explaining, introducing two concepts. One is self-cleaning iterations.

59:28The other one is regression focal loss to look at where during the iterations in inference time, where the model is failing or has potential to fail because we don't know, you know, because this inference time, but we have indicators, you know, kind of this confidence course like this course to go change the way that we are waiting the loss in those areas and and also we are handling ambiguities and also we show that you know these methods are i mean the networks if you do that you know you don't need to iterate many times it is computationally efficient um it it can run real time you know kind of we have a real-time running version also on mobile devices.

1:00:18So those couple of papers are, well, the first one, the optical flow augmentation is the main conference paper. The side flow is a workshop paper. The next paper that I wanted to learn a little about is low latency neural stereo streaming, also a main conference paper. And it's a wonderful idea. You know, kind of the input is a left, right stereo. It could be multiple also. It doesn't need to be two camera, but stereo or video stream. That's the input usually. But other algorithms do is you can take the left, let's say, and then compress and then take right, compress separately. So obviously there's redundancy here.

1:01:06Some algorithms, what they do, they take the left frame and compress them. Based on the left frame, they compress right frame. So it's one way. And another way of doing, you know, do left T0, then do T2, then right T1, then T3. This interweave, you know, fancy things. But this paper talks about a parallel hypercodec between this parallel autoencoder architecture. and there's this bidirectional shift module, kind of designed to learn the correlation between these left and right branches. So the goal is to do compression together, you know, left and right is compressed together. At the same time, that's why it is low latency.

1:01:56And this is important low latency, Sam, because there are now cameras on the XR devices, headsets and also autonomous vehicles, you cannot just stream the original video, you know, bandwidth limitation and other things, you know, storage makes it, you know, infeasible. So you like to compress them, but you like to compress them with low latency, you know, very, very quickly. And so how you can do that, you know, how you don't create these dependencies from left to right, right to left or, you know, in a virtual fashion, but do it together and still achieve, for instance, bit rate savings. We show that you can save 50 % bit rate if you do it the way that we described in this paper on Cityscape datasets or, you know, 20 or more than 20%, 15 % on KITI 2012 or 2015 datasets.

1:02:54These are benchmarks. Yeah. And it only comes with maybe a little bit more 35 % additional complexity on top of the very simple single stream encoding. Yeah. This is a very interesting paper, useful paper, very practical paper. If you've got a left and a right stream and you just use conventional compression, you're kind of leaving the correlation between left and right on the floor, you're not taking advantage of that. And so you could do better from a compression perspective. And so that's why you want to have a stereo aware compression model. You are right. You actually said it better than me.

1:03:34You know, the thing that I was talking about is bidirectional shift module, this correlation between left and right to, you know, compute it mathematically and, you know, then take it out, minimize the redundancies. Exactly. That's what it improves the bit rate, bandwidth bit rate. But also the fact that we are doing it together, we don't rely on the first left frame compression, then wait for it, then do the right frame. That is more conventional approach. or do left stream than right stream. It's also kind of like, it is delay, you know, kind of. So this delay is important. We are minimizing this delay by removing redundancy.

1:04:27We are going to leave the remainder of the papers as an exercise to the listener. We're going to have them all listed on the show notes page. There are, as I mentioned, a bunch more. Or in addition to a number of demos we mentioned, or you mentioned in passing the demos that you're doing, are there any particular demos that you would want to call out? Yeah, absolutely. You know, please stop by this. Many of those demos are running on device, on phone, you know, kind of mobile devices. And one of them is something we did, LoRa. It's stable diffusion. LoRa is low-rank adaptation. For instance, I can take original model and make it to generate better faces.

1:05:13You know, it is fine-tuned on faces or it can generate, you know, kind of any kind of, let's say, cartoon characters, you know, better. It stylizes with some kind of internal style. But the challenge is that this lower, additional lower coefficients, I have a big model and a small model, lower coefficients, adaptations. If I twist them, if I add them, and then run them. If I change, then I change my mind. I decided to impose more of the style, less of the style. I cannot do it. I have to go back and refuse it again. And these are big models, kind of several billion of parameters. Doing that fusion and then loading it again to memory is going to cost me in terms of time.

1:06:04I cannot do it very fast. It's going to take a couple of seconds on device. So this first demo shows that you don't need to wait. You can really easily switch to change the degree of this, how much you're going to impose. And also, you can very easily switch from one lower to another lower and another lower on the flight. This is a very cool idea. The second one is a mobile, for instance, you can take a picture of yourself, anything, a thing, you know, and then any object, you can ask a question and then it will answer. For instance, you are taking a picture of, okay, what can I cook with these ingredients in this picture?

1:06:52And it will tell you what you can cook and then you can ask, is the dialogue model, how many calories will it be? You know, kind of something like that. And you can imagine, you know, kind of like, you can literally have a conversation with it. And it's running on the mobile. It's available as of now. This is something we recently developed. And there are other demos. For instance, the demos that goes to our autonomous driving stack. Very interesting. We generate new samples, augment the data. For instance, there's a scene and we put animals in the scene, virtual. They are not real. But when we put them, then we use that data to train our stack detectors that we show they are improved.

1:07:36And in some cases, it's very difficult to collect such data. But also things around, let's say, more core computer vision segmentation on device, segmenting anything and tracking anything on device. Or like I said, portrait relighting of the faces or generating avatars, you know, talking avatars on the phone or, you know, kind of AI assistant with the avatar face, you know, kind of, there are cool demos. Yes. All right. And before we wrap, you also, you meaning Qualcomm also coordinated a couple of workshops at the conference. Tell us a little bit about those. Yeah, absolutely. The first one is the first conference, you know, first workshop in this area.

1:08:25it is efficient large vision models, SCVPR. They are becoming very popular, but also, you know, kind of we more and more need them to run on edge devices. So we co-organize this workshop with many other, you know, companies and institution, academia. And please stop by, you will see, you know, how this efficient transformer architecture for large vision models or foundation models, generative models, multimodal models are kind of making their ways into such devices. And the second one is called omnidirectional computer vision. And this is a very important area because it applies to many use cases from augmented reality to any camera system for automotive surveillance, photography.

1:09:22And now these solutions that we look into in this workshop is focusing on such input coming from these omnidirectional cameras. And this is maybe the third or fourth issue of this workshop. Well, Fatih, thanks so much for taking the time to share with us all that you've been up to and what we can look out for at CVPR. Absolutely. I'm very excited about that. Thank you so much for having me, Sam. And this is a great podcast. Thanks for inviting me. Thanks so much.