Data Augmentation and Optimized Architectures for Computer Vision with Fatih Porikli - #635

0:07All right, everyone. Welcome to another episode of the TwiML AI podcast. I am your host Sam I'm Charrington, and today I'm joined by Fatih Parikhli. Fatih is a Senior Director of Technology at Qualcomm AI Research. Before we get going, be sure to take a moment to hit that subscribe button wherever you're listening to today's show. Fatih, welcome back to the podcast. Thank you so much. Thanks for hearing me, Sam. I'm excited to jump into our conversation. It has been a while since we last spoke. In fact, exactly or just about a year, we had you on to talk about all of the things that Qualcomm was doing at last year's CVPR, and you did such a great job, we are going to do the same thing this year.

0:47In fact, there's a ton of papers and research coming out of the Qualcomm AI Research Lab at this year's CVPR, and we'll be digging into those shortly. But before we do, I'd love for you to take a moment to reintroduce yourself to our audience and share a bit about your background and how you came to work in machine learning. Absolutely. As you said, I'm a Senior Director of Technology at Qualcomm AI Research. Qualcomm has many departments, so we are at the Corporate R &D. I'm leading multiple research teams and projects across three continents, multiple cities. I'm an IEEE fellow, and I was a full professor before I moved to industry.

1:30And I have been working in computer vision, machine learning, AI, maybe more than 30 years already. And as you said, it was a wonderful, very exciting year for Qualcomm AI research, Qualcomm in general, and also for AI and computer vision with many things that are going on at the same time all around the world. And CVPR was an excellent opportunity to catch up to excellent research results. Awesome. So you mentioned an exciting year and I mentioned some of the work that we're going to talk about. There were, what, eight papers this year from your team at CVPR? Eight papers at the main conference and we had also another four wonderful workshop papers.

2:18So altogether 12. We have more than half a dozen also demos and many other things that we presented at CVPR. And you're an author on, I think, a good half of those papers? Yeah, maybe more than half, yeah. Awesome. So you spent some time thinking about the best way to kind of share this work with our audience, and you've got some categories in mind. How should we think about these papers that you're presenting this year? Absolutely. When I look at the papers on a high level, I think it would be possible to put them under two categories, Sam. The first category would be making the best use of the data available, either annotated or unannotated, unlabeled data.

3:09And the second category would be novel architectures, better architectures, more efficient solutions. So I would be happy to talk about these two categories and papers. And also about the papers, even though I'm involved in many papers, the credit goes to the team members, you know, kind of it's the team efforts as always. And so when you talk about making best use of data, just contextualize a little bit, contextualize that for us and its importance to the work you're doing there. Absolutely. In the first category, I like to mention very briefly, and we can go to details kind of as you like, four papers.

3:49The first one is district flow, which is an optical flow motion estimation solution. Data for motion estimation, deep learning-based solution requires training, and you need ground data for that. It's very difficult and challenging to collect real data because it is not possible to label. So this DSTREC flow, what it does, it proposes a novel solution to augment data. So the amount of the training data would be much more than the originally available, what is available. And it does it in a way that the augmentation, diversification done in a semantically meaningful manner. So that was about the first paper, very briefly.

4:36The second paper, we are introducing a progressive random convolution idea. In that paper, we show that we can take any image and then kind of do these distortions. But these are semantically meaningful distortions in these augmentations. We reduce the influence of the non-local pixels in the receptive fields of these convolutional kernels. So when we do that, we observe that all Bayes' models would benefit from such effective and representative domain-based augmentations. So I talk about the second paper. The third one is Read Dear Trans. It is for gaze direction and head pose data augmentation.

5:27And it does it in latent space. It does latent to latent translations in high resolution, full-face images in an interpretable manner. And it also uses a generative adversarial network for high accuracy data augmentation without need for manual parameter tuning. And the last paper I like to mention, we call it as Deja Vu. It's a novel framework which leverages the conditional image generation. So it's a generative AI solution as additional supervision. So even using the same amount of data internally, it augments, redacts the images and then augments more samples for dense prediction tests, for instance, for segmentation, motion estimation, surface normal estimation, depth estimation, and at least two more accurate solutions.

6:22So these four papers are using, they aim to make the best use of the data, available data, either labeled or unlabeled data. Got it. So then the remaining four papers are in that second category of better architecture. So assuming you've got a set of data, how can you improve the architecture of the network to create better results? Absolutely. For AI, we need data, obviously, because many solutions are data-driven, but we also need capable architectures. For instance, deep learning models, these convolutional or attention layers, it's like human brain. So we need such architectures running on hardware, on edge devices, on mobile devices in many cases, and they need to be efficient.

7:08They need to be low power and low memory. So for that purpose, we have another four papers. The first one is X3KD. X stands for cross here. It is Y3 because it does cross model and cross stage. This is the architecture stage and cross test. For instance, segmentation detection type of KD is for knowledge distillation. This type of knowledge distillation is for object detection. And it takes data from multiple cameras and LiDAR. Those are the model T's. and then blends everything into a framework to come up with the best architecture model, which also reflects to the results. It set the new state of the art on many leaderboards, on large benchmarks that we tested.

8:02That was X3K. The second paper is ECO-TTA. TTA here stands for test time adaptation. is an efficient approach that improves continual test time adaptation in a memory efficient manner. Reducing memory is crucial because most of the time such adaptation, test time adaptation, runs on H devices, limited memory capability devices. And also if we keep updating the model over and over again, it may cause drift. This long-term adaptation often leads to this problem, which is coined as catastrophic forgetting. So this is also a challenge. It hinders the adoption of the TTA solutions for real-world deployment.

8:49So we show that we can change the architecture to make it better by incorporating meta networks to adapt the original network without inflating memory requirements or spending a lot of compute. So that was the second paper. The third paper is dense network expansion. And this is for, again, adapting the network for new tasks. In this case, it could be new tests, could be recognizing a new class. If let's say we have a classifier and we already can detect, let's say, 1 ,000 different class of objects, we can segment them. But we want to do 1 ,001. We want to increment the number of classes. So this is a new test.

9:38That's just an example that I wanted to give to make it a little bit more clear. So this algorithm, dense networks expansion, allows trade-off between accuracy and the model size, and it can use, reshare the features that are already learned for the previous tests. So that is the second paper, CVPR paper, that I wanted to mention. And the last one is very interesting also. It is about 3D part segmentation. Here, this is a generative AI solution. It uses language models, but it does in a zero-shot manner, which means that we don't have any training data to do very refined, fine-grained 3D par segmentation.

10:24For instance, let's say we have a class of chairs, and now we want to go beyond chair segmentation. We want to segment the arms of the chairs or the small, you know, kind of like wheels of the chair. You can imagine there is a bird class and we want to now segment the beak of the bird or kind of other things, feet of the bird. But we don't have any such examples and we want to do it automatically. So this paper presents a solution which can do it in a zero shot manner without any data. It's very intuitive. So these four papers with 3D part segmentation, we also dive deep into these better architectures, better learning frameworks.

11:08And of course, we have more papers, workshop papers that I would like to talk about. Well, as you know, we want to dig deep into this research. Of course, we can't cover all these papers, so we'll be digging into just a few of them. And the first one that we'll talk a little bit about is the distract flow paper. I think last time you were on, we spent a lot of time talking about optical flow, in particular some of the research that you did and presented at last year's CVPR. What I recall you saying is that optical flows kind of emerged as this broad technique that is applicable to a pretty wide variety of video use cases and improving the way we do optical flow is you kind of at least see it as critical to making progress on video.

12:04Is that still your thinking around optical flow? Oh, absolutely. There is video. Optical flow is an essential task that we need to do to get an idea about the motion observed, captured in the scene. It could be the motion of the camera. It could be the motion of all the moving objects. It could be kind of motion of the swaying things and everything. We need to understand that type of motion so we can propagate what we learned in the past for the previous frames to the current frame. Let's say we want to do better super resolution or we want to do better compression, we want to do better detection, you know, better tracking, object tracking.

12:48We need optical flow. So optical flow is a fundamental task in computer vision, Sam. And we have been working, as you mentioned, in optical flow. We had another paper last year, CVPR, mean body paper, and this year also we have actually two. And just kind of taking a step back, describe the traditional setting for optical flow and the way the problem is represented. In optical flow, we have multiple frames. In video, we have multiple frames. Let's say we have the current frame and we have the previous frame, t and t minus one. And for each pixel, we want to find where the pixel was in the previous frame or from previous frame where it moved in the current frame.

13:34So for each pixel, we want to know how much they moved. You can think that we are estimating the motion in x and y directions, delta x and delta y. So this is a correspondence challenge. For each pixel, we have to now decide where it is in the previous or the next frame. It is a search operation. It's a very costly operation. Of course, AI-based solutions, they do not just do that type of blind search. Maybe in the past, they were efficient search solutions many, many years ago, several decades ago. Even myself, when I did my PhD, I had to dive deep into such solutions, conventional optical flow solutions.

14:23But more recently, what we are doing, we are training models, taking multiple images, let's say current image and the previous image or a set of images. And then network applies a feature extractor around each pixel. it tries to learn most representative, useful information, a mathematical model, through applying these convolutional kernels. And we end up with a representation, a vector for each pixel, each patch. And then we compute something called a cost volume, which means that, okay, how similar this vector, feature descriptor for a pixel, to another feature vector for another pixel in the other frame, the previous or next frame.

15:13So we create this cost volume, but then what we often do, either kind of directly find the correspondences, or we iteratively start with some initial estimation of the optical flow and gradually refine it to make it maybe more robust or faster. So this is kind of a high-level AI-based motion estimation that I mentioned. And the cost volume that you mentioned, it sounds like roughly a multidimensional error metric. Is that the way to think about it? Wonderful. Yeah, absolutely. That's a very nice way of saying that because we have X, Y directions. We have also two of them because previous frame, current frame.

16:02So you may think that it is four-dimensional tensor, more correctly mathematically. And when we are computing that tensor, we are going over one-dimensional or two-dimensional vectors as well. So it's a high-dimensional data structure. So this is one way of doing that. Of course, earlier AI-based solutions, again, deep learning solutions, what they would do, they would take two frames and pass it through a feed-forward network without any cost volume. but at the end they will predict the delta x delta y these motion directions for each pixel the cost volume based one the thing that i just mentioned it is more accurate but there is a big challenge you know how are we going to train such algorithms ai based solutions we assume that we can train such networks with lots of parameters with real data but motion labeling is very challenging.

17:01Of course, I can sit down and, you know, draw a box around a person if I want to do detection, but it's very difficult manually to say that this pixel in one previous frame image is this pixel in the current one, and we have to do it for millions of pixels. So obviously, we cannot do it manually, and we cannot put markers, you know, on everything in the scene and then track them by some other algorithm or manually. So this is a very challenging task. There is very limited amount of real data for training of optical flow estimation networks. And even that data labels, they are pseudo-labels generated by yet another, you know, more trustable algorithm that says, okay, this would be the optical flow direction, maybe with some manual kind of correction and everything.

17:56So this is a test really needs data, but we have very limited data except synthetic data coming from, let's say, game engines. So this track flow paper directly tackles this challenge. It, in a way, doesn't require, you know, any labeled data to train optical flow solutions, which is very innovative. That's what we showed at CVPR. What it does, it takes any video sequences, let's say, for autonomous vehicles, a video sequence showing one road, and then a related, relevant video sequence, and then combines these frames together, which is the distraction. But this distraction now is not like regular, you know, kind of predefined manual augmentation.

18:46Of course, there are many augmentation solutions. Maybe I should, let me just touch upon that. We can do color changes. We can, you know, rotate images, do some local transformations. We can make it blurry, hazy, or those type of things. But those are limited, and they are not semantically meaningful. So what we showed in the district flow paper, you are not limited only to that type of, let's say, transformative or manually defined augmentations. You can go beyond those. And that's what we are showing, that if you go beyond by inducing semantically meaningful augmentations, distractions, and then if you use that, and those are all pseudo-labeled.

19:31I mean, as I said, you don't need ground truth label for that. that you use that data for training these AI-based optical flow solutions, actually you make the model, same model, same architecture, more robust, more accurate. So that is the beauty of that paper. I mean, in a way that, you know, it doesn't require any architecture changes. Can you talk a little bit more about the nature of these distractions? When you first mentioned it, I envisioned just adding a separate video feed or frames from a separate video into your training data set. But it sounds like, well, we're talking about data augmentation.

20:11It sounds like you're manipulating your training data with this distraction video. Explain what the relationship is between these two and how this separate video feed is used. Yes, absolutely. So the baseline would be, I have a video, I have, let's say, some optical flow ground root, limited amount of it. When I'm augmenting, I'm rotating the input. So optical flow field also rotated. I'm doing some geometric or radiometric color changes, optical flow, then color changes. It should be constant. Yeah, it's traditional. So traditional data augmentation, like you're doing things where you can mathematically manipulate both the training data and the pseudolabels.

21:02But we are going beyond it. But we do now. We are taking the frames, they say, images or videos, very short video segments from related content. And it could be an outdoor scene, a traffic scene. It could be an XR environment, like indoor, like my office or room. So wherever we want to compute this optical flow, adapt to optical flow, we start with relevant videos. And then we take one frame from another, a completely different scene maybe, and then blend it, fuse it into the current image. So we have the T-1 of the current scene and now blended version of the current scene. So kind of maybe you will see ghosting like effects, but that ghosting is relevant because it comes from, let's say, another vehicle, another person.

21:58It's not just random, you know, mathematical deformations. This is actually coming from something real. And we know how much we induce, I mean, the degree of kind of this blending. and we also know how confident we were when we use original frames, T and T minus one, when we do optical flow and we focus on the confident areas. So we are making it difficult. Now we are putting some hallucination ghosting in front of the algorithm and asking, now you should handle this ghosting. Yes, I made it more challenging. I distracted you and these distractions are maybe, you know, kind of meaningful. distractions now, but I expect you to be robust against those.

22:45You should still be able to estimate the same optical flow when we had no distraction. So this is the condition. These are true loss functions, obviously, kind of the things that we use in training, which makes the model robust. It's amazing that that works. Yeah. There are lots of things going on, of course, like backward motion, forward motion, consistency. So it is pulling all the, you know, kind of novel ideas together in such, but that's what we want to do at Qualcomm AI Research also, you know, not just looking at an individual theoretical contribution, but pulling everything to solve the problem in the best way possible.

23:28This is a real solution for a real problem. And one of your workshop papers was demonstrating it or discussing how you would actually run it in a hardware environment? Absolutely. We call it as DIFT, Dynamic Iterative Feature Transform. Cost volumes, it deals with cost volume. So this workflow is about how you will train the data, you know, kind of where is the data? I mean, this is the training style, the learning style. Architecture could be anything. Let's say we have this cost volume-based architecture. Because of, like you said, high dimensionality of the cost volumes, it is very difficult, challenging to fit them, port them to an edge device, phone.

24:15You want it to run on the phone because you want to grab much better, higher quality, higher resolution videos of your loud ones, you know, kind of. That's what optical flow can do, or super-resolve everything, among many other things. So now we want to run such optical flow on device. Cost volumes are great. They are very accurate, but they wouldn't fit memory-wise. And also, let's say we can fit it. We just expanded the memory. Now we have a very costly solution, hardware costly solution, but it will be still slow. So this DIFT workshop paper that we show that the original algorithm, cost volume-based solution, First of all, it doesn't fit, but even if it fits on a kind of similar, let's say, neural processor, it will take maybe a couple of seconds for it to run.

25:07We show that we can actually run a few milliseconds, around 30 milliseconds or less, the cost volume. And we would get very accurate motion estimation, optical flow estimations. And to accomplish that, we did a lot of tricks. We have a mass resolution cost volume computation approach. We are doing everything just in time. If we do not need that part of the cost volume, we do not actually compute it. We also take advantage of the Qualcomm Snapdragon processor, neural signal processor, accelerator architecture advantages. So we optimize for that. So altogether, we show that we can improve the throughput, how much data we can process at the same time, eight times, and speed maybe many times, you know, maybe more than 30 times.

26:02Oh, wow. Awesome. So that's a great example of the work in that first category, making better use of data, kind of holding the architecture constant, evolving data augmentation technique to achieve. In thinking about the results you mentioned just now, the results of the DIFT work on hardware. Are there also results that you can talk about for DistractFlow itself as a technique? Oh, yeah. So we showed it on different benchmark. I mean, there are challenges. There are established benchmarks for optical flow computation. We show that on all of it, this flow improves the original architecture. As I mentioned before, it applies to any architecture, any model, because it's a training style, a better training algorithm by augmenting a bit more data.

26:53So we show that in the paper, at CVPR, it improves everything consistently. That is the beauty of it. You do not need to change your solution, optical flow model, let's say on your design for a specific device, but it just performs better. It's all in the paper. Yeah, it's some of the KPIs called as EP and point error. So we show that we are almost helping in Sintelo and KITI datasets. Oh, wow. All right. So then, you know, moving to an example of a paper focused on the better architecture perspective, one that we can talk about is the X3KD, which is focused on object detection and perception broadly.

27:37Can you talk a little bit about the context for that work? It goes to a real solution again. I mean, in the future, we hope it. Of course, this is research work, by the way. And this is the real solution. And it's very important solution. This is autonomous driving. And in autonomous driving, these autonomous vehicles, there are multiple sensors, multiple cameras, four, six, even more. There are also radars. And then we collect data for such application use cases to make the vehicle drive itself on the road. The fleets that we have, they also have LiDAR data to get a 3D model of the environment and everything.

28:22So we have this data available, multiple cameras and LiDAR and radar and all type of model T's, IMUs, everything. And a lot of such data. We now want to come up with the best solution, most accurate solution, most efficient solution. So X3KD, knowledge distillation KD, does it cross multiple things. One is cross model T. In runtime, when we are running this algorithm, it will only use images, by the way, Sam. But in training time, since we already have LiDAR data and other type of modalities, we make use of that data. Of course, then how I'm going to take these images, which are in perspective view, because image just looks at the scene.

29:13But LiDAR is a point cloud and we map it onto something called a bird's eye view. So it looks like you are looking from the top to the street and behind the vehicle. So how am I going to take these images and map into this bird's eye view? And how I'm going to know beach points in one image relates to another point in another image? And also this bird's eye view representation coming from the point cloud, LIDAR point cloud. So there is this cross-model distillation. installation. Our task is to detect vehicles and everything in the scene. So this is called as 3D object detection. So we are trying to find the boxes, 3D boxes of everything in the scene.

30:01And we can, of course, do it accurately with LiDAR data up to some certain degree, but LiDAR is an expensive sensor. And when we are running this solution, we do not want to use LiDAR, rely on LiDAR, just use cameras which are more cost effective. And I think that would, the real use case also kind of directly does that. And that's one type of cross-knowledge distillation. We have this LiDAR detector, 3D object detector. We are asking the camera part, so we have a LiDAR part and camera part to learn relevant features, feature representation. We have an encoder, decoder, of course. But also, when it is mapping to Bursar view that I mentioned previously, it does in a similar way with the LiDAR, because LiDAR Bursar view is much easier and more accurate than mapping things from an image to Bursar view.

30:57We do not have 3D model to do that in runtime. We could, but it would be maybe slow. And then dedections are consistent. So this one way of teacher is, let's say, LiDAR and student is camera. This one way of cross-model knowledge distillation. The second one is cross-stage. These stages are the architecture stages. So every layer of one student network layer, we are aiming it to be consistent with the teacher, more capable, you know larger network for efficiency so we do not need to spend a lot of power and we do not need to wait longer time we are in a way that we have this very good solution big architecture and we are distilling a smaller version with you know same performance so this is small but very capable it has all everything all the performance kpis of the larger network so this is the cross-stage.

32:01A stage could be also not just network architecture, but module itself, because we have encoders, decoders, and attention modules. And the last one is, of course, I mentioned we want to detect 3D objects, but we can also do 3D object segmentation or 2D object segmentation in images, 3D in LiDAR data. So we want this detection also be consistent with the segmentation results. And Sam, we do all of it before offline, before we run it. In the runtime, we only get images from these multiple cameras. Network is already optimized with these multiple different type of knowledge distillation to run it very efficiently and fast.