Generative AI at the Edge with Vinesh Sukumar - #623

0:07All right, everyone, welcome to another episode of the TwiML AI podcast. I'm your host, Sam Charrington, and today I'm joined by Vinesh Sukumar. Vinesh is head of AI ML product management at Qualcomm Technologies. Before we get going, be sure to take a moment to hit that subscribe button wherever you're listening to today's show. Vinesh, welcome to the podcast. Thank you, Sam, for providing me the opportunity to be in your talk show. I'm super excited to jump into our conversation. As we all know, AI has been evolving very quickly, and we're going to dig into how that evolution is impacting AI at the edge and the various use cases that are supported by edge technologies.

0:49Before we dive into that topic, I'd love to have you share a little bit about your background and how you came to work in machine learning and AI. Yeah, sure. Absolutely. I've been in the AI space for, I would say, give or take about 10, 15 years. Started my career working for Jet Proportional Labs. Those days was mostly looking at image classification, object classification, but under the nomenclature of computer vision. And those days, it was all about looking at big charts and able to classify content based of the captures. With time, transition mode towards consumer electronic field, like mobile cell phones, working on cameras.

1:26and then with time transition over to PC platforms and then to automotive platforms. And then finally at Qualcomm, had the opportunity to look at AI ML more horizontally across many verticals from mobile all the way to automotive. And the challenge with AI is what you know today is obsolete tomorrow. So you need to be a student at heart all the time and learn new things. And that keeps me quite interested while working at Qualcomm. Well, I think that theme of change is going to be a big one in our conversation. In our previous chat, one of the things you mentioned that struck me was just how the evolution in use cases and deep learning architectures as just a couple of examples, all these things impact the hardware architectures that are used to support these kinds of use cases and technologies on the edge.

2:18Can you talk a little bit about what you're seeing from a use case evolution perspective? Yeah, absolutely. A great question, by the way. Historically, when you look at AI ML on the edge, to a large extent, it was quite small in nature. It was very much evolving around image and video, wherein you want to look at image enhancements, image modifications, background segmentation, classification, detection, those kind of use cases. but as the technical acumen a lot of contributions from research scientists ecosystem acceptance of ai started to morph you could see that the evolution of ai use cases started touching other modalities from image and video moving on to text moving on to linguistics moving on to commerce and many of the segments so as such when you put this totality into picture historically to a large and all the AI algorithms was mostly convolution heavy, which was mostly reflecting on CNNs, because it was quite good in nature at that point of time.

3:24But as you get in more modalities into the picture to support use cases or create key experience indicators, now the investment is moving towards, I would say, transformers, as an example. Then moving on to generative content, which has been quite a large interest these days. Recommendation engines, which has been quite popular in social media and in commerce platforms. The question really becomes, is your architecture generic enough to be able to support new and upcoming architectures? Is it only meant for convolutions or can it go beyond convolutions into complex architectures like large language models and recommendations engines, which have been quite popular these days?

4:05And or do you want to focus on architectures which can do very specific tasks? They have been excellent in performance, excellent in power efficiency, those kind of stuff. So these are some of the questions and debates we ask ourselves. We try to enable both. And depending upon the vertical of interest, vertical alignment is a specific form factor of interest, could be mobile in nature, could be automotive in nature. We try to make sure those investments kind of scale accordingly. Yeah, I'd love to have you kind of dig into some of those trade-offs. When I think about what I know about the way Qualcomm approaches enabling AI on device from many prior conversations, one of the themes is really understanding the architecture and optimizing around that architecture, which you're suggesting here, maybe an architecture or maybe a device has some special accommodation for convolutional neuron networks that allows them to operate quickly and with a lot of efficiency.

5:08And then the predominant architecture shifts and there's something else that you need to think about. At the same time, a lot of the conversation around optimization is a bit of a lower level than that. Optimizing matrix multiplication and things like the resolution of your integers and things like that. So maybe talk us through kind of how you've approached the balance in the past. And when a shift like this happens, do you just optimize around the new thing or are you kind of creating something that works across the new and the old? Yeah, that's a fair question. This is something, you know, we get constantly challenged by our partners as well.

5:48So historically, the way we're trying to approach these use cases or applications is we try to understand what are the key performance indicators. They could be applications which are anchored on latency, anchored on performance, anchored on quality of service, anchored on power efficiency, or all the above, right? So as such, when you have a certain use case, you try to get into a certain system decomposition. The decomposition really means is, if I invoke a certain application, I'm able to meet the KPIs of interest at an application layer. That kind of answers the question by itself is, what is needed from a software perspective?

6:21and what is needed from a hardware perspective. Now, the hardware is obviously a little bit more challenging just because we want to kind of think two or three years in advance, right? Do we have the right investments in place to really push for it? And software is a constantly evolving topic to say what kind of optimizations is necessary at the library level, at the kernel level, at the compiler level, to be in a position to generate machine-readable code that is easy to execute. So let's start off with the hardware perspective. From a hardware perspective, you know, I would say some of the key most important elements is do we have the right data types?

6:53In Qualcomm's perspective, our position and push has always been we want to move towards fixed point as we believe it gives you the highest performance, quite leadership class power efficiency, and also a very small memory footprint compared to that of a floating point representation of the same model. So as such, we continue to push the boundaries of where we invest in data types. Historically, it was float, and then we transitioned to int8, and we're now moving towards int4 representation right so that's what one perspective the second thing is what would be the investments necessary from a compute standpoint from a local tightly coupled memory standpoint with expectation that you could store the bias functions within the tightly coupled memory rather than making round trips to DRAM because you want to see on power efficiency as an example right so there's always a question of you know what kind of compute is actually the right amount of investment.

7:44So here we start looking at use cases and then try to understand what is the state-of-art architectures really supporting those kind of use cases, right? And then we try to construct the investment and compute around that. Now, last but not the least is elements around bandwidth. Do we have compression schemes that would really push for high inference per second? You know, how is the register spacing established? What kind of sparsity or intelligence needs to be established as part of the hardware so that you don't really spend cycles of time in trying to do zero-based computations. So all these are feature enhancements that we really push for from a hardware perspective to really drive for leadership.

8:21Then you go towards the software level. Now, software level is obviously complicated. Here, we can start off with quantization. Quantization is a metric that we heavily push for, wherein we can actually provide the opportunity to our partners to transition towards a fixed point in 8-bits and 4-bits for both activation and bits. We have a tool called EMIT, and we're putting a lot of energy on AMET, which is Artificial Intelligence Model Efficiency Toolkit that provides the flexibility to our partners to not lose an accuracy while getting higher performance. Then we move on to our AI stack, which we recently announced a couple of months ago last year, which we call as a Qualcomm AI stack.

9:00As part of it, we have the runtime, we have tools, we have libraries, and the low-level API structure that is flexible enough to execute this model or design application on various OS. Could be on Windows, could be on Linux, could be in Android, those kind of subsystems. There's a lot of emphasis on how do we optimize on performance. Now, performance is a very broad term. So we look at, if I happen to have concurrency, how can I pack a single graph that is made up of composite, smaller graph into our hardware mechanics and then execute that as an example? Second thing is, if I have to securely store the model, In this case, what kind of encryption schemes needs to be supported to make sure somebody doesn't hack into the model of interest?

9:42The other could be is what kind of preemption schemes I would have if you happen to have concurrency. Can I support all these graphs simultaneously without compromising on quality of service? So all these are investments that we really push for working with our engineering staff. These features get updated on a regular basis. really make sure we can push for much larger deployment of applications, which are quite complex in nature across all views. And one good thing is, since we have a unified Qualcomm AI stack that works from mobile all the way to automotive, the lessons learned from automotive get propagated to mobile, and from mobile get propagated to auto as well.

10:17That way, there is a lot of handshake of good information. That way, the applications can be deployed across all firm factors with ease. You mentioned mobile and auto. What are the specific use cases you're seeing in each of those domains as the shift from convolutions to transformers takes hold? Are there any in particular that jump out as kind of driving the trend? great point so historically when you look at mobile as an anchor point and you look at it's a consumer buying pattern to a large extent people are buying phones because of good camera so which means most of your ai analytics is revolving around image or revolving around video right so they are to a large extent convolution heavy now people are slowly you know trying to bring in text as an input and then fuse image with text where transformers is becoming a quite an important element of interest and then try to see, can I use the sequential data that is embedded as part of a transformer network to understand context and make a much better prediction?

11:17So we're kind of revolving around text or an image and quality, but with different classes of networks, which are not necessarily convolution heavy, but more towards transforms, right? So that's, I would say, to a large extent, an anchor point for mobile. Now, obviously, in mobile, there's a bigger challenge is, you know, it operates on battery. So our expectation is we really give this performance at much lower power so that you don't drain the battery as much as possible and have a much smaller memory footprint because you're operating within a mobile device. But as you move towards automotive, it's a completely different space.

11:48Here, concurrency is a big deal. You're trying to work with 15, 16 different sensors. Could be camera sensors, LiDAR, radar. How can you fuse them together to actually provide you a quality of service? Latency is a much bigger deal. here the physical input tensors of the resolution of the camera is much bigger right either you have one megapixel three megapixel five megapixel eight megapixel on the mobile side the input tensors are very small in nature they're probably 512 by 512 256 225 by 225 kind of scenarios but on the automotive side because you need to have a much larger field of view you want to make predictions or decisions much more stringently you take a much larger input tensor so the physical compute requirements is much more intensive compared to that of what you would see from a mobile environment standpoint.

12:34And not to mention is you're going to see also the ask for one AI model that does multiple stuff. It could look at drive policy as an example. It could look at parking stack as an example, but you have the same infrastructure to really make a lot of those predictions. A different set of challenges from an automotive standpoint compared to the turf mobile, But having a unique and, I would say, a consistent Qualcomm AI stack, both in the mobile and automotive front, has really helped us really accelerate, I should say, new operators, new complex models, so that our developers can really use this investment and then drive innovative applications.

13:09When the dominant mode of transportation is autonomous vehicles, they projected that the compute requirement was pretty much on par with all of the data centers today. Have you seen that paper by any chance? Not necessarily, but I think in the gist of what they are probably claiming, as you're suggesting, is the compute requirements within ADAS space. historically when you look at automotive there's an infotainment site which is poorly in-cabin entertainment purposes and then there's an ADAS site. ADAS obviously goes by L1 to L5 depending upon the levels of autonomy. Right and I believe they were specifically referring to getting to full distributed fleet of level 5 autonomous driving fleet.

13:55Yeah so as you start doing that to a large extent having both infotainment and L5 level of autonomy services is quite complex in nature, right? And one layer on top of it is the fleet management, is how can you have multiple trucks, as an example, or multiple robotaxis being efficiently managed from one control center. And all this information handling is going to be quite complex. And I wouldn't be surprised if you're really looking at an infrastructure that powers the data centers today, because I think that's where you're going to go towards as fully autonomous vehicles or fleet management takes a little bit more, I would say, broad acceptance by, I would say, by consumer space.

14:33Yeah. So it sounds like that level of scale required doesn't surprise you given what you know about ADAS and those requirements. Yeah, absolutely. I would think, you know, we've been doing that to some extent historically as well. And you look at our history, we started off small with a much smaller footprint occupying in the mobile space. With time for the last five to six years, we've kind of expanded to adjacent business units could be AR, VR, compute, IoT, and automotive. And with each adjacent BU's getting a lot more traction in AI, the requirements have actually scaled up much, much more than we had expected.

15:13So to answer your question, I've been surprised not necessarily, we've been planning all along. We just have to make sure is the time to market, the time to really accelerate solution deployment can really be much more faster because the rate of innovation that's happening, for example, in automotive space is enormous, right? I see a lot of L2, L3 kind of vehicles coming out this year and next year as people start transitioning to level four and level five, which is full autonomy at that point of time. We have the necessary infrastructure to take actions based on the compute investments and software investments you're going for.

15:45And that's what we've been trying to push for to make it happen. Yeah. So we talked about mobile. We've talked about automotive. A third area that historically has been interesting for edge use cases is enterprise and kind of things like distributed cameras. Is that still an interesting use case or growing use case? Yeah. So when you look at firm factors as such, automotive is one extreme. Enterprise could be seen as something like a PC compute devices, as an example, could also be seen as personalized devices like wearables, hitables, as that market takes over. And PC as a firm factor is gaining a lot of popularity within the enterprise segment for good or bad reason post-COVID, only because work from home has become quite important and the need for AI in PC is just soared post the pandemic situation.

16:47A few examples in this case would be, how can I improve video conferencing as an example? How can I improve streaming as an example within the enterprise space as well? In streaming, historically, when you look at it, there's been enterprise streaming, game streaming. Enterprise streaming always used to be on a reference platform, but post-pandemic, now people want to be doing it from their own homes. The question has really become is, are we in a position to support an infrastructure that supports enterprise streaming? And what kind of firm factor will we push for it? And PC has been a fantastic firm factor to really drive it.

17:21And most of our engagements moving forward, we're going to see a lot of big announcements come from that front and how we're really positioning, I would say, PC as a firm factor to drive enterprise scale experiences. That's interesting. I hadn't thought about PC experience in that question. I I was mostly thinking of these use cases like, I guess, what you might call more embedded use cases, like distributed cameras and distributed sensors and that kind of thing. Do you see a lot of growth in those use cases? Absolutely. I would probably position it this way. When you look at AI inferencing to a large extent, it's very generic in nature.

18:00What applies for me applies to you in any given context. There is no personalization element to it. but the question really becomes is can i collect more data about the user to drive personalization and this could be coming from sensors registered to the user could be cameras registered the user as such if i get more pieces of information it could be visual in nature could be textual in nature could be audio specific information can i make that inference on the edge for the user more optimal right and this is something that's gaining a lot more popularity but the biggest challenge has been as part of this use case enablement is how much data is good data?

18:43What data is good data? How do I enable some level of supervision to make sure the data that's been collected is also labeled properly? Because if we don't have labeled information, I can't engage in retraining. If I have to do retraining on the edge, what kind of constraints I would have on the hardware and software side to be in a position to encourage learning or retraining? So all these big questions usually come up and the ecosystem in general is trying to understand is how can they scale moving forward and this is one such fantastic example as collecting the right amount of data and then driving towards personalized inference moving forward.

19:25Interesting that touches on themes of data-centric AI which is a big conversation in the industry kind of shifting from masses of lower quality data to small amounts of higher quality data and putting a lot of care in place around how that data is curated so as to make more efficient use of it in training and tuning, which seems right in line with the kind of thing you'd want to worry about if you're dealing with edge-type devices. I would say absolutely. Industry as such is kind of morphing away from a model-centric view to a data-centric view. But as you transition to a data-centric view, the question really becomes, can I have investments across the entire picture of MLOps from data annotation all the way to data monitoring?

20:17And especially on the edge, if you're going towards more user-based personalization, it's extremely critical to have elements of collecting the right data, labeling it, filtering it, and training on the edge. And then once you enable inference, you want to also monitor it. are you seeing some kind of drift what kind of drift are you suggesting and can i make any compensation based on the drift so yes absolutely becoming more critical how we go make it happen on the edge is always a big question mark because the physical structure of edge devices is very different not all devices can support it and the question really becomes is can i run partially on the cloud can run partially on the edge but these are big questions that the industry is facing today and i'm pretty sure the next six to nine months will have opportunities out there where people are going to say, guys, AI is actually becoming more meaningful to me because now I'm engaging in more extensive MLOps investment rather than a very toned down, generic, with no common sense AI inference that exists today.

21:16When you think about what that MLOps footprint looks like, kind of this edge aware MLOps footprint, what's different about it from the way folks are doing MLOps kind of in non-edge environments? So when you look at the ML Ops as it exists today, historically, there are two or three scenarios. One, you have a pre-trained model. The pre-trained model either has been available in an open source community or somebody's already developed it. You take the pre-trained model, you quantize it, you fit it to the envelope of an edge device, and you invoke it and you get it running. Now there's a drift. You see drift over time.

21:55Nobody monitors it. Nobody cares about it. And it's already completed. The second category I would say is wherein you don't have a pre-trained model. You want to start from the beginning. And this is a classic example like an infotainment use case within an automotive segment. An automotive segment, you want to look at, let's say, a driver monitoring system. You want to understand the emotional state of a driver to take certain actions and maybe take control over him because if he's not in the right state of mind, you don't want him to encourage driving. But understanding the emotional state of a driver means you need to understand or make accurate predictions, but for that you need access to data.

22:31Today you don't really have too many data points. So here there's a lot more emphasis on creating synthetic data to kind of really create a scenario that would show the emotional state of a driver. Synthetic data was not popular, I would say, three or four years ago, but now it's becoming extremely popular because you don't have enough data to create the right model. So once you happen to have that, again, then you get a pre-trained model, you emphasize, you get it running, and you're fully functional. So the first two is, to a large extent, mostly popular in the industry today. But the third portion is, again, is how do I constantly learn?

23:04How can I make sure that the model I've developed from a synthetic data source or a pre-trained data source can evolve with time as I understand the user much more diligently moving forward? That doesn't happen. But the question really becomes, how can I make it happen? I think over a couple of years ago, Tesla kind of introduced this fact wherein they used to take elements about the road and the scenarios that they're trying to evolve and work on. And then they used to send that to the cloud, optimize the model, and then invoke it back to the device. A similar concept, Google talked about it, what we call as federated learning a couple of years ago, similar concepts.

23:41What we're trying to see is if that information, which is very prevalent to the user, you don't want that information to leave the device. So what this infrastructure that existed before in the cloud, can I bring it to the edge and constantly make the model learn so that it commits less and less mistakes moving forward. And on top of it, can accommodate user-specific information. So that's the big difference moving forward from an industry standpoint. You spoke a bit about these metrics that you are tracking and caring about when you're looking at edge cases and trying to map them onto hardware architectures, things like latency and bandwidth and all these things.

24:20To what degree has each of these architectures kind of drive their own distinct usage patterns? Like we used to think about the kind of the cloud as you want to kind of provision in a hybrid type environment, at least like provision for your sustained usage locally and then be able to burst to the cloud for peak usage. Is there that kind of scenario or thinking in place when you're thinking about edge? I would say to a large extent, when you look at edge by definition as a very wide footprint, the footprint could be in terms of compute, in terms of bandwidth, in terms of memory, in terms of power, which really would put a constraint of what kind of inference is possible on the edge.

25:06That's one thing. The second thing is cloud to a large extent provides you a unified experience, meaning that no matter where you are, as long as you have good connectivity to the cloud, you're able to get a certain inference profile. But the challenge is you need to have that strong connectivity to the cloud. Now, the third scenario which you mentioned, some kind of combinations of hybrid, wherein can I run portions on the edge, portions on the cloud? Absolutely possible. To some extent, it already exists today. A classic example would be if you're trying to do video conferencing, as an example, BlueJeans, Teams, all these folks, when they enable language translation or language transcription, They really invoke automatic speech recognition, neural machine translation, and text-to-speech.

25:51These are heavy compute workloads, and most of them are resident on the cloud. The reason they're trying to be resident on the cloud is you want to be supporting multiple languages to be able to translate English. So by definition, it's a very large library file, which does not really fit an edge envelope. So there are going to be these situations wherein you have to partition your workload between edge and the cloud. Now, the question for us as people are working on the edge is how can you make edge more competitive so that you can enhance user experience? And I'm pretty sure, yes, the generative AI use cases, chat GPT-like use cases become more prominent.

26:25The question only becomes is, should I be running these models on the cloud? Going to run portions on the edge and portions on the cloud, but these are all good discussion topics that is happening in the industry as we speak today to really drive that partition and make edge more relevant compared to that on the cloud. Yeah, literally as we're recording this, One of the things that is capturing a lot of excitement in the community is folks are downloading this new LLM, the meta Lambda model, and running it on their local machines, their MacBook devices, and running the 7 billion parameter LLM and getting results in decent time.

27:04And I saw a post just earlier today about someone who ran this on a Raspberry Pi. And I think this is an indication, an early indication maybe, that people are going to want to see these types of models running on device. And certainly we're going to hope that the performance is better than the 10 seconds per token than the person saw on the Raspberry Pi. Absolutely. Absolutely. I mean, this is going to be inevitable, right? Like people are going to understand the fact is image-to-image translation or text-to-text translation, especially for complex queries, if it takes like 15 seconds to respond, you're not going to get the best user experience, right?

27:43So the question really becomes, can I bring it somewhat manageable and can we get it done? So do we know what that transformer-specific architecture looks like and how it differs from the convolution-tuned architecture? Yeah. So when you look at large language models, by definition, it's an evolution of the transformer itself. To a large extent, most of these models are kind of basically divided into three portions, the encode portion, the decode portion, and the unit portion. But in the expectation, as you can understand the context, understand the relationship between the sequential data that exists as part of the model itself, and make an accurate prediction.