Stable Diffusion and LLMs at the Edge with Jilei Hou - #633

0:07All right, everyone, welcome to another episode of the TwiML AI podcast. I am, of course, your host, Sam Charrington. And today I'm joined by G-Lay Ho. G-Lay is a VP of Engineering at Qualcomm Technologies. Before we get into today's conversation, be sure to take a moment to head over to Spotify, Apple Podcasts, or your listening platform of choice. And if you enjoy the show, please leave us a five-star rating and review. G-Lay, welcome to the podcast. Hey, Sam. Very nice to meet you here. Yeah, I'm super excited to dig into our conversation. we'll be talking about, well, a pretty broad variety of things because you cover a broad variety of areas, but we'll be particularly digging into the work that you and your team are doing around generative AI on devices.

0:55But before we jump into that topic, I'd love to have you share a little bit about your background. Sure, Sam. I'm specialized in information theory and signal processing. So this is actually about 20 years ago when I obtained my PhD degree from USSD. And then I came to Qualcomm right afterwards. So one thing I should remember to note is the founding father of the information theory, whose name is Klaus Shannon. He happened to be one of the early pioneer in the area of machine learning too. And also many of the machine learning, modern machine learning practice and achievements actually are inspired.

1:33by information theory. I'll give you one example. Many times people using a term called a KL divergence, which is a divergence between two probability distributions. Actually, that concept first came through in the area of information theory was introducing that context. And then later on, it's become a very, very popularly used when people using the training objective when they're trying to train neural networks, for example. So there's a kind of a match made in heaven, so to speak, between information theory and artificial intelligence. Yeah, clearly. And that's particularly the case at Qualcomm, because a lot of what you do from an AI perspective is focused on applying machine learning algorithms to compression, which is also kind of a natural place for information theory.

2:19Right. I'm curious, you know, I've spoken to many researchers there at Qualcomm, but you've had an opportunity to be instrumental in the forming and development of Qualcomm's AI research group there. Can you tell us a little bit about that experience? Sure. You know, a few years ago, we made a few acquisitions of the AI startups. And from where, you know, we assembled a few very smart kind of, you know, mind people in the machine learning area, and many of which actually very influential machine learning research community. When we assembled this the talent team together and it's become only become natural this is around 2018 and we're starting this initiative we'll call the Qualcomm AI research and the key right is about to drive and accelerating the cutting-edge research to helping Qualcomm to accelerate the machine learning adoption at a company as well as contributing back to the research community as a whole.

3:16Maybe adding to this very early on we have a very clear focus in terms of our research agenda I primarily focus on the power efficiency for on-device AI experience and also think about personalization. So when we think about power efficiency is largely think about bringing machine learning workloads to the device where we can run very efficiently and with a very good user experience. And personalization is more coming down to the idea of make sure the machine learning models can be adapted to your personal need and personal preference. For example, a lot of the speech and audio use cases fall in that category.

3:55And I have to say, with all the earliest research we have done, one of the clear highlights is we have a building the world class quantization research team. So quantization research team is a primarily boils down to if people today, right, using FP32, FP16, or even people can talk about FP8. But I think we'd be able to quantize them into more uniform kind of data grids. So where the data can be presented much more efficiently and also even from a computation, from an implementation point of view, it will be the most power-efficient data format we can implement on the device. And later on, as we're touching on more on the topic around Generely AI, we'll be able to say a lot of breakthrough in terms of the ability we are able to bring all these use cases to the device.

4:42Actually, quantization and pre are very essential for us. Lately, we have expanded our team. I think Qualcomm AI Research, we probably have a size between 200 to 300 people in our range. So we cover very broadly when you think about fundamental research, when you think about platform research, platform research largely alluding to bring all the efficiency technique to our AI compute platforms. And also is it coming down to applied use cases, where you think about a lot of vision use case, even a lot of wireless use cases, we can bring state-of-art solution to all the different technology and the verticals.

5:19So generative AI is certainly a bit of a buzzword nowadays. Is that a new area of research for Qualcomm AI research? Or is it something that you've been working on for a while? So when people talk about generative AI, right? I think it's a subclass of AI where you're able to generate or even regenerate the new content. The new content can be text, can be image, can be video. via resampling or sometimes called reconstruction too. Even though the term of a GNRI is a bit new, but the technology behind it, what people call a generative modeling, actually has been around for quite many years. For example, the familiar terms people think about GAN, think about VE, think about autoregressive models, those technology or kind of machine learning practice has been around a few years now.

6:11So why, you know, GenoEI just all of a sudden becomes so popular in a little sloppy way to explain, right, is really boils down to people realize actually just with the same kind of generative methods. But if you are able to feeding this model very efficiently with the internet-scale data, where you can just scrape the data from the internet, think about text data, think about image or even video data, and then being able to feeding the model through and also allow the model to have a massive capacity to absorb all the data in a sort of a very knowledgeable way. I think that's the situation sort of where the breakthrough is happening.

6:51And then general AI just brings a lot of disruptive benefits to the consumer and also to the enterprise industry altogether. One thing maybe I can add into this is, as I mentioned, when we think about general AI, I think in a more broader term, I think our research group has been working in those areas for quite a few years. So one of the key areas we have made a lot of investment is the topic of data compression. So when you think about data compression, you can think about the terms of speech compression, think about image compression, or think about video compression. One thing early on when we made an investment in this area is because if you think about today, right, the data traffic on the internet today easily can be 80%, 90 % are all coming from image and video itself.

7:38So if we're able to using AI approach to further improve the compression efficiency of the image and video, I think this is going to provide tremendous value to Qualcomm and also to the industry as a whole. I remember this is like probably four years ago. We have a jump into this. And one of the early thought was saying, hey, look, this is essential. You think about you have a sender. You have a receiver, right? Transmitter and receiver. And it's about on the transmitter side, you have, actually, in this case, in general, we are applying this very popular technique called a VAE. It's called a variational autoencoder.

8:12So you have on the transmitter side, you have an encoder, right, which is trying to compress all the pixels down to latence, which is in a much more compact space. And then on the decoder side, then you will be able to decompress and reconstruct the samples. So in that sense, you can think about the decoder essentially play a very popular role as what we're discussing today. From a latent, you can reconstruct or resample to regenerating the image or video samples. In terms of the concept, in terms of the practice, it's very similar how the diffusion model is running the denoising step in terms of give you the reconstruction of the samples.

8:51When you think about another use case of a stable diffusion. This is back in 2019. And also, I believe our team back then in 2019, in ICCV 2019, one of the premium conferences in the machine learning, we have published one of the early papers in terms of applying the deep learning end-to-end approach towards video compression. Actually, that paper is getting very high citation, as it's a very popular amount of research community. So then soon afterwards, we're looking at a variety of techniques and how to improve and how to add new benefits to this. As we speak today, I will say we are able to improve the efficiency to the level.

9:31It's probably at least on par or even slightly better compared to some other conventional technique. But more importantly, I think this kind of technique is bearing a lot of utility and flexibility. You know, for example, think about a video stream, right? It could be your social media video or it could be, let's say, a recorded film coming from Netflix, for example. If you're able to apply the neural net-based approach, then essentially you're able to customize, even personalize the content in terms of the codec itself to make sure the codec can be specific, customized to the video itself. So in essence, we are able to achieve even better efficiency and also make sure the compression is done in a way, even customized to the video scene itself.

10:16So a lot of potential. And also we are the first one to bring this kind of AI-driven video compression concept to our device, to our mobile device. I think, remember, this is about a year ago. We are able to demonstrate a CVPR last year. on the HD quality, the end-to-end AI driven video compression scheme. So this is getting a lot of positive feedback from a research community as well. It's an interesting analogy that you draw between generative AI and the diffusion model, stable diffusion in particular, and compression. You can almost imagine thinking about what if some compression model where you've got your transmitter and you convert that into text and text is easily compressible.

11:04And then you use that text to generate a new image. If you can do that perfectly, that would be a great compression scheme. Right. I think that's why a lot of research opportunity in the area is endless for us. A lot of cool opportunity ahead of us too. So let's talk a little bit more about stable diffusion and the work there. You spent some time working to get stable diffusion running on a device. Why is that important? One thing we feel motivates us a lot in terms of running Stability Fusion and, you know, on the edge, coming down two folds. You know, on one side, Stability Fusion model itself, right, is an open source model.

11:40So that's why it's very, very popular. And being able to put an effort into Stability Fusion and also contributing back to the whole AI community, we felt this is a really big thing, you know, we can help all together. So on top of which, I would truly feel bringing Stable Diffusion to the edge, to the device, there's quite a few benefits I can elaborate here. Now, first of all, right, it's really coming down to privacy. You know, privacy coming down to, for example, if I have a personalized prompts, which I probably hesitant to share to the cloud. But in this case, if we're able to keep the Stability Fusion all running all together, right, in a device, so my prompt and my content all, strictly speaking, stay on the edge, I think this is a highly, highly sort of beneficiary from a consumer point of view.

12:29But then it's also coming down to cost. For example, we know when we discuss and communicate with all the hyperscaler, all the cloud players, they did express a lot of concern, this kind of large, massive model running from the cloud point of view. in terms of per inference or in terms of when you think about running millions or even billions of inquiry for such kind of use cases, the cost just becomes prohibitive from the cloud player's point of view. So in this way, we are kind of distributed, right? And also getting on all the devices, a billion devices, to amortize the inference cost is a huge benefit to everyone in the AI ecosystem.

13:06And also lastly, we also fail even reliability point of view from a performance because you don't have to send in the cloud. Sometimes the cloud can run out of capacity, but on my device, I always have the capacity, always ready to process my personal needs. Yeah, so I think we see a very exciting way bringing a lot of value in terms of bringing Stability Fusion to the edge. Very cool to see that happening and I get a lot of positive feedback here. And so from a research and engineering perspective, what were the key challenges in making that happen and getting it to run on a device? Sure. So we can share a few insights about this question.

13:44And first of all, we'll see the challenge. It clearly is coming down to model size. So before stable diffusion, the typical machine learning workloads, deep learning workloads we're running on the device, it's more or less, let's say, less than 100 million parameters. Stable diffusion alone is like 1.1 billion parameters. It's almost like 10 times, right, in terms of workloads. And additionally, on top, you remember the Stability Fusion, you need to running through the denoising steps. Adding the default stage, you don't have to running through, let's say, 20, 50 steps. That's all kind of adding up in terms of complexity and the challenge, how to, you know, being able to run in this workload on our device.

14:23So one of the first questions is to handle this kind of workload in terms of robustness and also the automation we have to make sure maintain to letting this model can run through our AI software tool chain. This is a really good challenge. It's a really good stretch, but it turns out our software stack is pretty robust to handle this at the end of the day, and we're very happy to see that. The second challenge clearly coming down to the inference latency. This would require a full stack research and optimization, make sure from a model, system algorithms, to software stack, and eventually to the hardware silicon processor itself, end-to-end.

15:04We have to look at everything to be working very nicely together. Among which I would say, as I mentioned earlier, I think one of the big recipe we are being able to throw into this is about a quantization research. Remember when it came out, it's like FP32, FP16. If you just run them natively, even assume the model size you can support, this is clearly going to take even more than a minute and being able to run it on a device. So then probably a user wouldn't be able to have a patience, right? To wait that long to say, hey, what is my final image coming out after a minute or two? So then what we've been able to do is without retraining the model, simply by applying the post-training techniques, and we are able to quantize the model efficiently, in particular on the weights, we are able to quantize to 8-bit weights.

15:52But then we realize that simply if you just do a simple, naive post-training quantization, we realize that the signal-to-noise ratio, or in terms of quality of the pixel you generated, actually is not good enough. So from that point of view, we realize we have applied more advanced PTQ, post-training quantization techniques, what's something we call Adderound. So Adderound in a simple way you can think about intuitively speaking. Most of the simple way, every single weight, when we think about quantization, usually we're trying to quantize to the nearest neighbor individually. But the Adderound essentially, instead of looking at each of the weights individually, trying to create the quantization more across the board in terms of per channel, even per tensor basis.

16:40So this will give us a more holistic view in terms of all the ways when you're going up and down, there's a more systematic way to think about on a bigger picture, what would be the better way to apply rather than applying on a single weights. So it turns out the Adderound, in general speaking, we realize if we think about quantization, right, to 8 bits, for example, So Adderon actually magically can give us virtually. It's still 8 bits, right? But just because of applying the Adderon technique, we're able to add in probably one more bit. But this one more bit is magical because one more bit, even virtually, roughly speaking, if people understand the term of signal-to-noise ratio, it's adding us about 4 to 6 more dB.

17:23So 4 to 6 dB is very crucial in terms of able, allow us to getting very efficient data format without retrain the model altogether. So that's why we are able to very quickly, within a very short time, applying the Adderon technique and I'm able to release a demo at an MWC event back in the late February timeline. Yeah, so just to recap there, the two key challenges were, one, being able to work with this large model using your existing kind of tool chain and workflows. And then the second is some more specific challenges that arose in trying to achieve the desired performance levels and in particular the use of quantization.

18:06I think this question is related to both of those challenges. Did the ultimate solution from a quantization perspective, was this kind of an out-of-the-box push-button application of the tool chain or was there a degree of research effort involved in achieving the result. So this is a really good point. By the way, this is something I can make a small advertisement here. We actually created an open source toolkit. It's called AMET, called AI Model Efficiency Toolkit. So the intent of which is trying to democratize and providing very easy to use quantization toolkits for, let's say, data scientists or can be ML engineers being able to use them with the convenience and ease.

18:53Let's say, take the post-training quantization as example, we have more turnkey flavor quantization features is available in AMET. Actually, for example, developers can just pick up and use in a very relatively easy kind of format. But on the other hand, I have to say Adderong is a bit more advanced technique. You have to apply essentially like a localized optimization procedure to all these models. But of course, we are packaging this localized optimization procedure into the quantization float itself, sort of letting things easy to do. But on the other hand, I have to say, because it's still involved as a local optimization procedure, it still takes some time and being able to run through the optimization procedure itself.

19:41So actually, one of the very essential efforts we're trying to work on is when we're trying applying these kind of advanced techniques to large or even massive models, we're trying to reduce the optimization time significantly. So actually, in essence, you know, we're trying to bring more efficiency to this kind of advanced technique, and hopefully they will show up in the AIME very, very soon. This required the local optimization is beyond the current capabilities of the toolkit, but maybe something that can be generalized and rolled into the toolkit for future use cases. Exactly. I'm wondering if you can elaborate on that a little bit and how specific to the diffusion model, the local optimization that you had to do was, you know, or conversely, how generalizable do you expect it to be or what aspects do you expect to generalize?

20:36You're more specific pointing out the add-around technique or... For add-around, yes. So let me say the following. For example, add-around, we, when we apply to the stable diffusion models, I cannot get into too much details, but I can talk about a high level. We probably take about hours running time to apply the Adirond optimization procedures. But our goal, actually, we are probably within line of sight. So we are targeting to reduce this optimization time from hours to minutes. And hopefully with this application of the more advanced version of Adirond, we'll be able to make sure is become more turnkey, right?

21:13Bring more benefits to all the developers. So can you talk a little bit about, I don't think you actually mentioned the results that you arrived at with stable diffusion. You were able to get that down to 15 seconds or so. Is that right? We're able to getting down to less than 15 seconds. I think our record is like around 13 seconds right now. Without going into details, we are actually working on the even more efficient version. When we think about all the efficiency technique we can put around, I think we're very hopeful we can bring this down to less than five seconds, hopefully in the next few months.

21:52We'll be very happy to report back to Sam by then. And without going into those details, are you expecting it to be in the same domain, additional work on quantization, or are there new techniques that you're looking to apply? We look at a combination of different kind of techniques here. So quantization is still part of it too. Got it. And to what degree do the challenges that you saw in working with stable diffusion and getting that to run efficiently on device apply to other types of generative AI models like large language models? That's a really good point. So in general, actually, one of the nature of a generative AI, right, as I mentioned earlier, because essentially is a training on Internet scale data.

22:35So one of the sort of a property of these models, they have to be really large. and being able to absorb all the knowledge coming from the internet skill data. And then in this space, actually, I can divide this kind of chain of 8i model largely to two different categories. You call LLM, right, large language model. The other category I call LVM, it's called language vision models. So you can think about the diffusion model is largely a state under, state of diffusion, right, state under the LVM in that category. So within that category, I would say a lot of the technique, when we're looking at it from a quantization, from a software compilation, from a system hardware software co-design point of view, the practice we're doing from a stable diffusion is very, very directly leverageable, applied into all the other kind of LVM models.

23:29There's one very popular model lately just come out called SAM. Actually, it happened to be here with your own name. It's called SEM, Segment, Anything, Coming from Facebook. Very early preliminary results. We realized, you know, technique we're doing for Stapy Diffusion is very directly applicable to the SAM model coming from Facebook. So then the LLM model is somewhat different. Some are different in a sense. LVM model, the most computation. And when we think about signal to noise ratio, when we think about the quality of the regeneration is you have to go in back to the pixel space, right? So that's why there is a more clear analogy between stability efficient and with all the other LVM models.

24:13But coming back to LLM model, it's all kind of, or I should say, largely is a text to text. So when you think about text to text is a more in abstraction form, you think about it's from a token to token. So it turns out the signal to noise ratio, when you think about the noise handling, when you think about the regeneration of the token itself, it's a very different kind of a beast as compared to LVM. But still with that said, why is it so different? Think about these things as all being vectors, you know, representation at some point. So very different in the sense of vision model, the division model, because eventually at a pixel level, I need to have in-depth representation in terms of grayscale and also RGB colors.

24:58So in essence, those will have a demanding, eventually, final pixel output. I need to have enough bit width to represent. But if it's coming down just a simple text, and then the bit width, I can be a lot more aggressive towards in terms of compression, in terms of quantization. So in essence, for example, a very popular data format, when we think about the steep diffusion, what we came up for the MWC demo, right? That's just like 8-bit for weights and 16-bit for activation. But for the LLM model, because it's a more abstraction form of what you think about language tokens, language symbols, actually we can compress much more aggressively down to 4-bits.

25:38Even though we have a very early stage, we believe we have a potential maybe further reduce even beyond 4-bits. So I'm just saying LLM in general speaking, because on one side, I think this is clearly benefits to us because on one side, we felt there's more value to bring smaller bit width to LLM compared to C-B-Division or other LVM models. But on the other hand, generally speaking, right, you think about LVM typically is in a range of the model size between one to five billion. But when we go to LLM, right, if you think about cloud version, I get more than 100 billion parameters. But even on the edge model, the very popular LMA model from Facebook, they're starting from seven billion too.

26:18So what I'm saying is that clearly the quantization to smaller bit would have more incentive to enable us to bring all these large language models to the device. So I think this is where actually we believe one of the very important areas. We have to make sure to stay ahead of the curve. Eventually we can bring the value to the industry. And also make sure we're on the frontier to bring all these large language models to the device. We are very confident we'll be able to do so. And we probably look at a second half of the year as we're working across the board between research, engineering, and product team.

26:55We're going to bring a lot of new announcement to the industry. So just to make sure I got that, I think my takeaway from that last bit of your explanation is that both the LLMs and the LVMs are challenging to run on device, but for different reasons. For the LVM, it's because essentially your feature dimensionality is so high with the pixel representation. On the LLM side, it's the size of the model owing to the size of the training data. Right. Again, just kind of sharing with you, and also the insight we have obtained, looking at both LVM and LLM, right? On one side, this attention module is a very sort of important kind of constituent block, right?

27:42In terms of underneath the hook and being able to support both LVM and LLMs. So in essence, being able to putting out the best computation support from our AI compute platform for Softmax for attention module. We believe we're in the best spot, at least from the edge, from the device front. But on the other hand, we also realize the computation nature between LVM versus LLM is also very different, meaning the compute density for the LVM is potentially much higher on a per-weight basis as compared to LLM. This is also trying very different requirement when we think about this, the software stack, when we think about the graph level compilation technique, applying to LLVM and LLM somewhat can be different too.

28:30But on the other hand, we have also very good recipe when we think about automated version of a software compilation from both the algorithm and also from a software practice point of view. We are very confident we can push on both frontier and putting the most efficient form of implementation of the both category to the device in very, very near term for us. Yeah, it's tempting to think that since transformers are becoming this kind of unifying architecture across all different modalities that you would just have one problem to solve. But it sounds like you still have very distinct problems to solve for each of these new models that you want to bring to the device.