Large models on CPUs

0:06Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related technologies are changing the world, this is the show for you. Thank you to our partners at Fastly for shipping all of our pods super fast to wherever you listen. Check them out at Fastly.com. And to our friends at Fly, deploy your app servers and database close to your users. No ops required. Learn more at fly.io.

0:43Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist building a tool called PredictionGuard. And I am not joined today by my co-host Chris, but I am joined by an amazing guest who is an expert in all things model optimization and efficiency and running on CPUs, which is super exciting. I've got Mark Kurtz, who's director of machine learning at Neural Magic. Welcome, Mark. Thank you, Daniel. Thanks for having me on. Yeah, yeah, of course. So let's maybe just start out with a kind of like state of model optimization right now. So first off, could you kind of describe like when you're talking about model optimization or that set of tooling.

1:36What do you mean by that? And how does that fit within maybe the things that a data scientist or an AI person would want to do? So whenever we're looking at model optimization, we usually focus on a few different techniques. But the ultimate goal is to make the overall model smaller and faster, right? Neural networks known to be very large models, especially compared to more traditional machine learning. And it turns out the size of those models is the important part in terms of exploring a large dimensionality of a space, but actually doesn't use all of those pathways at inference time. So what we specialize in is specifically pruning, where we're going to remove connections within that network, quantization, where we're going to reduce precision of those connections in the network.

2:24So going from, you know, the typical FP32 down to Ent8, and then additionally distillation, where we're taking larger models and trying to teach a smaller model to mimic the capability and the functionality of that larger model. So it's kind of, you know, overall high level. And yeah, it's a very exciting space right now. It's kind of exponential in terms of the number of research papers that are constantly coming out on a topic. Everybody's very excited about sparsity specifically, mainly because you can turn these large models and get rid of up to 95, even 97 percent of the weights are actually useless in these.

3:02Obviously, you can use that for a lot of efficiencies around performance and energy. And that's specifically where we've been focusing in at Neural Magic and what I've been focusing in on my work. Awesome. Yeah, that's I definitely have felt this problem. So I'm sort of asking this question maybe for others out there that maybe haven't felt this problem as much. Why is it important to make models smaller or make them more efficient? How does that fit within what enterprises or even users running smaller applications? Why is that important for people, I guess is the question. Generally, there's going to be two cases that we're looking at in terms of deployment.

3:46And one would be an embedded space where we're running on the edge and trying to work there. So generally you want real-time latency and optimizing the accuracy as best as possible. So if you're using an object detection model, you want to make sure that, you know, for example, you're on a security camera, trying to draw object detection and make sure that you know when a person walks in a frame and whether or not that's alarming or not versus a dog or something like that. So in general, on that edge application, what you can do is use a larger model, remove a lot of the pieces from that larger model so you can keep the accuracy of the larger model, but take up the space of the smaller model.

4:23So significant improvement in terms of accuracy on that edge device while still maintaining, you know, the constraints that were set for you in terms of memory and latency that you need to request back with. And then the second one would be on the server side. And that's generally where we're looking at, you know, more throughput based applications and potentially also latency if they're shipping the data up to some server to be processed either on NLP or computer vision. But overall there, what we're looking at is, especially whenever people get into larger deployments on ML and neural networks, the cost significantly shifts, not from training, but to deployment.

5:08you're for a lot of larger enterprises that are actively deploying 80 90 percent of their costs is purely in deployment on these machines so what you can do is take the exact same model that you have reduce again the amount of a compute that you need to run it so that one it'll run faster but two ultimately what that means is that it's going to run significantly cheaper right and we have a cost savings on the order of you know 10x 20x even larger if you're really trying to specialize and optimize. There can be a significant reduction once you're at that scale. I would say definitely if you don't have anything deployed yet, don't worry about optimizing the model.

5:48Worry about getting a use case that works and something that you can prove out. As soon as you go into deployment, model optimization is a great thing to start because it's essentially just free performance that's left on the table that can significantly affect your bottom line. we've mostly been talking about kind of like model size and optimizations and i do want to get sort of down and get into the nerdy stuff around like how some of this works but before we do that i'm also curious about this element of deployment on gpus versus cpus it seems like some of what's indicated at least in like the tooling that you're building is like the potential to take a large model which might require a GPU at inference time and potentially run that on cheaper commodity hardware that only has a CPU maybe doesn't have a GPU like what is the state of that now and like how far can you push that or maybe also like how could people best think about that in terms of like when and when that might not be possible I guess as you said we specialize almost entirely on CPU performance.

7:03And in that, actually our latest ML inference results on MLperf have come out. So in that we show that we're running faster than T4s and A40s and things like that on this commodity CPU. So server-based CPUs, stuff that you have in your laptop, desktop, things like that. And it's very surprising that what's thought of as these little CPUs can outperform the GPU. And we see this generally across every domain that we've tackled. And that's been across image classification, object detection, excuse me, segmentation. And now we're working in the NLP and NLG space and actively coming out with that. But overall, we're seeing the same use case where these models are overparameterized.

7:50We can take away a lot of that compute. And what that means is that you can actually get the CPU and the GPU about equivalent in terms of compute throughput because with the sparsity and the dynamic setup of CPUs, we can run and skip all those zero multiplications, right? So significant reduction compute, they're about even, but then the CPU has a unique cache hierarchy, which means that we can reuse that cache more often than what you can get on a GPU. L1 and L2 being extremely quick, faster than GPUs main memory, and L3 being about equivalent. So overall, what we do on our performance optimization is skip all the compute to get even and then use that cache hierarchy as efficiently as possible on the CPUs so we can get faster memory access than even you can get on a GPU.

8:36And we pay a little bit more by doing a little bit more compute by doing that. But overall, it works out that you can actually beat the GPUs and that setup with just pure software. I anticipate this is maybe a question that you get sometimes, but it's like you hear so much about the necessity of GPUs for running these large models. Do you find generally practitioners are just unaware of this like possibility of running these large models on on CPUs? And like, how has that been for you and those that you work with that are actually doing this amazing work and have these awesome tools? Like, how has that been in terms of overcoming that barrier of perception that people have?

9:21It was a barrier that we hit, especially early on, a couple years back, where it took a lot of convincing to even get in the door to talk to anyone because they just didn't believe what we were saying. Now it's gotten to be quite a bit better, especially with the newer software that's been pushed out of the newer chipsets on CPUs. They're getting a little bit more even in terms of GPUs, so it's, you know, within a stone's throw to try and match GPUs. So people are a little bit more accepting of it. but yeah whenever we show them the numbers generally the first reaction is well let me try that on my hardware because you guys got to be doing something weird let me let me try to do that let me replicate that and as soon as I do then the next question is okay well how do I do this to my model and that's usually where the tricky part comes in is model optimization hasn't always been the easiest thing to do it can take a lot of research to enable new architectures and things like that.

10:18But that's what we've been also specializing on at Neural Magic is making all the research that we're doing, being able to put that into open source, and also building out a SaaS platform on top of it so everyone can easily play with hyperparameters and get something that is consumable. But I would say that's probably been the biggest gap in terms of trying to get people off of GPUs onto CPUs is the model optimization that needs to take place first to be able to run faster than the GPUs. You talked a little bit about sparsity, which I want to dive into over time. And I want to get to those tools and open source stuff.

10:53You mentioned like these large models and people are probably used to hearing these numbers, you know, 3 billion, 7 billion, 13 billion, like however, and up from there, like these models have in terms of numbers of parameters. Could you describe a little bit what you mean when you say 90 to 95 % of these connections, or maybe less than that, but a high percentage in some models have no impact on the actual forward pass or inference in the model. Could you describe a little bit more by what you mean by that? Yeah, definitely. And I'll take two steps to doing that. One is just covering kind of the 90, 95 % class, at least where we've been able to get to on those.

11:35And the second is looking specifically at large language models. So for the first one, whenever we're looking at getting rid of 95 % of the weights, let's take ResNet50 as an example. This is our toy benchmark model. This is essentially what we prove out all of our technology on because it's a common feature in MLPerf and for most performance tests. So what we can do coming in is looking at those convolutional layers. It has, I forget how many million parameters within it, but it's definitely not the 3 billion, 7 billion or up on top of that. But within that, we can actually zero out. So what we're doing is taking all, imagine taking all those parameters, dumping them into a giant array, and we're just going to zero out the ones that are not important.

12:18And figuring out the ones that are not important is part of the research. The easiest assumption is just saying that the weights that are the largest are the ones that you want to keep. So the ones that are furthest from zero are the ones that you want to keep. Generally, you can think of this in two ways. One is that as the model's training and being regularized, the weights that don't matter are going to move towards zero. And then the other thing is during that forwards pass, the weights that are higher magnitude have more an effect on the output, right? And everything else is going to be noise in between.

12:49So we're able to essentially get rid of just our, whenever I say get rid of, I mean setting those parameters to zero within 95 % of them. So you're left with 5 % of your weights that are non-zero. And that's actually all that you need to preserve the accuracy on ImageNet for ResNet 50, for example. And some quick kind of intuition in terms of how I've been able to think about this and why it works and things like that. You can see as we increase the size of our dimensionality in our optimization space, what we're doing is, and there's a few research papers out on it, that we're able to connect more of the local mins.

13:26right so the optimization process will slowly converge further and further down because more of the local men's are connected generally though there's only a few of those pathways that you actually need to connect those local men's so all that we're doing is we're following down that most optimized pathway and removing everything else around us in terms of that dimensionality so it's kind of one of those things that as you're training it's slowly selecting the weights that matter that gets you down to that local men and there's very few so the important part was that large dimensionality of the optimization space but not every direction mattered right so then we can get rid of it and then diving in on the llm side and large language models we actually have a recent paper that came out from one of our principal research scientists dan alistar called sparse gpt and that's where we're looking at taking opt and bloom models all the way up to 175 billion parameters and be able to optimize those and remove as many weights as possible all in this case in one shot so just using the model without any retraining we're able to get rid of around 60 percent of the weights without doing anything and there's a new paper out of cerebris actually that was looking at the llm story and they're able now to get to 80 sparsity on these llms with retraining so that's kind of the research direction that we're headed down now is proving out how optimized we can make these models.

14:52Because there's also a lot of interesting stuff that happens with the large language models, specifically because it's generating one token at a time, very latency bound. And that means that it's a lot of memory access to load those weights. So if you can quantize those and then get rid of half of them, you're already at anywhere from a four to six X speed up just on your inference times. And that's generally where we're focused and looking at currently to try and get those LLMs to run faster. The other thing to call out for those two is, you know, 7 billion parameters and 175 billion parameters.

15:26Those don't fit in a single GPU. So now you have, you know, clusters of GPUs to serve one model. And a lot of that compute is just completely wasted because all that it's going to is trying to maximize the memory on the GPUs. For CPUs, you can throw a few terabytes on there and it works out fine. So that's the other thing to call out with the LLMs in terms of GPU versus CPU.

16:01This is really interesting, Mark. I want to follow up on what you were just talking about, which I think is a really, it's sort of a subtle point, but it's really interesting in that I think if I understood you right in what you're saying like let's say that I have one of these large models 175 billion parameters or whatever and even for inference I have the necessity to have multiple GPUs just to load that model into the memory of the of the cards whereas on a CPU you can have terabytes of memory what I'm assuming is like you could load that in as long as you're able to execute it quickly, which I guess is the other piece.

16:41So am I right? You sort of have to have both like the ability to load it into memory and you have a bit more space in that on the CPU side, but then you also have to be able to execute it very quickly, which I guess is like why you would think about both space and sparsity. Is that an accurate way to put it? As you said, you know, you have a total space you need to take up and then a minimum latency that you want to respond to the user out, right? And that's going to set the constraints for your hardware. And for CPUs currently, at least for the smaller models, you can get to a usable speed on those.

17:18If you've seen like llama.cpp, they're doing int4 and things like that on smaller models and they're usable, but they're less accurate. So what we're trying to do and what we're actually working on right now is making sure that we can get that GPU class speed while maintaining the large memory advantage of CPUs. So you can deploy this 175 billion parameter model on something local and you don't have to worry about data privacy, anything like that. It's just there working and available and highly accurate for you. I definitely heard people that I've talked to who have tried various optimization techniques and have maybe been dissatisfied with the performance hit that they're getting, not in terms of compute, but in terms of actual model performance or accuracy or whatever.

18:04Does that performance hit often come about because of the quantization that's maybe part of the optimization techniques? Or are there multiple sources of that performance hit? How should people think about that? I think the biggest thing there is honestly just the amount of choices that people have to apply and not knowing when to apply them. because generally for quantization, for example, you can apply quantization to pretty much anything at end eight for both activations and weights and have it recover. But there are definitely cases where, for example, we've been quantizing efficient, that's quite a bit on our image classification side.

18:48There's one or two layers in some of these that are extremely sensitive for whatever reason to quantization that you can't quantize those. So removing those, then you get 100 % recovery, right? So it's a lot of these kind of little things that researchers know intuitively in terms of having used this constantly. What will work? What won't? But that's not really coded into software anywhere to make it easy for people to use. So generally, they'll go through, try and quantize, and there's no feedback loop. There's no methodology. It's just, hey, I was able to apply it in one shot, but it lost 5 % accuracy.

19:22Talking about quantization. but if you do you know a quantization or a training scheme generally you'll recover all of that back and it generally works completely for that same thing on pruning pruning you'll definitely see more of a drop and it's much more of a requirement to do training aware on the pruning side at least to get to really high sparsities but you definitely will see this kind of the choices that are made in the hypergrammeters that are chosen those can significantly affect the recovery and the quality So generally, I'd say, you know, if they were seeing drops in performance, it's primarily because of those choices and those issues and just the wide breadth that's available right now and not knowing how to narrow it down.

20:02And that's what we're actually working on. That makes sense. Yeah, yeah. That's good for people to myself. I want to develop a little bit more intuition around these things, because like you say, sometimes you're just like, oh, here's the command that I run on the command line. and like I get this file out that is smaller, right? But I don't have a great intuition about like, similar to hyperparameter tuning, right? Like that takes time to figure out like, okay, how should I think about changing my learning rate if this happens or if that happens, that sort of thing. You mentioned one thing which I think would also be good to kind of clarify and help people understand is like training aware optimization optimization versus just non-training, I guess, optimization, or I don't know what the counter to that is.

20:53Could you talk about those, like how they're differentiated? Some people might guess what that means, but like, how are they differentiated and how does that work out in practice in terms of how you would optimize a model? Technically, we have three categories generally available. One, and the two you're going through is one training aware. And then we have post-training or one-shot, which are kind of interchangeable. And then we additionally have on sparse transfer, which is something that we've been pushing a lot because the research has worked out quite a bit for it. So I'll cover all three of those in a little bit more depth.

21:24So for training aware, what we're doing is taking the exact same model that you're wanting to deploy in the exact same data set it was trained on and continuing the training process further. So while we're continuing that training process, we could be continuing it. This is where the hyper parameters come in but generally it'll be about half the time that it originally took to train it we'll train it for that much longer and as we're doing that training we're iteratively pruning away or we're applying quantization or both and the reason we're continuing that training is because as we're iteratively applying these optimizations we're slowly moving the model away from its local men right and it has to adapt and adjust back so by slowly doing that and training over some time we allow small jumps that the optimizer can recover from and adjust the remaining weights for rather than doing it all at once.

22:15And the all at once piece is where we get into post-training in one shot, where we're not going to try and retrain the model at all. We're going to take a small calibration data set, and then we're going to use some heuristics or algorithms to figure out, using that calibration data set, how to optimize that model and remove weights or quantize. So the most common case would be static quantization. We're using a calibration data set to figure out the activation ranges for each layer, right? And once you have the activation ranges for each layer, you can set up a simple quantization scheme to say, given that it's going from, this layer is going from negative six to six.

22:57Now I need to fit that range into an intake scale of zero to 255, right, and be able to map that in. So that would be a simple post-training or one-shot application. And then the final one is sparse transfer, which works exactly the same as transfer learning or fine-tuning. It's just we're starting with a sparse model to run through, and that's a lot of what we've pushed up in neural magic into our sparse zoo are these open source sparse models that we have you know sparse birds and res f50s and yolo v5s things like that and you can just take those plug in your data set and transfer over to it so the sparsity mass stays in place and it just adjusts the remaining weights to fit your data set we have a few papers out on that as well that shows that sparse transfer works just as well as regular transfer yeah that's really interesting.

23:51I guess this would somewhat depend to like, if I'm just thinking of like the average practitioner out there, right. Probably in a lot of the space that I work in, in like the larger language model area, then like, I'm not going to be able to retrain one of these large models on the original data set, right. Or even half of that or for half of the epochs or whatever. But these other things are certainly things that I do all the time, right? Like fine tuning, transfer learning. So it's cool to understand that there's options out there. Am I correct in assuming that like you mentioned the zoo, the sparse zoo that people can find on your website and we'll link in here too, that like researchers, your team, practitioners, whoever are the people out there are also putting in work to actually release some of the things that are going to be able to do.

24:48of these sparse models publicly to the community so that I can take those and then maybe do fine tuning on that. Or maybe it's just good enough from what's released in the community. Could you tell us a little bit about that community and what's being released? So on the open source side, pretty much everything that we have on the sparse suit currently has either been from our lab. We have a few that are from Intel's lab up as well. And some hugging face examples and things like that, primarily because a lot of the sparsification research is all built around a few models like ResNet 50, BERT, and things like that.

25:27They don't expand it out past those models to prove out their algorithms. So we have the best of those. And then, yeah, that's exactly what our team is working on. And what we get some community contributions in every once in a while for sparse models that people have generated or transferred, right? So our goal is to able to push these up so that as you said anyone can come in from the community and be able to pull those down and get value out of those models so you can think of them as sparse foundational models rather than the dense foundational models

26:10mark in addition to the sparse zoo which is really cool i know that neural magic is producing some pretty uh interesting and useful tooling otherwise as well in terms of actually doing some of this optimization themselves. And I see certain things, DeepSparse, SparseML. Could you describe a little bit about if I'm a practitioner, I have a model and I want to do optimization, what does that look like for me right now with the tooling that's available? So we have SparseML, which is our open source model optimization framework. It's built primarily on top of PyTorch. And then we have integrations with Torch Vision, Hugging Face, YOLO, Ultralytics, YOLO v5, pretty much all the common repos that most people are using.

27:00We've already integrated with, so you can use our integrations and just plug in your model and go along with that. And then the other part is that we have recipes, and I'll go through recipes in a second. But we have those kind of pre-coded integrations, or you can create your own integration. Usually it only takes a few lines of code. we've done all the hard work in terms of making sure that whenever we want to optimize a model that you just have to wrap the optimizer in PyTorch. Essentially wrap the model and the optimizer, which is what our code handles, and then it's going to go through and handle the optimization past that.

27:29So there's no coding really from your side or from the practitioner's side on implementation. And then the other part is then coming up with an optimization recipe that people want to use. And what that's going to lay out is saying that I want to prune from this epoch to this epoch, for example, and then apply quantization and target these layers and at this sparsity level, things like that. We have automated ways to generate those as well as examples on the sparsity and in other places. That's generally what it would look like is, you know, get up and running. We definitely recommend checking out the sparsity first, seeing if there's anything that you can do to just transfer the model onto your data set, because those are the quickest and fastest ways.

28:09and then otherwise if you do have a specific model architecture that you're looking at then you can start going down this integration pathway and generating your own recipes and that's the current state that we're at the other thing that i wanted to call out is that we're working on a saft platform right now to make all of this more intuitive so you have a ui to be able to predict where that model is going to end up before you start optimizing it and then additionally actively benchmark it across your different deployment scenarios so this is called sparsify we've had an old kind of alpha state for a while that was a downloadable and we're actually going through alpha testing right now so anyone that is interested in trying that out definitely reach out we're going through alpha currently looking to go to beta and probably next month two months and then ga following up after that so definitely check that out as well and yeah yeah that's great um yeah i i love how you're thinking towards uh usability as well around these things because i i I do see this as something that kind of blocks people on optimization a lot because they get stuck.

29:13Like you say, it's like, well, what recipe do I use here? It seems like there's a million options. Like, where do I start? So that's really awesome to hear. Just so people can understand, like, there's options available in the Spar Zoo. I'm kind of scrolling through there now. There's a lot, even ones that I know that I use, like Distilbert, which is fine-tuned on squad. I use that all the time for question answer. And apparently I should use the sparse one because, yeah, that would help out a lot, both in terms of compute and speed. Let's say that a model is not there. And in particular, people, of course, are interested in all of these things being rapidly released all the time.