Generative models: exploration to deployment

0:28Welcome to Practical AI. and database close to your users. No ops required. Learn more at fly.io.

0:42Welcome to another fully connected episode of the Practical AI podcast. In these episodes, Chris and I keep you fully connected with a bunch of different things that are happening in the AI and machine learning community. And we talk through some things to help you level up your machine learning game. My name is Daniel Whitenack. I am a founder at Prediction Guard, and I'm joined as always by my co-host, Chris Benson, who is a tech strategist at Lockheed Martin. How are you doing, Chris? I'm doing great today, Daniel. How's it going? It's going good. You know, this week I was, well, it's been an interesting couple of weeks for me in that I was at the Intel Innovation Conference out in San Jose the week before last.

1:30And then this week I was at the Go Programming Language Conference called GopherCon and taught a workshop there. And so that was really enjoyable. So two weeks in sunny California or mostly sunny California, I guess. That was really cool. So maybe even just highlighting a couple of cool things that are happening in those communities at Intel, there were a couple of things that were highlighted that might be of interest. One is, it seems like Intel is really diving into the idea of AI-enabled applications on your local machine, which I know is something we might talk about a little bit in this show in particular.

2:15That is like, hey, if I want to build a desktop application that people actually run on their laptop. And I want that to run stable diffusion as part of the application and not, you know, reach out over the network to some API. How would I build that? And what would those sort of like AI PCs is, I think, what they're calling them, what would those have to look like? And they're thinking about that with some of their processors, which is interesting. And then on the data center side, they had a bunch of things, including announcing the Intel developer cloud, which is cool because you can go on there similar to other cloud environments and spin up either a VM or actually connect to a bare metal instance that has their latest generation of processors, including these Gaudi 2 processors, which are from Habana Labs.

3:13They were acquired by Intel, I forget when, but they have, so they would be sort of on the data center side. You're running accelerated workloads on these. And we're actually running some of our prediction guard stuff on these Gaudi processors and seeing really great performance. So those are a couple of things highlighted from there. And yeah, I don't know. Have you heard those themes in your conversation as well in terms of either new processors, advances in data center technology or this kind of local inference side of things? I have quite a bit actually. And I'm certainly not an expert on microelectronics by any stretch, but I have friends who are and listen to them closely when they talk.

3:57There's a bit of an ongoing revolution on the microprocessor side. And so, you know, many of us that have been in the AI world for a long time. There have been, for instance, GPUs from NVIDIA have been kind of a core to that. But there's a lot of chip types that have been coming out by a number of different vendors to compete with that. Famously, Google was probably the first one well-known with their TPUs, Tensor Processing Units. But there's all sorts of specialized chips and chiplets that are coming out that are enabling these types of things. So I think Intel is definitely one of the global leaders in that and looking forward to having it'll be nice when everyone's laptops and phones and everything are all completely equipped with everything they need.

4:44Yeah, yeah. It's super interesting, especially for use cases where it's like your personal assistant, AI enabled personal assistant that really is tied to you personally. Applications like that, I think you'd want to run a lot of those things locally and not be sending a lot of that data all around. So that's kind of interesting. They also talked a lot about confidential computing, which is an interesting topic that I think maybe some of our audience, at least, wouldn't be familiar with as much from what we talk on this show about. But it is very connected to the AI world in the sense that if you are running kind of secure workloads through AI models, whether you're doing that on NVIDIA chips or other chips like we've talked about, there are ways and toolkits to enable you to actually secure the environments that you are running those models in and actually provide attestation to know that nothing has been tampered with inside of those kind of secure environments.

5:56So I'm going to surprise you. I actually know quite a lot about that. Those are trusted execution environments. Let's just say I've touched on those quite a lot. I think Intel's version is like TDX trusted. Yeah, it's something. They have a couple of different versions. That's the one that's out in the marketplace right now. But yeah, it's the idea of ensuring that when you normally, if you're running a program for audience, if you're running a program and it has to transit, obviously, from system to system, every system has a processor it's processing on. And even if you're running encryption at the application layer, you have to unwrap that encryption for the processing to happen in the chip.

6:33An adversary, you know, if it's on the order of a major nation state, has the ability to steal unencrypted information that had been encrypted in transit straight out of the processor memory. And Intel and other vendors are starting to push trusted execution environments and products and services around that which protects and guarantees the safety of that data inside the processor something i've spent some time on actually yeah it's super interesting and i think even the cto and his talk had like a t-shirt that sort of had a venn diagram kind of thing between like security and ai and at the intersection of that is a lot of you know what he talked about this sort of idea that, hey, whatever hardware you're running on, if you can combine AI workloads with these sort of trusted or confidential computing ideas, that can be very powerful and take care of at least some of the security and privacy concerns that people have with AI workloads in general, which is cool.

7:41So yeah, the two are converging in a big way because while trusted execution environments, which are referred to as TEEs, have been around for years in processors, now that we are having large federated workflows, which is really classic on cloud-based AI jobs, where you're distributing an AI inference or training across many, many systems with very, very important data that you would not want to get into an adversary's hands, that federation is really kind of pushing AI and chip providers together in that way to guarantee that. We didn't see lots of workloads that would be falling in that category until we hit the AI space, and it's chock full of them.

8:22So I keep remembering things that happened over the past couple of weeks while I've been traveling and people have mentioned, But one maybe other noteworthy thing for people to be aware of on the more of the infrastructure side, which I think we will talk a little bit more about in this episode, is that Cloudflare announced their workers AI. And I think this is the latest in this sort of series of serverless GPU solutions. So these worker AIs are Cloudflare's version of the serverless GPU type environment that we've talked about with things like modal or base 10 or banana. And there's a lot of these coming out, but I think it's worth noting that a very large player like Cloudflare is now kind of dipping into this serverless GPU space, which I think also signals that we'll be kind of seeing in the cloud side more and more push towards serverless GPU workloads and environments that support that.

9:29Interesting. Very interesting. Well, that's a bunch of infrastructure and confidential infrastructure and computing and security stuff that has crossed our paths in the past couple of weeks. But one of the questions that you asked me leading up to this recording was about things are moving so fast. And I think deploying and managing an AI workload may look different now than it even looked six months ago. and it's been a while since we talked through the kind of developer or technical team perspective on how you might if you want to use one of these models that's coming out all the time so mistral ai's model just came out the ones that received huge amazing amount of funding just earlier in june and now they have their first model out it's released apache too so you can download it so the question is, let's say you want to use one of these great models that's coming out these days and you want to host it in your company's infrastructure or even just play around with it as a developer.

10:41What does that look like currently? Because there's also, along with these models that are coming out, new tooling that's coming out all the time. So what does that look like these days and what are the various options and things to consider as you're interacting with these models and considering even hosting them yourself or integrating them in your own infrastructure? That's a fair question because it's been a while since we talked through some of the infrastructure, I think, Chris. It has. And for what it's worth, I'm going to brag on you for a second since I know that you would not do that to yourself.

11:16With Daniel being the founder of Prediction Guard, this is a topic that he is a global expert in, really, really knows what he's doing. And as we were talking about, I've had so many people asking me these questions that Daniel was just talking about lately. And I was like, well, you know, one of my best friends is a real pro at this. So thank you. If you can kind of start walking us through, and this is a moving topic, as you just pointed out, it's changed in the last few months and, and we'll continue to evolve over time. But yeah, if you can start walking us through what that looks like today.

11:50You know, we're in the beginning of the fall of 2023, something that might help the rest of us for at least the next few months. And maybe one note on this is I'm also getting these questions all the time. And like you say, I'm deploying models all the time with PredictionGuard. I think a lot of people, if you're a developer or a infrastructure person, you just have that natural desire. Even if you end up using a model that's behind some API that's hosted by someone else, it can be useful and instructive in building your own intuition even to just try deploying one of these models, see what's involved, see how they run, that sort of thing.

12:31It's also kind of worthwhile from my perspective to experiment with different models before you say, you know, lock yourself into a certain model family or something. it's relatively easy now with the tooling to get somewhat of a sense of how these different models perform and build up that intuition for yourself even if you end up using a model that's behind an API. I mentioned I was at GopherCon this week and that was some of the questions that came up to taught a workshop on generative AI and that was a good long discussion in there that people had a lot of questions about was hey let's say I didn't want to use one of these APIs how do I pull down a model and use it.

13:14So yeah, let's jump in. Let's first maybe talk about something that I know that we've touched on before, but just to emphasize here, where can you get models? And let's say that we're putting aside for a second, the kind of closed proprietary chunk of models. These would be ones from like OpenAI, Anthropic, Cohere, et cetera. they have their own APIs. They host those models. Let's say that we're interested in either an open access model, but it could be either an open and somewhat restricted model or an open and somewhat permissively licensed model. And we've talked about that on the show too.

13:58For example, there's models that come out that are licensed for commercial use or non-commercial use or research purposes only, but let's say you want to use one of these open access models. The first question that might come up is where do I find these models? The best place that you can find these models is on Hugging Face. So if you go to the Hugging Face website, just huggingface.co and you click on models, you'll see that there's at the time of this recording around 345 ,000 models on Hugging Face. A few to choose from. Yeah, yeah, a lot to choose from. And think about this, those of you that are familiar with GitHub, right?

14:40How many GitHub repositories are there? There's a lot of GitHub repositories that are someone like tried something in an afternoon and uploaded something to their GitHub repo, right? It doesn't mean that's the most useful thing for you to use in your workflows, although you could kind of learn from it, maybe. It's similar on Hugging Face there's a lot of people that might like oh I tried fine-tuning this model and now I uploaded it to my repo on Hugging Face and similar to GitHub one of the things that you want to look at just as a practitioner is look at how many people are downloading the model look at how many people are hearting the model or you know liking the model and you can filter by those things.

15:28So if I click on model, I can then click on a filter like the task that I'm interested in, a computer vision task or an NLP task or an audio task. And then I can look at both the trending models and how many models were downloaded, filtered by things like licenses and languages. So yeah, I think the first thing to be aware of is just the landscape of models and where you find them and the best place for that currently although there are other repositories is by and far hugging face and go there and treat it similarly to github and that there's going to be a lot of there that might not be of interest to you but there's going to be some really great things there as well

16:22what's up friends there's so much going on in the data and machine learning space It's just hard to keep up. Did you know that graph technology lets you connect the dots across your data and ground your LLM in actual knowledge? To learn about this new approach, don't miss Nodes on October 26th at this free online conference. Developers and data scientists from around the world will share how they use graph technology for everything from building intelligent apps and APIs to enhancing machine learning and improving data visualizations. There are 90 inspiring talks over 24 hours. So no matter where you're at in the world, you can attend live sessions to register for this free conference.

16:58Visit Neo4j.com slash nodes. That's N-E-O, the number 4, J.com slash nodes.

17:10Okay, Chris, I'm on Hugging Face and I see a bunch of different models that are potentially available to me. and I can click on, for example, object detection and see that the trending model that I'm looking at is from Facebook, DETR ResNet 50. Seems like people have used ResNet quite a bit. 603 ,000 downloads. And so maybe that's a good place I want to start if I'm looking at object detection. If I go to, let's say, automatic speech recognition, up at the top would be OpenAI's Whisper model, which is a great choice and released openly that you can use for speech transcription. If I go to, for example, text generation, which a lot of people care about these days, the trending one right now is this new Mistral 7 billion model that we mentioned earlier was just released.

18:13So let's take those as our kind of example. Let's say I want to run something like OpenAI Whisper, or I want to run text generation with Mistral, 7 billion. Or there's even a range of sizes of models, right? The 7 billion model from Mistral, Falcon, 180 billion was released recently. So one question that I think people have is, how do I know which model might serve my task well? And one thing I'd like to recommend to people is even before you try to download the model yourself and run it, you can go in and click on these models. Like if I click on Mistral 7 billion version 0.1. if you notice on the right hand side of the hugging face model card for that model a lot of these models already have a hosted interactive interface that you can just click the compute button and see the output of the output of the model so it's kind of like a playground that you can see a bit of the output of you can do the same thing with a lot of you know computer vision models or audio models and then below that you'll see a little thing called spaces using mistral 7 billion or if you're on whisper spaces using whisper these are little demo apps that are actually hosted within hugging faces infrastructure where people have actually integrated mistral 7 billion.

19:49And a lot of these are kind of just a simple input output interface. And so even without downloading the model, if you're just trying to get a sense for what these models do, you can click through some of these spaces that are using them or just look at that kind of interactive playground feature and just try, you know, upload some of your own prompts or upload some of your own audio or whatever that is to see how the model operates. I think a lot of people might miss this if they're just scrolling through. Let me ask you a quick question. When, if you're looking and you're trying to narrow down, you know, which model you want to pick, we've talked on previous episodes about some of the concerns that go with different sizes and such.

20:32So are there some models that, unless I have a very large infrastructure available to me, many, many GPUs, for instance, that I should probably disregard? Is there like a minimum and maximum practical threshold that would say that I have some hardware, but not everything that I would dream about that I might want to go for? So there's kind of an answer to this and then a follow-up. One is for this sort of transformer language models, oftentimes if you go much beyond 7 billion parameters, maybe pushing it up to kind of 13 to 15 billion parameters, you're not going to be able to run it very well.

21:14just by default by downloading it and running it with the kind of standard tooling on anything but a single accelerated processor like a GPU. And even then, most of the time, not on a consumer GPU. However, the follow-up to that is that a lot of people have created open source tooling around model optimization that may allow you to run these models on consumer hardware or even on CPUs. And I'd like to talk about that here in a bit, that a lot of times you may want to consider this sort of model optimization piece of your pipeline when you're considering how to run the model because sometimes the sort of default size and default precision of the model might not be best for you both in terms of your needs in terms of performance or in terms of the hardware that's available to you but i would say in this phase of like what model is going to be good for me go ahead and put that sort of hardware concern although it's important put it a little bit to the side and focus on which model is giving me the output behavior that I want, right?

22:29Because you have a certain task in mind, right? And if you could figure out, hey, this model kind of does what I want, and it seems like it's giving pretty reasonable output. And then you find out, oh, well, I can't run it on the GPU that I have, or I need to figure out how to run this on a CPU, then And then that kind of narrows down the type of tooling that you're going to have to use for optimization or you might not need to optimize at all. So kind of start with the smaller models and build up to something that fulfills the behavior requirements that you have by just using some of these demos, using some of these spaces.

23:07And then think about, OK, I've now figured out I need Falcon 180 billion. So what does that look like for me to run that in my own infrastructure? Then there's kind of a follow-up series of things that we can talk about related to that. Gotcha. Thanks. So I was kind of getting ahead of myself then a little bit in terms of worrying too much about hardware first. Yeah, yeah. I think the question, well, maybe it's because I come from a data science background, right? My data science experience always tells me start with the smaller models and work your way up to the bigger ones until you find something that behaves in a way that will work for you and then figure out the kind of infrastructure requirements around that.

23:54Because if you start smaller and work to bigger, it's going to be easier to work with that smaller model infrastructure wise and latency wise and all of that. But some people do have really complicated sets of problems where they need a really big, like, let's say, you know, I want to produce really, really, really, really good synthesized speech or really, really good transcriptions from audio. I'm going to need maybe a bigger model than a really, really small open AI whisper model. So it has to do with the requirements of your use case as well, I would say. okay so let's say you identify a model and you're you've kind of picked what you want to do where do you go from there yeah so let's say that you've picked a model and let's take the first case where it's a model that could reasonably or you think it could reasonably fit on a single processor a single accelerator or by your own sort of infrastructure constraints you need it to operate on a single accelerator and even if you don't have those infrastructure constraints i think one recommendation i often give is it's just way easier to run something on a single accelerator or a single cpu so i personally recommend to people even if it's a bit larger of a model convince yourself that you can't run it on a single accelerator or a single cpu before you make the jump to spin up a gpu cluster or something like that it's just a lot harder to deal with even with good tooling some good tooling around that side which we can talk about so yeah let's say that you found a model i don't know let's say it's our mistral 7 billion model You should be able to run that on a single instance with an accelerator or GPU.

25:53I would then look at that model and depending on the type of the model, oftentimes in the model card on Hugging Face, hopefully if it's a nicely maintained model in Hugging Face, then it will likely just like a readme and GitHub, it will likely have a little code snippet that says, hey, here's an example of how to run this. What I usually do in that case is I just spin up a Google Colab notebook because I want to see how this thing runs and how many resources it's going to consume. So I'll spin up a Google Colab notebook. If people aren't familiar, Google Colab is just a hosted version of Jupyter notebooks with a few extra features like you can have certain free access to GPU resources.

26:44There's similar things from like Kaggle and PaperSpace and DeepNote and a bunch of others. So spin up one of these hosted notebooks and just copy paste that example code in that notebook and try a single inference. And oftentimes what you can do in these environments is if you look up at the top right corner of Google CoLab, there's a little resources thing and once you load your model in you can actually look at oh how much gpu memory am i taking up right how much cpu memory am i taking up and that gives you a good sense of hey i loaded this model in i performed an inference if i just do nothing else like the the most naive thing i can do then i'm consuming 12 gigabytes of gpu memory or something like that And that kind of tells you if you don't do any optimization, then you're going to need a GPU card that at least has 12 gigabytes of memory.

27:43And so maybe you use like an A10G or you could use an A100. That might be a little bit overkill in this case. But one of these with maybe 24 gigabytes of memory, you have a little bit of headroom there. And you can say now you've narrowed down not only the model, but potentially the hardware. where assuming you don't do any optimization, potentially the hardware that you could use to deploy it. So as of yet, I haven't spun up really any infrastructure. This is kind of my standard thing where I'm like, hey, what's the deal with this model? How do I perform a single inference? And what kind of resources am I going to need?

28:20It's a nice little cheat code equivalent of finding out what you're getting into, it sounds like. Yeah, yeah, for sure. And if you happen to have, the other way I've done this in the past is if you happen to have a VM or maybe it's just your own personal workstation and you have a consumer GPU card, if you have Docker running on that system, you could pull down a pre-built transformers, hugging face transformers Docker image and just run it interactively, open a bash shell into that Docker container and run an inference, just like I said, or spin up the model loaded into memory in Python. And then in another tab or another terminal, just run Docker stats and it'll tell you, you know, how much memory you're consuming and that sort of thing.

29:11Or run NVIDIA SMI or the similar for other systems or other processors that would tell you how much GPU memory you're running. So this is kind of a next phase that I do. The first is like, maybe what kind of model do I want? The second is how do I run an inference with this model? Then kind of is a whole branching series of funness, which is either you go down the path of saying, I want to optimize my model in some way to run it either faster or on fewer resources. Or I want to go down the path of saying, nope, this is fine. I can run it with the resources that I figured out it needs. And then you kind of move on to the deployment side of things.

30:07so

30:19okay chris let's say that we want to follow the path on our choose your own adventure that you want to do some model optimization on your model. Okay. The reason you would want to do this is one of two reasons. One is, hey, it turns out I crashed my Google CoLab trying to run Falcon 180 billion because I ran out of GPU memory. And turns out you need more GPU memory for that or multiple GPUs. And I don't either have access to that or I don't want to pay a bunch of money to spin up a GPU cluster and run the model in a distributed way. Or it's maybe even a smaller model and you want to run it either faster or on standard non-accelerated hardware.

31:09Like I heard a talk at GopherCon about a workflow where people were running a model at the edge in a lab to process imagery coming off of a microscope. And it was all disconnected from the public internet. So in that case, you just have a CPU, maybe you need to optimize on the CPU. So there's gradually more and more options that are out there to do this. Some people might have seen things like Llama CPP, which is sort of a implementation of the Llama architecture that's very efficient and allows you to run Llama language models on like your laptop or on like an, I think a lot of people were running them on MacBooks with M1 or M2 processors.

31:55If you want to kind of scroll through this set of optimization stuff, if you go to the Intel Analytics Big DL repo, that's Big DL, like big deep learning. First of all, the Big DL library does a lot of this sort of optimization or helps you run these sorts of models in an optimized way. But they also have this little note at the top which is actually a very i found it to be a very helpful little index as well they say this is built on top of the excellent work of llama cpp gptq ggml llama cpp python bits and bytes q laura etc etc etc these are all things that people have done to run big models in a smaller way i guess would be the right way to put it so bits and bytes is a good example of this Hugging Face has a bunch of blog posts about this where they've run the big Bloom model in a Google Colab notebook by loading it not in the full precision, but in a quantized way.

33:05But there's a lot of different ways to do this. And that's a kind of a good reference to see a bunch of those different ways. At some point for a future show, we should come back and revisit that. That sounds really cool. Yeah, yeah. And I think it probably deserves a show in and of itself. People might refer back to an episode that we had with Neural Magic on the podcast where they talked about the various strategies for optimizing a model to run on commodity hardware like CPUs. But there's a ton of different projects in this space, both from companies and open source projects like OpenVINO and Optimum and Bits and Bytes and all of these.