939: Mixture-of-Experts and State-Space Models on Edge Devices, with Tyler Cox and Shirish Gupta

29:58What I really like about these models, and we're talking specifically about IBM's Granite 4.0 family of models, is that they're covered under, you know, the standard Apache 2.0 license. They are a hybrid architecture model combining state space models with transformers. And that really changes the game in terms of long context performance and memory efficiency. And these are enterprise grade. And why do I say that? Because yes, one, they're open source under the Apache 2.0 standard license. They're trusted. They can be trusted. Open weights, open training data for full customizability and deployment flexibility.

30:42And to top it off, they're performant, right? They're near the top of the leaderboard of the standard Helm IFE valve for open weights. And actually, bookended by 100 billion parameter plus models on both sides, right? So these models really punch well above their weight. And in my opinion, they're ideal starting point for fine-tuning and customizability for enterprise deployments. Tyler, why don't you share some more about the models themselves?

31:16Jon Krohn:Yeah, let's talk about, so that was a really helpful overview, and it's great to hear that it's doing so well on the Helm. I'll have a link to Helm in the show notes for people who want to understand more about that. And so this is specifically at this time, this is the fourth version of this model family. So it's the Granite 4.0 model family. And I understand. Fourth major version. Fourth major version. Right. Yes. Thank you for getting, yeah. It's good to be a software people on this call to get me right on my model versionings. And so within the Granite 4 model family, there is coincidentally four different variants.

31:53Jon Krohn:And so maybe you can fill us in on the details of those four variants and why you might choose one of these models for a particular use case, Tyler. Yeah, yeah. And there's a reason I interjected there. And it was that what IBM's been doing with Granite is really exciting for enterprises. They're delivering some pretty consistent results. And I'll start with the bottom of the family and highlight that. Right. So Granite, the three series had a 3.0, a 3.1, a 3.2, a 3.3 version, all improving on the same model. The first model in the Granite 4 family extends and improves on that again. Right. So that's the same familiar pure transformers model architecture with new knowledge and improved performance.

32:45So it's a great 3 billion parameter dense transformer model. Then from there, they get even more interesting. So the Granite 4 model family has a, the other three all have a H tag. So they are Granite 4 H models. That H there stands for hybrid. And specifically, it's a hybrid state space model. So what the IBM team has done is they've blended state space models, which I think we'll talk about here in a minute. So I'm going to leave that definition hanging and transformers to get great long context scaling specifically. Right. One of the big things with attention mechanisms built into transformer models, one of the big limitations is memory scaling.

33:34There is a quadratic scaling law that means if you double the input context, you are quadrupling the memory required to run. For example, your KV cache in most scenarios, what a state space model, one of the important properties it has is you get linear scaling. So with the Granite 4H family of models, you have sub-quadratic scaling there. So the first of the Granite 4Hs is a Granite 4H micro that accompanies the Granite 4 Micro that I just talked about. So that is also a 3 billion parameter dense model. But now we have hybrid state space. Then the last two models in the family, Granite 4H Tiny and Granite 4H Small, are both hybrid state space models like the Micro, but now they are also a mixture of experts.

34:27So these are sparse models. They have a total parameter count of 7 billion for the tiny model and 32 billion parameters for the small model. They have activated parameter counts that are lower than that. So it's a 1 billion. So on tiny, you've got 1 billion active parameters. On small, you've got 9 billion active parameters. What that means, right, for those of you who a sparse or mixture of experts designation is a new concept, that means that they have a knowledge and accuracy profile closer to their total parameter count, but they have a throughput performance profile that is closer to their activated parameter count.

35:08So what they're doing is they're picking portions of the model dynamically for each request and really at a token level to answer and kind of retain only the most important pieces of the model in answering a given request. So really, really interesting blend of technologies in that Granite 4 family.

35:34Jon Krohn:Yeah, the mixture of experts approach is certainly something that all listeners should be familiar with. If you've been listening to this show for years, you've already heard about it a number of times, but if you haven't, you definitely need to know it in our space. It's important because it provides exactly that kind of power that Tyler was talking about there, where you get the power, the capabilities of very large models, or yeah, like as you said, all of the available parameters in the model that you choose, but you get the speed and cost performance of a much smaller model because you're not using all of the parameters in the model on any given call.

36:12Jon Krohn:You're using a subset like, you know, it might be common to use like an eighth of the parameters in a given mixture of experts model when you're actually running at inference time. So yeah, really cool approach. And so my understanding is that, you know, you mentioned earlier on in this episode about the Dell Enterprise Hub on Hugging Face. And so all of these granite models are available easily through that infrastructure, right? And so, you know, you can give me an answer to that while also telling me how Dell optimized these granite models to run on AI PCs and workstations. Yeah, so Dell Enterprise Hub is our partnership with Hugging Face.

36:55If you go to dell.huggingface.co, you will see our model catalog there. That's what we're talking about. What we've done in Dell Enterprise Hub is we've, number one, all of the Dell Technologies portfolio devices from AI servers through workstations through PCs, they all show up there, right? So if you say I have a Dell Pro 14 premium, what can I run? Dell Enterprise Hub is a great place to go. So inside of there, yeah, we have the Granite 4 family of models. We had day zero support from AI servers to AI PCs on there. So great place to go to get started. For optimization, so like I mentioned before, we go in and we run all the models on our devices.

37:51We figure out what the best set of inference time parameters are to fit different footprints of different devices, whether that's context length or some of the other things. And we'll bake that in, right? So we're presenting an API interface to applications. we're picking some of the best parameters for different devices so that you don't have to worry about whether you're on a 16 gigabyte AIPC or you've got access to something with an RTX Pro 6000 Blackwell and 96 gigs of VRAM. We'll make sure that you're getting best performance out of that model.

38:30Jon Krohn:Nice. That sounds like a useful feature. Now, let's dig more into what you're talking about there. We dug a little bit now into a mixture of experts. Now I want to dig into state space models and hybrid architectures, which you touched on, Tyler. So, you know, the Granite 4.0 release introduced a hybrid architecture that combines state space models with transformers. And that sounds important too, but maybe, maybe I've gone too far. Maybe we should start with digging a bit more into what state space models are first. Yeah. So state space models are a really important family of models, even pre-AI usage.

39:15So if you go back to the 1960s, these are being used in spaceflight control. They've been used in population studies and economic modeling for a long time. The core construct is that you have an input signal that you map onto a hidden or latent state space with one set of equations. And then you have a second set of equations that translate that state into an output that's observable. So there's lots of different ways that you can construct a state space model to represent different problems. um what we what uh what's what's happened over the last um 10 years or so is that state space models uh were investigated is part of the deep learning um kind of revolution uh is is how do you construct uh your your matrices for state space models to better model um different tasks without as much um kind of classical uh feature engineering approach to it right um so that that was one important thing.

40:23But then there's a couple of researchers at Carnegie Mellon and Princeton who really kind of drove this home over the last five years or so, right? So there's a great set of papers. I invite everybody listening who wants to learn more to look up the work here on Mamba and structured state space models and things like that. There's a series of papers from 2021 all the way up to 2025, still working on it, that introduced some really great optimizations inside of state-based models to make them appropriate for sequence transformation and language modeling tasks. So you get this evolution of structured state-based models, your S4 paper, and then you go into structured state-based models with selection and computation by scanning your S6, which turns into, hey, that's a lot of S's, sounds like a snake.

41:27Now we've got Mamba. And the Mamba architecture really optimizes states-based models for the kind of compute profile that's needed to be relevant for for the language model tasks that we're applying here. And so some great optimization work that happened there, some great mathematical insights into the matrix properties of those. And I won't be able to do those full justice here, but really, really interesting work. That kind of accumulated into the Mamba and Mamba 2 language Bush model blocks that IBM pulled in to the Granite 4H series models. Right. So composition here, you've got a nine to one ratio of Mamba layers to attention layers in the Granite 4H family.

42:29So quite a bit of state space model in that hybrid. Those are, like I said earlier, those are linear context scaling. They are the Granite 4H models are no positional embedding. So they have in the data set out to 512K context represented in the samples. They're validated out to 128K. The IBM team says, theoretically, you should be able to push it past that, right? So some really great long context performance. I think one of the key measurement points in the release notes are if you take eight sessions at 128K context on a micro, so a 3 billion parameter model, you get about 15 gigabyte of memory usage versus about 80 on a pure transformer architecture, right?

43:27So some really great context reduction, which means that on more constrained devices, on the edge, you can make use of more useful context in RAG workflows, in multi-term workflows, and things of that nature. Just to close here, Shereesh had mentioned IF eval. That's a really great benchmark for instruction following. right, structured output tasks. The other thing that Granite scores really well on is the Berkeley for function calling leaderboard, specifically BFCLV3. It shows up in the top five as we sit here recording, among a bunch of other frontier models and hundreds of billion parameter models even at, for the small, 32 billion parameter footprints.

44:26So really, really punching above weight class there.

44:29Jon Krohn:Really cool. Thanks for all those stats in there. You really packed them in. And so it's kind of like a key takeaway, basically by having these post-transformer architectures as part of these granite models. And so it sounds like specifically they've gone with Mamba as their particular state space model. It allows us to have better performance over long context windows, like you're talking about half a million input tokens, being able to handle that. And when we handle that kind of large amount of context, that large number of input tokens, we start to become worried about memory and about compute performance.

45:09Jon Krohn:And so over these very long context windows with these state-based architectures like Mamba, we're able to have scaling that isn't the quadratic scaling that we were used to with transformers. And what I mean by quadratic really quickly for listeners is with the transformer architecture, which is still the predominant model architecture within LLMs today, as you increase the number of input tokens, the compute and the memory increases quadratically. So by a squared factor, basically. And that starts to add up really quickly. And so, yeah, so we're getting sub-quadratic memory and compute performance, which matters a lot in these very large input token situations.

46:06Jon Krohn:So really cool. So you get a lot of power, a lot of flexibility with these Granite 4.0 models because they incorporate the state-based models like Mamba into it. And so I guess my final question for you on this, and it was actually kind of where I opened around this, was I was talking about how the Granite 4.0 models are actually a hybrid that include, I think you mentioned there, a nine-to-one ratio of Mamba versus traditional transformer attention heads. Why, what's the advantage of having both? Yeah, it's a great question. So one of the things we do see with Mamba is it has great global context performance, but it loses a little bit of the sensitivity of local context that attention players are really known for.

46:52Right. Now, one of the other interesting things that happened right after IBM launched the Granite 4 models is there was a great meta fair research paper that compared some of the performance of a pure transformers architecture, a pure Mamba architecture, an intralayer hybrid and interlayer hybrid. Right. And so the intralayer versus interlayer, that's looking at running through attention layers and Mamba layers in parallel versus interlayer, which is what the Granite 4H models are, which is running through them sequentially. Right. So I think one of the recommendations from that team was build more more of these hybrid models that the the training efficiency, which we didn't talk about, and the inference scaling efficiency is really great without a lot of tradeoffs on accuracy, given some of the the Mamba improvements.

47:54So I would expect to see more of these type of hybrid models in the future.

48:01Jon Krohn:No doubt. No doubt. It is the future indeed. Some, yeah, a post-transformer architecture will be the dominant architecture in the future. All right. So changing topics here a bit from all this really cool technical stuff to kind of the real world implications of this. If you are an enterprise and you're trying to take advantage of AI, there was a recent study, I talked about this on an episode in the past, probably several times, but I did a whole episode, episode 924, on this MIT researcher claim that 95 % of enterprise AI projects fail, where fail means that it does not deliver a return on investment in production.

48:44Jon Krohn:So either it never makes it to production or it's never profitable in production. It's never successful. It never meets your success criteria in production. And so can you speak to or provide examples of what typically goes wrong in enterprise AI projects and how our listeners perhaps leveraging things that we've already talked about in this episode can prevent those kinds of failures? That's a great question, John. And honestly, you have done a stellar job covering it in that podcast episode that you referred to. But, you know, my take on it, I think some of the most important things that everyone that's on their AI journey will do well to sort of ground themselves to is it starts in a very cliched manner.

49:35It does start with setting clear goals for what outcomes are desired, right? And that includes very specifically what is the desired type of response and accuracy of said responses. And I think we're still very reliant on humans for ensuring that outcome. In other words, you just can't proceed unless you have humans in the loop and expect to have, at least during the training phase, and expect to have good results. In fact, I think having humans in the loop even during production as feedback baked in to the solution itself is vitally important because you want to account for drift. Right. But another cliche I will share is don't apply Gen AI for the sake of it.

50:35You know, use the right tool for the job. I think I've seen plenty of misuse where everyone just wants to apply Gen AI to a problem that is very well solved with traditional. I call it traditional. It's funny, but machine learning. Right. Solid, solid machine learning deterministic outcomes. So those are the basics, right? I think beyond that, you just want to really be very deliberate about the problem you're trying to solve, right? So again, problem, goal, tool for the job. Sounds pretty basic. You could actually take that response and apply it to literally any, you know, even a classroom discussion on how to do a project or how to manage projects.

51:22So I think those are the basics that you need to ensure. And then as you go from there, there are obviously going to be very use case specific, task specific best practices, which need to be uncovered and adhered to as you go along the journey.

51:43You know, one other thing I will mention here is that it goes back to some of the pain points that we anchored on earlier on in the conversation about the problems that we were solving for with the Dell AI factory and the entity formerly known as Dell Pro Ahas Studio. And, you know, I'll defer to Tyler there to sort of talk about what are some of the best practices that, you know, customers can take advantage of with the solutions that we're providing. Yeah, so I think taking advantage of the device through our solutions means that you've always got a enterprise-ready model at your disposal, right?

52:31There's 2 million models on Hugging Face. They all do different things really well. We're trying to bring the ones that we think are really great for a lot of key enterprise use cases ready to go. We also make sure that they're performant on our devices of all types. So when we pick models, we try to make sure that they've got great coverage across different silicon types. So they're really good building blocks. A lot of times you'll start down a pathway with one model and you might hit a dead end when you try to scale up or scale across different device categories. It can be a little bit challenging to figure out what the right pathway is to getting everything hooked up the right way.

53:25Jon Krohn:All right, that all sounds great, Tyler. Sharish, you mentioned to me before we started recording, and you mentioned actually a little bit in today's episode as well, how Dell PCs perform on benchmarks, how Dell AI PCs perform compared to previous generations, particularly around compute, battery life, efficiency. Do you want to fill us in a bit on those key benchmarks? Yeah, absolutely. I think it's important for customers to understand what they're actually getting when they invest in the newer hardware, right? So, for example, the Dell Pro Plus with Intel Core Ultra 200 V-series chips, there's just tremendous benefits, advantages over, say, an N-2 non-AIPC chipset like the Core Ultra 14th gen, right?

54:27So I'll just give you some examples. 88 % more battery runtime running Microsoft Teams meetings compared to the non-AIPC on the, you know, the Dell Pro Plus with Intel Lunar Lake. 4.8X higher graphics performance. So very important to think about the GPU as well, the iGPU, which is tremendously, you know, is shown tremendous gains gen over gen. And even within the gen of Intel's latest chipsets, Lunar Lake, with its system-on-chip architecture, is just tremendous in terms of its gains for the iGPU, not to be ignored. We've talked so much about NPUs, but I think if you look at the overall CPU, GPU, NPU as a whole, it's just a tremendous value proposition.

55:24Almost 10x on-device AI CPU performance and 4.2x AI GPU performance on that same device relative to the non-AI PC. And then if you really want to talk about specific models, because that's what your users are going to be most interested in, the same device with Lunar Lake provides greater than six times higher performance on Mistral 7B and Llama 3.1 for TextGen and up to 5.6 times higher performance for ImageGen with Stable Diffusion 1.5. Just showing you the breadth of the AI performance and the battery and, you know, runtime and power consumption improvements for these devices relative to some of the non-AI PCs.

56:17So I really encourage people to look at these new chipset architectures and the new devices from a holistic lens and not just focus on the NPU itself.

56:30Jon Krohn:Yeah, so it sounds like we've gotten to a point where companies like Intel, with the way that they make their CPUs, as well as obviously GPU providers, neural processing unit providers, and then even companies like Dell, the way that they package all of that up with something like an AI PC, you are optimizing everything for AI workloads, for training AI models, for real-time inference on the edge, on AI PCs. And so that's how you're getting the big multiples that you just went through in dozens of examples of, Sharish. Yep. Nice. All right. So final kind of technical question for you. Historically, AI transformation was a multi-year, huge undertaking for enterprises.

57:16Jon Krohn:But that kind of timeline is irrelevant these days because, you know, if you think about a multi-year timeline, we have no idea what the capabilities are going to be like a few years from now. We can't be thinking about a solution today that's going to be ready in multiple years. How does the Dell AI factory accelerate the timeline so that we're talking about weeks or months instead of years getting to deployed AI models? Yeah, that's a great question. I think part of it is you've got to simplify how you think about it, right? If you can't get, if you're down in the details on everything, then you're slowing down.

57:55So picking the right technology partners matters. It will help you go faster. The second thing is you need to build on technologies that will scale and help you, not just at deployment time, but six months, a year, two years down the road. We all know AI is not sitting still. The model you deploy today is not going to be the same one that you deploy in six months or a year. Having manageability built in means out and control that, right? If you want to go do an update to your application, give your user five or 10 or five, 10 % performance or two X performance, right? We don't know what tomorrow brings.

58:40Then you need to build on top of the right tool sets.

58:44Jon Krohn:Awesome. Really enjoyed this interview with both of you today, Sharish and Tyler. Sharish, we've had enough book recommendations from you already. Tyler, what have you got for us? Yeah, so I read a bunch of different things. I use it as kind of escape, right? Whether that's fantasy, science fiction. One of the things I think is something that, a story that a lot of people will love is one that is being adapted, right? So Project Hail Mary from Andy Weir is my recommendation. It's a great, great fun story. Touches on a lot of interesting environments and excited to see the adaptation. I think next year is when that will come out to film as well.

59:37So make sure you read it before the theater.

59:40Jon Krohn:Yeah, yeah, exactly. All right, thanks for that recommendation, Tyler. And we might as well just stick with you for a second. How should people follow you after this episode So to get more insights on, I mean, yeah, you went into huge technical detail on models, on capabilities. Where can they get more insights from you after the show? Yeah, I'm on LinkedIn. We'll put the profile in there. I will say I'm not a heavy poster, so don't expect to see a ton from me. But you can be sure that you'll get the latest updates on the Dell AI Factory from my feed. Excellent. And Sharish? Same. I'm also on LinkedIn.

1:00:19My link will be in the show notes. And that's, you know, I'm maybe a little more active compared to Tyler, but still not one of your prolific, you know, everyday prolific posters. So, but yeah, LinkedIn is a great place to follow and I appreciate the connect.

1:00:39Jon Krohn:Fantastic. All right. Well, thanks for that. And thanks for this whole episode again, Sharish and Tyler. and yeah, I feel like it's probably not going to be long. We don't have it planned, but I wouldn't be surprised if we're welcoming one or both of you on the show again in the future for more. These are the only kinds of episodes where we dig in detail on how people can be getting inference or model training on the edge. And so I always learn a lot in them. I really particularly enjoyed all the conversation today around state-based models and mixture of experts and the granite release, all really cool stuff.

1:01:19Jon Krohn:Thanks, guys. Thank you, John. Thanks for having us again, John. Yeah, it's always a pleasure to be here. And I was delighted to have Tyler join me for this episode because his technical depth is just awesome. It's awe-inspiring. I feel humbled every day when I enter the innovation lab where he sits and just can't get rid of my imposter syndrome. Like, you know, I don't deserve to be here. This is the place where genius, you know, way beyond my years thrives. So I just enjoy working with him and his team every day. interesting episode for sure in it Tyler Cox and Sharish Gupta covered state space models like Mamba and how they use selective mechanisms to process information more efficiently than transformers making them ideal for edge devices where memory bandwidth is the bottleneck they also talked about mixture of experts architectures that activate only specific subsets of parameters for each task allowing large models to run on resource constrained devices by keeping most parameters dormant.

1:02:25Jon Krohn:And they talked about how the Dell AI factory simplifies deploying and managing custom AI workloads across enterprise PC fleets by abstracting away silicon diversity, simplifying deployment, and providing enterprise-grade manageability, accelerating AI transformation from multi-year timelines down to weeks or months. As always, you can get all the show notes, including the transcript for this episode, the video recording, any materials mentioned on the show, the URLs for Tyler and Sharish's social media profiles, as well as my own at superdatascience.com slash 939. All right, that's it. Thanks to everyone on the Super Data Science podcast team, our podcast manager, Sonja Brejevic, media editor, Mario Pombo, partnerships manager, Natalie Zajski, researcher, Serge Massis, writer, Dr.

1:03:09Jon Krohn:Zara Karche, and our founder, Kirill Aramanko. Thanks to all of them for producing another excellent episode for us today for enabling that super team to create this free podcast for you. We're deeply grateful to our sponsors. If you're ever interested in sponsoring the show yourself, you can find out how to do that at johnkrone.com slash podcast. Otherwise, you can support us by sharing the show with people who would like to listen to it or watch it, review the show on your favorite podcasting app or on YouTube. Subscribe if you're not already a subscriber, but most importantly, just keep on tuning in.

1:03:41Jon Krohn:I'm so grateful to have you listening and I hope I can continue to make episodes you love for years and years to come. Until next time, keep on rocking it out there. And I'm looking forward to enjoying another round of the Super Data Science Podcast with you very soon.