Qualcomm's Durga Malladi: HBC vs HBM, Dragonfly AI 250, and Winning on TCO

0:00Durga Malladi: Aloha listeners, Austin Lyons here with ChipStrat and with me is Durga Malladi, IEEE fellow, which is pretty cool, and EVP and GM of tech planning for Edge and Data Center. So we're here to talk high bandwidth compute. But first, before we get into that, which is a great topic, I want to know more about like, tell me what your job title means. Tell me what you do at Qualcomm. So I run all the IP and the technology planning for what we need to do across all the products, ranging from data center all the way down to edge solutions, including devices. So that means the IP, if you kind of think about it, is what should we be doing in our CPU portfolio in, let's say, three, four years downstream, in 29, in 30, in data center, versus what do we need to do in devices, in handsets, in new agentic devices.

0:56And then rinse and repeat the same exercise for what do we need on GPU? What about NPU? What about modem? What about sensing hub? And so on. So these are all the different IPs. So that's the planning exercise, which kind of runs the roadmap behind the scenes, but it is across all the business units.

1:11Durga Malladi: Nice. That's pretty cool. I like that you see across all the different technologies so that it's not like CPU running, not together with GPU or not together with different products. Yeah, that's pretty cool. Okay, so let's talk high bandwidth compute, HPC. So the way I see this is obviously Qualcomm is trying to enter and is in a competitive space with inference solutions in the data center. You've got GPUs, there's XPUs, and Qualcomm has the Dragonfly lineup, AI200, 250, 300. it feels to me like HBC is Qualcomm's big technical bet to differentiate and bring something different to market as compared to what everyone else is doing.

1:57Durga Malladi: So, you know, of course, I want your thoughts on that. But also then tell us what is this HBC and how is it different from what others are doing? Absolutely, yeah. So when we reentered the data center business last year, and I actually ran it for one year, like last year. One of the things that we really focused on is, you're absolutely right. There's a lot of players already in the market and they have some pretty incredible solutions already. So it wasn't one of those moments where we said, okay, let's also jump into the fray. And it's not like a me too moment or anything of that sort. But it was like, what is it that is going to be differentiating us from the rest of the ecosystem there?

2:40and what is the really, what is a really good tough problem that we really want to solve because that actually gives us the benefit of, okay, that is different. Let's take a look at Qualcomm. And we latched onto the fact that, and we had an R &D project that was running internally for a while on what is called as near memory computing. And the principles of that are very simple, but before that, what is the problem that we are trying to solve? And it turns out that if you kind of go back in time, maybe over the last six years or so, you will see that the compute capability, like if you just take any of the GPUs and say generation over generation, how much is it improving?

3:20It improves quite a lot, actually. It's like at least a factor of two, sometimes a factor of three. But the memory bandwidth wasn't actually improving at the same league. And so the natural next question to ask is, so what? What does that do? Well, you know, if you kind of go back to some of the basic principles of how do you do compute, well, you have all your data sitting somewhere and you have to fetch the data to do the compute processing. But at some point in time, what happens is that your compute is so good that you've kind of tilted the scales quite a bit. So you might be just idling, you're waiting for the data to actually show up.

3:59So memory bandwidth becomes a bottleneck and it's called the memory wall. And you see that a lot more with AI inference, especially in the decode portion of it. And so what we did is our HBC architecture is based upon what is called as near memory compute. Instead of treating memory and compute as two distinct concepts, what if we actually do a lot of the compute right next to memory, like literally right next to it? It does two things. One, the latency comes down quite a bit. And second, we can reduce the power consumption quite a lot. So today in data centers, HBM, the bandwidth is very high, but it's expensive and it's a power hog.

4:43So we actually wanted to make sure that we can solve it in a different way. And one thing led to another. And as we started talking about this with a lot of the hyperscalers and a lot of other partners, They loved the idea. And as we announced it, we actually announced a couple of names in terms of some of the partners that we have. So it's been a great journey on HPC. But as we briefly mentioned in our investor day in June and today at Snapdragon Summit, in fact, Cristiano left a teaser out there. But let me just tell you, you know, what is the point of this? HPC for us is a technology, but it's not just for data center.

5:27It actually applies to everything, all the way from data center to servers and accelerator cards and automotive and PCs and XR, and yes, in smartphones as well. But it actually works in a different way in some of these things. But the principle remains the same, which is you're doing a lot of computation right next to memory.

5:47Durga Malladi: That's good. That's a great explanation. And I like how you mentioned that HBC is something that can be applied across from the data center all the way to the edge. And that's a good organizational design, as you're talking earlier about how your job is to see IP across all that. So that's pretty cool. Okay, so what I heard from you was the problem is compute is increasing quickly. Memory bandwidth is not increasing fast enough. So all this extra compute can't get fed quickly enough. And maybe instead of trying to just increase the bandwidth of the memory we use today, you guys stopped and said, wait a minute, why have to shuttle all this data?

6:23Durga Malladi: Why shuttle even more like wider buses or even faster speeds? Why not just bring it really close to the compute? That's correct. Exactly. Exactly. And keep in mind that it's a principle that, you know, as you start bringing down the power consumption quite a lot, just going one step further on devices, power consumption matters quite a lot in devices, especially as you think of all the agentic AI-based devices that you see these days. They're still evolving, by the way. There's a lot of new players who are coming to the market. In fact, quite a few of them have very little background in manufacturing devices or making devices.

6:59They come in from an AI-first mentality, and they're coming out with these devices. And the job of agentic AI devices is to be kind of being constantly on because they're working all the time. So that power consumption benefit that we talked about is going to be key, we believe, in unlocking the full potential of agentic AI devices.

7:22Durga Malladi: Yeah, that makes sense. So it's not just the memory bandwidth problem, but the also added benefit of, because you're not shuttling this data so far horizontally, now it's just going vertically, which we'll get into. There's also a big power savings. Yes. Of course, we know that power is the big constraint. Okay, so then, you know, to someone who's listening to this the first time they've heard of high bandwidth compute, are you sticking memory on top of an accelerator? Or tell us more about like the stacking and the separation. What is being sent kind of across the wire horizontally versus vertically?

7:55Yeah. By the way, there's a really cool animation that we showed at Invest today. I wish we had that, but I'll just use my hands as props. Okay. So the way it works is that the memory is like, you know, you have these individual, you know, the DRAM. It's called the stack DRAM. You're basically stacking DRAM one on top of the other. And there's a lot of wafer-on-wafer bonding techniques that are used on that. So this is a hardcore process technology, by the way. In fact, it has nothing to do with compute at this point in time. But then right underneath that stack DRAM is what is called as the logic dial.

8:26That's where you put in all the compute. And depending on the use case, you might actually say, I want some specific logic to actually reside over there. There might be yet another processor that is still sitting, which is away from it. But you've brought in a lot of the heavy computational needs, like the ones which really need that memory bandwidth. You put all of that right next to memory in that base dial. And that HBC, therefore, is a combination of both. The stack DRAM on top plus the logic die that is underneath or the compute die that is underneath. And now you've kind of done a bulk of the work just with that alone.

8:59There might still be some residual compute that you want to do. And that could occur in a separate processor over here. In some sense, HPC kind of is like, it's a bit of like a co-processor because it's doing a lot of the compute. But it's not the only thing. You might still have another main processor that's running out there. But it removes that complexity quite a bit. So in short, it is literally right underneath that stacked DRAM that we're talking about. That's where all the computers are.

9:29Durga Malladi: Okay, okay. That's helpful. So anyone who's familiar with HBM will say, well, this sounds a lot like HBM4E with a custom base die and stacked DRAM on top. So tell us the differences. First of all, is it a different class of DRAM that's being stacked? And then how should we think about the difference between the accelerator at the bottom versus a custom die? Yeah, in fact, this is exactly one of the things. When we first came out with the concept and when we talked about it, I went to Korea and some of the other memory vendors in the U.S. and in other parts of the world. And to be honest, I half expected people to say, no, wait a minute.

10:08No, we're doing exactly the same thing. I don't know what you guys are talking about. We have a pretty robust roadmap coming in. And actually, what I heard is the diametric opposite of that, which is how can we work with you closer on this? Because, yes, this is indeed the right way of actually doing it. Because one thing that happens is there might still be ways of, you know, the basic principles and the concepts look similar. But that wafer-on-wafer bonding that we are talking of, and we still use a lot of LPG-DDR-based interfaces towards the main processor, which is sitting out there, which means we still keep the power consumption extremely low.

10:44There are evolving things that are coming up in HPM as we move forwards. But at this point in time, we see that if you're a memory vendor, even if you are working on HPM and you're looking at customizing HPM a little bit further, going towards 4E and beyond, I think Stack DRAM is here to stay. And you might have noticed a couple of announcements that have been made even by the memory vendors. They have different terminology for that. It is their equivalent of HPC, but actually they are partners of ours. So we expect that these two technologies will co-evolve for a period of time, and we'll see what happens after that.

11:22Durga Malladi: Interesting, interesting. Okay, so then, which by the way, have you guys disclosed anything about the layers, number of layers that you're stacking? We have not yet provided all the numbers. In fact, stay tuned on that front because, you know, you haven't asked the question, how's the silicon coming along and so on? because we've talked a lot about it in terms of PowerPoint presentations and most people are actually taking our numbers for what they're worth. But we will be showing some proof points over the next quarter or two as the silicon is back in the lab and it's coming along very well. So stay tuned.

11:58Durga Malladi: Okay, awesome. I love to hear that. So you'll have silicon coming back soon. Looking forward to that. Looking forward to see the benchmarks. That really turns it real from just PowerPoint. Back to the stacking of the memory, talk to us about the thermals. Does that become a constraint for that base die? I think it is one of those things that, as Qualcomm, when we come in with, you know, we are not, when we talk about IP, for instance, it's not just about what we are doing for our CPU or our NPU logic and so on. We have a rich heritage of both packaging technologies and 3D stacking and a lot of these things that we do.

12:42We work very closely with the foundries in terms of how these techniques need to come together. That has played in our favor as we brought out HPC. and one thing to mention is that in this space, you know, you have to think through carefully as to which product are you designing this for and based upon that, you have to think about the right kind of packaging so that you don't start hitting other constraints. Like it's not the power maybe, but then you start hitting thermal constraints. So we do have a lot of experience in that space. So depending upon which product it is, we have dimensioned the compute in such a way, compute plus the memory, which is dimensioned in such a way that we avoid the thermal problems.

13:30And so that is something that we've paid some careful attention to.

13:33Durga Malladi: Nice, nice. Very promising. Okay, so the Dragonfly AI-250 is the first product that will have HPC in it. That's correct. And that's coming in 2027. That's correct. I saw that compared to the AI200, they both have the same amount of LPDDR in the card. It's like 768 gigs. 768 gigs, yes. And yet the 250, which is the first to have the HPC, has 18x the effective bandwidth, I believe is what the website said. So again, just for everyone, can you remind us again why that HPC in the stacking is 18x the bandwidth as compared to in the 200, where presumably the LPDDR is a little bit further away? So first of all, the raw memory is completely different concept compared to the actual memory bandwidth in itself.

14:19So this is LPDDR, which means we can actually pack. It's a very dense card that we have. So it's 768 gigabytes. You know, a lot of the other things that we see in the industry, they have like 288 gigabytes per card. This is like 768 gigabytes. which by the way if you kind of think through that 768 gigabytes per card means if you take a typical order of magnitude trillion parameter model and it's all like a mixed precision fp4 you can fit a trillion parameter model in just one card that is like seriously mind-boggling we have shown that we have shown some proof points associated with that but that's just the memory capacity.

15:00The memory bandwidth is where as we bring in HPC, then you have this combination of the underlying logic plus DRAM stack. So this actually is a slightly different concept. This is not about the capacity anymore. Using the same capacity, you've just boosted the memory bandwidth significantly. That's where there are two independent concepts. But another way of looking at it is if you take an AI200 card and it has a certain performance, certain exaflops per rack or exaflops in a given rack. And of course, we have disclosed that, you know, our racks are like 160 kilowatts. When you go from AI-200 to AI-250, the amount of memory inside, it's like 41 or 43 terabytes or so, that doesn't change.

15:46But because of the fact that there's a huge memory bandwidth increase, the number of exaflops in the rack actually goes up substantially because of that. because the decode performance improves quite a lot. So you see, the total number of, like the amount of capacity that you have in the rack is the same. But now you've dismantled that memory wall with HPC, and that's the reason why our overall performance goes up significantly with AI-250. Okay, so let me reflect it back to you,

16:17Durga Malladi: and there's a part where I'm a little fuzzy on. So we've got more bandwidth in the AI-250, which of course would mean better decode performance. So just like more tokens per second. That's correct. How does that, that unlocks more flops because the flops are not underutilized. They're not like sitting waiting. Is that what it is? That's right. In fact, because of the fact that you're running it that way, your TCO also becomes better in the sense that if you just take in, because the tokens per second per watt goes up significantly because your tokens per second has gone up. It's the same power consumption, by the way, same 160 kilowatts.

16:52But then in addition to that, then when you take a look at the total, it's called the tokenomics, where you also normalize it by what's the cost associated with that, it comes out to be even more favorable compared to some of the other things that you see in the market today. So in that sense, every time we have talked about the benefits of AI 250, we have talked about not just tokens per second, but tokens per second, tokens per second per watt, and tokens per second per watt per dollar, or sometimes even tokens per second per dollar. When you put the entire arithmetic together, then you see why it becomes actually far more favorable to go with this.

17:28Durga Malladi: Nice. Okay, so I was going to ask you then. So when you have talked to hyperscaler partners, ultimately, is there one of those that's most compelling? Is it about the TCO or about the tokens per watt or kind of all of it? I think increasingly these days, it's about the TCO. increasingly. It used to be about raw tokens per second and tokens per second per watt. But I would argue that TCO already kind of captures that. Because if you kind of, you know, think about it, what are all the things that you need to worry about? Well, okay, if I buy one of these racks, first of all, how much am I paying for it?

18:02So that's like one variable. Second, what's the power consumption? And then you have to figure out for that power consumption, what's my utility rate? And there is some number associated with that. The third one, of course, is how many tokens per second does it really generate? Like, how effective is it? And you put all of these three variables together, and I'm discounting things like water that you need for cooling and so on. It's the TCO that is something that is emerging as the most important, but it does take into account like these three big variables.

18:36Durga Malladi: Yeah, yeah. Yeah. No, I think that makes a lot of sense. And just like, the simplest way for me to think about it is like, okay, if I have a billion dollars to spend and I have like a gig, not even a gig, a hundred megawatts, if I can buy X racks from some provider that fit within the power envelope and my budget, or I can buy two X racks from someone else. Well, even if the throughput's a little bit different, you know, generally, if you have two X the racks, you're going to get more, more throughput. So that makes a lot of sense to me. So I guess, let's see maybe last question when so there are other ai asic startups who also because they're coming in and needing to do something different they're also making different technical bets and usually like the skepticism that people get with them is they say okay that's great um can you actually manufacture it can you actually get allocation now obviously qualcomm has is one of tsmc's largest customers and you have way for allocation but i also know that you have many different business units from data center to edge who all want the same wafers.

19:38Durga Malladi: So like, talk us through, like, how does that work? And I assume that, you know, you do have the advantage there because you are manufacturing things in production at scale. So, you know, that's one of the places where it is true that it's not enough to just have the, just the technology in itself. That's why we've been placing careful bets in terms of making sure that, you know, our wafer allocations, we have like an excellent set of relationships with all the foundries and also with the memory makers. And so as they have come up as extremely valuable partners and across the entire portfolio, not just in one domain, those relationships do matter.

20:15When we go in front of a customer and say, yeah, we are in a position to supply you. How much do you want? You want, okay, these many megawatts or maybe a gigawatt. This is what you want. Yeah, we have the supply chain lined up for that because we do have those relationships across all the different businesses. And that actually helps us quite a lot.

20:36Durga Malladi: Yeah, sure. Totally. That makes a lot of sense. All right. Well, that's it for today. All right. Durga, this was great. We will have to talk again once you guys have more to share about the actual measured performance of HBC. All right. Sounds good. Thank you. Thanks. Thanks again.