3D Stacked DRAM for Inference: d-Matrix's Sudeep Bhoja

0:00Every token an AI model generates comes with a cost, moving data from memory to the circuits doing the work. That movement takes time and burns power. When memory can't keep up, adding more compute gets you only so far. Dmatrix has worked on inference since 2019, before the generative AI boom. It started with analog in-memory compute and then moved digital, always focused on getting data from memory to compute efficiently. Today, co-founder and chief technology officer of D-Matrix, Sudeep Boja, joins us to trace that journey through Raptor's 3D memory architecture, how to connect multiple racks together, and what the future of inference will look like with their next generation lightning architecture.

0:44Let's get into it.

0:50Welcome to the Semi-Dope podcast, where we have conversations with some of the best minds in the semiconductor industry. I'm Vik Shekhar and I've worked in the semiconductor industry for like 20 years. And today I'm an independent researcher and the founder of SemiExponent. I also write a sub-stack newsletter called Vik's Newsletter. Welcome to the show, Sudeep. How are you? Good, good. It's a pleasure to be on your show, Vik. I've always been a fan of all your work and it's great to be on your podcast. Thank you for coming on. We have so much to chat about today because I am quite familiar with Dmatrix.

1:25I've covered it on the Substack before and we've spoken at conferences. There's so much exciting technology to get into, so we should just get right into it. Let's start right at the beginning because the story here is very interesting. And that's because Dmatrix actually was founded in 2019 and that was well before the AI thing even took off. So what was interesting is like you and the CEO, Sid, were both at InFi before starting Dmatrix. And it's really interesting because right after that, actually Marvel acquired InFi. So that was really cool to see how you guys decided that you're going to go and do inference architecture that early in the boom when something as exciting as Marvel's acquisition of Infi was in the works.

2:21So how did you foresee all of this happening? Of course, we did not know the acquisition was in the works at the time. but we are working at Marvell in the data center business unit and the interconnect products we're building were being used by the hyperscalers, Google, Microsoft, Facebook at that time, Amazon. And we were seeing that the AI training clusters were pulling our interconnect solutions And they were, in fact, driving speeds and feeds, which was a change from what it used to be before that. And then we realized that they were training these models, these amazing models. ImageNet had come out in 2012, and there was breakthroughs after breakthroughs already.

3:21and late 2017, the Transformer paper had been released. And we decided to focus on inference because we realized that the problems of inference were very different from what everyone else was working on at that time, which was training models. Training models is about high batches, compute, right? because it's a compute-bound problem. You want to pack as much compute as possible. Whereas inference is a memory-bound problem. Latency is very critical. And so that's what we focused on. That is a unique decision to make at that time because if you actually see all the other, I'd say, pre-ChatGPT companies, whether it's Grok or Cerebrus, all of them went for the training problem first because all of them wanted to go and train these models.

4:20And I don't think at that time, the industry realized how big these models are going to get and how many chips are going to be required. So a lot of them did start with the SRAM-based approach, which actually D-Matrix also has in their technology and we'll get to that. But clearly you went straight for inference. So that is a unique problem that you identified early on for sure. Yes, yes. So specifically, we recognize that once you train these models, you have to run inference on them. And the characteristics of inference was very different from training. To do an inference, you have to read all of the weights for every token and read all of the users' KVCache.

5:09So if you want to keep latency small, then you can't batch too many users. And that means that the memory subsystem is very pressured. There is not sufficient memory bandwidth to feed any sort of compute. And having worked in the semiconductor industry for many years, We recognized that the compute itself was very cheap. It was in the femtojoules per operation. The data to feed the compute that had to come from HBM or other memory subsystems was in the picajoules. So there was this three orders of magnitude gap, which is the technical problem that we wanted to focus on and solve. And that turned out to be a very good match for inference.

6:01So that power delta that you identified told you that this is the problem you need to solve. So who was the first person you convinced that this problem is worth solving? Who decided to fund the effort first? Yeah, so the initial check was written by Infi. So the company we worked at before we started Dmatrix. and so they were the first investor in D-Matrix in the seed round. And then on Series A, we had Microsoft M12 and Playground Global as our lead investors. Yeah, it's fascinating. It's fascinating you could get infeys like your first investors. It's amazing. So now they're like an entirely optics play anyway at that time and that's what their Marvel acquisition is all about.

6:57So it's nice that like D-Matrix got started out of that. And it's a great story. But your first chip like looked nothing like what your technology looks today, right? So what was the first kind of chip that you went after as soon as D-Matrix was founded? Yeah, so we built, we wanted to build a proof of concept silicon before we scaled the company in our seed round. So we essentially went after trying to co-locate compute and memory together. After many years of transistor scaling, we knew that the compute is very small, but the wires had not scaled at the same rate. So it was data movement that was a problem.

7:43So our initial idea was, can we take an SRAM array and add compute into the SRAM array itself? So we took, unlike a traditional SRAM, where you only activate one word line at a time, we ended up designing an array where we activated all of the word lines. And so that was a pretty good operation for a matrix calculation, matrix vector-specific calculation. And then we activated, instead of the bit line, we used an analog scheme there using capacitors to do the accumulation of the bit line. Okay, awesome. So just for some background, SRAM generally is arranged, let's say, in a grid. And you can activate it with word lines and bit lines, right?

8:37So you can store something in an SRAM cell by applying the right voltages at these levels and reading out with the right sense amplifiers on either side, and you can get what bit was stored or what was not stored. That's typically how SRAM is used to store information. It's been around a very long time. Typically, there are like six transistors that build up this unit cell, which you access with bit lines and word lines. That's the basic SRAM approach. But you were not using SRAM as a storage unit as much as you were trying to compute with it, right? So you are trying to do analog compute using the six transistor SRAM cell.

9:22Yes. So the weights were being stored on the SRAM cell. And then we added a few transistors to convert that into a multiplying bit cell. So if you insert on the word line, if you were to send activations one bit at a time, then you could essentially multiply right at the place where the data is stored. So there's no movement of data. And then instead of the bit line, we ended up putting capacitors on top of the SRAM array and shorting the bit line. And the capacitors would essentially perform the additions. and then an analog to digital converters would convert that value to digital and that's how we would complete the matrix multiplication.

10:12Awesome. So at the heart of all of this is the multiply and add function, the MAC operation that you're doing, but you're doing it now in the initial version of the D matrix, the first version, it was all analog and it was done by holding the weights in the SRAM cell and using an extra transistor to do the multiplication and then a capacitor to do the charge addition of all of this. And then the result, you read it out and then you have your MAC operation, which means now you can multiply vectors together. That's correct. That's exactly correct. So in the original versions, we have digitized a lot of it.

10:50Obviously, the storage is binary, right? So it's a bit serial, bit parallel architecture. We had removed the DAC function that traditional in-memory compute circuits had by sending one bit at a time. And only the accumulation was analog. So that was the first test chip that we built, the proof-of-concept chip. But that proof-of-concept chip was a full quad, if you will, of the Corsair product itself. It had a RIS-5 processor. It had all of the architecture built around these compute engines. It had a software stack. It had a runtime. So we were able to prove out the whole stack using these basic fundamental building block elements that did matrix, vector multiply, SIMD operations, reshapes, data movement, and so on.

11:52Yeah. What was the first version of this chip called? What did you call it? It was called Nighthawk. So that was the chip that we taped out in our seed stage of the company with very few people. And then we realized, along the way, we realized that it was very difficult to maintain precision of the bit line. So customers are very used to seeing 32 bits of precision. And so we had to maintain that level of fidelity. and occasionally we would see a bit error that we worked very hard to correct, but it was very hard to maintain the little bits of precision across parts, right? Across when you scale this product.

12:37And that's the reason that we went down later on, on Corsair, we went down the digital in-memory compute approach. What we did is we just took that analog bit line and digitized that bit line. by using an adult tree. And we had worked on low-voltage circuits, and we looked at the energy consumption of that versus analog, and then we decided that it was far easier to productize the digital in-memory compute. So that's nice because you have actually, the first version is actually, you do have an analog in-memory compute chip that was like complete with its RISC-V processor. It was a whole chip, the Nighthawk chip, Technically, if the whole 32-bit precision weren't an issue, you would have become an analog in-memory compute company to begin with.

13:28But then now you had to solve the problem because you just can't get that level of accuracy because of these elements constantly varying. Is that it? Yes, exactly. So that level of fidelity is very hard to guarantee with analog techniques. and it's not that the neural nets really need that level of fidelity, but it was very hard to convince ourselves and convince the customers that it's okay to drop a few bits once in a while. And so it was just an engineering trade-off. You could do it using analog, and people still are continuing to work on that problem. But if you want to maintain fidelity, and if you can do it with digital techniques, it was far easier to productize that.

14:24Okay, that's a perfect segue to go into the next product you have because clearly when customers wanted that level of precision, you can't do it with analog compute. Now we can argue whether that is really required, whether analog in-memory compute is still viable. There are people working on it, but that's not our focus because we've got a lot of interesting stuff to cover. So clearly you then said, okay, that's good. Analog worked. It's not like it didn't work. I think there's a very important distinction to make. It worked, but not well enough for the customers at that time. I think everything evolves eventually, right?

15:03But today, like the fundamental method of computing you have is digital, right? So now you have digital in-memory compute. So can you walk us through what the transition is now so that you have it all digital, how that looks? Yeah, yeah. So it's still the matrix multiply units are exactly the same, which is you have the SRAM array, and as part of Corsair, we have all of the word lines that we activate, and we perform this bit serial, bit parallel, you know, multiply and add. circuitry in the tensor engine piece of it. And instead of an analog accumulation, we put an adder tree and that's how we are able to perform matrix multiply in digital and we get to very, very good energy efficiency by doing that.

16:08I see. Okay, that's good. So now your entire compute is still digital in-memory compute, but now you use entirely digital techniques, so you get the kind of accuracy that people were looking for. Correct. So we got to 32 bits of accuracy, so we maintain FP32 precision on the accumulation data path and so it's just like any other digital engine from that perspective. It's just that the computation happen right at the bit cell. So there's no moving of data inside the tensor engine. And for that reason, those tensor engines are very efficient. They get to 25 to 40 tops per watt. And depending on the voltage and the frequency we choose, some of this we presented at Hot Chips last year.

17:06Nice, nice. I had one question though, because if you have FP32, and that was probably the initial implementation of what inference, people thought inference should look like, but today we have other formats. We can do FP16, FP8, FP4. Do we still need that level of accuracy when it comes to these operations? Or could you get away with something far less accurate? Oh, yeah. Even on Corsair, only the operands, we supported block floating point format. So we supported something called MXint8 and MXint4. So those are like 8-bit and 4-bit formats that shared an exponent across a whole vector. Now it's part of an OCP standard.

18:00So the precision, we support 8-bits and 4-bits. but only the accumulation. So once you, you know, the activations could be in 8 bits, the weights are in 8 bits but when you add a lot of them up, the accumulation is now in 32 bits. That was what we used. So we certainly don't need 32 bits for weights and activations. I see. Okay, that's a good distinction to make. I truly learned something today. The thing with this, digital approach, which is now your Corsair platform, is still completely SRAM-based, right? And so what that fundamentally puts you in is now in the class of the other SRAM-based accelerators, which all have a fundamental limitation, and that is capacity.

18:53So how much capacity does Corsair have, and what is the implication of that limitation? Yeah, so here's Corsair. So this is a PCIe card, and we have 2 gigabytes of SRAM on this card, and we also have 128 gigabytes of LPDDR on this specific card. we can go up to 256 gigabytes. The SRAM capacity is the, it's more capacity than any other chips, right? So on a single card, we're able to get two gigabytes. And SRAM is the fastest memory there is. So for models that fit within its footprint on chip, it is blazingly fast. It's faster than any other memory. This episode of Semi-Doped is presented by Crusoe, where you can fine-tune and serve open models without managing a single GPU.

19:55When we sat down with Crusoe's serverless fine-tuning product team, we asked the question every AI user wonders, what will you do with my data? Crusoe has a simple answer. They don't touch it or see it. Your data is yours to fine-tune models with or simply store for future use. Their fine-tuning is designed so that both your training data and the fine-tuned model itself remain your property. Your training data is encrypted, so even Crusoe's cloud engineering team can't see it. You can deploy it on Crusoe Cloud in one click or download the checkpoints and run it anywhere you'd like. The fine-tuning process is fully managed, so you upload a dataset, pick an open model, and Crusoe takes care of the GPUs, orchestration, and all the infrastructure management.

20:39You pay per token processed rather than per GPU hour, So run a few versions, compare them, and download the one that actually works best for your use case. Create a new Crusoe account for free and get$5 in credit to start testing today. This part is super important, guys. Support Semidoped by going to semidoped.com slash Crusoe, click on that link in the show notes, type it in yourself, and you'll get directed to a sign-in page. Sign in with Google or create a new account, and Crusoe will immediately create that account for you with$5 in it. Boom. Again, just click that link in the show notes, sign in,$5 for you, and you support Semi-Dopes.

21:19So thank you for your support. Go check out Crusoe. Try to fine-tune your own data. Again, the link to try serverless fine-tuning is in the show notes. So hit that now, click through, and make sure you claim that$5, give it a try, and support us. Back to the show. So that Corsair PCIe card you're holding up essentially has not just one chip in it, right? Like how many chips does it actually have? Yes. So we have a, you know, you take all the sheet metal out. This is what the card looks like. So there are two chips. So this is, you know, one of the chips. Here we have taken out the lid of the other chip and you can see the four chiplets.

22:05On the second chip, each of the chiplets is half reticle. So there's a sum total of two reticles on each chip. And together they form two gigabytes of SRAM capacity. But more importantly, they can deliver 150 terabytes per second of memory bandwidth, which is orders of magnitude more than what is available using HPM even Blackwell GPUs are

22:4118 terabytes per second and this is 150 so this can deliver a lot more bandwidth because it's all on chip and at the back of the card you can see all of the LPDDRs so there's So that's what we use to store as bulk storage. Is it the black chips that there are the LPDDR in that image? Correct. Black chips are the LPDDRs. Okay, okay. So that's interesting that you have both SRAM and LPDDR, and we'll get to what the benefit of that is. If you have two reticles worth, I'm guessing this is a TSMC leading edge process. What node is it built on? It's TSMC 6 nanometer. 6 nanometers, okay. That's a good node in a sense SRAM kind of has stopped scaling much beyond that.

23:42So if you go down to like maybe 2 nanometers or 3 nanometers you don't get as much SRAM capacity anyway and then you get the benefits of better wafer cost at 6 nanometers and therefore you still get 2 GB SRAM out of this. That's exactly correct, yeah. The SRAM roadmap like you pointed out has stopped scaling from, you know, so it's, you only see maybe 10 % better density as you keep shrinking process nodes from 6 nanometers to 4 nanometers, 3 and two and so on. So six nanometer is a sweet spot in terms of SRAM capacity. Yeah, yeah, awesome. So I think it's useful to put what this PCIe card is capable of, because it has only 2 GB of SRAM.

24:39Clearly, to run a model that's bigger than that, you're going to have to put more of these cards together, right? Yes, exactly. Yeah, so we use it for many different applications. One of them, so first of all, we build up whole racks, right? We take a server with eight of these cards. For example, we have a Supermicro X14 server with eight of these cards. And we also have low-intensity networking that we built called Jetstream. and we use that to build up an entire rack that we call squad rack with an Aristar top-of-the-rack switch or any Ethernet top-of-the-rack switch that can host the entire model, all in SRAM.

25:28So per rack, we can get 128 gigabytes of SRAM. So that's pretty good. So for sure, you can run probably not like the biggest models out there at 128 gigabytes, but you can run some quite capable models at blazing fast speeds in a single rack. Correct. Exactly right. So there are different parts of the workload that we disaggregate and run on our cards. So we think of our deployments as heterogeneous deployments. There's still a GPU. And we accelerate the GPU, we turbo boost the GPU. And for example, one use case is something called specularity decoding, where we have a draft model that guesses what the target model running on the GPU and helps accelerate the GPU's target.

26:24For example, the GPU could be running KimiK3 or GLM5.2, a very large model, but since the GPU is running at one token at a time, it'll be relatively slow. But if a drafter that's running on Corsair on the SRAM can guess, let's say, 8 or 16 tokens at a time and provide that guess to the GPU, the GPU can verify that in a single shot. So that's the specularity decoding use case that we have talked about. That's one application. The other application is something called AFD, attention feed-forward disaggregation, where we take the experts. The experts are very memory-bound. And we move the experts to Corsair, to the SRAM chips, but we leave the attention and the pre-fill on the GPU.

27:21So that's a second use case. And the third use case is you leave the pre-fill on the GPU and only have Corsair cards to the decode. So there are different applications, each one disaggregating a portion of the workload that works well with these fast decode, limited capacity chips. Okay, that's a great description because the framing that comes out of all this is that you have to use this chip where its strengths lie and its strength lies in providing an enormous amount of memory bandwidth, 150 terabytes per second, which is like 10 times higher than what GPU HPM based systems give you. And I believe there is also a benefit that comes from not having to move bits as much and that saves a lot of energy as well if you don't have to shuttle bits far away.

28:21right so we'll get to the energy efficiency aspect but because of this capability of corsair and sram and the limited capacity you can actually use it in unique places so the speculative decoding was one example you gave where i'm just trying to like explain to myself so i understand it as well is that you use a smaller model the drafter model you mentioned is usually a much smaller model than like a Kimi K2 big, you know, trillion parameter. This is not a big model. My understanding is the speculative decoding drafter model doesn't even have to be that accurate. So it's okay to use a small model because you're going to be spraying out like eight tokens and the GPU will decide which of these eight tokens are the correct one because by the process of verification.

29:10So the idea is that you have to keep the GPU fed with speculative decoder and the drafter, which is being really fast, which is where the Corsair fits in, so that the GPU can run in parallel verifications. It can run 8 to 16 tokens in parallel verification as long as it gets enough tokens from SRAM to verify, right? Exactly correct, yes. So some of the work we did with Gimlet, we have a 1.6 billion draft feeding a 120 billion target model. So typically it's a percent or less. It's a much smaller model. For example, in the Kimi K3 work that we're doing, we would use 4 billion drafter for a 3 trillion model.

29:59So that's a very small draft compared to the target model. And the draft would be able to get many tokens right. And it's always exactly the same as what the GPU would output because it's always verified by the GPU. If at all the drafter makes a mistake, the GPU tells it what the mistake is, and then the drafter corrects and continues from there. So there's no difference in the output tokens that GPU produces compared to, you know, suppose it was not running any speculative decoding compared to running speculative decoding. So using these rejection sampling approaches, we make sure it's exactly the same accuracy.

30:50We don't lose any accuracy. And what is the bandwidth benefit you get by using this drafter verifier approach rather than just running decode on a fast GPU? Yes. So the fast GPU, it would be generating one token at a time. And if you were to get, let's say, five tokens correct on the average, right? Let's say the drafter produces eight tokens or 16 tokens. and feeds that in a chunk to the GPU. And on the average, let's say we get five out of eight correct, you would get much better speedups. So we were able to demonstrate three and a half to four X speedups on the GPT-OSS model. So there are two places how you could use it.

31:42One is you could get to regimes you could not get with the GPU. So in the work we did with Gimlet, for example, we could get to 1 ,600 tokens per second per user, something that the GPUs could not get to. The GPUs could only get to, say, 400 tokens per second per user. So we get this 4x higher interactivity. Another use case is for the same interactivity, you can get much higher throughput. So both modes are possible. and it's up to the user how they want to trade off throughput versus interactivity. And we essentially shift the whole Pareto curve to the right. Yeah, nice, nice. You did mention two other forms of disaggregation, which is like attention feed forward disaggregation and then just disaggregating the pre-fill and decode.

32:37Those are, I think, reasonably well explained and we can't possibly get into all the details, but the broad idea is that even the process of inference has many ways to break up the problem. And you could use SRAM-based inferencing for parts of them where it really benefits the most. So all of those things, just like the speculative decoding example you gave, should speed up the inference and move the perito frontier of the interactivity curve in a good way. So these are all ways you can use the SRAM approach. This is a very good setting the stage for what comes next. But before we do that, we have to address what the LPDDR on the card is meant to do.

33:21Yes. So there are obviously in the performance versus TCO curves, there is you can have super fast these SRAM chips like D-Matrix or Cerebrus or Grok. They're super fast but you sometimes require a very large footprint. So that's at the top of the tier. Then there's the mid-tier, which is essentially all of the GPUs, and that's where the bulk of the market is. And then there is another tier where you can store large models very efficiently on LPDDR, but you may not get the interactivity advantages of the GPU or, of course, the SRAM, right? But you could use that to reduce cost. You could use that to shrink the number of cards because you can pack a lot of cards together, a lot of memory on one card.

34:22For example, the card I was showing, you can put 250 gigabytes, and this is commodity LPDDR, the same LPDDR that is found in cell phones. And it doesn't require three times as many wafers to produce the same bits as HPM. So it's much cheaper because of that. So we cover both one and three. And obviously, we don't use any HPM. On Corsair, we don't touch two, but we cover the other two. So you have one approach to go really fast and another approach to go really slow. The really slow approach gives you more capacity, but not as much speed and that's useful for some workloads maybe like offline batching you could do process all your day's work at night and then next morning you can come back to inference results, you can run those cheaply on widely available DRAM, although I don't even know what that means anymore, is DRAM even widely available?

35:26I don't know nobody gets, there's no such thing called cheap DRAM or widely available DRAM anymore but anyway, it's DRAM okay it's not HBM which is even harder to get a hold of possibly yeah so it serves two different areas of high speed or high capacity because you don't have HBM you don't have the medium speed medium capacity tier but you have the other two extremes so this is good like I think the next step we should go into is actually the meat and potatoes of all of the D matrix stuff that's been going on at Hot Chips, which brought all the excitement. And it was all over Twitter after your talk, and I was at that talk as well, which is basically 3D DRAM.

36:15And I think it solves this high-speed tier somehow by increasing capacity as well. But before all that, I think we should explain what 3D DRAM means in this context. because there's a lot of confusion as to whether this is actually 3D DRAM from like the transistor level is now 3D because that is the holy grail of DRAM. And that's the way you can increase DRAM density beyond what we have like plateaued at today. So just like we discussed SRAM density, DRAM density also has kind of plateaued. People are pushing it, but it's not improving by orders of magnitude. So this is not that kind of 3D DRAM.

37:00So we should set the stage of what actually 3D means in this situation. Yes, yes. So this is our Pavehawk proof-of-concept silicon of 3D-stacked DRAM. What we have here, what we mean by 3D DRAM is an inference accelerator where one layer is logic, which is where the compute happens, And another layer is the memory, DRAM, and those are vertically stacked. So unlike HBM, which sits 2D, so you have a DRAM that sits next to a GPU, here we vertically stack compute and memory together, DRAM together. And that's what we mean by 3D DRAM in our context. It's different from bit cell level 3D. So this is specifically around how to build accelerators.

38:02Yeah, this is helpful because now we can discuss basically why we had to stack DRAM on top of logic. And I have maybe several questions around this. The first one is, we will directly be able to increase capacity because DRAM is about an order of magnitude better density, bit density than SRAM is. So immediately you will go from the hundreds of megabytes per chip. In your case, you have multiple chips that give you 2 GB of SRAM in the Corsair, right? But now, what kind of capacity do you get by just putting one chip of DRAM on top of your logic chip? Yeah. So we get 32 gigabytes of DRAM. In fact, we put the DRAM in this specific construction.

38:59We have DRAM at the bottom and logic at the top. And this is what we mean by 3D DRAM. And you can see the cross-section on the right where they're connected using micro bombs. And the power to the logic die is delivered through the TSVs. The logic die itself in this case is a TSMC N4 logic die. and the bottom die is a custom-designed DRAM. And we're using 36 micron face-to-face stacking. And this, so going back to your question, Vic, so we can pack 32 gigabytes of capacity at 100 terabytes per second of memory bandwidth. So essentially, we got 16 times the capacity of SRAM while we retained SRAM bandwidth.

39:49And in this particular demonstration, we measured 0.37 pJ per bit, because the bits only go down and traverse very short distances, as opposed to HBM, where you have, you know, it has to go down the entire stack, and then it has to go sideways, and then it has to traverse, you know, tens of millimeters on the logic die. And so all of that power adds up and you will end up burning about 2 to 3 picojoules, 2.5 picojoules to be exact, on the HPM itself, HPM4 itself, but another 2.5 to 3 picojoules on the GPU. So that 5 picojoules became 0.37 picojoules. And that's the big advantage of this technology.

40:42So for the same power, you can... So if the total power is the same, then you can deliver a lot more bandwidth, in this case, 6x bandwidth, by using this technology. So to put that in perspective, if you took your previous Corsair card, which had 2 GB of SRAM, you need 16 PCIe cards, which you probably have to put in across two different racks. I think you can put only eight PCIe cards in a single rack unit of Corsair, right? Single server. Single server, yeah. And then you're going to have two full servers to get 32 GB of SRAM. Now, Corsair required all that while Raptor, just by putting DRAM on top of logic, no, no, other way around, and I want to talk to you about that, you actually put logic on top of DRAM.

41:50You ended up getting basically two servers worth of capacity while retaining the bandwidth that you had in SRAM. So you got, you had lunch, you know, you got, what is it? The statement goes, you had free lunch and you ate it too. Yes, this is very rare in engineering. Yeah, that's great. And the energy efficiency too, even though you ended up stacking, the distance that the electrical signals have to cross between the compute chip and the DRAM chip are so close by that you don't spend as much energy shuttling bits between memory and logic as you would in HBM because you have to, in a GPU environment, you have to send bits from the HBM all through the advanced packaging substrate, this COOS packaging, right?

42:50And it has to go several millimeters into the logic chip or the XP or the GPU or whatever. Now you have a micro bump that is like, what was the size of that thing? Like 70 micron? 36 micron. 36 microns each. It's a pitch, but like the height of the bump itself is like something like tens of microns, right? So instead of going millimeters, you're going tens of microns. So the energy required to shuttle bits is now one-tenth of what you would have to do with HPM. Or one-twentieth even, based on your five picajoules per bit number. Something like that, right? Ten to twenty times more. Exactly right.

43:31So, yeah, because vertically, of course, you have to make sure that you're not going to move a lot of data on the logic die. So we custom design and pitch match the compute engines directly above the DRAM banks. So we call it micro banks. And we design a channel such that three micro banks fit underneath the compute engine. And they're pitch matched. and so you have to co-design the compute and the DRAP. Okay, I'll get to that detail in just a bit but is there a specific reason you chose to put compute above logic, above the memory? So I think there are two ways to do this and I do remember you explaining this really nicely and at the hot chips talk and I wanted to get back to you on that on the podcast which is basically there is two options.

44:26One, you could have the compute chip on the bottom and put DRAM chip on top, which seems to be kind of the approach that Cerebras wants to do at wafer scale or Qualcomm wants to do with their high bandwidth compute where you put DRAM on top of logic. The other approach, which is what Dmatrix has, at least on the Raptor version of this, is logic that is sitting on top of DRAM. So there are definitely some trade-offs here that we need to talk about and why you chose this approach. Yes, yes. So, you know, you can pick your poison. And so on the left picture here, you see the logic on the bottom, the DRAM on the top.

45:13And, of course, since the liquid cooling core plates are on the top, the heat has to conduct through the DRAMs and the logic is the hottest die here and it has to traverse the DRAM. So the DRAM temperature will rise and thermally this is not as good a solution as one on the right. One on the right, the heatsink is right on top of the logic die because the logic is on the top. It doesn't have to conduct to the DRAM. So the DRAMs don't see a temperature gradient as much of a temperature gradient as on the left. So from a thermal standpoint, the solution to the right is much better than the one on the left.

46:04But from a power delivery standpoint, the solution on the left is better than the one on the right. So those are the fundamental trade-offs. Yeah, so putting logic on top of DRAM means that you can just put the cold plate right on top of the logic chip and extract heat before it upsets the DRAM die too much. DRAM really doesn't like heat, does it? And so the problem with that approach, now the poison you have chosen here, is you have to deliver power to the logic chip through the DRAM die, which is a power delivery challenge because you want to make sure that there is no power drop across these vias going through the DRAM die to the logic die.

46:53So clearly you decided that thermal was a bigger problem to fix early on and you deal with power delivery later. That's correct. That's correct. So for one high stack, we found that we could, in all our modeling, we could reliably cool the DRAM to be less than 95C for power densities of the order of 0.5 to 0.7 watts per millimeter square. And so that, you know, that's the power density of our logic die. So we could solve both the power delivery problem and the thermal problem using this approach. In the future, I know right now you have micro bumps connecting these things together. but in the future do you think that going to maybe hybrid bonding I'm sure it will help in the bandwidth aspect because the pitch can get closer than 36 microns it's going to get much much lower than that which means you can put far more parallel connections between the compute and the memory chips but do you think that helps the thermal aspect if you go to hybrid bonding in the future?

48:06Yes, it does help the thermal aspects especially if you do multi high stacking but the fundamental problem is still reducing the energy consumption on the DRAM and because you still have some routing on the DRAM and it still takes like in our case 0.37 pJ per bit so if you can pack with Wafer on Wafer you can get a lot more IO than say 100 terabytes per second You put maybe 400 terabytes per second. But if you don't reduce that IO power from say 0.37 pJ to 0.1, the total power that you'll dissipate just in the interconnect is so high that that is no longer the limiting factor. Yeah, that's a very good point.

49:05And to put some more light on that, The way you can easily calculate how much power this kind of a chip at a given memory bandwidth will consume is if you take the bandwidth, let us say 100 terabyte per second, and you multiply it by the number of picajoules per bit, it automatically spits out the power consumption in watts, right? So if you go four times higher in the memory bandwidth by doing some techniques, you want to be sure that you reduce the energy per bit four times to keep the same level of power dissipation at the chip level. Otherwise, what ends up happening is if you boost the bandwidth, but you're still spending energy shuttling bits, you're just shuttling more bits, and that's going to cost you in terms of power.

49:56So it will go up by four times if you keep it constant. It's a very interesting point, as when people try to make faster and faster interfaces, make sure that you compensate in the picajoules per bit so that otherwise your GPU will suddenly start. In your case, you probably have a few hundred watts here in terms of your power delivery. Let me do the math right here. 100 terabyte per second times what is it 0.37 picajoules per bit right so that's a few hundred watts right yes so 100 terabytes so you have to convert it to terabits so 100 times 8 times 0.37 that I think is about 280 watts just in the memory subsystem to move those bits into the compute unit And now if you start putting stacking more, maybe that number will go up.

50:55If the number, let's say, even remains the same, but if you want four times the bandwidth, then that will end up burning upwards of 1 ,000 watts. So we have to first figure out from an accelerator standpoint what is the bandwidth we desire, how much power we want to spend on the memory subsystem, because there's still compute you have to spend power on, interconnect you have to spend power on. Typical rule of thumb we use is 60 % on compute, 30 % on memory, 10 % on interconnect, and then figure out what technologies can help you get there. I see. There's like a lot of power planning here that goes into it.

51:39And so it's interesting. It's only 10 % that this whole interconnect budget goes to. I wasn't expecting it to be that low. When I say interconnect, I mean chip-to-chip interconnect. Chip-to-chip, yeah. Yeah, yeah, yeah. Okay, I see, I see. Okay, that's interesting. And just one last detail before we get too much into the weeds. You mentioned that you line up three banks of DRAM with the logic. And so each logic tensor core now has its access to its own slice of DRAM memory. there's no contesting between tensor cores. They don't fight for DRAM allocation. Is that how I have to look at that? Yeah, you can look at it that way.

52:23In the Raptor architecture, we have 256 tensor engines per chiplet

52:33and 768 micro banks. So each tensor engine has its own three micro banks. Of course, a tensor engine can access a different set of microbanks too, but not at full bandwidth. And in addition, we provision about 8 % or 9 % additional microbanks as spare, so we can tolerate failures. And that's very important, especially when you get to wafer-on-wafer techniques. You cannot test anything. In the construction we showed you, before you put the logic die on top, you can test it. It's micro-bond, so there's test infrastructure. Yes. And only long-good parts are then tested, and then that's what is used for the stacking, chip-on-wifo process.

53:27But if it's hybrid bond, then how do you test any of these things? So you have to make sure that you come up with new techniques to do that, and the industry is still trying to figure all of that out. Yeah, yeah. That's cool. Now I think I want to move on to the really exciting part of the news that recently came out from Dmatrix, which is the use of the Raptor chips in the NVIDIA NBLink Fusion ecosystem. Like, what's the idea and what are you thinking going into putting Raptor chips into the ecosystem? So, like, basically, what is the benefit of that? And really, how easy is it to integrate Raptor chips into the NVIDIA ecosystem?

54:14Yes. So, obviously, Rack is our product, right? At the end of the day, like you pointed out, the models are very large. And let's take an example, like Kimi K3. Kimi K3 is the largest open source model out there. 2.8 tera, right? So it's a huge model. And it's got 1 million context. Obviously, it won't fit in one chip, right? In GPU or otherwise. So the whole idea is to put many of these chips, in our case, you know, 144 of these R4 packages. So we have essentially the Raptor package looks exactly like this. So there's four chiplets in a package. And there's 144 of those packages inside a single rack.

55:03And for that, we chose the Vera Rubin rack. And we use Envy switches. So we have 18 compute trays and nine switch trays. And each compute tray has eight of these packages. So that's how we get 18 times 8, 144. So we have 144 Raptors inside a VR rack, Vera Rubin rack. And so what we do is we keep the Vera CPU the same, all of the networking, the CX and the Bluefield networking the same, and replace the Rubin GPU with Raptors. and instead of four Raptors, four Rubens, we have eight Raptor packages. So some total of 144 chips that's all-to-all connected and this rack then can talk to a Ruben rack because they use the same networking technology.

56:12and so then you can going back to my previous disaggregated inference examples if you want to do PD disaggregation or AFD or run full models all of that is possible now within the NVIDIA rack yeah so you just have to pull out all the Rubin GPUs there will be like 4 of them put in like eight Raptor chips. You keep the Vera CPU and you build a rack with 18 compute trays. Each one has eight Raptor chips. That's 144 chips. Each one, what is the total capacity of the 3D DRAM in a single rack? 2.3 terabytes of capacity. So that's pretty big to hold an entirely large model all by itself, actually, if you wanted to do that.

57:16Correct. So we were showing at Hotchips that we could hold a Kimi K3 model at 4-bit precision and have sufficient space for many users, KVCAS space, and we could hit 1 ,000 tokens per second per user. So it's a performance metric that you can't achieve with traditional memories, right? So 3D DRAM, because it's such high bandwidth, you can get these levels of interactivity at high batch, all in one rack. In a single rack, we can get to 3D model at 1 million context at 1 ,000 tokens per second per user on Kimi K3. And on GLM 5.2, we could get to 3K. So this really fast decode speeds. And in the NVIDIA racks, you can do multiple racks.

58:14So there is an NBL 576 option where you have four racks, and the four racks are connected using optics, optical scale-up. And using that now, you have not just 2.3 terabytes of capacity, but 2.3 terabytes times four, right? So 9.2 terabytes of capacity, which is sufficient to, you know, at FP4 precision, serve even a 10T model. So it can serve out the whole frontier at unprecedented speeds because of a rack level, rack scale solution, as well as the fundamental technology that delivers much higher bandwidth. So you're talking about basically putting four racks of Raptor chips, each with 144. So that would make it, two of them would be 288 and four of them would be 576.

59:13Hook them all with optics because now you are any form of very high speed interconnect, whatever it is. And then now you have a large scale-up domain with basically nearly 10 terabytes of SRAM-like bandwidth, but with 3D DRAM capacity, which can serve any model, large frontier model of the future at ridiculous speeds. Exactly. Seems like a winner. Awesome. That's great to hear. that's great to hear I'm excited to see what you know what all that comes comes up and what kind of benefits it provides in the future because stuff is still evolving and I'm sure that there are so many people trying so many different approaches I like one approach you know of what you've described clearly seems to have like a bright future from what I'm looking at so that's that's very interesting how does the whole software stack work though so if you put in a rack of Raptor chips next to an NVIDIA rack running GPUs, all of that stuff runs on CUDA.

1:00:28How does the software work? How do you do the disaggregation? Yeah, so Dynamo and frameworks like that would handle the disaggregation, right? SGLang or VLLM, any of these frameworks would partition the work, even today, between, say, a pre-fill cluster and a decode cluster. Same thing happens here. But underneath that, when it comes to generating code for our chips, we have our own software stack, right? We provide something called a model builder SDK. And the model builder SDK, essentially, we expose a set of primitive kernels, you know, mat models, MD, various shapes, and so on, and a way to stitch together models using those primitive kernels.

1:01:21And we can, you know, a programmer can choose to write new kernels if they want to, because we expose the eyes of the chip, or just to take these building blocks and compose models. And we provide a lot of example models. And the AI kernel generation tools have gotten so good these days that using these examples and using these agentic harnesses, you can quickly come up with a new model and perform a new model in a couple of days. So that's the loading stack. And then we have our own execution stack where the device runtime, there's a runtime that runs on our chip, and that runtime runs on the embedded RISC-V processors that are part of each of those chiplets.

1:02:10It's the same on Corsair. It's the same on Raptor. And everything essentially is targeted on that RISC file. Today's sponsor is Meter. You've got the GPUs and the power. Now, who builds the network? And who do you call when it breaks? With Meter, it's one call. Tell them your GPU or server count, and they design, deploy, and run your front-end storage and management networks. Meter builds its own networking hardware, switches and all, and writes its own network OS. So that call goes to the team who built it. Running a new cloud of clusters in a colo? Go to semidope.com slash meter. Links in the show notes.

1:02:56I see. Fascinating. What comes next after this? Like, clearly, if you can have one DRAM stack, what are you going to try next? Put two, three, four? What's the roadmap on that? Yeah. So we are trying to build up to a 4i stack. Now, you know, and so on Lightning, which is our next product, we have our logic on the bottom and then four DRAM on the top, and we're trying to build up to that. Now, we have obviously lots of challenges in terms of thermals, in terms of the bandwidth we can deliver, in terms of the availability of hybrid bonding. As you know, hybrid bonding requires this, you know, the hybrid bonding of DRAM and logic hasn't yet been successful, mainly because of contamination and warpage.

1:03:56these are fab grade processes that you know you need you cannot take a wafer into a clean room and contaminate for example a logic fab right so all of these are challenges that need to get worked through and so we are working through all of that but more importantly from an architectural standpoint if you cannot test any of these things and you're just going to hook this up how are we going to solve this problem How are we solving the yield problem? How are we solving redundancy problems? If you look at even Raptor, we do refreshes that are eight times faster, four millisecond refreshes, compared to 32 millisecond refreshes, precisely because of these thermal concerns.

1:04:44And now, what is the solution for multi-eye stack? So stay tuned. I mean, we have solutions to many of these problems, but it's definitely a work in progress. In terms of the packaging aspect itself, do you think that the four DRAM die stacked on top of each other, can just that be hybrid bonded? And then you use micro bumps to the logic die. Is that like a feasible approach? That's definitely a feasible approach. Because with micro bumps, you get, let's just do some math here, right? So if you have a reticle-sized die, so it's 27 by 33 or whatever and let's say it's 25, even 20, 25 micro bumps, so you can get 1000 by 1000 quite easily right, 1 million micro bumps and if you run each micro bump at a gig that's a million times a gig, 1 petabits per second 125 terabytes per second of memory bandwidth just on one reticle.

1:05:54And so that is sufficient bandwidth. Now, so certainly you can do that. You can just hybrid bond the DRAM and then micro-bond that to the logic. That's a possibility.

1:06:09And so there are so many options here that I'm sure the industry will explore many paths here. Yeah, this is interesting because in my memory of how I know this, how chips have been tried out. I don't think anybody has really gotten to this point of stacking a four-chip memory stack on top of logic. Is that a fundamentally new thing the whole industry is trying to do? Has it ever been done before? And if so, to what extent? Or is D-Matrix Raptor the only example where you have logic and memory stacked currently? Yeah, so in terms of a full accelerator, we believe we are the first to stack a full logic accelerator with micro bumps across the entire surface area on top of a DRAM interposer.

1:07:12and obviously there are parallels in the custom base die world with the logic on the bottom and then even 8 or 12 or people even talking up to 16 stacks of HBM. So similar technology, but it's not the same, right? It's not trying to pack a lot of compute under the base die. So the power densities are different. The sizing is different. It's 100 millimeter square versus, you know, what I showed, half-retical chips. Yeah. And so the trade-offs are different. And I think these trade-offs are better because you don't need to go as high. So yield fallouts would be better if you restrict stacks to one to four high.

1:08:03Like we talked about, the energy benefits are obvious. once you explain it, everyone can see it, that you don't traverse millimeters. And traversing one millimeter on silicon is 0.35 picajoule per bit per millimeter. So it's really expensive. And so I think to us, it's an idea, like we said at Hot Chips, it's an idea whose time has come now. And the benefits are so obvious that everyone is going to... We believe this is the direction of the future. Yeah, yeah, yeah. Exciting, exciting. So now with four memory stack, you were mentioning how you can get a 10 trillion parameter model running with like a 576 chip domain.

1:08:54But now with like a four high stack, you have four times as much in a single rack. And so you can run a 10 parameter model in one rack. Is that the right? Exactly. That's the right On Raptor compared to HPM4 So what we lose is on capacity Because we are not on the latest and greatest HPM4 Technology And so we So if we were to use that technology We would get much better Capacity, megabyte per millimeter square And so But in terms of bandwidth per millimeter square or power per millimeter square, we are, what does a magnitude better? So at the end of the day, we are trying to generate tokens given a limited power budget.

1:09:44You can only get one gigawatt or something into your data center and your token production is proportional to bandwidth. So if you look at this, for a given power and the fact that you can get so much better 20x the bandwidth per millimeter square and 13.5 times better power means that you should be, you will be far more efficient in this token factories or does it make it more efficient than other technologies? And to your point, if we were to use more leading edge process and stack it, you can get 4 to 8 times the capacity. So on a single rack, you can get to 20 terabytes of capacity and that should serve any model, right?

1:10:35Any model that we can visualize in the near future. Yeah, any model for the next two years. I don't know. He's joking. Models are getting ever bigger. So any model for the next two years. Famous last words. I want to like end by asking you a more generic question. So to me, I've always said this in like many previous podcasts in like articles I've written, it seems like to me that inference world right now is kind of the wild west. Like making training chips seems to be like there's like one or two methods to do it. Most of them are GPU based. But you know inference is like it's all over the place.

1:11:17Like you've got wafer scale engines, you've got like SRAM only accelerators, you have like SRAM like Corsair, you have SRAM plus LPDDR and then now you have 3D DRAM and then you've got like other companies who use all three tiers. They use SRAM, they use HBM, they use LPDDR. Like everybody has their own approach. So in the future, say a year from now, do you see any kind of a convergence into what is the right architecture we should be doing? What is your take on that? It is what we've been doing, what we've been discussing. So, I mean, to us, it's fairly obvious, right, that the most, the densest memory today is TRAM and so if you want to pack more in terms of because it's 1T, 1C you can get 40, 50 megabyte per millimeter square versus 4 or 5 on SRAM so you get much denser storage You can get the bandwidth by stacking.

1:12:31So with this heterogeneous integration approach where you stack, the benefits are so obvious. And specifically the metrics we called out on bandwidth and power is so compelling that it's only a question of when, not if. So we feel like in five years, this is the wave of the future. Everyone will be doing 3D stack DRAM. There are challenges. There are engineering challenges to solve along the way. And we'll all figure out. There's a lot of problems to solve and we'll figure out how to solve them. But we believe this is the end state of inference. That's amazing. That's amazing. I'm only going to leave you with one contest that I think somebody in the future is going to come over and say, hey, why don't we stack high bandwidth flash on top of logic?

1:13:23That'll give us even more capacity, right? then you open up a whole new world of problems and then we can all work on it for another five years this is great yes yes exactly talking about wild west that's it's a really exciting space there's so many new ideas there are so many avenues to explore I'm sure creative people will find new ways to solve problems it's just a really really exciting time to be in this space yes absolutely Thank you so much for doing this podcast with us. As always, whether it's your talks or meeting you in person or on the podcast, I always learn something new. So I hope our listeners learn a whole lot of stuff about Dmatrix and how everything has evolved from your initial analog in-memory compute days far before, you know, even GPT was a thing.

1:14:17You know, you've started building something for inference, which is clearly like, you know, well over half a decade out. And you have like working silicon that is now going into racks. And, you know, it's linking up to NVIDIA's ecosystem. And in the future, you're going to have even more memory per chip. And so we can build even more models and bigger things and smarter things. So that's all very exciting. So thank you very much. Thank you. Thank you, Vik. Always been a fan. Enjoy your writings and enjoy your podcasts. it's great to be here and talk about you know what we love to do so thank you for having us alright that's it for this episode of the Semi Dope Podcast if you enjoyed it leave us a 5 star review on Apple Podcasts it's very important rankings are always important we always want to rank higher than somebody else so get that and you can always watch this on YouTube or any podcast platform of your choice and if you enjoy it share it with a friend and we also have a daily newsletter on daily.semidope.com where we keep track of all of the daily news so check that out as well catch you on the next one