Ray Wang on How AI Is Causing DRAM Prices to Surge

0:00UKG. Their HR, pay and workforce management tools help business leaders empower their people. because when work works, everything works. Learn more at ukg.com slash work.

0:35Policy and the people shaping the European Union right now. And from London, I'm looking at what all that means for markets, money and the wider economy. We've got reporters across Europe and around the globe feeding in as stories break. So whether it's geopolitics, energy, tech or markets, you're hearing it while it happens. It's smart, calm and to the point. And it fits into your morning. You can find new episodes of the Bloomberg Daybreak Europe podcast by 7am in Dublin or 8am in Brussels, Berlin and Paris. On Apple, Spotify, YouTube, or wherever you get your podcasts.

1:14Bloomberg Audio Studios. Podcasts. Radio. News.

1:30Hello and welcome to another episode of the Odd Lots podcast. I'm Joe Weisenthal. And I'm Tracy Alloway. Tracy, you know what I'm worried about? I don't know if I'm worried about it, but it kind of feels like the direction things are going, which is that like all energy, industrial commodities, anything that we use for any purpose is just going to like feed the AI beast. And us humans, we're just going to get left out in the cold. They're like, nothing for you. We got to like feed, we got to feed it all to the AI. And maybe in 10 or 20 or 50 years, the AI is so powerful to just decide, why do humans get any of this?

2:07We just thought we should keep it all to ourselves. A legitimate concern, I would say. I mean, to some extent, we are already seeing this crowding out effect, right? So energy prices in certain areas have been going up. And most recently, memory chip prices. So this has been all over earnings recently. You had companies like Apple saying that because of a crunch in memory chip supply, they might have to either raise their prices or like cut down on the amount of phones they use. Nintendo. Yeah. Like if you pull up a chart of Nintendo shares right now, they are just getting hammered. And supposedly it is all because of this memory chip shortage.

2:45By the way, do you remember, do you remember like buying your first PC in like, you bought one before I did, I think, but mine was in the mid 90s. Yeah, yeah. I think that's about when I got one. Do you remember how much memory those things had? Oh, like nothing. Yeah. Like I looked it up. Something like less than 10 megabytes for a bunch of them. You know how much PCs have nowadays? Actually, I have no idea. I think it's, let's see, 16 gigabytes. Yeah. Isn't that crazy? It is really crazy. I remember, like, do you remember zip drives? Yeah. Oh, my God. And there were all these special, you know, obviously various memory peripherals that you could buy.

3:25and they came in sort of dedicated cartridges and stuff like that. I think they were called ZipDrive. Yeah, I think, right? They were ZipDrive? I don't remember. Yeah, yeah, it was. Yeah, that's right. Buying additional memory storage, et cetera, used to be this big part of the consumer process. I'm actually glad, you know, my son really wants a Nintendo, but if they can't produce them anymore because they can't get the memory, then I'll say, sorry, no video games. That's so mean. You're going to have to sit him down and be like, I'm so sorry, son. There's this thing called the supply shortage.

3:52I'm going to use this as a learning opportunity to explain how global supply chains actually work. Totally. You know what the other, the one interesting thing is, if you look at spot DRAM prices, which we have on the terminal, this is, I think, maybe the interesting dynamic. There's a lot of AI related things on the terminal that have been surging for several years, most notably some of the big chip companies, NVIDIA, et cetera. If you look at a chart of spot DRAM prices, the big surge wasn't until late last year. And then since then it's gone nuts. So for a long time, even as this AI story, the AI meme, the AI industry propagated, people were really paying attention to that area.

4:30And then suddenly it's just gone like – Haywire. Haywire, yeah. And so obviously huge supply demand imbalances. We need to understand what's going on because I think the stakes are – it's getting so much. As you mentioned, a lot of companies are losing out pretty big. Others are making a lot of money. We sort of need to get a handle on what's going on with memory. Let's do it. I'm really excited to say we really do have the perfect guest. We are going to be speaking with Ray Wong. He's an analyst at Semi Analysis. And he's recently published a brand new report on exactly this topic, talking about a memory super cycle and so forth.

5:04So truly the perfect guest, as I mentioned. He comes to us from Korea. Ray, thank you so much for coming on Odd Lots. Hey, nice to be with you both. And thanks for having me. So many questions. Really appreciate you joining us. Let's just start with the basics. Like, what is it about AI or what is going on? What is the core idea such that demand for memory in various forms is absolutely seem to be booming right now in a way that must have caught the supply side of the equation very off guard? Yeah, I think, you know, to really look at kind of the sort of imbalanced supply demand dynamics, really, we want to look at like, you know, what happened a few years ago, right?

5:41Because, you know, your capacity to check like your investment a few years ago. and what's happening in 2023, 2024, it's sort of a down cycle, right? Because remember, during the COVID, all the companies are trying to flock in to buy tons of DRAM, right? And there's a lot of purchases there, right? And there's sort of like up cycle, right? But it's like a very, very short up cycle because like when COVID gradually getting better, right? You actually don't need that much DRAM. And back then, the company will be like, hey, you actually don't have a sustainable demand, right? So they don't want to over invest to expand their capacity.

6:16And when you go into the down cycle, your capex typically be very constrained. Very like, you know, company from their perspective, they try to be conservative. So that leads to 2024 and 2025 that your incremental wafer capacity that goes through DRAM is actually quite limited. But what's happening on the demand end is actually the demand was accelerating so fast, right? You know, one way to look at it is to look at NVIDIA's earnings, right? And if you, when you're on one side, your demand is accelerating so fast. On the other hand, on the supply side, your supply just couldn't catch up. And there's more nuances behind the supply, right?

6:53Because before, they are just supplying all the commodity DRAM, like DDR, LPDDR, right? All the memory that goes to PCs, goes to laptops, goes to your mobiles, goes to Nintendo Switches, right? But what's happening in this AI is like, there's a new thing called HBM. Think about it as like a specialized memory directly for air-asservator. And that memory is extremely DRAM wafer intensive. To give you a sense, right, on the same wafer basis, you can produce three more bits if you do commodity DRAM, but you can only produce one bit of HBM, right? And that ratio will actually go in higher when you go to HBM4, HBM5 in the next few years, right?

7:36So in that way, on the same wafer basis, you can only produce that much HVM. And right now what's happening is like, hey, this HVM is super, super profitable. And why not we dedicate more wafer for HVM? So here's the problem, right? You only have that much wafer and you try to do everything. So that's pointing out a lot of the supply going to the commodity dealer, right? So we have sort of both things is happening. And essentially it comes down to like, I think really, you know, since start really kind of people are aware. it's like probably Q2 2025 that people realize, hey, like, you know, we actually couldn't buy enough, like enough DRAM, both enough commodity DRAM and also HPN for their demand.

8:16So you mentioned the C word just then. Oh, wait, not that C word. Commodities. You mentioned commodities. And when I'm reading these articles about a memory chip shortage, people use commodity type language, right? So you hear super cycle a lot. You hear commoditized chips versus, I don't know, other types of chips. Are they customized? I don't think so. But anyway, how much does this particular industry actually resemble a commodities-like industry? And then maybe explain a little bit more why historically it has been really, really cyclical. Yeah. So I think a couple of reasons, right? Fundamentally, the cost per bit for DRAM was actually coming down every single year significantly.

9:00Even in the most recent year, it was also coming down, but the extent that they are coming down for every single year is actually slowing down over the past 10, 20 years. So the cost was keep going down. So the cost is hard to become the most competitive part. Another thing is there's a committee that set up the standard for DRAM. So you are doing similar products. You're just having, oh, this is kind of a little better power efficiency. The performance was a little better here and there. So it's hard for a memory supplier to significantly differentiate their products compared to their competitor.

9:39So essentially, what's the things that will compete in the end will be the market prices. And then when you're selling the similar products and you are targeting a similar audience, that's sort of like a commodity, sort of semiconductor commodity products to me. And I think those are the two main characteristics I'm seeing for DRAM. And I think that's a little different for HBM. Tell me, explain this. So there's commodity DRAM and then there's HBM. HBM is a type of DRAM? Is that what you're saying? Yes. Can you explain this further and like sort of, yeah, walk us through what the commodity version and why there is this thing called HBM high bandwidth memory, what that's all about?

10:22Yeah, yeah. So essentially, the emergence of the HBNs really go down to a kind of model layer. Because this is when you try to scale and try scaling your AI model, right? Your thought is increasing significantly every single year, right? Because your compute was increased a lot faster. But your memory bandwidth was actually sort of, it's increasing, but it's very limited. Especially if you are using the memory from kind of traditional commodity DDRN, then the bandwidth is actually very limited. So from memory supply, they are thinking, hey, how can we expand the bandwidth to supporting the memory that's going to scale the AI, right?

10:59So let's consult HBM. So the way they do HBM is, so they are stacking, multiply 8 or 12 or even in the future 16 deer and dice together, right? So that making you essentially having higher bandwidth and have enough capacity to support AI developments over the past few years and in the next few years. And that's very different. You are stacking all these dice together compared to, hey, we just have one DRAM product as a chip. And that also comes down to the point that HBM manufacturing, also the packaging, both front end and back end, is a lot more complex compared to commodity DRAM. And that's why I say commodity DRAM is more like commoditized, but I think HBM is less so because it's not more complex, right?

11:45And for supplier, you can actually differentiate either from the front end or back end technology. And when your yield is better, right, it gives you better margin, better ability to compare, compete with your competitor, right? And, you know, stuff like thermal, things like that is the thing that will really stand out, right? And that's very different commodity DRAM. Getting more and more layers on HBM, it kind of reminds me of the razor wars where like, you know, Gillette would unveil a three blade thing. And then the next day someone has like 12 or whatever.

12:28UKG. Their HR, pay, and workforce management tools help business leaders empower their people. Because when work works, everything works. Learn more at ukg.com slash work. Hello, I'm Michelle Hussain, and for more than 20 years, I was at the BBC. But all the time I was delivering the headlines, I wanted to go further than the news of the day, to spend more time with the people shaping our world. And that's what I'm doing here on this podcast. Speaking to people from Nigel Farage, to tech journalist Kara Swisher. And the tech industry is running wild. You know, they've gotten what they wanted and they've seen a huge run up in their stock prices.

13:15This will be a place where every weekend you can count on one essential conversation to help make sense of the world. So please join me, listen and subscribe to The Michelle Hussain Show from Bloomberg Weekend, wherever you get your podcasts. You certainly ask interesting questions.

13:38So if I'm a company like Nintendo, how am I actually purchasing chips? Because I think this is going to be important when we talk about what's going on right now. But do I have like a forward contract with a major supplier or do I have, I don't know, a warehouse full of chips that I bought previously? How do they actually manage that? So there's a couple of ways they do it, right? They will do it in terms of like kind of they will do it different way, like different environments and also different vendors, right? So it's hard to say, hey, this is exactly the way they do it. But typically, I think you can think about like, hey, this is amount of the chips we're going to purchase in the next four months.

14:16And this is amount we're going to commit. And we're going to sign the contract pricing whether this term will be four months or for the LTA, you can go as long as like to a year or even two years. So let's see what the way you think about it. But typically, you'll be like, hey, we signed this contract for four months. And by the end, sorry, for three months, for a quarter. So by the end of the quarter, you negotiate a new contract pricing for the next quarter. So that's typically the way it works. But the problem for right now is for consumer electronics, it's like the pricing probably is sort of, I would say, secondary.

14:48Because if you couldn't secure value, you couldn't even make any money, right? So number one is securing value. Whether that, you know, make sure that fits to the demand outlook you are seeing. Two is whether you can negotiate the best pricing you can. But given the overall pricing environment was still volatile, was still asking so much, it's hard to not take in some kinds of margin impacts because essentially, you try to get a chips, right? And you cannot just get too much away from your vendors. Yeah. I want to get more into the demand destruction element because I think this is very important to think about where the equilibrium is going.

15:25Just to clarify, what is it about AI specifically? Is it in the training? Is it in the inference? Where specifically in the process of building out AI services does this voracious demand for DRAM come from? So honestly, to be very honest, I think it's everywhere. Like, right, you know, AI training, you need tons of HBM. That just everyone knows, right? You also need a lot of the CPU DRAM. And that's LPDDR and the DDR that are going through the server. You also need HBM, right? So that's on the training part. In an inference thing, you also need HBM, right? And you also need the CPU DRAM to go through those workload, right?

16:04But the main one will be the HBM. And even the inference thing, right now, they are separating to pre-fill and decode. And I think 80%, I think right now, the most important thing will be decode, right? And decode is super, super memory-bound and memory-intensive. And HBM, the memory importance of there will only increase, to my understanding, especially given the long contents window continue to expand and the inference usage and adaptions continue to going up, right? So I think those are the typical way I would think about those, right? And the last thing is I think a lot of people talk about this day is about agentic AI, right?

16:38Agentic AI, you want to power those, you need a lot of CPU, you need a lot of CPU-based server, right? So what's including the CPU server? There's a lot of DRAM in there, right? And a lot of agentic world load also means you need a lot of inference as well, right? And that goes back to my previous point. You need a lot of HBM and DRAM, right? So I will say like kind of DRAM and I got HBM and DRAM is kind of everywhere. And we kind of talk about storage, but storage is all everywhere, you know, in doing this process. I have a basic question, but why does AI need a lot of memory at all? So you just explained, you know, it's important for inference, it's important for training, whatever.

17:16But like, why? Yeah, I think this literally like goes down to the model layer, right? Because essentially when you are trying to train, let's say you want to train the model, right? And then you just need a pure tons of data. You need to address tons of data there to train your model. And then for the inferencing part, you need one data after another to kind of compute one after another sort of a chain of sorts, right? Yeah. So to do that process, how can you keep the previous data, your process to keep it in there and to process to another data that's coming out after, right? So that's why you need a lot of memory.

17:48And go back to my previous point, right? The long context window basically means when the things that you are processing longer and longer, how can you have enough memory to process those? And one of the good examples, if you use TrigyBT, you guys definitely remember, back to TrigyBT coming out, people are just like, hey, what's the weather? Write a poem. People write now, or last year, people are doing like, hey, can you write a 20-page report how cancer can happen on people, for example. They will give a 20-page report and they will wait like five minutes, right? And so if you think about the process, like they do all the calculations, all the researches, and also give you the output tokens that the answer you get from your prongs, that's significant way longer than the prongs that you are given, right?

18:37And that's also a lot longer response that you are getting when you go back to the GPT-3. And remember, the usage we know back to GPT-3 was, I forgot the monthly daily users, But right now, I think the user for GTPT was 800 million, right? And we are not included the users for the Gemini, for Cloud, for xAI, right? So if you multiply those effects, I think the memory demands is very significant, and it's very clear to me. Yeah, it's really interesting. I saw some people were talking about this, but it really strikes me in my own usage of AI, just basically how much more I use it as capabilities have grown.

19:15Like it was sort of cute in late 2022. It's like, oh, get it to write a poem about this. And it's like, OK, then I would go, you know, several weeks sometimes without having anything to query. And then, you know, these days, like I'm using it all the time, like trying to do things with code or scrape data, et cetera. So overall token consumption has grown massively with increased capabilities. All right, let's talk about the demand destruction element. Any commodity is going to be, the price is going to be a function of supply and demand, and eventually prices get high enough such that certain forms of demand just are no longer economical.

19:53And it's like, you know what? We're just not going to make as many video games or cameras or whatever else uses a memory. Are we starting to see any of that yet, such that certain uses and consumption of memory? So Joe doesn't have to buy a Switch for his son. Yeah, that's right. Yeah, I actually bought a Switch recently. So the price. You should resell it. Are you going to resell it? I don't know. We'll see how the arbitrage there works. But I think we're already seeing a lot of impacts, actually. But it kind of vary in different kinds of products, right? In PC, we're surely seeing that. Dell, Nanovo, Anser, Asus, they are all having a price hike for kind of different products, right?

20:36And on the demand side, you're also seeing that, I think, for the Chinese smartphone market, a lot of the analysts, a lot of the research firms, a lot of the companies were saying they are cutting off their smartphone outlook. For example, I think MediaTek recently left earnings. They are saying they are cutting off the mobile to, I think, 12 % or 10 % to 15 % of the 2026 outlook. That's very significant, right? You're saying this year is supposed to be$100 million, but it will essentially go to$90 million. That's very significant. So I think we are already seeing that, but kind of very different places.

21:12Apple will be a great example here because they are saying, hey, we have all the price hike, but we actually managed pretty well. They are saying the real sort of meaningful margin impact will actually show up in the second half of 2026 instead of the first quarter or second quarter. I think those are sort of the things I'm seeing. So it's definitely already showing out in the market, right? I think what you can expect this year will be two things. One is sort of demand destruction in PCs, right? The border consumer electronics. Or the display, whether it's the memory display, that's very straightforward.

21:45But also the camera, right? The camera is also very important. I'm hearing some of the display on the camera as well. And, you know, also on the sort of what we call a delay launch, right? Because you don't want to launch your new products that have higher price than you initially thought, which will impact your new launch performance. Which usually your new launch product has better ASP, better margin compared to your previous generation products. I think those are the things we're going to see in the coming months, in the coming quarters. I think it will show impact and people will start feeling it.

22:16Joe, do you think we're going to get people stripping old Nintendos for memory chips? Like, remember how during the commodity super cycle in like the early 2010s, I guess, there were all those stories about people like stealing electricity wires. Yeah, it happened again in 2021, too, I think. Yeah, we might get some of that. Well, actually, on that note, is there like a short term fix for this problem, either in terms of design? Maybe you can make the products more efficient in some way so that it needs less memory or recycling old memory chips. I don't know. Yeah, I think, you know, from the demand side, it's a lot difficult, right, from an OEM perspective, right?

22:55There's only that much you can do, right? You can dispatch, right? But I don't think that addresses the fundamental issues, right? You are just dispatch your products and so you can ship. But there's this double-age sword, right? When you downgrade your products, that actually looks bad compared to some of your peers who doesn't dispatch, right? And make your product less competitive that impact your final sort of quarterly or annual shipment. I think what needs to be addressed or what can be done is probably on the supply side, right? So supply side, we mainly talk about a couple of vendors, right?

23:28The Micron, Hynek, Samsung. And for legacy ones, right, you have Winbound, Nanya, and China, like CSMT, and GHICC, right? Those people. So I think the most direct way right now this year, because this year, the really challenge is there's a clean room constraint. So clean room constraint is basically you have limited space that you can put all those equipments and start manufacturing chips in the fan. So you have clean run shortage in all three major memory makers. So your incremental wafer capacity coming online for 2026 will actually be quite limited. So in that sense, how can you produce more bits, more DRAM bits, while having only limited incremental wafer capacity?

24:08The only thing you can do is non-migration. So in DRAM process now, you have the most advances. right now is 1C, going down 1B, 1A, and 1X, blah, blah, blah. I think right now they are trying to do it to migrate as much as they can to 1B and 1C, which on the same wafer basis, they will produce a lot more bits compared to the legacy node. So doing that way, on the same wafer basis, you can produce more bits. So that should produce more supply even though your wafer capacity this constraint. But the challenge is two things. One, how fast you can run about this non-migration. Non-migration is typically difficult.