WFE to the Moon (Still), Semicon West and OCP Insights

0:05Ben Bajarin: Hello, everyone. Welcome to another episode of The Circuit.

0:09Jay Goldberg: I am Ben Bajarin. Greetings, programs. I am Jay Goldberg.

0:13Ben Bajarin: Well, this was a double event week combined with some juicy earnings and a lot of busyness on both of our parts. This week had Semicon West in San Francisco and OCP in San Jose, which next year Semicon West is going back to Arizona. And OCP is going to San Francisco to be in the – On different weeks. Thankfully, different weeks. Thank you. Yes. So we can all be insane in now multiple weeks instead of one giant insane week. There were also three analyst days or three analyst days-ish buried in there too.

1:04Jay Goldberg: Like it was an impossible week.

1:06Ben Bajarin: Impossible week is an understatement. But we were there. So we're going to lump this into a couple categories. We're going to talk about semi-cap slash wafer equipment, which kind of will combine ASML, earnings, TSMC, conversations we had at Applied Materials and their investors day. And then we'll talk about OCP on the back of that and the major themes that we saw. So, you know, one thing that stood out to me, and I've sort of been saying this fairly frequently for, in fact, we've titled multiple episodes this, but I do find the semi cap part of this cycle very interesting because when you listen to those making the goods or building the clean room, you hear a very interesting side of the story.

1:58Ben Bajarin: You hear them talking about what's going on in new equipment, where innovation is needed, where that wasn't necessarily the innovation pipe for tooling and machining before. We'll just say in the kind of monolithic designing a mobile and a laptop type chip that as we push radical limits and try to develop these giant AI accelerators and all of the things, including much larger and getting larger networking ASICs. So let's not forget that networking ASICs are going to have a whole lot of advanced packaging on them and be bigger, not just GPU XPUs. You hear this interesting theme from Semicap saying, we're seeing the challenges in scaling this industry to meet the demands of those who want to use our fabs as fabless or use our tools in a fabless semiconductor and are investing like no real cycle we've seen before.

3:02Ben Bajarin: And that obviously manifests itself in commentary from ASML, in commentary from TSMC, and really in this commentary from Applied Materials, where you could see the underlying, I guess, this new foundation emerging for the semiconductor industry who will be able to build very different types of silicon as we scale clean room space and we see innovation in these tools that we haven't seen before. And I just find that kind of like an interesting, regardless of what goes on in the market and people's emotions about finances, the semiconductor industry is laying a foundation for a very different shape of capabilities of silicon on the back of Semicap.

3:56Ben Bajarin: And I just, for all things considered, I find that an interesting observation and not really one we've seen in some time. Perhaps the only historical precedent is when we were chasing Moore's Law regularly and doing node shrinks on an 18-month basis. Material and testing and lithography was a big part of that. And now it's like it's back in full force and everybody's talking about having to go and innovate, solve problems, and even do new machines that they didn't have to think about before.

4:31Jay Goldberg: I'm trying to refrain from talking about the market. because there's a lot of bad news in the markets this week. And your point is that that won't matter because there's so many interesting things going on in semiconductor land right now. And I think that's a fair point. I think there is... Like, semi-cap is, to me, is a lot of fun because there's so much innovation and there's so much nerdy technical detail there. And ask a semi-cap toolmaker how they lay individual atoms of some molecule down one by one. Well, not one by one, but evenly. It gets to some really interesting conversations. So there's a lot of cool stuff happening in semi-cap technology.

5:22Jay Goldberg: There wasn't a lot of news, I felt, out of Semicon event. I don't know if people break news at conferences anymore, but there wasn't a lot of news. but what little I was able to see of it, it was pretty fascinating. There's a lot of really interesting things taking place.

5:37Ben Bajarin: Yeah. Yeah, I think the underlying just theme in general was, you know, the industry is gearing up for growth, right, depending on whose numbers you believe it will cross 300 billion in total TAM somewhere around 29, 30, 31-ish. I mean, I seriously doubt it takes longer than 2030, But that's most people's lines, which is, again, a structural changing of this industry where that kind of tooling is being invested now in a whole lot of new cleanroom space. But like I said, I think this came from Applied and not that ASML is in all of the pockets that someone like Applied is. But even their commentary was the same in terms of their need to ramp and build and bring on as many new equipment as possible because their customers are basically lining out the door for ASML machines through the end of the decade.

6:40Ben Bajarin: It is just a structural change in Semicap. And I just think there, you know, I pointed it out, but I think there's a lot of implications to that observation that haven't been wrestled with around the shape of the semiconductor industry just being drastically different going forward than really anything that we've seen before on the back of all of this new innovation we're seeing and how these things are made and tested and how wafer yields are done. you know, advancements in wiring, like all of these things that the industry didn't really have to do before. You know, they kind of did, right? I'm not saying like it didn't want to innovate.

7:24Ben Bajarin: It's just these are different problems. And semi-cap, you know, wafer equipment, the testing, they're all gearing up to meet very challenging demands that didn't exist before in designing semiconductors. I agree.

7:39Jay Goldberg: I think there's a few news items I think we can just hit real quick that I thought were worth noting. One is a little ahead of Semicon, TSMC, said they're moving to a larger reticle size, which is – the reticle is the size of – whatever, look it up. The reticle is the size – the maximum size of a chip on a single wafer, right? And they're going to bigger reticles, which sounds – in some sense it sounds kind of small, but it's going to cost, I don't know, tens of billions of dollars of R &D and retrofitting and everything to get that to work. And it's important because the chips are getting bigger.

8:20Jay Goldberg: We want bigger and bigger chips. And we're – a light constraint on some of this is reticle size and you start having bigger reticles, you can do some very powerful things. So it sounds like a relatively small change But it's going to cost a lot to get it to happen And it's going to enable Bigger and bigger, better, more complicated chips So I thought that was cool I spoke to someone who Had gotten orders from TerraFab Right? So Elon's out there talking about building TerraFab A lot of us are kind of scratching our heads going What does this mean? Like nobody really knows what it's going to entail TSMC might rent some capacity Intel might manage it maybe none of those things but like it's actually they're actually putting in orders for tools they won't get those tools for a long time but they're putting in real orders for tools so it's that's

9:09Ben Bajarin: there's some some surrealness there i heard there was some sourness though over this news like some people were like why are you giving him machines when this whole thing could be a farce when we're going to use these machines like there was some drama that came out of somewhere i forget where

9:25Jay Goldberg: Yeah, but they don't understand. He has flying cars, so.

9:30Ben Bajarin: He has cars that can do very fun tricks. Yes. Yeah. He's Elon Musk. That's why, I guess, why he's going to get some ASML machines.

9:40Jay Goldberg: Yeah. I mean, yeah. Like, I'm sure the other side of the conversation is, he's Elon Musk. Why isn't he getting more machines? I have no stake in this. I'm just saying there's something actual tangible happening there. It's the first time I've seen anything real happening there. Applied Materials had the grand opening of their R &D center, their Epic Center in Santa Clara. Ben and I were both there. It was a very interesting event. They had a lot of very high-profile people. The CEO of Hynyx was speaking, and I got a phone call while I was there. It was an important phone call that I had to take.

10:15Jay Goldberg: And I was afraid to leave the room because I don't want to anger the memory gods. So I waited and called the person back, but it was like very high-profile people, applied materials. It was very cool that we could go there. I would just have one sort of constructive criticism is more tools. Like we would love to see more tools than were on display. It would be like to me that's like seeing – I would love to see a room filled with all the tools and robots running around. And that would be fun. So next year maybe. so you know there's a lot happening in this space i think the other the other thing is tsmc reported this week and i i've i've given up trying to track what expectations were i guess it wasn't what people thought i don't know numbers look fine to me um but i they they said a few interesting things on their call right they they talked about terra fab 2 but they kind of played it down uh i I thought the most fascinating thing they said is they actually have their own employees going on site to data centers now around probably the U.S., maybe around the world, checking to see how far along these fabs are, which I thought was very fascinating.

11:31Jay Goldberg: Right. Here's a here's a foundry. They are multiple layers removed from data centers, but they actually have people doing site inspection, trying to figure out what's what's real, what's going to get built on what time frame, which is. Is a is a big change from the world of the past. Right. I remember the days not so long ago when TSMC would just call up the cell side and say, what's your forecast for handsets? what's your forecast for PCs, and that would be their modeling. They're much more engaged now, as I think they have to be in trying to figure it all out. And then everything else, I think, from their earnings is pretty much as expected, right?

12:10Jay Goldberg: All the timelines are the same. No big changes there.

12:14Ben Bajarin: But it's TSMC, so it's good to see that they're chugging along. Yeah, I thought the data center point, to be honest with you, was probably the most interesting. But as a broad thematic for TSMC, I just, again, right, I'm trying to, Ben's not in the market. Ben doesn't recommend stocks or try to really analyze the stock market. I'm trying to keep these things separated because TSMC is the one company. I mean, I could argue there's many. But I would just say historically how this company operates and its management operates, this is a very conservative company. And this is why as a bellwether, having them go and say the things they're saying, that they are talking to customers, they're in the market, they're doing these data center checks to make sure that we have the power to energize the gigawatts that they're going to make and sell.

13:14Ben Bajarin: and they're still investing. They didn't announce a CapEx increase, but people assume that they will before the end of the year or in the early quarter in order to build more factories, more clean room space. This is a company that cannot risk having factories sit empty or underutilized. We've talked about this before. So the fact that they're doing this, to me, has to be the most bullish cycle for demand because they would not risk it. They just, that's not how they operate. And again, I think that plus my points on Semicap, just you have to wrestle with an entirely new foundation of the semiconductor industry being built on the back of what people are convinced and have deep conviction is, is a growing demand profile, not a overbuilding profile, I guess is my point.

14:13Okay.

14:14Jay Goldberg: I'm not quite there yet, but we can debate that on a different day. In terms of demand, there's a lot of – yeah, I'll save it for another time. But I certainly agree there's a lot of – there is a lot of demand right now. It will stick around for a while. And we need – I think the way I look at it is we underinvested. The industry underinvested over the last – call it five years. And now we're playing catch-up at the very least and have to get someplace very new. I think the other interesting thing from TSMC was they said that they're converting N5 fabs to N3. Right. And we touched on this a little bit last quarter.

14:59Jay Goldberg: I know Ben Thompson on Stratechery talked about this a lot. That's a new thing for them. Usually they would just keep all the N5 fabs running and then build new N3 fabs. and they're actually retrofitting the N5 to N3, something that's new for them, three nanometers in big demand. I guess it's going to be that way for a while going forward.

15:23Ben Bajarin: Yeah, which raises the question I think we've either asked or noodled about or discussed in private, what happens to all of the demand for five nanometer and below? Because it is still there. There is still demand for, let's just say, trailing-ish edge. And as TSMC does this, that creates opportunities for others, likely global foundries, who again is raising CapEx and moving to new nodes and benefiting from this cycle that once wrote off is now seizing an opportunity as well.

16:01Jay Goldberg: I think the – if we talk about the trailing edge mature processes, that's a different conversation. There's a lot of factors pulling on that market. Again, a topic – a bigger topic for a different day. But I think trailing edge is going to be less exciting economically.

16:25Ben Bajarin: Oh, for sure. Right.

16:28Jay Goldberg: But it's still needed. Yeah. Demand is still very strong. There's an awful lot of that capacity coming on stream everywhere, especially in China. And how that all works out is still to be determined. Right? And certainly that wasn't like – for all of the new fabs coming online in China, nobody really talked that much about trailing edge at SemiCon. I think people wanted to talk about new things.

16:56Ben Bajarin: I will say there was –

16:57Jay Goldberg: I will say there's – go ahead.

17:00Ben Bajarin: I was going to say that's what – new leading edge or new nodes is what drives new equipment. So it's like that's kind of why I think Semicon West talks about it. It's good for equipment makers.

17:12Jay Goldberg: But I've been reading up about this. Trailing edge equipment is in shortage too now. In fact, in China, there's a very healthy secondhand semi-gap tool market. And that has gotten so tight that people are stealing equipment. You're a broker. You sell used etch tools. You have a warehouse somewhere. Somebody pulls up a truck. Oh, my gosh. Presumably with 10 guys and a forklift and steals equipment. That's how bad, how much demand there is for this equipment.

17:47Ben Bajarin: The crime has moved from robbing banks and other things to robbing semiconductor equipment. Yeah. That is the day we live in. Yeah. Yeah. Interesting stuff, I mean, for sure. But like I said, I think there's a lot to noodle on around the shape of the semiconductor industry post-28, 29 on the back of all of this additional capacity and new tooling, innovation and process and radical limits and packaging. and all those things, it's going to be super interesting to see.

18:31Jay Goldberg: I think there's a couple other things I picked up and that I think are worth mentioning. One is ASML is apparently going to have a new tool to do hybrid bonding. Right. Which is interesting because we usually think of ASML as strictly a lithography company, but they have other products and they're getting into hybrid bonding. I haven't dug too deeply into this, just haven't had time, But my sense is that they have a ways to go. It took, you know, it took BESI 10 years. AMAT took 10 years to get to where they are. So I think that there's still a little bit of learning there. But it's interesting that ASML is branching out in that way.

19:10Jay Goldberg: Certainly hybrid bonding is here to stay and can be more and more important. But that also leads to this sort of interesting debate that's taking place in the memory world, which is how high do you want to stack HBM? And this is like the one place, like there are a few, like I said, Applied Materials had an analyst event and they had a breakfast for broad audience and they had a separate Q &A for the sell side. And digging into those weeds, this is the one place where I see a little bit of light in between my view and what Applied is saying. because they seem to be very bullish on this idea that we're going to hybrid bond HBM wafers to each other to get increasingly high stacks of HBM, which is very, very different than what we heard at Hot Chips back in August where people kept saying, no, no, no, don't do that.

20:03Jay Goldberg: And when I say people, I mean like end customers who are buying HBM. They're like saying, don't do that. Don't stack it higher. Just give us more of it. And Applied seems very, you know, pretty certain that we're going to get stacked HBM. I don't think I agree, but like, you know, they're – they have good reasons for thinking it's real. I'll comment on this only as an observation that is a, you know, let's watch for this because a question did come up from our friend Vivek Arya on this of B of A and he basically asked them, you know,

20:41Ben Bajarin: this point, right? What's going on with 12 high 8, high 4, et cetera? And they made a comment that I think is worth watching and thinking about, which was essentially that it depended on the workload, that there are workloads and they erred toward a training workload, which has and can, I guess, better utilize higher stacks of memory for its advantage. And then there's other workloads, call them light inference workloads or maybe medium reasoning workloads that while they could use 12 high are not going to use it at capacity effectively for, you know, call it sustained use. So I don't know the answer to this because I think there's all sorts of variables.

21:24Ben Bajarin: But what I would say is I think this is a model point. like where models go can either necessitate higher stacks of memory or not like that's a that's a software the software has to take advantage of it if it can't take advantage of it then it doesn't make any sense if it can take advantage of it then i can see right a point now again is that every workload is it 20 of workloads that can take usage of 12 or more is it more or less like that we don't know but but i think that was kind of how they answered that i've heard others deep in the weeds in this use say say a similar point um so i would just say dig into that and keep an eye on the workload and whether the workload can demand a a higher stack of memory or not um as a greater proportion of how it's used on a on a on a utilized on a daily basis by the masses.

22:25Jay Goldberg: Okay. I'll, you know, we'll, we'll, we'll see.

22:32Ben Bajarin: Before we go into OCP, I wanted to share an anecdote that I shared with you and I want to share with everybody. I, I witnessed a fairly fascinating event in San Diego this weekend as I was walking in the neighborhood with, my child who goes to school there.

22:54Ben Bajarin: As any good technical person or analyst does, you walk past somebody chatting and they say a couple buzzwords that catches your ear and you hone into it. I heard Data Center get talked about by these two gentlemen who were having a conversation. One guy was pretty upset about this and the other one was talking reason and trying to talk about the benefits of Data Center. Having a good, a calm conversation, but clearly some emotion involved. And, and I turned around and just kind of looked and the guy who was, you know, in, in the, you know, pro data center camp had all these pamphlets and it hit me that this person is evangelizing data centers door to door in this neighborhoods of San Diego.

23:38Ben Bajarin: And I thought this was fascinating. I really wish I would have gone back and been like, you know, give me your story. Who's behind this? Like, is this the energy companies as AWS? Like, who's paying you to do this? But I just, like, sign of the times again, right? Like, that weird timeline we're in, that there are apparently people door-to-door evangelizing the positives of data centers to communities because of the community backlash. So I thought that was very interesting and entertaining at the same time.

24:15Jay Goldberg: I wonder if that guy will still be doing it after the election.

24:21Ben Bajarin: It's a good point. I mean, I don't know. I mean, I think as long as there's this kind of hostility, I can see these types of evangelistic things trying to just help communities because where else are they going to hear? I mean, think about it. But you're right. It feels like an election issue because this is the kind of thing you do to educate people who you want support for for things. This is playbook 101 of how to change minds. But it's just interesting to me that that's the tactic. Because again, where are you going to hear it? You probably won't see a reasoned argument on Facebook or Instagram or the news.

25:03And so it's actually an interesting medium to try to have more informed conversations.

25:08Ben Bajarin: So you're right. I don't know, but I found that an interesting tactic that's going on to attempt to soften communities around data centers. You know, the show Peter Dore, it's not a religious person, right, or somebody selling something. It's a, hey, did you hear about how good data centers are for our city and your technological future?

25:31Jay Goldberg: Brother, can I give you the good word of Sam Altman?

25:35Ben Bajarin: Of data centers. Of our intelligence future. Okay. Anyway, I thought that was a fun anecdote. Okay. And no one in my family is in technology, so I was the only one that appreciated this situation. But there you go. Okay. Let's go to -

25:53Jay Goldberg: That's why you have a podcast. That's right.

25:55Ben Bajarin: That's why I texted you immediately. Yeah. Okay. All right. OCP. I actually really like wait wait

26:05Jay Goldberg: before you start talking about the show can I just flex a little bit day two of OCP I got street parking really yeah I got street parking

26:22Jay Goldberg: it was the best guys it was the best it was so crowded

26:27Ben Bajarin: yeah it was I mean I mean, just to even give you more of us for anyone who's been involved in an event, you can sort of judge how hard it is to get an Uber by like the density of something. And I had to wait 25 minutes to get an Uber. In fact, I didn't think I was going to get one. I was like, this guy doesn't want to drive me 15 miles away because he wants to just – they just want to do his little short drives. Like I'm going to be stranded here and have to call a family member to come pick me up from downtown San Jose. and that was how busy it was it was it was it was it was gtc level busy which was pretty interesting yeah not quite but but pretty close and gtc was almost 30 000 people so um all right so i actually really like this show um for a lot of different reasons you know the open compute project is really something that is standard space which i think is good so like you talk to a lot of people on the floor, you talk about rack design, you talk about cohesiveness, and they're talking about, you know, that this is an OCP spec.

27:32Ben Bajarin: And so it makes sense, right? You want an industry to have, you know, cohesity across its ecosystem for support and growth and standards, even though there's no standards in optical yet, but we will probably get there. But I like, like, having these kinds of conversations, especially around some interesting pockets, some around networking, because again, there was a lot of what's going on in ethernet, what's going up, what's going on in different types of networking that is standardized, as well as what was going on in liquid cooling, which I thought was super interesting because 100 % that actually does need more ubiquity than it has today.

28:15Ben Bajarin: And then the 800 BDC conversation that was starting to come up with, again, pockets of fragmentation but but these things need to be standardized so data centers can install them operate you know all of the things but i like again right my point is this show is around people trying to build together on standards on a on a baseline way of doing things so that there is interoperability as far as that can go one way or another with differentiation so that's like the center of it and when you would talk to people and their themes of like yeah it's ocp compliant or we're working with ocp to make this more of a standard for things like the one i thought was most interesting which is the shift from copper to optical back planes which once you go to optical in a rack whether that's compute or asic you're not going to have copper on the back side of that you shouldn't uh you're going to say you want to slide this tray in with an optical back plane it It has to align up once you slide it in.

29:14And so these were interesting discussions talking about future rack design for scale up and scale out type of networking compute domain.

29:23Ben Bajarin: So that was just one example of rack compliance, size, standardization, the way in which you plug things in. Anyway, that's extending now also to liquid cooling and 800 BDC, which is positive, positive for the industry.

29:44Jay Goldberg: I, so just so everyone knows, we're recording this late at night, so, and it's been a very long week, a very long weekend for me, so I'm a little extra crunchy. Spicy. I liked, spicy. I liked OCB. I thought this was a great conference. I think it's, it was very good to attend. I know some people were frustrated that there wasn't a lot of news.

30:10Ben Bajarin: Yeah.

30:10Jay Goldberg: But just the attendance and just the quality of attendees was really, really good. Like I would just be standing in a booth for whatever random company and the CEO would show up and start talking to people. And like not investor meetings, just like he was coming into either his booth or some other booth to talk to somebody. It was a lot of – it wasn't a pure marketing fluff event like some conferences have become. there was a lot of substance there and a lot of really good people there who knew what were going on. Good, I mean like people who knew what were going on. You could go into a booth and ask a technical question and get a serious technical answer from most of the companies, which I thought was very useful and very important.

30:58Jay Goldberg: I'm not a big believer in OCP ever becoming a standard. Most people I talk to tend to sort of – they view OCP as being useful for other things. But I don't think anyone ever really adheres to the standard itself that comes out of OCP. It's a good guideline. And my example for this is like OCP really started off by standardizing rack design. That was an important thing where everybody's like this is how wide the racks are going to be. This is what a U is. Like, I mean, we had some of that before, but this is kind of, there were some, they weren't really standards. They're just like common patterns that people adopted.

31:36Ben Bajarin: Yeah.

31:37Jay Goldberg: But then like the number of companies I talked to who had these really weird sized rack trays, it was mind boggling. Because we're now moving to like, you know, NVIDIA, Kyber, and AMD Helios are double wide racks. And people are starting to say, oh, well, if we can double the width, let's do this. And so I started to see server trays that were sort of one and a half times a server tray width or two-thirds of – like all these weird configurations started to show up. And like there was a Chinese company that makes trays, and they're like, you just give us the measurements. We'll make it whatever size you want, which is – right?

32:16Jay Goldberg: Yeah. So –

32:18Ben Bajarin: Yeah, and in the grand scheme of yes, there are a handful of companies who can do that. The other example that we talked about, so the Microsoft Maya tray was a different size tray. Those who have their ability to do those things, yes, but then they're still in those rack size, whether that's the connectors or the ports or whatnot. There's still some – you've got to be able to scale these things, right? You've got to be able to meet demand. So there's some level of – it's a – you design to a spec. And then there's other parts where you've got, you're right, flexibility for those who have. But something, again, like I think 800 BDC and to some degree, right, direct chip liquid cooling, which you talked to everybody who was there.

32:59Ben Bajarin: We're talking about the need to co-design the liquid cooling plate to the chip design, which is true, right? You've got different sizes. If we're going to push reticle limits, right, it's got to push sizes. You design to that spec size. In case in point, a lot of those examples that you saw in a liquid cooling sized template was a Broadcom Tomahawk 6 chip. It was the exact same size because they are building to the spec to cool, liquid cool Broadcom Tomahawks, right? So it's like – it's just things like that that the industry needs to scale. It needs to design to a spec. And so there's areas where you do that and then there's areas where there's flexibility.

33:41Ben Bajarin: So I agree with that. I will say, though, I have one concern. And, you know, Ben's going to play, I don't want to say I'm pessimist, but something. Think about what we're literally on the back of right now, right? You're talking about the future of the data center and you go, yeah, it's going to be direct to chip, single phase liquid cooling, designed to spec. It's going to be – which includes these insane manifolds, which I took pictures. You should see on Twitter. and it's literally, it looks like a car engine to a degree, right? That's cooling anywhere from a couple hundred megawatts to, or a couple hundred kilowatts to the largest one from Delta, three megawatts, the largest, which was a huge, giant cabinet.

34:35Ben Bajarin: But you talk about the future of the Air Center and you go, it's going to be optical. It's going to be, like all optical, It's going to be liquid cooled. And then it's going to be 100 VDC. I don't see how anybody in the next couple of years can do all of those transitions at one time. Like, you're going to need to pick your transition. Which one am I doing? Am I starting with liquid cooling? Because that's already an entirely new design. Am I going to then figure out how to add optical in there? Again, another design. And then I'm going to throw in these sidecars and have to do my step change of 800 BDC, another transition.

35:13Ben Bajarin: It just seems very unbelievable for everybody who's like, look how big this market's going to be for all three of these things, that those transitions happen at the same time. That three – those three fundamental transitions happen at the same time. I just don't see the reality of that.

35:31Jay Goldberg: So let me start off by saying It was very noticeable that Consistently the most crowded booth At the show for all Two, three days Was the Delta Electronics booth Which is confusing to me But like it makes sense in the context of that show Everybody wanted to see this Mammoth cooling system But yeah Did you notice that was the most crowded booth

35:56Ben Bajarin: I got elbowed in the chest Like multiple times I had to throw elbows to get anywhere near this thing.

36:03Jay Goldberg: It was packed the entire time. I've seen other booths that crowded, but usually that's at like 530 and they're serving free alcohol. This booth was packed from the beginning of the day until the end. Everybody wanted to look at it. I think, though, that your point about timing is actually my – I walked away from the show. That was the one sort of biggest concern I had on all of this is just how long some of this is going to take. co-packaged optics is going to take a while. 800-volt data center is going to take a while. Cooling is – all these big liquid cooling systems are going to take a while.

36:39Jay Goldberg: It is – it's a lot. I think you're right. There are going to absolutely be people who are going to pick and choose. We'll do one. I mean the problem is some of them you have to do all at once. Like how do you have an 800-volt data center that's not liquid cooled? You need – you need – same thing with co-packed optics.

36:57Ben Bajarin: Optical, right, for example.

36:59Jay Goldberg: Yeah. So I think that is partly why Delta was so crowded is I think most people were realizing they're going to have to go. If anything is going to get accelerated, it has to be the liquid cooling stuff and all the other things are going to be contingent upon that and have to wait. But I definitely, like I spent a lot of my show talking to people about 800-volt data center. I spent a lot of time in TI's booth and in ADI's booth and looking through all their parts. And in both booths, I got elbowed by someone who was almost certainly a spy, like for another foreign country. Same. 100%. Spies were abound.

37:40Jay Goldberg: There were a lot of spies around.

37:45Jay Goldberg: And, yeah, there was – for me, it was interesting because I saw for the first time like real products. one of the companies I talked to was explicitly saying that they had a design win with an NVIDIA system they all have this sort of part all these analog companies have this part that fits it's very small it's a lot of chips this system, it's a lot of chips but they put them on a very very small board and that's so it can fit in a compute tray and these guys were pretty clearly had won some kind of design more than a qualification, I think they actually were starting to talk about volume for their part in NVIDIA systems in the future.

38:30Jay Goldberg: But like timing and all that is 27 a little bit and really it's 28 for 800 volt data center. It's big, it's coming, it's just going to I think we will very likely see growing pains where people start doing it and then they realize, oh, we didn't think about this or we're having faults on that and we have to slow it down we're gonna have some fits and starts here yeah i i want to

38:57Ben Bajarin: come back to the size of these things because what that got me thinking about which is actually a networking point when you talk to everybody about you know the reach of copper slash optical what's going on you know east west and you tell them talk about you know we're going to go up to you know much faster speeds you know needing to go east to west but picture picture that distance conversation is largely being had without a three to four foot wide manifold cooling system sitting in that rack in that row so you're sort of like okay yes you want to cluster four or five or how many of these things together but you're also in row going to have a very large cabinet and then another one when you want to do an 800 vdc sidecar like okay you're running out of space for for all of these racks they they take up more space so it's sort of like if if data center space is the equivalent of clean room space and you're trying to get your dollar per square foot you have just introduced two pretty large things that that solely exists to keep it cool so you can again crank more tokens per watt out of that thing but you're losing rack space by adding these gigantic manifold coolers and 800 bdc sidecars to the equation of your rack

40:24Jay Goldberg: row okay i'm picturing somebody two years from now saying hey we we decided to go with copper we we use all this fancy electric active electric copper because we thought you know we needed that extra distance we need that cable that could go two meters but we forgot about the sidecar we actually need three meters and so and so it turns out we need optical after all yes yes that's somebody that's gonna happen somebody's gonna be like oh we forgot about the sidecar and we gotta

40:57Ben Bajarin: well and and again as cool as that delta three gigawatt one was that was probably eight feet wide like that's not a small cabinet to be in a in a row and sure the as you can see i'm gonna hear my point though the benefit is one of these that can cool more means you need less of them so i get the benefit it's just it's just another networking topology you have to consider in the grand scheme of your square foot as you add these very large manifolds so again hence you need networking it's going to add distance to your compute pods. And so you've got to deal with all of these things. You know, the other thing I heard from a lot of the vendors on just again, a point about OCP was maybe a little bit more.

41:49Ben Bajarin: I mean, I don't want to say this doesn't happen at Computex and probably the best way anybody should think about the show is kind of like a mini Computex, not quite as big, but a lot of the same kind of cool stuff thematically vendors. But I heard from the vendors that there was a lot more deal making happening at this show that they were engaged with customers having real engaged ordering types of decisions like what do you need um that wasn't necessarily say the same kind of case at at at computex so i thought it was interesting on the back of the vendors actually got a ton of value from this um and i thought again like i i heard I won't say who, but I was at a booth for a very popular technology doing CPO, and I saw one of the largest OEMs, data center rack OEMs, basically say, we need to talk.

42:48Ben Bajarin: Let's have a conversation. We need to do a deal.

42:51Jay Goldberg: I'm like, hey, okay.

42:54Ben Bajarin: My snooping ears win again. But that's what was happening here. Like people were going and seeing things being, yeah, I didn't know that existed. Like let's have a conversation. I probably want to use this in my technology or in my solutions.

43:10Jay Goldberg: I'm not sure how much of you anonymize that, but yeah. No, it was a real show. Like there was a lot. I'm not surprised to hear that there was jailed in making complaint because all the principles were there.

43:20Ben Bajarin: Yeah.

43:21Jay Goldberg: Right? And I don't know. I still haven't gone through the sessions, the technical sessions. I haven't either. I made it a point not to attend any because all the videos would be online. But I keep hearing the ones that were most crowded were 800 cooling and copper. Everybody wanted to talk about copper. And I think that was to me the sort of big mystery that came out of the show was – right? There's this, just as a refresher for everybody, optical is coming because copper, we're reaching the limits of copper. And the problem is every, we've said this before, we've had this conversation before.

44:07Jay Goldberg: We're like, oh, that's it. That's it for copper. It's time to go to optical. And every single time somebody pulls out another trick to make copper go a little further last a little bit longer. There is a very strong preference for using copper. It's just much simpler. And I think there were a few of those at the show too. From what I can tell, they're not perfect solutions. They get pretty complicated pretty quickly. So you don't really have this, it's not really true copper anymore. But I think to me, that was the big mystery was this debate about sort of the timing and sequencing of how much further can copper go?

44:46Jay Goldberg: Can we push it to another generation? When does co-package optics available? Are we going to do something in between like NPO first? I thought I was going to walk away from the show with a very clear answer on that, and I think it was the opposite. It just got much blurrier for me. Yeah. Although I did think of a really good analogy for copper versus optical. Can I share it?

45:10Ben Bajarin: Mm-hmm.

45:12Jay Goldberg: Okay. Let's say you got some money and you want to buy a nice car. You can buy a brand new, call it Alfa Romeo. It goes really fast. It's beautiful. It's the newest thing, right? But it has a lot of electronics built in. It's very complicated. You don't know how reliable it's going to be. You could also, instead of buying the fancy new Alfa Romeo, you could buy a classic 68 Mustang, right? And you know that that Mustang, you can fix it. Like you can get on YouTube and figure out how to fix it yourself. You have all the parts for it. It's very straightforward. It's not going to last forever, but it's still, you can get a lot of, you can go pretty fast.

45:53Jay Goldberg: It's not going to go as fast as the Alfa Romeo, but it's going to go pretty fast, right? So that's copper versus optical. Take it a step further. You've got a lot of money in your fast car. You're going to buy the Ferrari, right? You can buy a Ferrari. It's going to go really, really, really fast, much faster than the Mustang could go, right? But it's very complicated. It's hard to buy. There's only one way to buy it. So that's co-packaged optics, right? You've heard reports that it's actually much more reliable, but you don't know that, and you know that, like, fancy modern sports cars aren't super reliable.

46:34Jay Goldberg: You saw on a Facebook post that it's, no, no, these new Ferraris are super reliable. You don't know if you can trust it. That's co-packaged optics. That's my metaphor.

46:45Ben Bajarin: Yeah. And I think when you see these in person, to be honest with you, like when you see a NPO slash CPO solution, and really I'll just go with CPO because A, there's a Tomahawk solution that shows this. It's still serviceable. right you can still clip the lasers on but it's basically over it's it's on top of the edge of that surface area and then same for compute it you look at it in your you like you realize how much more challenging that is than any form of its copper alternatives in in complexity and how how it sits on that chip like it's it's it's it's a head scratcher honestly every time you look at it as to how are you really going to scale this.

47:34Ben Bajarin: But again, we're pretty sure we'll get there. At this point, I don't know how much I'm going to say. People are trying to solve hard problems. We'll get there.

47:46Jay Goldberg: We'll get there because eventually the Mustang will wear out. It just can't last forever. Eventually, we will get there to go back to optics.

47:53Ben Bajarin: Let me make a contrarian point on this, though. Let's just say that copper can get to, I think, what is it, right? People have confidence in 800 gig, and I think they're talking about it still in 1.6. But let's say that these compute domains we talk about actually take off. And you're now able to have three, four, five racks together function as one compute brain, able to run a massive model in parallel, which is really the main reason why anybody is trying to do compute density in a rack so you can run a bigger model than you can run today in just that one rack because that's largely how this works.

48:40Ben Bajarin: It's served and inferenced from one rack's compute. And so you can now bundle four or five of those together. Well, why, if that's the case, would I want to increase my GPU density in one rack at which point maybe copper is going to be okay in north to south for much longer as you can connect scale-up pods in east to west and run those larger models across three or four racks instead of trying to do it in just one. And if that's the case, I think you and Broadcom actually has a very strong point of view here about how long they see Tomahawk being able to increase with their 30s to do copper in a rack.

49:20Ben Bajarin: and you'll still get more jet in bed city than you have today. So you could maybe go to 500 GPUs. But even, you know, you're doing, theoretically, you're doing 144 today too, because right, that's 72 packages of 144 GPUs. And so, you know, say that you can scale that a little bit more and now you're, again, you're networking all these racks together. you could make an argument that maybe I don't need that much density in one rack and then therefore maybe copper stays around longer.

49:58Jay Goldberg: I always like a good contrarian argument. I see your point. I think though they're just getting so big, right? I mean people are talking about million node worlds. That's on people's roadmaps. I just don't see how you do that without optical. Right? It just gets... Yes, yes, yes.

50:24Ben Bajarin: But that's east to west. I'm saying like what if... The assumption was copper goes away north to south. What I'm saying is what if this scenario is you get that density by these fully connected nodes and therefore you can keep... There just becomes a reasonable limit of how many GPUs you have in one rack, and you can actually maybe still do that in the true scale-up north-to-south domain. And then you connect that to the one next to it with optical, which is going on today. But that's a bigger brain now. It's not just limited to the GPUs per rack.

51:01Jay Goldberg: Don't you think the GPUs themselves are getting so big and so fast that the data rates overwhelm copper?

51:09Ben Bajarin: But that's the theory. But again, the thesis for that was I want to pack as many GPUs into one rack so I can serve a large model per one rack. My point is if you could do that on two or three and four racks easier with more reliability and you're increasingly scaling up those domains, maybe the argument is not. Because think about like we have this conversation about Kyber. Invita didn't show Kyber. So people are like, oh, what's happening to Kyber? Does it have the same problems? you are really adding a whole nother level of complexity when you start packing that many gpus into one rack and so i'm just saying what if the engineering trade-offs aren't worth it just network two or three or four of these together and you get the same thing and then you can let you you can have these engineering problems not be so hard as trying to put 500 something or more gpus in just one rack

52:01Jay Goldberg: I like that because it tees in nicely with what we were saying before You can't do all these things at once You're going to trip So maybe that's the way it goes Copper stays around for north-south a little bit longer Maybe Although I will say the other big news That came up from the show Was NVIDIA talked about network they had a networking session and they showed their ruben ultra yeah deck and they said yeah pod yeah and they said that would that would have scale up co-packaged optics yeah and so that that brings everything forward because that would be next year yeah right ruben ultra is next year most people i'd assume that co-packaged optics wouldn't come until fineman which is in 2028.

52:53Jay Goldberg: And so NVIDIA seems to be hinting that it will come sooner than that. Now, personally, I don't think that's going to be the volume networking stack. I don't, it's not possible. Like there won't be enough parts.

53:07Ben Bajarin: Yeah.

53:07Jay Goldberg: Right. It's a mix. It's one mix. I think it's sort of like they're putting their toe in the water. They're saying it's going to be capable of it, but I don't think many people will actually deploy it just for no other reason than you You just won't be able to buy them, the parts. Yeah, absolutely. But this does – it does sort of lay the rest. There was a big debate earlier this summer about, oh, is NVIDIA going to go with NPO for Ruben Ultra?

53:28Ben Bajarin: Yeah.

53:30Jay Goldberg: Yeah. But again, that's the networking switch and that's still east to west, not north to south. Yeah, yeah, yeah. No, I know, but they're still – they're still there in a hurry.

53:40Ben Bajarin: But if they pull that off, I actually think it makes my point stronger. That you just put a massive pod together through CPO switches east to west, and you left copper north to south. And you now have a giant scale-up domain.

53:58Jay Goldberg: That's better than my conspiracy theory, because I can't share my conspiracy theory on air.

54:04Ben Bajarin: In a public forum? Okay.

54:07Jay Goldberg: I'll tell you about it afterwards, but it's not going to make you feel good about the world. Okay, great. That's amazing.

54:13Ben Bajarin: All right, well then, we're going to leave you on that cliffhanger, everyone. And we look forward to talking to you next week. Have a great one, and we'll talk to you later.

54:22Jay Goldberg: Thank you, everybody. Tell your friends, tell your agents, click like, subscribe. Thanks for listening.