EP 190: NVIDIA and Marvell Earnings, Hot Chips Hot Takes

0:05Ben Bajarin: Hello, all. Welcome to another episode of The Circuit. I am Ben Bajarin.

0:11Jay Goldberg: Greetings, programs. I'm Jay Goldberg.

0:14Ben Bajarin: Well, we have a lot of topics to get through this week. It was a hot week because we had hot chips, which we'll get to, and we had NVIDIA earnings and Marvell earnings and I'm sure others.

0:27Jay Goldberg: I'm in an un-air-conditioned hotel room right now.

0:32Ben Bajarin: Jay's going to start sweating profusely. And because we're talking deep silicon today, for anybody who watches this, I have my TPU shirt on. Here's my glorious TPU shirt since they did talk about TPUs at Hot Chips. All right, let's kick it off with NVIDIA earnings. Yeah, the stock did okay. kind of a little bit down on this, what, Friday, August 28th that we're recording, but was up more than it was earlier in the week, but nothing monumental. I would say the reaction was positive. Folks remain optimistic that NVIDIA continues to grow. But what was your top line take?

1:19Jay Goldberg: the the so the quarter and the guide were were pretty much as expected probably um not good enough they were how do i say this they were beat expectations but everyone sort of expects them to beat expectations so that was fine what i think really interested investors though is they gave revenue guidance for 70 revenue growth next year and i think that's that's what people really keyed keyed in on, this has been a big, big topic of conversation over the last few months. People have been trying to figure out like, oh, revenue growth is going to decline. What's it going to decelerate to next year?

1:57And no, it's actually, it's going to grow 70%, they say. And Jensen kept pointing out they could grow more if they could get the capacity.

2:08Jay Goldberg: So pretty bullish signal.

2:10Ben Bajarin: Yeah. So let's talk on this for a second. I guess when you do something like that, you kind of set your expectations right in stone. So people will just do what they do numbers-wise, assume what that growth rate means. And so now the upside question becomes where can they eke out any additional surprises? And I talked to folks who were like, oh, they're probably being super conservative. It might be more than that. They realize it's a supply-contained environment. So they're leaving some room for the upside. But you're right. They're key in saying it would be a lot more. Jensen was asked this on the call, and he was like, how much more would it be?

2:55Ben Bajarin: I think it was our curry. And then he's like, it would have been a lot more. I was like, okay. So great. But you are supply constrained. And he was sort of like, we've got to go figure out how to get more supply, which supply doesn't grow on trees. So I don't know where that comes from. But yeah, I don't know. I guess my point is like, where do they go from here, right? They've just guided to that. So like, how do you paint a picture going forward other than you're executing on your 70 % growth call?

3:29Jay Goldberg: Yeah. So I mean, the typical investor relations playbook is you under promise and over deliver. And so people say, oh, he's saying 70%. It's really more like some number that's bigger than 70%. and I think at this point they have a fair degree of certainty around their business because they know pretty well what the orders are. Especially because we're talking about data center construction next year, that's all that's getting lined up now. And I think to some degree it's going to come down to TSMC and what capacity they can get out of TSMC. That's going to be the big swing factor, and we probably won't know that until early next year.

4:11Um,

4:13Jay Goldberg: right. So, uh, and then, yeah, I mean, I think there's other things they can do here. They're, you know, they're going to start selling LPUs from crock. Uh, we'll see how the market feels about those, but like those aren't as capacity constrained. Uh, and so maybe they get, you know, a little, a little kicker from that. There's a few other levers they could pull.

4:33Ben Bajarin: Yeah. They mentioned CPU stuff, uh, which, you know, obviously they, they're holding, holding their number, but, you know, they did announce a deal with Amazon, um, that includes Vera CPUs in a, in a shipment number, um, of, I think it was 2 million more GPUs. Uh, but that's a combination of Vera standalone CPUs and attached to head node. So do your mix, right. As you will, but, but they're associating value to that in the CPU side. And I imagine others, right. But, you know, part of it too is, um, I guess, I guess what I'm sort of curious about is like, So all the supply chain work we do kind of seems like 2028, which is not – which fiscal 2028, which is not NVIDIA fiscal 28, but – Calendar 28.

5:22Jay Goldberg: Calendar 28. Yeah, calendar 28 is when pockets of capacity can start to open.

5:29Ben Bajarin: And the other part of this that I think is interesting is, you know, you just look at the lag from order to data center and kind of those giant orders that people started planning a year ago, like really started ramping this CapEx wise, start to show up in energized gigawatts in 2028. So it's kind of like supply chain capacity, again, will still be constrained, but it will be better at TSMC with substrates, with all the components that they need, with wafers. But I kind of feel like the following year could actually be bigger because of an unlock of capacity that I think they're going to remain constrained in for calendar 2027.

6:16Ben Bajarin: which is hard to like again that's just nutty like to predict my point though is they will still be capacity constrained in calendar 28 but i think less so than calendar 27 that's that's what

6:30Jay Goldberg: i wanted to say yeah they they said they're sold out for this year and they're it likely sold out for next year as well and like 28 is every you know that that's that's everybody's everybody's guessing right now um a lot of capacity coming online you know it's so far away who knows

6:52Ben Bajarin: yeah who knows but but this is but this is where i think this gets super interesting which we were texting about this is where i think intel comes into the picture and you know intel's going to start to have foundry capacity around that timeline nvidia is i mean they're not they're as supply constrained as anybody. However, I think they have the most ambition to utilize every bit of existional capacity. They like the most hunger because they know that's going to impact their bottom line immediately. Again, true of everybody, but supply chain scale favors NVIDIA at this point. I really think the picture is being clear that NVIDIA is most incentivized to help Intel execute here and that they are customer zero cleaning the pipe for Intel the same way that Apple did for TSMC.

7:47Ben Bajarin: Because again, it behooves them so much to have Intel ready to go to scale for them to be, because if it is that high of demand that I think they're the ones that can move this needle for Intel. In fact, I was joking that Intel should just full port all internal products right now to TSMC and just give all that capacity to NVIDIA because they'd probably make more money in Foundry immediately. But I think they're too far down the path of where first-party products go. But you get my point. Like, they are going to be – NVIDIA invested in them. They're going to be the ones I think NVIDIA champions to help additionally meet their scale beyond what TSMC has.

8:33Jay Goldberg: Yeah, I agree. NVIDIA and Intel makes a lot of sense. But there were a couple of nits on the NVIDIA call, as bullish as they were. I think there's a few questions people had. Like the big one that I know a lot of people are talking about is gross margins. So they guided down gross margins, not this current quarter, but the Q, what, the January quarter?

9:04Ben Bajarin: I wish they would ship this to a calendar year. Their Q4, yes.

9:09Jay Goldberg: Their Q4. Gross margins go down from 75 % to 72%, and they stay down at that level, the low 70s, for the next three, four quarters. And I think that's weird. Like I had to – because we know in the press there are reports that they've increased prices. Like they sent those letters out last week. So they put in place all these pretty significant price increases, and still their gross margins are going down. which is they said it's it's partly mix and it's partly memory it's mostly memory is what they're saying and um i just don't know what to make of that like memory so it sounds like they're not able to pass on as as much memory markup as they used to so even they're feeling the pressure there

9:55Ben Bajarin: so the memory guys have the leverage so i i think it's actually not just memory i think again right Every part of this supply chain, PCBs, analog, everything, everybody is increasing prices. And I think, again, you're exactly right. And this has been sort of our question for Apple too, right? And Apple doesn't have 70 % margins on hardware like NVIDIA does. But there's sort of always this kind of challenge of what really is your pricing discrepancy? And I think they're sensitive to that, to be honest with you. I think they're sensitive to it too because perhaps others, competitors, namely AMD, might not have to increase prices as much given what they've already secured for a two-year run ramp.

10:47Ben Bajarin: So I agree with you. I think that they're willing to absorb some of that. I mean, they used to be very clear, right? We pass these costs on. It sounds like it's a little bit more ambiguous. but you know i i say that i say this to make the point and we argued this a long time ago when you know i was pointing out right you don't just you know the upside for nvidia in a capacity constrained environment is not you know all of a sudden greenfield shipments it's increasing prices but not at the cost of right gross margin because it's hardware cost it's increasing asp meaning that you could sell this for 10 million a rack instead of five but but but get that margin not not because your components are that are that expensive so so i don't know it's it's an interesting thing i mean i don't know i don't i didn't hear a ton of concern over that i mean they're still gonna continue to ramp but but i agree with you i think they just are gonna have to eat some and that is what it is when everybody in your supply chain is raising prices on you

11:47Jay Goldberg: it's it's i think it's significant here because nvidia is able to mark up those memory costs those memory prices, to their corporate margin. And no one else can do that. And I have to wonder if the hyperscalers, you know, they're now going directly to the memory companies. I think that's what's going on. It's rather than NVIDIA necessarily losing pricing, negotiation, and leverage with its customers, they're just seeing more of the memory business sort of go around them. And that's really high-margin business for them. So, you know. losing that is i mean it's not it's not it's not what i'm saying is it's not a huge deal but i do think like memories become so sensitive people are going to push back on the i'm paying extra

12:32Ben Bajarin: for that wherever they can yeah okay so let's table the memory bit because something came out of hot chips that's relative to this that i want to talk about uh when we get to there um but you know so so the other the other thing i was actually intrigued about and i was thinking about this relative to the Amazon deal. And I had forgotten about this, but I shouldn't have forgotten about it. But remember at GTC, I don't know, two or three years ago when you and I were in the Jensen QA and somebody asked him something about like, you know, you guys sell racks or you sell whole compute systems and whatnot.

13:11And he was like, we sell,

13:14Ben Bajarin: I'm going to botch his words, but basically he was like, we sell components to a data center. You can buy all of them or you can buy just some of them and we'll continue to sell them to you. And I think this is important because, you know, we get kind of lost in like what a NeoCloud would buy, right, which is a full NVL 72 system. So that's everything, right? Soup to nuts, all NVIDIA. Then you have what Microsoft and Google and Amazon buy, which is parts of that. Maybe not the networking, maybe just the CPU and just the GPU. and I just find it's interesting the way that that, and I say that to say that in my brain is a little bit harder to model because in a rack-based environment, let's say they were selling, you know, Grace Blackwell, whatever it was, between five and$6 million whole system, it's hard to say, well, how much was the CPU content value of that, right?

14:06Ben Bajarin: Or what was the GPU content of it? You could estimate it's whatever, you know, 40, 50 ,000 GPU, but that's a bundled price, right? But then you go and you sell a standalone CPU or standalone GPUs or standalone networking. Your markup could be a little bit different there because it's not a bundle. It's an individual component. And so I just say that to say I'm intrigued by areas where they may not be getting the memory. The hyperscalers are getting the memory and the PCBs. They're just providing them the components to go build their racks. And I just wonder if there's ways they can leverage pricing there that they haven't before versus the whole rack sale, which is where everybody kind of gets – the whole NBL solution is what NVIDIA typically gets benchmarked by.

14:56Ben Bajarin: My point is this individual component size, there might be variations in pricing too they can do there that we're not entirely seeing.

15:06Jay Goldberg: so they they sort of touched on this because remember last quarter they reclassified their revenue segments and now they say data center and they break out within data center they have ACIE which is which is enterprise and neoclouds and sovereign and then they also have hyperscalers and I don't think they actually flat out said it but they kind of implied Colette implied that the hyperscaler is a lower margin. And clearly, like, they want people to model it this way, is the longer segment lines. And I think it's safe to say that, you know, when you sell to a major customer like Amazon or Google who's buying in huge volume, you know, they're going to pay a little bit less than a neocloud who is buying in smaller volume, is buying the whole bundle, which is, I don't mean that in the antitrust sense, but just the whole package with the networking and the memory.

16:05They're smaller customers and they're much more dependent on NVIDIA for existing.

16:12Jay Goldberg: So you're going to pay a higher margin. They're going to pay a higher margin. And I've been tweaking my NVIDIA model a lot lately, and I'm trying to hone in on that component of it. I haven't quite nailed it down, but there's clearly a gross margin component difference there.

16:29Ben Bajarin: yeah and i'll and i say that again just to say this is how difficult this is to model right because i see you know friendlies on the buy side or sell side be like this is how many racks are going to sell and i was like well yes that's racks but there's also individual components that is a growing part of their pie well not necessarily a growing part but a key part of their pie and what they sell to hyperscalers and so anyway i just thought that was interesting to remember they'll sell you components as well not just the entire rack and you're right that should impact how you try to model this yeah i think the other interesting thing on the call was they talked

17:07Jay Goldberg: a lot about uh their financing programs they're putting in place right and they directly they directly you know addressed the question is it circular financing they say it's not um we see

17:17Ben Bajarin: it a different way was the words exactly.

17:20Jay Goldberg: Yeah, yeah. And it's complicated. There is news out today. I think the journal is reporting that - It was the Wall Street Journal, yeah. Yeah, it was the journal. It was reporting that this$500 billion SPV platform they're putting in place is getting reviewed for antitrust reasons. And it doesn't sound, at least from that article, it doesn't sound like the Department of Justice is actually investigating them. It's just that as they're putting that syndicate together, they're realizing that they have to be sensitive to antitrust concerns because of that word bundle, right? If, you know, we're going to provide you the financing, but you have to buy our product, that gets pretty close to antitrust law.

18:05Jay Goldberg: That word bundle really is a third rail. And so I don't know. That's all complicated, but I added it up. They have$579 billion in commitments. A little bit more than half of that goes to the supply chain, so buying lasers and substrates and TSMC capacity. But they're going to add – there's another$100-ish billion for a couple other things, and then there's this$500 billion platform they're talking about, and then presumably there's a lot more out there. So it's – I mean it's – and they also issued a bunch of debt, right? So in one quarter, they've sort of – they laid it out pretty clearly. It's a big number in terms of outside liabilities and obligations, some of which they may not have to pay, but it's a big number.

18:57And so I don't know. That part makes me – I've talked about this before.

19:03Jay Goldberg: It makes me nervous.

19:05Ben Bajarin: But I do think it's interesting because, like we talked about last week, right, that the balance sheet has entered the equation for competitive advantage, and they spated that specifically. And I just – I think it's a vector. And one of the points about – right, that I think they're trying to make around why they had to do some of these deals they did is really the credit rating, right? These companies need to go raise money. And we pointed this out, I think, last year. I think you said, like, this is why they're going to help finance these people. They need them to be able to go get money, so they basically have to backstop them to help their credit rating.

19:39Ben Bajarin: Great. We get it. That makes sense. The nugget, right, you talk about that is interesting and very hard to model. Like, you could think this could be billions of dollars of revenue sharing with the NeoClouds, but maybe, you know, like, that's not guaranteed. So it's hard to work that in even if there's upside. So, I mean, I guess even if NVIDIA wants you to like noodle on that, I would say don't. Like just assume it doesn't happen. But I think the point, the meta point is just the way in which they're going to need to use their balance sheet in order to help the ecosystem grow and, again, be able to raise money, build power, land power, shell, et cetera, et cetera, et cetera.

20:26Jay Goldberg: um so yeah but i mean i guess there's only a handful of companies that can do this right

20:31Ben Bajarin: let balance sheet be part of the competitive advantage and i think it was just interesting that they made that point clear um you know it is a competitive advantage and we're and we're

20:40Jay Goldberg: going to use it yeah i mean to their credit they are disclosing this pretty pretty well um Um, but it's, it's just big numbers and, um, it's, it's, and, and it's going to get bigger. So, you know, I guess, I mean, that's what they, I mean, you have to, I always think of this in the context of they are in a strategic competition with the hyperscalers and they, and NVIDIA is going to will the NeoClouds into existence. And now it has to use its balance sheet to keep them going. Right. And to the people who say that's circular financing, I think it's unclear. There's a valid argument that they put out, which is this is just up front.

21:23Jay Goldberg: This is an expensive business. You have to invest heavily up front. But the returns are such that, like, we're just helping them get started. And these are viable businesses long term and can pay all this back. And, you know, at the moment that that works. But at the same time, they're also talking about prices going up, basically doubling in the next couple of years. Makes it pretty challenging for the NeoClouds. Yeah.

21:48Ben Bajarin: Which means they don't rely on video more. Yeah, which is true. I also think that they're, like everybody, continuing to double down on this. What's their content opportunity per gigawatt? Which, again, I tell everybody, you don't just go. you don't just go there's going to be 30 billion or 30 gigawatts activated in 2027 and nvidia is going to get 40 40 billion per gigawatt like that's not how it works they're not getting all of that but in an environment where right they're the larger host or beneficiary of that right they're they're keep coming back to the to the content opportunity in in 40 billion per gigawatt and, uh, and growing.

22:29So, um, yeah, I mean, it's interesting.

22:32Ben Bajarin: I, you know, I, I think the supply chain stuff in my head is just kind of the most interesting about where they can move the needle there. Um, you know, not just wholesale continue to increase prices and get gross margin as a part of that, but, um, where might they be able to eke out, right. Some supply chain stuff. So it's going to be an interesting next, I would say, calendar 27, honestly, because their guide for I think it was$109 billion-ish for next quarter. Certainly they'll do that, how much they beat. But I think how this shapes out and can you tease out bigger growth, outsized growth above I think where most people landed like – I want to say it was between 630 and 660 billion, which I think most people were adjusting their numbers to next year.

23:25So like it's got to be like, oh, we're going to crush that.

23:28Ben Bajarin: It's going to be like how they figure out to get them to 700 or above is an interesting exercise. But it's going to have to come from supply chain somewhere. Yep. Okay. Any other nits that we're missing? I want to cover any other nitpicking. No?

23:50Jay Goldberg: A trillion dollars in debt and obligations is a pretty big nit.

23:57Ben Bajarin: it's rough out there in the supply chain folks like we said lta waterfalls you got to secure everybody to make sure you've got it um that's this came up at hot chips too man like everybody making silicon we had so much supply chain conversation so we'll talk about that all right let's go to let's go to marvell i'm intrigued marvell to me like uh i this it's i'm gonna i'm going to start by saying I assume the sentiment is because they have an investor day coming up and they want to save a lot of stuff for that in early October. And so they didn't show like, this is how we frame the market opportunity in dollars, TAM, SAMs, whatever the things.

24:46Ben Bajarin: You know, the big things, obviously, this$120 billion deal with Google, which is potential revenue, not guaranteed. And a lot of investors trying to figure out, you know, give us some idea of the conversion schedule of this and, you know, what we can think about. And they just didn't want to do that because of this investor day. So I kind of feel like it was like, great, good posture, sounds all wonderful, but not enough clarity. So you're just gonna have to wait until investor day and reactions were what they were. Like, because it's still positive, like CXL, all the things. oh the other thing i was gonna say i because we talked about this a lot so i'm glad that they have finally officially chosen their lane of xpu attach and so they were very very clear you know like we're not gonna not do monolithic compute tiles if they don't come but like that's not gonna be whereas like two years ago everybody's assumption was you got to do the whole thing or you're dead right or you got to do more in compute or broadcom eats you and then they kind of kept saying xpu attach xpu attach xpu attach and now it's like firmly our lane is xpu attach and and i do think that's a positive it makes you a much more uh companion to the entire asic ecosystem not having to win the whole program but you can do well with your lane of everybody's program i think that's a good positioning um tougher to model but i'm just i'm i'm happy that that they've picked a lane um but yeah other than that it's like yeah anyway give me what was what was your take i i think the

26:16Jay Goldberg: The stock is down 10 % today almost. And I think the big – the very simple summary of what people didn't like was they have the unfortunate position of reporting the day after NVIDIA where NVIDIA is talking about 70%, at least 70 % growth next year. And their number was 60%, which in absolute terms is fantastic growth for Marvell, for any company, see 60 % growth. but it's still just mathematically they're losing share to nvidia uh it's just very very hard and and to your point i'm sure like they're going to have their investor day in october and they're going to they're going to lay out a much more positive picture but i think for the moment it's like just nvidia's growing 70 they're growing 60 and then you have to deal with all the complexities of like what do these guys do and where are they selling and what is this google deal going to look like and it's just it's uh yeah yeah no i get that like like i said it's it's hard because you have the clear execution timeline position of of nvidia and then you have like it it actually sounds good on paper but i need to know exactly how to like convert and model

27:40Ben Bajarin: these timelines for you guys. So it's like conviction, like looking more optimistic, but not like a guarantee at this point versus where like NVIDIA tells you, like you said, they're going to grow 70. They're going to at least grow 70 and probably more. We don't know, right? The same confidence level for Marvell. But I do, like I said, I like that they have their lane. I like that CXL has come into the picture, even though again, right? CXL is not going to add a gigantic amount of revenue to them, maybe couples of billions of dollars, not tiny, but decent. But yeah, the Google one's kind of the big one, right?

28:16Ben Bajarin: If that can convert more of that$120 billion in a quicker timeline, that's a very different shape of their business. But we don't know, and they didn't provide color on that.

28:26Jay Goldberg: Yeah, they were certainly very excited. I mean, their tone was very positive. Matt was very bullish, I would say, in his comments about that opportunity they face. at Google. My advice, if they're listening, is we know what day NVIDIA is holding earnings next quarter. Don't do it. Marvell hasn't announced it yet. Put your date two days before then. You never want to follow. You never want to follow NVIDIA. It's a tough act to follow. So that's true.

28:59Ben Bajarin: Okay, anything else on Marvell we're missing? I know that was kind of, I mean, like I said, we will, I am sure, talk about this after their investor day and kind of try to parse out the market opportunity in economics. But okay. All right, let's talk hot chips. you know it was interesting hold up for anyone who knows

29:27Jay Goldberg: Hot Chips is an academic conference like professors set up by the electrical engineering department at Stanford they have professors it's the kind of thing where you have academics PhDs presenting academic papers there are posters outside but it's an academic conference held in the heart of Silicon Valley and so the people who attend and speak at it tend to be people from the major companies doing the most advanced work. And that is super interesting, but it comes at the tradeoff of, you know, in an academic context, you're probably going to present your best work. When you have all the big corporates presenting, even though you have the technical leads, they're probably holding stuff back a little bit.

30:10Jay Goldberg: But it's a super interesting conference. The biggest complaint I've heard about it lately is that it's the amount of investors who show up is sort of, as everybody frustrated, but it's, it's very, it's a very interesting conference, but it's very, very technical. Yeah.

30:25Ben Bajarin: Yeah. So, you know, my, my favorite part about this is always like the way, you know, the hot chips guys have always sort of framed this. Like whenever I talk about it with them is, you know, it's engineers presenting to engineers showing how they did things, how they solved problems, what the problems they face, things that they don't know, things that are unsettled that they're trying to solve. Um, and to be honest with you, one of the things I really like is a lot of the questions come from competitors. You know, you would have like a Google event and you had somebody from Marvell or D-Makers be like, hey, how'd you solve this problem?

31:00Ben Bajarin: I'm curious how you think about this. Like, it's amazing. Like, you're like, this is great.

31:06Jay Goldberg: Or the best part is the Huawei guy shows up every year and asks incredibly sensitive questions to every single presenter. And sometimes they answer him.

31:17Ben Bajarin: I actually didn't hear the Huawei guys much. This year was owned by a D-Matrix employee who like literally just got up almost every time and asked questions. And people were like, is this marketing stuff for D-Matrix? The other fun thing was Etched was there as a sponsor, which was cool because like I got to ask them some more detailed questions. But they had this rack, this black box cabinet, and it was locked up. and everybody's like, what the crap's in this thing? Like, I'm assuming it was empty, right? Because obviously they want to sell racks at some point. They sell individual boxes at this point, not rack fabrics.

31:55Ben Bajarin: But I'm like, sitting there, I'm like, why is there a, is Houdini in there? Is Houdini going to bust out of there? Like, give me something. So there's a couple of startups. Samsung was there, which I do want to talk about when we talk about memory and custom-based eyes. But let me give kind of like a high-level view. I do think it was one of the best hot chips in a long time, despite what others thought. There was an inordinate amount of investors there. I had two group dinners with both of them. All of them, admittedly, I'm not technical. I'm kind of just here to check it out. Like, explain to me what this is.

32:34Ben Bajarin: but because there is some presenters who kind of took a non-technical route and I think because they were just trying to tell a story knowing maybe it was a more diverse audience and then there were others who went super deep in the weeds really really cool stuff across the stack each year it varies last year there was also Memory, Interconnect um a6 and cpus this year was all three of those but like in years past it was transistor design it was next generation foundry so you know it it kind of changes but this year like last year lots of interconnect networking cpus a6 memories um was essentially the the core um but but my take and i wrote about this and this is sort of the high level framing um that i gave is there was a a lot of agreement on the problem inference at scale and serving inference at scale is a computationally complex problem how the asic works how the memory works how the software moves between those two workloads everybody agrees but what was really interesting was you listen to you know the the dmatrix guys or the sambo nova guys or even nvidia and amd and And everybody who presented, even the memory guys, everybody kind of had like a different point of view on how they think is best to go solve this problem.

34:06Ben Bajarin: And so there was disagreement in how we go and do these things. And so I wrote about that this week because there's an academic term called the era of ferment, which I will sort of just briefly explain, right, for everybody. but it comes from a 1990 paper called technological, technological, technological discontinuities and dominant designs, a cyclical model of technological change by Anderson and Tushman. And essentially it is an era of ferment is when a period after a major technical break, when an entry tests competing approaches before one common architecture emerges. And this is why you're seeing, you know, a lot of diversity.

34:50Ben Bajarin: Now, interestingly, like people who were either technical in the weeds or other engineers or have a bias to a particular one of these problems was sort of like, hey, you're only doing this in SRAM or on HBM because you can't get HBM. So they're like, so you're hacking around this, not because it's the best way to do it, but because you don't have a choice, right? And that may or may not be true. But the reality was like you have all of these kind of competing approaches on inference, on memory, on ASICs, on the design, even to some degree the interconnects. And I just thought that was interesting.

35:29This is the first time in a long time I've seen this dynamic of we all have strong opinions and they differ about how we should go solve inference at scale.

35:41Ben Bajarin: and anyway so that was like high level um kind of kind of my take and like i said i haven't seen this in a while um which just to me was kind of interesting and the way i frame this too is they're all even the startups right even the memory guys you could you could be clear that you know they're showing next year they're showing two years where they're showing things that they haven't even solved yet like conceptually we're going to go do this but we realize heat's a problem. We're going to go do this. We realize yield's a problem or we can't manufacture this yet. They would acknowledge those changes, but they're making technical bets right now in some of these architectures for the future in two years.

36:23Ben Bajarin: And we have no idea what the future of models or anything's going to look like in two years. So it's just interesting to look at bets they're making and compare and contrast those with others, again, in an environment that is highly unsettled. And so we don't know, you know, we don't know what the standardized inference architecture will look like. I think it will emerge. And then again, you know, say that some of the startups are successful in proving that out, then, you know, my concern is one of the bigger vendors just takes that design and does it if that's true. But anyway, the diversity of technical approaches, I found fascinating and thought was super interesting.

37:00Jay Goldberg: Yeah, it was very interesting to see a number of companies four or five companies presented their cpu architecture and as i so i in full disclosure i couldn't attend this year for family stuff but i was reading through all the papers and reading all the following content and it was really striking to me the number of cpus that were presented and gpus that were presented and just how different their architecture was. NVIDIA has these big cores and ARM has much smaller cores and just the sort of number of nitty gritty features they're including. There is a lot of experimentation taking place. And, you know, my read on that is sort of what you're saying, which is we don't know what inference architecture is going to look like.

37:47Jay Goldberg: And you say it's going to standardize. I'm not sure that we go down that path. I think it could remain very fragmented and very different. um it just it is it is really striking how sort of radically different some of these technical approaches are um so much variety it was a fun show i wish i could have gone yeah yeah well and

38:07Ben Bajarin: not just like like some of them were like novel like you talk to the engineers in your real life they're they're really swinging for the fences with this one you know like it could work but but it's also really hard, you know? And so, so, you know, I, I, I, I agree with you. I don't know it. I think at least an agreement. So my take again, just, and I, and I talked to everybody I could, I had meetings with six of the leading startups in this was like the, the challenge is there's a software tax to all these different approaches, right? If you're going to put the KV cash somewhere else in memory or storage, or you're going to move it off direct, the software has to do different work, right?

38:49Ben Bajarin: And so because it's not just a clean, it might be efficient from an algorithm standpoint, but like some of the other agent orchestration and whatnot, like there's a little bit more work involved. And so there's a clean way to do this. Certain vendors do it and the software, and then there's a really hard way to do this, which again might be right. If X, Y, and Z goes according to plan, X, Y, and Z, right, might not go according to plan. So, So – but again, like the idea that everybody is making technical bets I think is super, super interesting. And I'm assuming we will have some bespoke variety by the custom hyperscalers and then a couple of merchants.

39:27Ben Bajarin: But I think you probably agree with me. 30 or 40 startups don't survive the day. But maybe the industry learns from what they tried and apply it in their architectures because maybe it was philosophically or technically sound. They're just not going to be the ones that pull it off. Somebody else will.

39:47Jay Goldberg: Yeah, I mean, this is the dilemma. It goes back to what we were talking about before about NVIDIA sells racks. They don't sell servers. They don't sell chips. They sell racks.

39:54Ben Bajarin: They sell systems. Systems. Platforms. They would like us to say platforms, Jay.

40:00Jay Goldberg: I'm not saying it. Systems. I'll go with systems. I'm not giving in to platforms. They sell systems. And if you're a startup that sells, like, you can only do so much, right? And so your biggest ability to differentiate is going to be at the chip level. It's right. And how do you then design a whole rack, whole system? And I think that's that's challenging because then you have to factor in, you know, networking. Right. If you're a chip company, you could have the world's greatest XPU AI accelerator and you lose all your TCO advantage because you're using the wrong networking stack. and I think that's right and so it's gonna be tough for a startup and I think it makes it very hard for them to sell and then let alone like everyone's competing with balance sheet now too which you obviously can't do as a startup so yeah I think it's gonna be tough but I do think I do think some of them will get acquired for for I won't put a number on the valuation but I think some there there is a likelihood that they these get acquired yeah agree because they are coming up with some really innovative things and agree with the market as hot as it is like i think the big the big players are going to recognize that they need rapid time to market and pick these things up but yeah but not 70 of them or however many there are yeah there's no no correct um

41:29Ben Bajarin: okay anyway i thought that was interesting memory i want to spend some time on uh because the theme of all memory providers, which, which again was also the theme of future memory summit. So it's just carrying on with detail is custom, custom HBM based dyes. And, you know, it was, it's, it's interesting. Like you listened, I listened to this, the Micron didn't give any platform details on custom based dye, but both Samsung and SK Hynex did. And it was interesting. They kind of point out like, this is why it logically makes sense to go to custom-based eyes. And, but this is why it's hard, right?

42:07Ben Bajarin: It's a thermal problem. They got to deal with hotspots. It's once you have that level of a, of, of an ASIC, uh, in terms of, of, of wattage, it, it gets harder. Um, so they're trying to solve right that problem, but, but the thing that all three of them kept mentioning, in fact, and, and Micron CEO said this at the conference they were at, um, I want to say earlier in the week or last week, I can't remember, that they think about like their work in custom-based eye more like they, like you should think about custom ASICs, that it is a unique design customized for that customer. And because of that, his point was, and I just raised some questions to me when talking to the Samsung and the SK guys is that it's harder for one vendor to support three memory players, maybe even two memory players, when you've gone down that degree of customization in BaseDi with that person's memory and your customization on their BaseDi platform.

43:09Ben Bajarin: And that just got me thinking like, okay, so what if Micron's BaseDi platform is more efficient, leads to better efficiencies with Micron memory than Samsung's does? And now NVIDIA is sitting they're going, I'm not saying this is happening. Nobody freaked out about this. I'm just saying this is an example. But then you have performance differences between vendors for the same accelerator. That's possible. And I don't know how to reconcile that with then the point I was going to make with the NVIDIA one. How does that also not increase the cost to do this for memory that you're actually going to customize a base-type platform?

43:51Ben Bajarin: They're going to charge you more for this. It makes all the sense in the world and optimizations and customizations to do this but it's going to be more expensive so so so in a custom based eye world like memory doesn't go down in fact it probably goes up and it's going to behoove one to two vendors memory vendors per customer because i agree it would be very hard to support three custom based eyes for one gpu i think that's an

44:16Jay Goldberg: unreasonable thing to expect this is a this is a tricky one you think about it if you're a memory company, you know, the knock on memory companies is, oh, it's just a commodity. It's interchangeable. Yeah, it's interchangeable. Yep. Right. And so the memory companies probably see a lot of appeal to some form of customization. So that memory isn't totally fungible and it's much harder. You get customer lock-in. But at the same time, like that sort of commoditization of memory is valuable to them because it's interchangeable, because it's standard. right everything standardized and you they risk losing that if they start going everything custom um but i understand the appeal it's i think it it leads to complexity down the line that it's hard for them to think through because the appeal of being custom is so big

45:09Ben Bajarin: i agree i also i'm not saying everybody uses custom based eye but i'm saying like i guarantee you NVIDIA and AMD probably will and maybe somebody else. But just take your biggest customers, right? Maybe even your XPU providers who might do this for good reason. My point is exactly what you said is that it becomes more design in, it becomes more co-optimized before the full stack, but it's also less fungible. And I would argue might have performance variations that vary. Like I, again, I just don't like, it would be, it's a weird world to say my GPU with micron memory or with Samsung memory behaves differently, performs better than my SK one, right?

45:53Ben Bajarin: Like that's a weird environment to be in if that played out even in that fashion, right?

45:59Jay Goldberg: So anyway, everybody walked away like custom-based is going to be a

46:05Ben Bajarin: thing. We get it. Yes, it benefits defenders. But my point is in no world does that help memory costs it it changes it but it's certainly less fungible for the memory guys so i just thought that was like the talk of the show everybody was like this custom based i stuff's gonna be incredible and i was like yeah there's also some interesting trade-offs you know with it uh you know if it's gonna be a thing which it seems it will so anyway that was that

46:35Ben Bajarin: crazy yeah and the other stuff like like samsung was showing zhbm and and then and then uh you you know, more 3D stacking. I was like, like, that's great. If you get there someday and you can commercialize this and scale it, wonderful. But that's, that's way far out. That's not, in fact, I won't get into it too, but like a guy, I can't remember where he was from. I don't know if this was from a high bandwidth flash company or if he was just giving a presentation on high bandwidth flash, but like parts of it sounded positive, but then it was actually kind of, kind of, kind of negative. Like he had these slides, like, eh, we got to solve these problems.

47:08Ben Bajarin: and did you think about it this way? And people were like, I don't think he's showing positivity on High Band with Flash. That one got some good conversation in the slack. But anyway, yeah, it was a great show. Like I said, it was about as busy as I've ever seen it. I mean, people couldn't sit in the main auditorium. They had to go upstairs. There was overflow outside. Yeah, it was big.

47:37Jay Goldberg: and it's just going to get bigger next year, I'm sure.

47:40Ben Bajarin: But anyway, it was a good show. And lastly, I just say I like these shows and this just doubles down on my point of this era of ferment concept. My standard view has always been if you just understand semiconductor company roadmaps, you can predict the future because software performance is directly aligned to where compute goes. And that's why I like going to the show. It was just interesting that like models all have to go in a very certain way in order for all these different approaches to have some degree of success. And that's very hard to predict. So it's like my – this confusion doesn't help my prediction because it's so diverse.

48:25Ben Bajarin: I'm not sure what it helps me predict. But that's why I like going to the show is it's a glimpse of what's computationally possible in the future.

48:34Jay Goldberg: Did you listen to the Waymo presentation?

48:37Ben Bajarin: Yeah, I was in the Waymo.

48:38Jay Goldberg: Because I thought that was super interesting because this sort of talks about, touches on what we were talking about before in your age of ferment, which is Waymo had a big discussion on their approach to silicon for their autonomous cars. And it's very clear that their inference workload is really, really different than what a lot of other people are. So certainly what the frontier labs are dealing with. Like, you know, they use like, you know, there's this big move to FP4. It's a sort of less precision because it's more efficient. and you don't lose much. But you can't do that in a car because you actually need super high degrees of precision because you have all these different sensors that are very, very fine-tuned.

49:16Jay Goldberg: And that means that their silicon for their workload is going to be really, really different. They're moving against a trend that's been pretty common. Everyone else is moving down. They don't want to do that. And I think that sort of speaks to there are going to be more of those. We didn't talk about it, but like, back when they announced earnings, AMD acquired Talus, which is a startup that sort of claimed to be able to do custom silicon for every individual model. And they had a really interesting approach where they sort of had lots of different metal layers in their chip design that were standardized and just the top layer or two they could switch around for each model.

50:01Jay Goldberg: So in theory, allowing very, very rapid iteration of custom silicon and for you know if you their pitch was give us your model and we'll we'll optimize the chip for you and that's a real that's a really challenging approach for a lot of reasons we don't have to get into but amd acquired them we have to assume for some for more than just maybe they just wanted a bunch of engineers but i think that they're they're attuned to this that they're going to have to do more semi-custom work for yeah for so everyone's gonna have a different model.

50:34Ben Bajarin: You bring up a really good point actually. And the Waymo guy like kind of talked through, like, actually this was the most fascinating part of this conversation because I had not thought about any of this. Right. But, but he was making points about, you know, what a, what a, what an autonomous taxi has to do and know is different than your autonomous car that you're operating and parking in a parking lot. And, and, and what he showed was, you know, this, this car needs to know, for example, can it park right here? And this is a sign with a pole. And it had like eight different when you could park their signs, not on Tuesday, not from 8am on Wednesday.

51:10Ben Bajarin: And he was like, and it needs to know if it can stop here. And I was like, dang, like you got, that's a lot of image processing, what day of the week it is, like, what time is it? How far is my next ride? Like, it has to solve different things, right? Then a, you know, your, your passenger car. And I just thought, but, but that's, again, that's them saying our problem is unique. Our problem is also somewhat bespoke and we need to go solve it a unique way. And so exactly like you said, I just thought that was super effective. The other thing they showed was they showed the back, they showed the back of one of their cars from like really early on.

51:49Ben Bajarin: Like it was a rack. There was a, there was a rack of, of, of compute, of like a little data center in the back of this car in like a wooden slash metal box. To be an engineer on that product, be like, yeah, how are you going to do this? Well, you know, we got to figure out how to get a computer in the back seat. We're going to put this rack. It's connected with all these crazy fibers. Like, anyway, it was super, super interesting. But it also makes sense why they feel like, like you said, they need to do their own models and they need to do their own compute with their own ASIC, which they outlined.

52:21Ben Bajarin: Because again, they're trying to solve a problem relatively unique to their vertical and you've got all these other things they got to deal with, right? Safety, compliance. Um, you know, anyway, I could go into the other presentation was how I could go, but we'll, we'll, we'll stop, but you're right. Not novel approaches, uh, for the time being.

52:43Jay Goldberg: All right.

52:45Ben Bajarin: That was a good one. Uh, thanks for listening, everybody. Uh, we'll talk to you next week when hopefully it won't be uh i guess i was crazy but you never know fall's gonna be

52:58Jay Goldberg: crazy thanks for listening thank you for listening everybody leave us a review tell your agents to leave us a review tell your friends