FULL INTERVIEW: Dylan Patel Says We’re Still Underestimating AI

0:00What's up man? Good to see you. Good to have you. Data centers in space, what you got? When are we going to space? Coming in hot. Me, you, the International Space Station, let's break it down. You know, the space tourism industry is quite a fun one, right? Yeah. Would you go? Would you do the Blue Origin thing where they blast you out past the Carmen line? It's good enough for Katy Perry. It's not good enough for you. What's going on? It's like you're in free fall. You're not actually in space. Oh, it doesn't count? Shots fired. Carmen line denied her. I want to be like going around for days. Oh, OK.

0:30I want my bone density to start to atrophy. I truly want to feel the negative effects of space. Yeah, yeah. It's not enough just to go up and back. I think I would do it. It's like 90 seconds, right? Yeah, but it's better than hanging out on Earth. But all the cool stuff that astronauts do, right? Like, you know, put water and then they're bubbling and then you try and drink the water. Soon they'll be unplugging the GPS and plugging it back. Oh, yeah, yeah, yeah. That's how you pay for your space tourism. You got to go on the sea of ships. 90 seconds of servicing a SpaceX satellite. One 90 second trip at a time.

1:04No, but people were wondering, you know, TPUs, NVIDIA going on the Starlink V5 or whatever something gets up there. It feels like this will be something more like a Tesla silicon chip, an AI chip. Like, do you have any insight into like, what the process, if you wind up figuring out how to heat dissipate, if you wind up figuring out the costs, what might the chip look like? So I think, you know, everyone freaks out, oh my god, putting stuff in space is expensive. But if you look at like starship launch costs and they keep falling, you're like fine, right? I think that's not, by the end of the decade, the cost of space launch will be fine.

1:42The heat dissipation, I mean, it's a challenge, but you just put a massive, massive, effectively radiator and it's fine, right? By the end of the decade, you'll be good. I think the big challenge is that chips are just really unreliable, right? And so how do you deal with a couple things, right? Satellites can only be so large before they start needing a lot of support and structure before they tear themselves apart. So when you look at the launches, these things are shooting out tiny satellites, and many of them. Okay, so you can't have a big, fully connected cluster of chips. And then on top of that, how do you deal with any random error?

2:18On Earth, you have techs running around the data center, unplugging stuff, putting in spares, things like that. What do you do in space? You RMA it to the factory where they might unsolder it and re-solder it and then test it and it works and go back out. Sometimes it is just trashed. That's the challenge to me. I feel like maybe the pattern we should be looking at is how often do the Tesla self-driving chips need to get serviced? Because that's the team that would probably be building or bridging the gap there. The Starlink satellites, they go down, but the service works. like you're you're just relying on some sort of like you know 90 % uptime stuff's coming down but most people that are in a way mo like the chip keeps working right most people that are in a Tesla self-driving like they're not like you don't hear about Tesla owners being like I love FSD but I'm constantly in the shop getting my my custom silicon chips unseated and reseated right I mean it's it's also a function of like the complexity of a chip right sure um you know if if a chip is twice as fast yeah and let's say the bit error rate right Like how often a bit flips is the same, then it's erroring out twice as often.

3:28But let's say the chip is 10x as big, right? And so when you look at like a Tesla FSD chip, very, very good, very, very efficient, still like relatively inexpensive and cheap compared to, you know, a big old GPU or TPU or whatever. Those things are extremely large. And, you know, again, like if the error rates are the same, then it fails 10x more. But in fact, the error rates are a bit higher because they're pushing these things to the absolute limit. whereas Tesla does have some level of like, well, first of all, the Tesla car has two chips, sort of redundancy already built in, right? Maybe you do that on the satellite, but then there's more power, more...

4:04Yeah, right, so the whole lore, right, of it is, you know, effectively power is free, right? And solar panels, you look at the cost curve of solar panels, you look at the cost curve of satellite launches, you're like, this is free, this is great. But power is less than 10 % of the cost of the cluster, right? so so like it's that 90 % you're not saving anything on yeah and in so far as much as potentially a hundred times the hassle yes yes there's this whole like you know like if you look at NVIDIA GPUs right when you first turn on the cluster about 10 to 15 percent of them fail RMA in the first two weeks Wow and then that's fine like you have to reseat them whatever and like the industry knows how to deal with this right and in over time like hoppers now at 5 % but But Blackwell's still 10 to 15%.

4:51Actually started out higher than that. And when a new generation comes out, it's gonna be higher than 10, 15%. It'll have its curve gradually decline down. But who's gonna, are you gonna test it and burn it in on the ground? Or are you gonna say 5 % of my chips or 10, 15 % of my chips are trashed? Because someone can't go up there and do these things? Or am I saying, oh, I need robots who can do all this stuff in space, and now that's an additional engineering problem when sacks of meat are actually very cheap? Yeah, sacks of meat, yeah. Speaking of invidia, We haven't talked since the Grok acquisition.

5:20What does that look like in the bulk case? Like if it's a good, if the next version of Grok is a great chip, is it sitting next to the, you know, H200, H100s in the rack, GB200? How does it fit into the actual, like what NVIDIA deploys? Is it just a separate chip? I think it's a big vibe shift from NVIDIA, right? Before they were like, all right, I got this big GPU. Everyone's gonna use this GPU. Software ecosystem of the GPU is so good. It's one size fits all everyone's trying to make all these specific point solutions, but we've got the thing that's good at everything. And then they had a vibe shift, right?

5:56They launched this thing called CPX, which is a chip made for pre-fill. With prompt processing, creating a KV cache, and also good at video generation and image generation. And that's coming out later this year. In the Grok press release, they were really talking about video generation as well. So yeah, you've got CPX, you've got the standard GPU, and now you've got the Grok chips, and they all fill a different niche. But really it screams, oh crap, we don't really know exactly where AI is going, which I don't think anyone does, right? I mean, it's moving so fast, the software is, the model architectures, etc.

6:25So we're just going to engineer solutions that are along multiple points of the Pareto Optimal Curve, and then one of them will win, right? And I think it's sort of like a big vibe shift from NVIDIA. Also, they just knew OpenAI was going to do the Cerebrus deal, so they freaked out. Got it. Yeah, get me up to speed on what makes Cerebrus important in the ecosystem right now. So, you know, you have people thinking like, oh, latency matters in terms of where our data center is. It doesn't matter at all. What matters is, you know, as we've moved from, you know, chat applications, which were like, or search response immediately, chat applications, let's say response takes 10, 20, 30 seconds.

7:03You've got agents, you know, I don't know, my cloud codes are working in the background for a long time, right? It doesn't matter where the data center is, but what does matter is that these streams of inference take, you know, 30 minutes versus 10 minutes, versus five minutes, and for a lot of people, I'm fine to spend 10x the price on something that completes 10x faster. And so Cerebrus sort of just makes a ton of sense there. So OpenAI, they've got these long horizons. There's Codex 5.2, extra high thinking or whatever, it's terrible. Can you guys teach them how to market? OpenAI, you have to sponsor this podcast.

7:38Yeah, yeah. We had two on yesterday, and I did actually ask him, I had the Codex app pulled up on my desktop and I was like, there are six different models and then there's another button that I can pick to it. Well, how many different products are called Codex now? There's a lot. Now there's an app, yeah. We actually have another guy on just to do branding. Lexicon branding came on the show yesterday talking about all the naming. Naming architectures. Naming architectures. It is complicated, but hopefully. You can tell he's just blood's boiling because all the AI companies just have the most chaotic Anthropic, Claude, Claude Code, but also you can use Claude Code for other stuff.

8:16Yeah, but yeah, I mean, with Cerebras, it seems like there is a value to it, but are they constrained on the supply side? Can they actually scale up to a Colossus-style data center that could actually speed up Codex, not just for one user, but all the users? So, I mean, Cerebras can speed up multiple users for sure. The question is sort of like where you use it, and that's where they have to like figure out where within Codex, right? Because there are times where Codex is running for like 10 hours, and sometimes you don't mind, right? Like, screw it, I put up this nice prompt, gone, work on it, refactor my code, do this thing, do this task.

8:51Other times I want this iteration feedback loop, so how do you expose it to the user without saying, hey, actually there's another toggle, so your permutation is 18 times. Well, hopefully like a really robust model router, but it feels like that's been a process. Yeah, so the OpenAI deal is like for 750 megawatts. It's not that much capacity on the order of what OpenAI has talked about. By the end of 28, they'll be at like 16 gigawatts of that. So it's just like the absolute cutting edge, the most price-insensitive customers in that specific use case of, this is the type of prompt that you need to return fast, then you'll get the speed up, potentially.

9:26Right, right. And they've got to figure out how to do it from a product, exposing it to the user, etc. But it's clearly something where there is demand, right? Like, I don't know, like Andre Carpathie doesn't care if he's spending a thousand bucks per agent per second or whatever, right? Like, you know, so whatever it is, these like super cracked engineers don't care at all. And then obviously there's like a long tail of like actually cost does matter for most people. And so all along that curve, they've got to have solutions, right? When did you first think that XAI might end up at another Elon company?

9:59I mean, this has been rumored for a long time, right? Like people are saying Tesla, Tesla, Tesla for the longest time. It's harder with a public company. Yeah, yeah. And then a bit ago, people were like, oh, SpaceX, I'm like, wait, this makes no sense. No, but there was a very coordinated narrative pump at the end of last year. And it was almost perfectly telegraphed. Well, there's a bet, right, between basically the head of compute of XAI and the head of compute of Anthropic. And the bet is what percentage of worldwide data center capacity is in space by the end of 28. And the bar is 1%. Oh, wow.

10:34And so the XAI guy is like really bullish. The Anthropik guy is like, eh. Yeah, a little slower. Yeah, yeah. But it's a really interesting bet. I take the under on 1 % by 28, because that's a gigawatt in space. Yeah. But it's actually not that crazy, right? Yeah. It's roughly 150 Starship launches. Yeah. We'll get them to a gigawatt in space. Yeah. So, you know, Starship hasn't worked yet fully. I was looking at the energy draw of the current Starlink fleet, and I think they're at like what is it 200 kilowatts or something like that so you you get a thousand of those 200 megawatts and like you're starting to be in the territory yeah so the v2 stars satellites I think are the only ones they've launched maybe they've launched a few V3 the V3s are coming soon and those are those are like a hundred X more bandwidth each right and more power and just more power and so what I'm just thinking of like can you scale this thing up at all it's like are they two orders magnitude off are they three or two this feels like they're like one order of magnitude off something that looks like an H100.

11:35I think the metric is like 50, it's either 50 kilowatts a ton or something like this per satellite for V3. Let's say from V3 to whatever the compute thing is, they double it again, get to 100. I think the V2s are like 25. So if you get to 100 kilowatts per ton for launch, it's only 150 or so Starship launches. I think that's so reasonable. Maybe not 28, maybe it takes 29, but like, you know, it's so reasonable. Well, the question is cost and reliability. What happens when the chip fails? How do you service it? That kind of stuff. How do you deal with having clusters be much smaller instead of these big clusters?

12:10Even for inference, big clusters are useful. How do you think about Google's response to Grok's, Eris, TPUs? Obviously, they're very successful. But are they forking that project to eat more of the Pareto curve? Yeah. Yeah, so for the longest time, Google's had one main line of TPUs, right? All made by Broadcom. And then sort of next year, they've diverged it, right? Where Broadcom makes a TPU and MediaTek makes a TPU. These two TPUs are focused at different things. And they're fabbed at TSMC. They're both fabbed at TSMC. Everything at the end of the day goes to Arrakis, right? I want to go there next, but everything goes to Arrakis.

12:50So fabbed by TSMC regardless, but both of these TPUs are focused on different things. Okay. And they've actually got a third project for another kind of TPU there. They also see this need to proliferate along the curve of like, hey, do I care a lot about super high amounts of flops, not that much memory? Do I care a lot about super fast on-chip memory only? Do I care about 3D stacking memory? Do I care about this sort of general purpose middle ground AI chip, which is what an H100, a Blackwell, a TPU looks like today? They're sort of like, oh, we need to hit the entire Pareto optimal curve. And it's like, okay, within this, there's training versus inference differences and and what numerics you want, and all these other things.

13:26There's so much complexity there. Everyone sort of is diverging their roadmaps once they're at a sufficient scale, I think. Yeah, is Google still way ahead on cross data center training? Yes. And are the other labs, like, is that important to the other labs to catch up there? Or is it something that will just naturally happen because everything sort of commoditizes? Or do the other labs need to sort of marshal some Herculean effort to like crack the code on what it takes and what Google's doing? Yeah, so it's a couple of things, right? In 2023, everyone thought that scaling was pre-training. Yeah.

14:00Right? You know, more parameters, more data. And that's very difficult to split across data centers. And has Google been able to do that? And Google's been able to do that to an extent, right? So what they've done is they've got, you know, they don't have the largest individual data center campus, but what they do is they do these like regions where it's like, hey, each data center's roughly 40 miles apart from each other. Sure. So in Nebraska and Iowa and then in Ohio, they've got these complexes, and now they're building one in Oklahoma, Texas. Got it. You know, these complexes where there's all these data centers pretty close to each other.

14:29So it's not really cross data centers, like across the world. Right. It's just across regions. Yeah, and then that makes a lot of the difficulties a lot easier. Yeah. Flip side is we've also moved to RL, right? Yeah. And majority of the time of the chips is spent generating data, right? Only doing forward passes through the model. Sure. And then you only send the final tokens that you verified sort of back to train on to the training, right? So then you end up with like, oh, instead of in pre-training scaling, you need to like synchronize all the weights every 10, 20, whatever seconds. When you're doing these rollouts, and especially as things get more and more agentic in training, you might not only need to send not the entire weights, but just the tokens that are relevant, so way smaller amount of data and way less frequently, right?

15:11Minutes at a time instead of seconds at a time. Yeah. And so you've got this like now, now it's become like reasonable where, oh, actually, multi data center training is completely reasonable. And people do this. People do multi data center, multi chip training. Sure. Right. You know, you do your inference on one set of chips and you do your training on another set of chips. So like Anthropic does this. I don't know if Google does this, but Google's kind of already got the cards. Yeah. OK, got it. Let's go to Arrakis. Yeah, talk about Arrakis. Just there's this debate. TSMC risk. Is that the bottleneck or is energy the bottleneck?

15:43I was doing back of the envelope calculations. It seems like we're using maybe like 1 % of global energy production or Western energy production on AI, specifically, workloads. And then we're using like 50 % of leading edge fab capacity on AI workloads. And so that feels like, okay, well, even if we all agree, and we say as a society we're going all in on AI, we can only double the AI chip capacity before we need to build more fabs. That takes years. whereas we could say everyone turn off your air conditioning we're sending the electricity to the data that is right like we have the ability to do so now we have create new I need my cat dancing videos but seriously like there's this debate over you know is TSMC the main bottleneck or energy the bottleneck how How are you feeling about that?

16:40Yeah, yeah. So sidebar before I answer the question because I think it's fun. You know, in the US it's insane to say turn off your AC for AI. Yes. Right? And the general public hates AI already. Of course. But in Taiwan, they've had droughts before and they've turned off water to entire cities. They're like, oh, you get water three days of the week. Whoa. And then the fab still gets supplied water. It's like this is, you know, you've got to understand the mindset. We are not ready as weak Americans to do this. Yeah, that's crazy. No, but at the end of the day, water and power are certainly less big of constraints.

17:14Now, you've got to imagine, like, you know, semiconductor industry is used to, hey, doubling the amount of transistors made every year or two. Part of that is Moore's law, part of that is more capacity. Whereas the energy industry in America wasn't. And so, like, initially people were, like, not creative. They're like, let's do these kinds of gas plants. It's like, well, no, now we've realized, you know, yes, there's three main manufacturers of turbines and then you've got for a dual combine cycle then you've got like IGTs but you've also got like medium speed reciprocating engines right like turns out Cummins can make like a million diesel engines a year and like those can make electricity like if I don't give a fuck and I put it in West Texas easy um so now it's more of like a regulation thing a supply chain thing power is not a constraint in insofar like that much right I think it certainly is a constraint still today um it was the biggest constraint in 24 25 data center capacity power uh because the industry was not ready.

18:04People have woken up, they've sort of been shocked to the system. Now you've got tens of gigawatts being deployed. Next year, 30 gigawatts are being added, and we think the power's there for it. What was it this year? This year is like, I think it's like 18-ish, 10-ish. 15 to 18-ish, sorry. So almost a doubling. Yeah, almost a doubling, yeah. And when you look at TSMC and the crew, right, there is not really oh this random you know there's 12 people making medium-speed reciprocating engines that you can now convert to make power at some random data center no no there's like there is a racket yep right there is one set of spice like you know there's a you know that's it right and so and then and then the flip side is like okay when you have 12 vendors everyone's got a little bit of slack capacity you know there's more likelihood you know you can people like oh turbines you can't get you can call a broker and you can get a turbine you might be paying 50 % more 2x more but you can get a turbine yeah right like You can't get a 3 nanometer fab.

19:01You cannot get a 3 nanometer fab, exactly. And so when you talk about what's the, you know, the baton got passed from semiconductor shortages in 23 to power and data centers in 24, 25. 26, we're still, we're swinging the pendulum, but it will fully beat semiconductors again in 27, right? And so we see this across the entire space of the ecosystem. It's not just TSMC, it's also memory, both. Because both of them have built at a certain pace. Now TSMC has been expanding at some rate. The memory makers, in fact, have just not expanded capacity. Basically, they have not built new fabs since 2022 because their cycle is so undulating.

19:38Yeah, and so when you look at it, it's like, oh, even if they wanted to double capacity, they need to build the fabs, right? And building the fabs, it is the most complex building humans make, right? The entire area of a clean room circulates itself every 1.5 seconds. What? And you don't even feel it when you're inside. Really? It's like that, and it's like parts per billion of particles, right? Like, it's actually insane how you could get coughed in the face by someone who has COVID and not get COVID. It gets circulated so fast it doesn't even hit you? It's like that meme of like the spraying when someone's talking, and then it gets circulated.

20:14So another sidebar is everyone knows COVID like really popped off in Wuhan, right? Wuhan also is home to China's largest memory company, YMTC. And so when they were like welding people into their homes, the people who worked in the fab still went to work. Wow. Because it's, you know, one, it's a national importance. But two, like these people are getting sick. This fab is like way too clean. Yeah. Sorry, Jordy. I want to talk about Oracle. They put out a post this morning that said, Our partners financing for the Doña Ana County, New Mexico, Shackleford County, Texas, and Port Washington, Wisconsin data centers are secured at market standard rates, progressing through final syndication on schedule and consistent with investment grade deals.

20:58Obviously, they were fast following their posts from yesterday where they said the NVIDIA OpenAI deal has zero impact on our financial relationship with OpenAI. We remain highly confident in OpenAI's ability to raise funds and meet its commitments. And obviously, everyone was looking at this being like, give me a cigarette. I like spoken. It's like bank run language. I haven't seen posts like this since the FTX era. Is it just bad comms or is there something worse? It's terrible comms. It's terrible comms. I told my Oracle contacts, I was like, who the hell is in charge of the Twitter? What are you doing?

21:31NVIDIA did something similar last year when the whole TPU mania was going on. Yeah, it was like, we're thrilled with Google's progress with the TPU. That said, NVIDIA chips are the only, you know. It's like no one asked you to comment. I mean, like, I'm sure a handful of people in your DMs are random, but that doesn't mean... It's sort of the lion shouldn't concern themselves with the sheep. And like, okay, NVIDIA is a lion. Maybe Oracle is a little bit more bumpy, but I think Oracle is like fine. People are just freaking out because, you know, OpenAI is peak. You know, people are peak negative on OpenAI right now because of how good Anthropix has been killing it.

22:09But yeah, I think it's just like kind of silly. Like they need to hire someone to do comms like a Lulu or something, right? Both NVIDIA and Oracle because what are you doing? How did you process yesterday in general? Jensen was clip farming. He was like, I don't know why he does these street interviews, right? No other CEO does those where they just stick 25 microphones in your face and the paparazzi is flashing. It's a great vibe. It's, you know, Jensen's not been as famous as other CEOs for as long, and yet he's so important now. And if you've like, if you know Jensen, how he's in meetings, I feel like there's two Jensens, right?

22:49There is like PR, like good at PR, just good at talking, good at like making people hyped up and believe what he's doing. He's great at standing on stage, holding up the chip, delivering like a sermon. And then there's the real Jensen, which is like a business killer. Yeah. And like actually just knows about every like aspect of the supply chain, right? All the way from like niche semiconductor, you know, design and manufacturing stuff all the way to like energy power data center. Like and then doing the business deals too, right? And so like you've got this whole Pareto, like a whole thing, a whole range of things that he's good at and he's a killer in.

23:23And clearly he's like he was in a meeting where he was being a killer and like negotiating like supply contracts or something And then he walks out That's hilarious That's my inference But I like it Yeah That's awesome And that's why he was like you know like he was like still a killer like no we never said we committed to 100 billion You know like and it's like I don't know where do you even get the 100 billion dollar number from And it's like well you did go on CNBC and like you know make a big deal out of it So I think people would assume that it was, but they did say in the press release, I remember these are early talks, but they just kind of jumped the gun.

23:58This was the height of the press release economy. Yeah, what's funny is Oracle stock peaked just a week after they announced the OpenAI deal. And so the press release of like, hey, OpenAI's gonna do this humongous deal, stock peaks. Same happened with a couple other vendors who announced deals with OpenAI or NVIDIA. A lot of these, they all peaked then and then it's been like NVIDIA, OpenAI Trade has been going poorly and the TPU, Anthropic, Google, Amazon complex has been doing well. It's quite interesting that this happened. There's been good energy back at home with the roommates. What's going on in here?

24:37I wanted to... One more thing. Over the weekend, it was drowned out by all the Justice Department stuff. but we have you just talked about Elon saying you can smoke a cigar in the fab no yeah yeah yeah this is I didn't realize that was related yeah that makes this yeah indoor heaters yeah we have indoor heater technology I know it's taking yeah what does the fab look like if you'd have no humans inside like that's probably his long-term things like yeah there there will be an optimist no one no like the number of people working a fab is like irrelevant like but but is it is it irrelevant because there's all these things you have to do when a human's in there because they sweat and they breathe and if you don't have to do that because it's a robot walking down even if it's puppeteered or tele-operated you you might be able to have different considerations I don't know if that actually affects well it's like a nesting of like it's a nesting of the cleanliness right for example you've got this wafer you've put like down let's say you put down copper yeah and now you're moving it from one area to another well it needs to be stored in a vacuum but the easiest way to store a vacuum like or an inert gas yeah and that's like the thing that's being transported in but then around that you want it to be super clean as well you If you don't then the copper starts getting oxidized, it affects our yields, all this sort of stuff happens.

25:49And so like you kind of want it to be a nested layer of like, well this chain this thing inside the EUV tool is super clean and then the thing feeding it is super clean and then the thing it sits in is super clean. Because that's how you get to like there's zero particles. Because like you know in the FOOP, in the transportation devices like parts per trillion and maybe FOOP, it's called F-O-U-P, front operated, front opening, I don't know, something pod. But it's called a foop. It's like the thing that moves and it carries the wafers. Sure, sure, sure. And then the fab is like parts per billion and you've got this nesting relationship so everything is super clean.