CoreWeave’s Brannin McBee on the future of AI infrastructure, GPU economics, & data centers | E1925

0:00You look at the existing cloud infrastructure that was built over the last decade, it was built for serializable workloads. It wasn't for parallelizable workloads. And it's like you're having to rebuild the cloud, so to say, and you're having to rebuild physical infrastructure at the pace of AI software adoption. It's a mind-blowing concept, right? Because AI software is being adopted at the most rapid scale of any technology that we've ever observed. I mean, we're building at, I think it's 28 data centers this year across North America. We're one of the largest operators of this infrastructure in the world.

0:34And we are unable to keep up with demand. And we really don't see that subsiding for years to come. This Week in Startups is brought to you by OpenPhone. Create business phone numbers for you and your team that work through an app on your smartphone or desktop. Twist listeners can get an extra 20 % off any plan for your first six months at openphone.com slash twist. Gusto is easy online payroll benefits and HR built for modern small businesses. Get three months free when you run your first payroll at gusto.com slash twist and Northwest Registered Agent. When starting your business, it's important to use a service that will actually help you.

1:24Northwest Registered Agent is that service. They'll form your company fast, give you the documents you need to open a business bank account and even provide you with mail scanning and a business address to keep your personal privacy intact. Visit northwestregisteredagent.com slash twist to get a 60 % discount on your next LLC. All right, everybody, welcome back to this week in startups. We've got a great guest for you today. You may have been wondering who's buying all of these nvidia h100s and how are people getting access to all of this hardware well there's a couple of companies that got to the hosting of ai and gpus early one of those companies is core weave they started in the space um i believe doing a lot of crypto where miners were renting gpus from their cluster and uh fortune favors the bold they were in the catbird seat when the ai revolution happened and everybody decided well they got to train their own models going to need a bunch of nvidia's hardware and other people's hardware and we'll talk about that today and they've since grown uh to a massive scale um for those of you who don't know nvidia's market cap has increased more than 7x from 300 billion to 2.3 trillion if you've been living under a buck and you haven't been watching this it's because people want access to these chips welcome to the program brannon mcb who is uh the cdo and co-founder what does cdo stand for chief development officer um so so my role is raising capital for the business i interface with um equity and debt participants for the company and help fuel the growth of the business And this is a very capital intensive business.

3:15You spent a lot of money on GPUs and setting up infrastructure. The company's been around for just under a decade. Am I correct? Yes. We founded the company in 2018. Got it. Am I also correct that you were supplying GPUs largely to crypto and Bitcoin miners and this cohort of individuals? Or that was the beachhead market? It was absolutely the beachhead market. So I can take a couple of steps back on our founding story. So it's myself, my two co-founders. We're from the institutional commodity trading sector. So we're risk managers by background. We're from hedge funds. Finance guys is probably the best way to look at it.

3:57But we were finance guys who were heavily data oriented. We worked in this commodity sector that you can actually solve for price. You can figure out supply demand. And there is a dollar per barrel of oil, so to say, that that solves market dynamics. So we've always worked with a lot of compute. We've worked with a lot of software. And the crypto space was interesting because it was a arbitrage opportunity. Right. There's there's a very discrete input price cost of power. And you could model the revenue very efficiently because there was no customers. Right. You were just participating in this this network.

4:37And thus, if you were just to sell the revenue from the crypto mining proceeds every day, you could effectively qualify it as an arbitrage opportunity. And that was interesting to us. But it wasn't as compelling as a large business because at the end of the day, all you're going to do is chase the price of power lower. right like that's that's the only advantage you can really extract unless you expand into other markets right so when we're looking at the cryptocurrency space it wasn't bitcoin mining that we were interested in it was ethereum mining and gpu oriented mining because a bitcoin miner basic uh things that are produced by entities like like that man they can only do that one thing They can only participate in Bitcoin, and they're very good at it.

5:31But a GPU, well, it can do lots of things, including running AI workloads. So we started in the crypto space, but it was always with this idea. And we had no idea how complicated an idea was at the time, but started with the idea that, well, you could do crypto and other things. Right. Started there. The other use of these is, I guess, running video games in the cloud. Is that correct? yes cloud video games become a real market or do people who are into video games just buy themselves an alienware dell whatever and be done with it i think it's more the latter um that they they use it more for that we certainly don't see that demand for video game streaming um i think a few of the hyperscalers tried to launch into that market and i don't believe that there's been a substantial demand.

6:24Got it. And so when we look at crypto, that market kind of fizzled right as AI was starting to boom. So you were able to sort of just navigate that? Or is the crypto still going on and people are still using your services for Ethereum and being part of that network? Or is that just too hard of an arbitrage now because people in China have stolen electricity, we hear, or their friend runs the hydro dam, so they run an extension cord, so to speak, over to their warehouse with a bunch of servers in it. And you're up against people getting zero cost of input electricity. It's a great question. And yes, I'm extremely excited that we're not involved in that cryptocurrency market anymore.

7:06We haven't been involved for a number of years at this point. We actually started making this transition into the cloud infrastructure market in 2019. We hired Peter Selenke, who was recently elevated to our chief technology officer position in 2019 to build a cloud for us. And it's not just plugging in GPUs and having users come access it. It's this really complicated software stack that runs the cloud. Or in other words, it's an orchestration environment that enables users to access and use our infrastructure. And it's one that it's very different than the way that the hyperscalers built it because they built for hosting websites and storing data lakes.

7:53And we built our cloud from a no compromises engineering solution for running AI workloads and highly parallelizable workloads. And there's engineering decisions you make in doing that that you wouldn't make for hosting websites. and that's allowed for us to, I would say, overperform in the market with a product that really doesn't have competition. We started in 2019. Yeah, the software layer to provision these H100s, A100s, whatever people are using, that's a key part of the puzzle that you have to build, you have to master. And AWSs, Google's cloud, everybody's Azure, they're all slightly different in using their own provisioning software or is there some open source standard there for doing all that?

8:41That's exactly correct. We use and contribute to open source as much as we can, but we have a proprietary orchestration solution that looks different than the hyperscalers do. My favorite analogy for this actually comes out of the automobile sector, where at the end of the day, everyone produces vehicles the same way, right from research design scaling servicing it's the same sort of product with different badges and different colors on it right and it's been that way for 60 plus years and then in the 2000s a company came along and said well what if we started with a blank slate and designed this process today and you know ultimately uh ford might have to produce vehicles like tesla does But I think we can all appreciate the foundational difference in the way that those vehicles have brought to market and the challenges that Ford will have to go through to get there.

9:42And I think it's a lot of the same for the hyperscalers, right? I'm not going to tell you a trillion dollar company with tens of thousands of engineers can't do what we do. But I will highlight the innovators dilemma that sits there because there's an existing product. You have to change everything underneath to run infrastructure like we do, and it's going to be hard to get there. And if we go through the economics of, let's just say, NH100, this is NVIDIA's state-of-the-art. It's not just a GPU, it's a rack, essentially. It's a platform to put many GPUs on. I'm not sure exactly how many the H100 holds, but it holds a number of GPUs.

10:25they go for like 30 40 grand each is my understanding so it's a um it's a server or node as we call it within a network fabric and a server has typically eight gpus within it um and then you put those into a cabinet and you put those into a data center and you bring power into the data center and you connect it to the internet right and then yes those things um that that price range was accurate for each one of those gpus in a server so a server can cost upwards of a quarter million dollars and so you rent out one of those h100 gpus an individual gpu in a server like that a cluster a node for four bucks an hour something to that effect yeah yeah yeah that that's right um where we specialize though is doing it at scale yeah right like we don't have many clients who just use one at a time.

11:22Our clients will use 10 ,000 at a time in a single contiguous fabric, which makes it a supercomputer. And it's interesting, this actually has become where we operate some of the world's largest supercomputers at this point. I think several of the top 10 now sit on our platform because of how large these fabrics are and how performant these GPUs are at these specific tasks. tasks so somebody fires up 10 000 of those i'm assuming they get some kind of volume discount so if it was two or three dollars an hour they're spending twenty thirty thousand dollars an hour on a job at one of those correct something in that range yes but i i will correct the the discounts it actually works inverse right because it's uh it's extremely surprising but it's because building a a single fabric of this size is so engineering intensive that not many people are able to do it.

12:21There's not a template. Not a lot of companies have gone out and done it. There's actually maybe three or four on the planet who are actually building fabrics of this scale. So you actually make a more scarce resource through scaling. You're kind of like decommoditizing the market through scale. It's not one GPU or 10 ,000 GPUs. It's, oh, 10 ,000 GPUs. That's a totally different engineering solution. Juggling multiple devices and apps to run your business is a mess. Open Phone is here to make it simple by simplifying your business communications with one easy to use app. Open Phone has rethought every detail of what a modern business phone should be.

13:04And here's the magic. It works through a beautiful, elegant app on your phone, or you can just use it on your desktop, making it super easy to get a business phone number for your entire team. And And you know how brilliant OpenPhone is? My teams use it every single day. My sales team loves it. My ops team, they use it all day long. And here's the features that we love. You can create a shared phone number, like customer support, with multiple employees fielding all the calls and all the texts to that one number. At my investment firm launch, we pride ourselves on replying to every single call or email instantly.

13:37And OpenPhone is the number one rated business phone on G2 for customer satisfaction. So here's your call to action. Super easy. OpenPhone. is already affordable. Starts at just 13 bucks a month, but Twist listeners get an extra 20 % off any plan for the first six months at openphone.com slash twist. And if you have existing numbers with other services, no problem. Open Phone is going to port them over easy peasy, lemon squeezy, no extra cost. Head over to openphone.com slash twist to start your free trial and get 20 % off. And how much of that cost, when we look at a$4 an hour cost, would you say energy is?

14:10Because these things are extremely energy reliant, right? They consume a lot of energy. So I'm curious how much of all this is energy. And then where do you put your, this will take us down the energy rabbit hole, but where do you put your data centers? And then where are data centers going to be in the future? Because these GPUs are taking a multiple of what CPUs would use. Yeah. So maybe you could explain that to us. Yeah. Yeah. It actually causes a pretty substantial bottleneck that exists in the market now is data center capacity. Right. And it's not square footage. of data centers. It's data centers that have enough power brought into them and adding more power to those data centers.

14:51And arguably, that's where the next bottleneck in this cloud infrastructure or GPU cloud infrastructure market sits, is how do we access enough data center space to accommodate the volume of demand that's coming in? So power, it's roughly about 10 % of our cost to deliver this infrastructure. The infrastructure itself is actually where most of the cost sits from a depreciation perspective. Depreciate over a six-year life on the infrastructure. The power side is... That's my background. It was in power markets and trading different electricity markets. So it consumes a lot of power. It has an immense amount of efficiency over CPU infrastructure.

15:38as well for what it's doing there. To run the same workloads on GPU versus CPU, it's actually more power efficient to run it on a GPU, right? If you're trying to achieve the same outcome, because you'd have to use so many CPU cores to get to the same solution. So yes, they consume more power on a density basis, but on a workload basis, they're more efficient. And this is, I mean, it's staggering. I was looking at one study that said, Like one of these GPUs at 60, 70 % capacity year is like the average American households, energy consumption. And that's just one of them. So this would be the equivalent of like, if somebody is using 10 ,000 of these, or I think Zuckerberg's going to be using low millions of these.

16:24It's like putting a million households online or something to that effect. Yeah. So. Yes, it's an immense amount, but it's also a, you know, it's a transformational technology. methodology i'd say that we're looking at and it's and its ability to unlock value from data is something that we've never observed before um it going to yeah so that speaks to justifiable like yeah i mean i'm not even i'm not even looking at this judgmentally like is it worth the amount of energy it's consuming i was looking pragmatically where let's assume it is where it's going to come from yeah let's say it's going to cure cancer it's it's going to find uh solutions for renewables or fusion that like we didn't even conceive of or the gains from it will be so extraordinary.

17:09It will obviously pay for itself and create an energy independent future. But what's happening in the industry today as people are buying these and looking for places to store them, you're looking to build up your infrastructure. Are we just out of energy? And where are people where are the nooks and crannies where people are looking to locate these facilities? I going to become like a place where people put these uh uh the plan is to put a nuclear power plant and these gpu data centers next to each other yeah is there any truth to that yeah yeah look i i believe that was microsoft or amazon is is effectively taking that nuclear plant and citing a data center next to it power i think you know beyond data center space right it is a national concern, so to say.

17:58There's been an immense amount building out renewable capacity over the past decade, which is fantastic, but it's also not necessarily the right kind of capacity, what you need for consistent demand growth. As you know, solar works when the sun's out, wind works when the wind blows. Neither of those things work for a data center or even necessarily for electric vehicles, for all these kind of demand areas. We need more baseload power. That's traditionally come from coal and natural gas. Fortunately, it's been more so from natural gas over the last decade because coal is quite dirty from an emissions perspective.

18:40And my personal hope is that it's more nuclear going forward, but it takes time to build nuclear sites. I think it's a decade for siting and build. In the United States, and we haven't built one in a long time. I mean, I think the last one broke ground in the late 60s or early 70s, and we haven't had one since. So this would lead one to believe that somebody with a lot of nuclear power and a lot of GPUs would have a massive advantage. It certainly helps. We've contracted a substantial amount of capacity for looking to ensure the growth profile of our business, but it is going to be a bottleneck for all other participants in the market.

19:26what about heat uh these things throw off a lot of heat um and you know some areas in the country are warmer than others is it are people moving these data centers north in order to get the cold air to just you know we've we've seen pictures of you know data centers that have open sides where cold air just blows right in or you know open doors essentially um because in other places if you were to put these gpus in texas i think you're going to be air conditioning them which seems doubly inefficient. So maybe talk a little bit about the heat these things generate today and if there's any hope of cooling them down without air conditioning.

20:04Yeah, so it's a great question. It's funny, like it takes me vividly and visually back to my crypto mining days where we did run those warehouses with the open sides and the giant fans and we were up north, could never run this infrastructure in those environments, right? From a security perspective, from a reliability perspective, it is mandatory to run this infrastructure in what's called a tier four data center environment, or sometimes even a tier five data center. And that's the highest classification in terms of reliability, redundancy, security, and environmental handling. So these are sites that you would see like Amazon or Google or Microsoft running within, Like true data centers that are meant for cloud infrastructure.

20:54And the way to think about the heat output, the other variable in there, which is wild, is actually the sound. They're extremely loud in these environments, upwards of 100 decibels, which is a direct derivative of the heat they're consuming. And then to move the air. But the way you look at it is the critical load around the infrastructure. So if it takes one unit of energy to run the infrastructure, it takes another 0.2 to 0.3 units of energy to cool the infrastructure and run the networking and everything else around it. The way that the world's leading sites handle this is just through forced air.

21:35Just move tens of thousands of cubic feet per minute of air through these highly contained pods. So you have all the hot air in a really small area and you're just jamming air through it. Right. And sometimes it's conditioned. Sometimes it's just air. But eventually it's going to be liquid. Right. We're going to move to an environment where you have direct to chip liquid cooling instead. And that efficiency ratio, call it 1.3, will drop to about 1.1 instead. So your GPU infrastructure will inherently become more energy efficient as we move to a liquid cooled environment. And we're working with leading data center operators, such as switch to, to facilitate and implement that movement for these upcoming generations of GPUs.

22:28when people say liquid cooled um most people have not actually physically seen that yeah unless maybe you're a gamer and you've seen your chip in a a tube run to the chip and there's literally liquid on the top of the chip that's cooling it um how do these uh when you say liquid cooled what could people envision uh of how these solutions are going to work are they going to just be like a bunch of racks in a in a in an olympic-sized swimming pool or is it just like little contained amounts of water on top of the gpus yeah so you're qualifying that correct there's two broad categories of liquid cooling there's immersion cooling which is the olympic swimming pool method downsized obviously and then there's direct-to-chip liquid cooling which is running the pipes to the chips um we will sit on the direct-to-chip liquid cooling side because it's uh operationally more efficient for us um that that's where we think that the sector is broadly going to go.

23:27If you think of immersion cooling, you're literally dunking a server into a vat of liquid. That liquid has its own problems with it as well. But let's say you had to go service that server. There's a node or a component that was wrong with it. Well, you got to lift it out of the liquid. And what happens then? Well, you got to wait probably an hour for all the liquid to drain out now before the tech can even get into it so you're extending these service times yeah materially and response times versus direct the chip liquid cooling uh you know pop it out and you don't have water containment issues you know things splashing across data center like it's it's a mess these are highly sterile and contained environments that even let us bring cardboard inside of the data center area because it's combustible right and you can have little particulates that float around the site and can accrete into the nodes um bad you don't want fires and data centers i mean when you are talking about that much air being pushed around that means any particulates in the air are going to get pushed around and so if you just had some very small amount of particulates floating around a room now imagine that room is changing the air every x amount of time the number of particulates is going to grow and then you're going to have a small fire on a chip which is just absolutely crazy do you think the demand is going to keep up are you seeing any signs of demand people saying okay we we built our 10 000 gpus we're making more efficient software making more efficient use of the chips okay yeah we're getting to a steady state we've bought enough so are you starting to see that with your customers saying you know what we've got enough gpus we got enough infrastructure right now or are they still in the begging, pleading, and doubling?

25:19What are you seeing from top customers? Are they doubling their capacity every year? Are they tripling? What's the field report? Yeah, I'll qualify it a couple of ways. So one way, we will increase our revenue by about tenfold this year. And we're already sold out of all of our capacity through the end of the year. So I have a build schedule. We have about 500 employees today. I'll be closer to 800 by the end of this year. That build schedule is fully booked this year already. We see that broadly across the sector. There's just an immovable wall of demand for this compute. A lot of it is being driven from this move from training the models to inference.

26:04And inference is actually bringing the commercial value out of training. So you want to go train a foundation model that takes compute to be built in the configuration that we build it in these 10 ,000, 30 ,000 GPU clusters. And then you got to go make it actionable, drive revenue off it and bring a product. And what we're observing is it might take 10 ,000 GPUs to train a model, but inference is linked to the number of users. If you go into chat GPT, for example, and query, that's spinning up a GPU. And now there's a million of you doing it, 5 million, 10 million. That informs the size of inference.

26:48So inference will really, truly be linked to the growth of this market. And we're seeing users who are using 10 ,000 GPUs for training need hundreds of thousands for their early stage inference products. So we don't see demand going anywhere but up to the right for this infrastructure. So while they may not need exponential use of, you know, GPUs for training, they'll get more and more efficient at that. And what they will need is those inference when people ask the query, that's inference, not training the model, but asking a question of the model. That is massively compute intensive. and in h100 if we were to look at that unit in an hour at full capacity how many queries you know i know it depends is always the answer but an average query like these things were costing a couple of pennies per query is that correct ballpark yes yeah that that's correct and i think that's the right right way to qualify it is is uh cents per query or dollars per query So you're getting in hundreds of queries within that period.

28:04And as you said, that will become more efficient over time as well. But it's just an unbelievable volume of demand. And when you step back and think about it, you look at the existing cloud infrastructure that was built over the last decade. It wasn't built for this use case. It was built for serializable workloads. It wasn't built for parallelizable workloads. And it's like you're having to rebuild the cloud, so to say, and you're having to rebuild it. You're having to rebuild physical infrastructure at the pace of AI software adoption. It's a mind-blowing concept, right? Because AI software is being adopted at the most rapid scale of any technology that we've ever observed.

28:46And you're asking people to build. I mean, we're building at, I think it's 28 data centers this year across North America. We're one of the largest operators of this infrastructure in the world. And we are unable to keep up with demand. And we really don't see that subsiding for years to come. Listen, as a founder, there are things I love doing, like building products or meeting with partners. hanging out with my team and dreaming up new ideas. And then there are chores that I don't want to do. I don't want to do HR. I don't want to do payroll. I don't want to deal with all that. So I use Gusto.

29:21Gusto is the best for payroll, for HR services, and for running a small business. It makes everything so much easier. Even a midsize business, man. I get a lot of portfolio companies that are pretty sizable using Gusto because it is designed for you, the small business owner. And payroll is something you definitely do not want to mess up. You got to get it right. And Gusto is going to make it perfect for you by calculating paychecks perfectly. Also payroll taxes, you got to get your taxes right. You can't make mistakes there. And you want to set up open enrollment, you want to be good to your people.

29:52Gusto handles onboarding, health insurance, 401k, time tracking, commuter benefits, awful letters, and they even give you access to HR experts. So Gusto takes all of this off your hand and lets you focus on important stuff, your product and your customers it's super easy to set up and get started and if you're moving from another provider gusto will transfer all your data for you here's your call to action because you're a twist listener and you're part of the family you're going to get three months free incredibly generous totally unnecessary thank you so much to our friends at gusto.com slash twist you must go to gusto again gusto.com slash t-w-i-s-t to get three months free thank you gusto team so then this would lead us to um lpus uh obviously using a gpu very expensive right um but grok my friend chamat's company um has this uh uh inference engine and these lpus are you starting to see those and that hardware stack emerge these language processing um units and do you think that'll have a good effect on the industry in terms of lowering cost and having purpose-built hardware for the inference moment yeah so so as opposed to the lord of the rings right where there's one ring to rule them all i don't think that there's going to be one gpu lpu one accelerator to rule them all nor do i think there's gonna be one model to rule them all either i think there's going to be lots of different models with different objectives right like models that do different things, whether it's helping drive a car or cure cancer or be an AI character.

31:33Models will do different things. And then there will be infrastructure that is most efficient for each different type of model. And I think that's why you're seeing entities like Microsoft, Meta, et cetera, who are focused on building their own silicon. They're not trying to replace the GPU. They're just trying to solve for different models that they're running internally. So I think the Groks of the world will absolutely have a place somewhere, but I also think that you'll see GPUs have this place. And what we're observing their place is at foundation models, at latest generation models, the most demanding and complex workloads will continue to sit on GPUs.

32:18And NVIDIA just has this unbelievable solution for iterating continually better generations of GPUs. And we think that those models will continue to accrete to NVIDIA's platform. So they're going to win the day, no doubt. uh nvidia when it comes to training the models inference you might see other folks carve a niche for themselves is how you would bet this emerges yeah yeah yeah i think inference will will have various levels of infrastructure that provide solutions for it um i will say you know if a model is trained on a100s it'll probably run inference on a100s as well like like kind of tough to make that architecture shift.

33:03And it's tough because of the software that NVIDIA has, their driver solution, CUDA. NVIDIA very thoughtfully open-sourced that driver solution in the early 2010s to support this sector and the engineers who wanted to work on these products. and it has become effectively a default solution across the market, right? It's similar to drivers for CPU, right? Everything was x86 for decades, right? And it didn't really matter if something was better or not than it. It's just what people use, right? Because there's an efficiency loss if you say, well, I'm going to go learn this other thing and just hope other people will use it or you could just use the thing that everyone else uses.