The Infrastructure Behind the Machine Age

22:59Erik Torenberg:Ben, I know you're thinking a lot about this and how this works in the organization. You think a lot about culture, of course. What are your thoughts here? Well, I mean, I think if you just look at us,

23:12it's like having a new kind of employee, and there's going to be a lot of them. And we have to, just like with our, we spent many, many, many years figuring out how to work with our kind of regular human employees, and now we've got these other kinds of employees. And there is a learning curve with them. So they can burn a lot of tokens and spend a lot of money and get nothing productive done. They can forget stuff. They can make stuff up. You know, they can have good behavior. They can have bad behavior. Like humans. They can create security problems. So, like, there's all those aspects to it, but, like, they can also be, like, super-duper productive.

24:01And so I think figuring out how to integrate them in, have them work nicely with the people that they're working with, the actual humans is all something that we're learning how to do. I mean, I don't want to sit up here and say, I've cracked the code, we've got this marvelous loop, and the whole firm is just completely automated now, and I'm going to slowly get rid of all the humans because I can. Like, that's not at all where we are. We're much more going like, okay, how do we make all our humans superhuman without, like, wrecking the place because the bots get out of control? So, yeah, and it's interesting because we've done, we've tried a couple of different ways.

24:41It's how best to get agents into the system, if you will. And eventually, it was Martin's insight, just treat them as people and get it done. And that's what we're doing. That's turned out to be the most durable way of getting this thing going inside of an organization.

25:00Erik Torenberg:I want to go back to the supply side and go deeper into the bottlenecks. We were talking about how, you know, in terms of the data centers, the chip architecture, system software, facilities themselves, that none of them were designed with AI in mind. What would it look like for them to be designed with AI? Like, what is sort of the mental model for thinking about what that could mean? Yeah, so, I mean, if you start with the statement that you just said, hey, original model of infrastructure on any of these models has to change. You can go category by calling and see varied breaks. And then you start unlocking the bottlenecks and the tone of these things.

25:42So eventually you have to get to a system where if you look at what an inference engine does, it takes up a lot of memory. It generates new tokens along with the compute. And so you can just think about how do I optimize all of this? What does the memory need to be? What does the compute need to be? How do they need to talk to each other? how much power does each of them need? And if they need all of this power, how do you code each of these, right? And then how do you put the collections of these things together? That is the exercise that's underway in the industry right now with a lot of the founders.

26:14So they're breaking down the problem into its fundamental comports and saying, what is the exact nature of the compute that's getting done? Okay, how can we make its multiplications? How do I optimize my computer and that kind of a scenario? and that they all need memory progressively to generate these tokens. What is the best way of hierarchically arranging this memory, right? And then how does the power consume? I mean, then you've got to connect it together. What are the ways of connecting it on the same chip but across chips and across data centers? How much power does each of these data transmission take?

26:49So you have to progressively break it all down and rebuild it from these fundamental building blocks. And that's what we see underway and that's where we see that generally. Let me give you an interesting mental model to think about how the landscapes change. So today to build a frontier model costs, let's say,$3 to$5 billion, right? And that's to train it. And so the inference has to pay back at least that, of course, right? In order for any of this stuff to be viable, let's say two times that. So let's say that now inference has to make$10 billion. So if you can save 20 % of efficiency on that, that's$2 billion.

27:30And you can easily build an ASIC for$2 billion, right? So we've actually gotten to this interesting point in the industry where it actually makes sense to build an ASIC per model just because the amount of capital investment in that model. And then unlike traditional software, traditional software has a lot of state and a lot of, you know, it's very dynamic. These models are fixed. The model weights are fixed. And so we don't know if the world goes to per model ASICs, but it gives you a great mental model of how you would evolve the architecture to be far more bespoke for these massive capital investments we're doing.

28:06I don't think in the history of the industry we've ever created a digital artifact with something like$5 billion that went directly into that artifact. And so, you know, like this, I think, is going to put the greatest demands on hardware that we've ever seen.

28:21Erik Torenberg:to that end rack power requirements are moving from roughly five to ten kilowatts to hundred to fifty kilowatts compute density is climbing something like 70x cooling is moving from air to liquid as a requirement what are the investment opportunities as a result of this well first of all when you get to that level of power per rack ac power doesn't work anymore so like that's a pretty wild thing. So now you're into DC power, which, by the way, also requires its own cooling and is, like, by the way, super fucking dangerous, which is kind of ironic because this was... Edison promoted DC power by claiming how dangerous AC power was and demonstrating it by, like, electrocuting animals and things.

29:13The horse, yeah. The horse, yeah. So, but he was right, but around his own kind of power, which is extremely powerful. It's the good news. So, you know, just starting with power, yeah, that's going to be, like, very, very different. I think with cooling, and this gets into, so, yes, we're going air cooling to liquid cooling. I think we're already at liquid cooling for any state-of-the-art data center. Like, that's already kind of a done thing. But it gets into, okay, you know, given the political environment and so forth, you, like, liquid cooling isn't enough. It's got to be eco-friendly liquid cooling.

29:57And, you know, kind of, DC power is not enough. It's got to be power that contributes to the power of society, not takes away from it. And so you have data centers who have been behaving badly, small percentage actually, probably 10%, wasting a lot of water. You know, not as much as pistachios or almonds and so forth as people demonstrate on the internet, but like they could be a lot more efficient with that. And then there are ones that, you know, kind of are parasites of power and don't contribute power back. Like, I think all that's going to end. It's going to have to end just because, like, we've kind of gone through a one-way door on that.

30:43So that requires, like, a level of engineering that, you know, many haven't invested in yet. So that's coming. And then, you know, like, if racks are that dense, there are other things that, like, the way the floors are designed have to support that kind of weight. You know, that kind of thing is actually for real. And then, you know, I think that there's, you know, you just need a lot of everything. And so there's going to be, you know, also by the things are really loud. So you have to build the data center with thicker walls or you're going to disturb the peace in the neighborhood, which is not going to be acceptable.

31:29Like I don't think any state's going to allow that. And so a lot of the ways people have architected and designed the buildings themselves are already completely obsolete. Like once we get to Feynman, a much smaller percentage of the data centers that we have today work. In fact, I mean, everybody talks about memory prices, but one of the fastest areas where prices are increasing is reinforced concrete. from their senders. The other thing that happens when these data centers are sending 800 volts to the WAC is it's become so dangerous, number one. But secondly, we don't have enough electrical contractors that have the expertise to deal with 800 volts inside the data center because this is high voltage.

32:19Only 2 % of electrical electricians in the U.S. have been certified on DC power. So, like, that gives you an idea. Now, Meta's got a whole program to train people up and so forth, which is great. It's like a new job core where they train people for free to do this job. But, you know, it's funny. AI is taking all the jobs. AI is going to create a lot of new electricians.

32:43Erik Torenberg:Yeah, I think we're just doing something in space, too. The guys that own the big cloud data centers, big data centers, they all... are furiously experimenting with robots, right, to do the work of assembling or putting servers into the data center, et cetera. And so you will see that increasing as a result of the evolution in AI. By the way, to be clear on the actual fund that we're raising, our focus is on computer science infrastructure. So anything a model runs on, that's computer science, right? So think, you know, chips, network, interconnect, storage, all the way down probably to the electricity.

33:25Erik Torenberg:Yeah. And say more about the robotics arm in terms of what we'll be doing versus maybe American Dynamism or how to think about that. Yeah, for sure. So, you know, again, we think that any platform that AI will run on, like one of the great breakthroughs that AI does is it allows computers to interact with the physical world, right? It can see, it can hear, it can talk, right? And this means new platforms, right? And the simplest way, people will say edge device, but that doesn't really mean anything, right? I mean, it could be a mobile device, it could be a CDN, it could be a laptop, but it also could be an embodied device that goes around.

34:00And so again, we as infrastructure-focused investors don't do heavy regulated industries or more verticalized industries, but any sort of computer science platform that's going to push AI further out, we're quite interested in. Yeah.

34:18Erik Torenberg:Going back to the data centers, by 2028, new data centers are going to need something like 44 gigawatts of additional power against maybe 25 gigawatts of expected grid additions. Hold on, hold on. We use that word gigawatt. No, it's like, oh, we'll have 100 gigawatts. Martin, what's a gigawatt? I mean, how big is it? It's multiple football fields. I mean, it's massive. It's 50 ,000 people. What do you mean? What is it? The equivalent is like 50 ,000 houses? 50 ,000 homes. 50 ,000 homes. I grew up in Flagstaff, Arizona, which is a town of 40 ,000 to 60 ,000 people, depending on the universities.

35:00We have less than a gigawatt of power consumption. So you can basically light up and air condition your entire town for a gigawatt. Yeah, I mean, this is... He's just throwing them around. No, but by the way, everybody talks about the gigawatt. There's very few gigawatt data centers that are actually up. I mean, we've got a long way to go.

35:17Erik Torenberg:To put that in, why can't utilities and hyperscalers just build faster? Oh, there's so many things. Well, there's, first of all, right now, you need humans to build them. So there's just like the regular construction. But much more than that, you need permits. You need access to power. that you can plug in. So you're either, you're doing a combination of you've got to get access to power, which is a massive kind of regulatory bidding struggle. There's very limited kind of amounts and things you can tap into in terms of natural gas, power grids, what have you. But then you also have to build your own power.

36:03And guess what? We've got shortages of transformers and turbines and everything that goes into that. So it's just, you know, like you've got to get all that stuff. This is not a software problem. It's not just like a bunch of engineers, can't you like work weekends and that type of stuff. Not that that works anyway, but there are real bottlenecks in this. And these lead times are not that easy to compress. And look, we have the best minds in the world trying to figure out how to compress them. But it's not easy. It's not easy. And the demand is not slowing down. So we're already behind. The demand is growing, you know, 10x a year right now.

36:52And the supply just can't grow that fast. By the way, it is so bad that right now, if we have new companies going for GPUs, it's often in Mexico or Australia or another country just because it is so difficult in the United States. Yeah, we're creating huge job, both job and long-term economic opportunity in other countries by banning data centers here. I think, look, the right answer would be to set a standard where a data center contributes back to the community. Like that power gets better, there's no noise, there's no water issue, and it's adding jobs. Like, that ought to be the standard. And then everybody ought to be just held to that standard.

37:35By the way, like, there are data centers that do that now. Like, that's not a, you know, like a futuristic dream or something. Rates, energy rates have gone down, like, every year they're there. And the reason is they provide their own power. They give power to the state during the day. And then at night, they borrow power from the state when the state doesn't need it. because the way power plants work is you're always generating peak capacity. And since a data center has steady capacity during day and night, and a city goes way up in the day and way down at night, that's a symbiotic relationship.

38:16Erik Torenberg:Zooming out, why do we think the, you know, we were batting around the name for a little bit. Why do we think machine age is a compelling term for what we're doing here? well listen let me let me take it right so the first one is i think ben's absolutely right artificial intelligence was the wrong word like we shouldn't have called it it's machine intelligence um you say more about that why is that uh because it's not how humans think necessarily right i mean it is a cache of how humans thought is a collection of humans thoughts but like today we don't know how to take a um uh an ai with no knowledge and put it out in the world and have it reconstruct language, right?

38:57Like, that's not what we've done, right? We've built something that can learn off of everything we've already learned and then use that in a productive way. And listen, AI is a general term that goes back 70 years in computer science formally that applies to many different things. And of course, it's got a lot of baggage, either from science fiction or from, you know, Nick Bostrom, who wrote about it or whatever. And so the first one is just an acknowledgement, like this really is machine intelligence. And then you want to emphasize the machine part of it. I I mean, there's kind of this deep irony, and this is from the software's eating the world people, that you've really come to a place where you pour money into something, and then you're limited by the actual machines below it.

39:37And so I think it is kind of a nod to the hardware component is so significant in this wave, and we want to acknowledge that. Yeah, I think that's what's going to create the next breakthroughs. Genitologies is the quality of the machines are made. So that's basically a reason to be named. It's also a cool name. machine age. That sounds good. Futuristic.

39:58Erik Torenberg:Yeah. Given how much has been spent on AI infrastructure to date and how capex intensive these businesses can be, are we past the point where new companies can break in at sort of material levels? You know, why not incumbents like NVIDIA, CoreEV, etc., just take the lion's share of these markets? They all do. Well, there's no question about it, right? But to our discussion earlier, when And you need fundamentally new innovations to keep the growth continuing or the pace of improvement continuing, whether it's tokens per second, per dollar, tokens per watt, tokens per rack, right? Or power. You take any metric.

40:44If you want to have a 10x on those metrics, you got to have new innovation. And new innovation traditionally comes from Boudin founders thinking about solving the problem from first principles in a different way, right? And that's what's needed here for the next jump in innovation. I mean, this is the law of markets, right? I mean, let's assume that the existing silicon incumbents are multi-trillion dollars in market cap, which is absolutely the case. Even 5 % of that is a massive private company. Massive private company, right? We're talking, you know, annual. And you could say, well, but NVIDIA could do that.

41:21They could, but why would they if they're focused on things that are in the 90%, which is also driving the same amount of growth? And you always ask these questions. We ask these questions during the cloud days, right? Like, well, wouldn't Amazon did this? You ask these questions during the Microsoft days. Why wouldn't Microsoft do this? This is a very natural law of markets is that once you get to a certain scale, there's tremendous opportunity for innovation at the margins. Yeah, there's a funny quote from our partner, Alex Rampell. He had this startup called TrialPay. And he was trying to sell it to or sell his services to Meta and then Facebook.

41:56And Dan Rose, who was the head of Corp Dev at the time, said, Alex, that's great. It sounds like you can collect a lot of silver bricks. But I'm like, I have so many gold bricks, I can't even pick them all up. So the last thing I'm doing is looking at a silver brick. And I think NVIDIA is in that position. 100%.

42:13Erik Torenberg:Yeah. We were talking about as it relates to the model providers that if you're, you know, in the sweet spot of what open air arthropic can do, you know, one of their main sort of interest areas, that might be a tough place to be. But anything outside of those maybe, you know, three to five areas might, you know. As markets expand, they fragment, right? And it happens all the time. And remember in the early days of Ford, there was 1913, there was the Rouge River plant. Literally, this was like made cars. It went like water, coal, and rubber trees, and out came cars. By the way, he bought a whole rubber tree plantation in the Amazon jungle.

42:52And there's a great book called Fordlandia. Because he wanted to own the complete vertical thing, where he created this city called Fordlandia in the Amazon jungle, which had like, was all Americanized, bandstands and ice cream and all this kind of stuff. And it actually worked for a while until he made people like show up to things on time. And then they were like, screw this, get the fuck out of here. So now if you look at the car industry, of course, there's multiple levels of supplier and there's a bunch of companies. And this always happens. So, you know, as markets expand, they fragment.

43:28And then once that growth slows down, they tend to consolidate. The consolidation can either be acquisition or it can be like new challengers rise up. And that is the everlasting cycle of private markets. Yeah, because the use cases are multiplying and there's no way, like if you're the biggest company, you can get to the biggest use cases, but there's so many use cases. And all, as Martin was saying, very valuable use cases that it's just very hard to get to in a great way. Yeah, even in France, it's just one simple architecture, right? No longer, it's just like, it's so complex now. So it's inevitable that you can optimize things in a different way.

44:07By the way, here's a very interesting thing. Like, people don't, often don't understand that, like, margins kind of fell out of the standard way of doing the technology with software. Right? Like, it wasn't really a technology problem. Like, once you got the business working, you tended to have pretty good margins because that's just kind of how software works. Certainly when you shift it, but even as a service. And that's not necessarily the case with AI. We may actually be entering an era where the optimization in the hardware is absolutely meaningful to the upside of the business in a way that we haven't seen in the past.

44:40So there's a lot of opportunity here.

44:42Erik Torenberg:Let's get deeper in talking about the types of companies we'll be investing in. Maybe we could start by either illustrating the subsectors or if we can talk about a few or a couple investments that we've made. I know there's some that haven't been announced yet, but Ravi, do you want to take us down? Yeah. I mean, the sub-sectors, as we've been talking about well, is every one of these categories, right? The obvious ones are compute chips. But these days, it's not enough to build a chip. You need to build a full system, right? And then, therefore, what goes into the system? There is potentially memory innovation.

45:17There is potentially networking innovation. There is potentially power chips, and so on and so forth. So each one of these categories are categories where you can see public company style companies emerging. And those are all the things that we are looking into. And then once you put it all together, there's a lot of software around it to automate all of these things, to manage these fleets, and so on and so forth. So that is another important area. So these things keep building on each other. Every one of these categories is important.

45:48Erik Torenberg:Talk about what's different about these kinds of companies from the usual companies. I mean, one thing you can tell from the companies we announced is their first rounds have been massive, you know, hundreds of millions. Is it a different kind of founder? Or what else is different as we think about just the practice of, you know, building and investing in these kinds of businesses relative to our traditional software? Well, I think the big thing is you hit on one of the big things, which is a lot of money goes in before they get to a product. And that's just kind of the nature of it. Now, that's true on big models, too, but I would say that's a little more of a known path, whereas this has got a little more risk and a little more money than some of the other things that we've done.

46:33And, you know, look, a lot of the chip founders are here from the past. You know, like the guys who know how to make memory, they're not young. Yeah, so it's, you know, that part is different too, but it's kind of exciting, you know. Yeah, the other thing about these founders, they all got to be systems founders. So what I mean by that is you can't just be a researcher or a great continuous artist, right? You got to be able to architect and design the chip or the system, whatever it is. Then you got to think about how is this thing going to actually get manufactured, right? Who's going to be supplying this?

47:16And a whole bunch of these downstream things, which normally if you're building software, you don't have to think about all of these things. So to really the best founders, and of course, Jensen is the Michael Jordan of this, right? They think the entire ecosystem right from the get-go before they start designing the chain, right? Because of the nature of the bottlenecks and all these things that have to come together. So that's a big characteristic, but it's different. There's two environmental factors that are important to you. The first one is the labs are so desperate that they will engage with startups.

47:52And so, like, we actually have quite a bit of signal early on because they're, you know, the labs are inking deals with companies before they actually have hardware available. And that's a big, big shift than, you know, five years ago, right? Like, you just didn't go and, you know, sell your kind of janky hardware thing to Google or whatever. So that's a shift. The second one is the capital availability has loosened up a lot. I think there's general consensus that, you know, it is the time to reshape this stuff. And so follow on around, there's a lot of capital available, which, you know, of course, you want to be investing in two areas where there's capital available.

48:23And so the atmospherics are also just different.

48:26Erik Torenberg:Patrick Colson, you know, remarked a few years ago, said, hey, it feels like there's less younger founders today in the way that Zuck, you know, in college, building next Facebook or Gates, you know, in the same way with Microsoft. And of course, you know, the Michael Trulls of the world, there's still some young founders building iconic companies, but it does seem, you know, to your point, that there's more older founders building these companies or less 20 year olds. I'm curious if it resonates and why. Well, I think it's Ragu's point that if you're building something that has like a very complicated supply chain, has to manufacture things, and is technically complicated, that, you know, some experience helps.

49:14And, you know, if you look at Elon or Travis Kalanick, their companies when they were young were software companies. It wasn't until they got like a lot, even those guys, the best guys, needed some experience in building a company, building technology and so forth to kind of graduate to the much more kind of complicated or I would say elaborate domains. You know, there's just much more, there are many more moving parts in these things. And so, look, when you're learning how to build a company, it's hard enough if you completely understand the product. If you don't completely understand the product and have to learn it while you build the company, that's just such a steep learning curve for a brand new entrepreneur.

50:03So I think that what we're seeing is you see Michael, on the one hand, who is a very young guy, brilliant, but what he built was kind of a pure software AI thing. And then on the other end, you have like an Elon or a Travis who's got enough experience. I think Michael could probably do that, you know, 10 years from now. But today, that would have been hard. And it's important to remember, like, it's been defocused by the entire industry and academia for the last 20 years, right? It just hasn't been the same opportunity. Like, it's been there, but, like, it's never been a growth area. The growth areas have been, you know, software, networking, things like that.

50:44And so I also think we just have a positive people coming out of the universities or having experience at large companies that have done this. I mean, there's just not that many. Like, you don't go intern and, like, build a chip. But a lot of that's changing now. like, listen, we're going to create a whole generation of founders that come from these new companies that will know how to do this, and they'll be hired in much more junior. I would say, actually, one of the greatest legacies of Elon towards this is, of course, he's created these great companies, but the amount of entrepreneurs that have come out of SpaceX that are changing the entire industrial complex may be an even greater legacy than the companies themselves, and I think we're going to see the same thing for computer science and hardware.

51:26As a matter of fact, one of our investments was started by two founders in the 20s, but if you go walk through their offices, you see the experienced people as well. So it's an ideal combination here. Yeah, it doesn't necessarily have to be the founder with experience, but that founder better be able to tap into that experience in a real way. Get the people with it, yeah. Yeah, well, and then be able to work with them and they have to be good and all these kinds of things. it's complicated.

51:56Erik Torenberg:Speaking of experience this is a big new fund we're launching and there's no new GPs we're sort of collecting it's because you guys have a lot of experience and the rest of the group in this field that has been kind of latent and dormant. Well it's kind of funny I think we almost had to be warned against it almost just because our backgrounds are from it's hard and I think the reason that we needed a reminder is because all of us have spent so much in our careers systems and hardware were kind of drawn to that And so, listen, we've been clearly invested in Harvard over the years, right? We're in SpaceX.

52:29We're in Adderall. These are very early checks. We're in astronauts. We're in Waymo. You know, so even early on, we did a number of those investments. But, like, you know, this is because it's so much in our DNA. And so I don't think this is necessary to increase the team of competencies just for us.

52:46Erik Torenberg:If this fund does what we think it will do, how do we see the world changing or looking like in five to ten years? Well, you know, hopefully America wins in the infrastructure game. And we have lots of, like, super eco-friendly, efficient data centers out there. Lots and lots and an abundance of chips and an abundance of memory and an abundance of power. And, you know, that would be awesome. And I think, like, we, you know, it goes back to, like, we really think America is a special place. And we're important not only to everybody here, but anybody in the world who wants to kind of make a contribution and do something bigger than themselves.

53:30It's kind of the best place to come with nothing and do something profound. So we'd like to keep that going. And I think that doesn't continue to go if we lose our lead in technology. I think we'll be in another era and it'll be another country. and maybe they have a different set of values around that.

53:51Erik Torenberg:Thanks, man. That's right. Great. For seeing me, it was fun. Our team, Ben, our group. Thank you. Ben, too. Thanks.

53:59Erik Torenberg:Thanks for listening to this episode of the A16Z podcast. If you liked this episode, be sure to like, comment, subscribe, leave us a rating or review, and share it with your friends and family. For more episodes, go to YouTube, Apple Podcasts, and Spotify. Follow us on X at A16Z and subscribe to our Substack at a16z.substack.com. Thanks again for listening, and I'll see you in the next episode.

54:44Erik Torenberg:For more details, including a link to our investments, please see a16z.com forward slash disclosures.