Gavin Uberti - Real-Time AI & The Future of AI Hardware - [Invest Like the Best, EP.356]

25:40And anyone that had an ASIC outperformed someone that had a GPU, outperformed somebody that had a CPU. So maybe just talk us through, everyone's heard those three terms probably, but may not know what the differences are and what the trade-offs are. What are ASICs and how does this work. Well, let me give it to you in the context of Bitcoin. In the very old days, back when the Bitcoin algorithm was first devised, people ran it on CPUs. And to go mine a Bitcoin, like you're saying, you have to solve a very hard math problem. And the only way to solve this math problem is by guessing and checking answers very fast.

26:13A CPU can do many things. It can do this too, but it's not very fast about it. CPU mining was originally the standard. People then said, hey, let's run this thing on GPUs. A GPU has several hundred mini CPUs scattered across the silicon. And because of this, it's able to go do many things in parallel. Each one of these cores on a GPU can do all kinds of different tasks. Once people wrote GPU miners, CPU mining became so bad by comparison that you'd lose money running it. But then people said, well, what if we took this a step further? On a GPU, only a tiny fraction of that GPU is used for mining Bitcoin.

26:53We don't need the caches. We don't need the IOs. We don't need the crazy instruction processing and L0 caches. What if we specialize a chip, put every transistor on that chip, so it would just mine Bitcoin? And when Bitmain and MicroBT did this in the 2012-2013 era, we saw an explosion in how fast people were able to solve these problems. The Bitcoin algorithm made them much harder to compensate, and GPU mining became unprofitable overnight. Because of this, GPUs have not really been seen in Bitcoin mining for almost 10 years. That's not to go say that GPUs were scrapped overnight. They found other use cases.

27:33They're flexible. They went to mine Ethereum or went to go do AI or went to play games. But not Bitcoin. Once you have an ASIC for something, it is just not commercially viable to run on GPUs anymore. I think the same exact thing is going to happen for Transformers. Previously, in like the 2010 era of Bitcoin, there wasn't really enough volume to justify spending$100 million to build a custom chip. But once Bitcoin became popular, there totally was. Then the chips came out, and everything else becomes no longer feasible. For Transformers, even 18 months ago, there wasn't enough demand to justify spending$100 million to burn that algorithm into the system.

28:16But now, with ChatGPT, GitHub Copilot, with Character, with all these models from Google and Anthropic, it does make sense. The first Transformer Asics are coming. Can we talk about this progression? So I'm just going to use, I'll stop saying ChatGPT just to abstract away, but let's say we're at model level four right now and we're going to go to five and six and seven. You referred to the need to probably build things that don't yet exist, not just chips, which is obviously what you're working on. not just training the models, not just gathering data, but actually the infrastructure, the physical infrastructure to do this.

28:51What do you think happens to data centers and power sources and things like bandwidth? It just seems so interesting that we're going to have to build new stuff to enable each successive progression of new model. So describe how you think that iterative thing might play out. I think there's a great analogy here, actually, and that is a semiconductor fabrication, the buildings that make microchips. These incredibly advanced facilities have incredibly complex supply chains. Many have heard of TSMC and the machines they buy from ASML for$300 million, and the mirrors ASML buys from Carl Zeiss. Well, that's just the photography machines.

29:31A huge array of complexity goes into this stuff, and that means that building a new fab costs$10,$20 billion. and I think we're going to see a similar thing for models of level 5 and 6. This whole chain will have to be redone. If you want to go and scale from 4 to 5, have the same multiple of compute over a 3 to 4 or a 2 to 3, you have to have 10 hundred times more chips, and that means 10 hundred times more power. This is not going to be a 20 megawatt data center. That's a 2 gigawatt data center. It's a data center that consumes the entire energy output of a big power plant. And I think this is also going to be very centralized.

30:12If you compare, say, the bandwidth of an entire undersea submarine cable, the bandwidth between a couple of GPU racks right next to each other, the GPU racks have double the bandwidth of a literal undersea cable. So there is so much incentive to centralized data centers. So I think we're going to see a few very, very large facilities that do all this. Now, the same is true of semiconductors, for reference. TSMC's fabs are all very large, very self-contained buildings. But so much else has to happen, too. You need way more power. You also need way more redundancy. If you have 10 GPUs, then sure, none of them are going to fail.

30:55If you have 10 ,000 GPUs, then okay, maybe one fails sometimes, and then you reset to a previous checkpoint, you try again. If you have 10 million GPUs or 10 million transformer ASICs, then you're going to have one failing every couple of hours. You're going to have to figure out how do I deal with one of these things crapping out without having to stall the whole thing. No one's solved that problem yet. Or heat. We've been able to go cool data centers in the 200, 300 megawatt range. What happens when the two gigawatts that go into that building are all turned into heat. How does that get pumped out?

31:31Is that just dumped into the atmosphere? What happens next? I really think the whole set of industries are going to evolve around building these massive, massive AI models. We've only seen the beginning of it. All that, which sounds so remarkable, that heat, that heat dissipation, the power, the bandwidth, number of chips, all of that, I think, if I understand correctly, is especially needed in training. Each marginal inference is less of a centralized, crazy power-intensive thing, although collectively, obviously, inference will get quite big too. We haven't really talked about training. And I know you're working on chips for inference to allow you to do orders of magnitude faster and cheaper inference, which will power the apps.

32:14But what about training? What is actually going on in training of one of these models for those that don't understand that process? Well, so first I want to go talk about inference for a second, and then we'll build into training. Because I believe that inference is going to be run in similarly large, power-hungry data centers too. And here's the reason why. When we run a transformer model, we have this huge number of parameters. And each one of these parameters is a number. And to use that number, we take in a number from our input, we multiply them together, and we add them to a running total.

32:47So every one of those parameters, in the case of GPT-3, 175 billion, is loaded from memory once and then used in a math operation once. It turns out that loading a thing from memory is way more expensive than doing the math. So how do we solve this problem? Well, we say that these weights are the same across one user or two users or four users or eight users. So we batch together a huge number of queries. Then we load in that weight once and we use it 16, 32, 64 times. And that's one of the really interesting things that a transformer ASIC can do. You can have a much, much larger batch, not 64, but 2 ,500.

33:30So we're able to go load that weight in once, pay the expensive price, and then amortize that expensive price over a huge number of users, making inference much, much cheaper. And now while this sounds good in theory, this does mean that you have to run that model in a place where you can have a huge number of users all kind of grouped together. So I think that means inference will be centralized in much of the same way as training. To go to training specifically, why are these machines so power hungry? Well, in inference, you have to run the model forwards. But for training, you have to compute how much each one of those little weights contributed to the final error.

34:07Do that, you have to run the model backwards, which takes about double the compute. Additionally, running forwards and backwards requires different kinds of network primitives, different kinds of connectivity than just running forwards does. So training is a more challenging problem. But the economics that make a transformer-specific inference chip make sense, I think will also apply to training. If you're spending, say,$100 million on an AI model, and it costs$100 million to build a new custom ASIC, then no matter how good the custom ASIC is, you're not going to make your money back. But if it costs a billion dollars to train your model, a custom basic costs a hundred million, and it makes you 20 % better, then yes, it does make economic sense to build that custom chip.

34:53If you're spending not$1 billion, but$10 billion, it is a no-brainer. You have to do it. Now, building that custom training chip is not an easy thing to do. Right now, going from conception to mass production of a microchip can take four or five years. And we can't wait that long for training our new models. So I believe that to support this demand, support these custom ASICs for training that we will eventually need, there has to be a whole redesign of the way semiconductors are designed. I'd love to understand a little bit about how the cycle time for new chip ideation design and then delivery has happened in the past.

35:37because from everything you're saying, it sounds like the iteration speed of chips themselves will need to go a lot faster than it has in the past in order for us to have all this great stuff that we've been talking about. And that that is a vector of innovation that's as critical, but certainly like extremely critical alongside everything else we've talked about. So how has that worked in the past? If I'm NVIDIA or something, I'm whatever company designing some new chip. talk us through like the major stages of chip design, maybe even starting with the thing trying to be solved, how you arrive at the problem to be solved in the first place.

36:13So that's step one. And step X is the chips are in a data center somewhere being used. And how long does each of those steps take? Like I'm sort of a novice on semis. And so I'd love to learn how it's done in the past. Then we could talk about how we might innovate to change that in the future. Well, there are a few steps involved in bringing a new chip to market. No matter what year you're building in, First, you have to say, okay, what's the problem we're trying to solve? And then from there, you'll devise an architecture and a microarchitecture, just sort of a more detailed variant of the architecture, of how your chip works at a high level to solve this problem.

36:47And as part of this architecture work, you're going to get a number of blocks. You'll say, okay, in order for me to do this giant matrix multiplication, I'll need a matrix multiplication block. And that matrix multiplication block will be made up of many sub-blocks that each do a multiply-add operation. It'll be made up of a control block that sends a signal to the full matrix block, and so on. Then you'll take these blocks, the number of blocks of chip has varies widely, but it can be as high as hundreds, and you'll give it to an RTL engineer who will write code in a hardware description language, kind of like a programming language, that tells the block how to behave.

37:25Then this block is verified. Because it costs so much to manufacture semiconductors, you have to make sure there is not a mistake in that RTL block for you to pull the trigger and tell a TSMC or whoever it is to begin cranking out wafers off their line. So a verification engineer will very carefully write a test bench to check that that block behaves the way that it should. And they'll also check that their test bench actually runs every single line in that Verilog file. They call that line coverage. And they'll go a step further. That Verilog, that code, will be compiled to a netlist. This is how each group of transistors talks to each other.

38:06And then you'll run toggle coverage on this. You make sure every one of those transistors will flip to either state. So that's what they call the front end of the chip. Now you have a netlist that comes out of that, instead of all the transistors and how they connect to each other. But now you have to put them on an actual silicon wafer and lay them out. That's a hard problem. So you'll give it to a backend engineer who will make sure that while both beyond the logic being correct, there's no issue where, hey, this block takes too long and it's not going to be done by the time this other block has to take its results.

38:37They call that timing closure. And they'll make sure that on a chip, especially a large chip, process issues, manufacturing problems in the TSMC line can make it so that one part of the chip behaves differently than another part of the chip, even things moving at different speeds. So they'll make sure that it works no matter what kind of wafer is made, whether it's fast or slow, what temperature it is. They call this getting all the corners closed. And a number of other checks, too. There's not going to be some weird radio frequency interference thing. There's not going to be some pool, some vacuum in the chip where other photoresist will flow in.

39:13They check a huge number of problems. And eventually, after this back-end process is all finished, they'll give it to a masked house, masked shop, who will take these designs of where your transistor goes and they'll say, all right, to make these transistors, we need to go and have silicon in these parts of the chip, but not these parts of the chip. If you ever heard of screen printing, the process kind of works like that. From this file, this GDS2 file, which says where there should be and where there shouldn't be a transistor, they make a photo mask that will shine light on the parts where there are a transistor and not on the parts where there aren't.

39:50And then those photo masks are sent to a fab that actually makes the chip, who then uses them to project an image onto those silicon wafers after coating them with a chemical called photoresist that hardens when exposed to light. So that means that after all this is said and done, you have parts of the wafer that are hardened in just the right pattern for you to put your transistors down on them. So this is a very long process, real production samples coming off an assembly line for modern semiconductors. And if we want to build the next generation of transformer-specific ASICs for training and for inference, this has to shrink.

40:30Now, there's been a huge amount of progress to make this faster, especially on older nodes. Google in particular has driven a lot of this. they have a series of open source efforts to automate parts of this backend process because really it's not that hard it's just checking rules and it should be done by a computer and if you're building on a node like those from it if you're building for old technology old semiconductor tech in like 2005 you can have the backend done automatically in a day instead of nine months by using one of these automated tools and we haven't gotten to this point with enormous modern-scale chips yet.

41:12It's possible in theory, and I'm very hopeful that more smart folks will begin working on this problem. And same thing for actually writing the chips, doing this RTL coding. Today, it's all done by hand. But for example, C and Python, we have seen language models themselves begin to help out with it. GPT-4 is very good at both those languages, as is GitHub Copilot because there's so much code in C and in Python available online. So having access to these AI tools can make a developer much faster. For a Verilog in these hardware languages, that's not true yet. While software loves open source, hardware really doesn't.

41:53So it's very, very hard to get all this code into your model as training data because it's all proprietary. and I'm very optimistic that somebody will be able to go to Synopsys and go to all the old chip vendors and say, hey, your code for CPU is from like 2005. That's probably not very useful to you anymore. Can I use that to train my AI model? And if enough companies are convinced to say yes, ideally to open source it, but probably not, maybe just to go train that model, then you can get a language model that was good at Verilog, as good at Verilog as it is at, say, C and Python, and use that to speed up writing some of this RTL development.

42:35What do you think the outside limits are on how fast this could happen? Where you said four, five, six years might be the start to finish time now. Where do you hope it is 10 years from now? What do you think the upper limit is of how fast this iteration time could be? I really want to go call out a lot of the Google-led Silicon ecosystem. I want to call out Global Foundry's, the Skywater Foundry, for open sourcing their PDKs to help make this process fast. Now, if you're a hobbyist who wants to build a chip on one of these older technologies, you can do it. From conception to putting it together, a few transistors, taping it out, weeks, months, even if you're starting from absolute scratch.

43:21That is so cool. They're able to do it for pretty cheap because they take all the hobbyist projects and put them together on one mask set. You make one photo mask that has all the chips on it. And I'm really optimistic that this can be scaled up. Now, it's definitely not easy. These hobbyist chips on old nodes have, you know, hundreds of thousands of transistors on them. And the amount of effort it takes to a computer to decide where to put these things is super linear in the number of transistors. Going from 100 ,000 to 80 billion, like you have on the NVIDIA GPUs, is more than 80 billion, over 100 ,000 times harder.

44:00So instead of being able to do it on one beefy server, you suddenly have a huge, huge data center working on this optimization problem to get it working. So I am optimistic there's a future where just like for a hobbyist building a chip on an old node today, we'll be able to get companies building chips on new nodes by spinning up huge data centers to do this back-end process and training these language models to help them write the front end. I think the upper limit really is a couple of months instead of the four or five years of today. A lot of work is going to have to happen to bring that down.

44:36And it's also a problem where the first chips that are made with some faster process probably aren't going to work. People don't like rocking the boat because of how expensive it is to build the chip only for it to come back dead. and once a few people do it, it'll be the obvious solution. But nobody wants to be first. Why do you want to try? I mean, why? Because it has to be done. Someone's got to do it. Someone's got to do it. That's the only way that we get to our$10 billion training runs. That's the only way that we get to GPT-6. That's the only way that we get these AI-generated books and movies and TV shows that would make the artists of today be impressed.

45:19Someone has to do it and might as well be me. It's interesting to think about in this future world now, the existing landscape of companies and stakeholders and how they'll start to interact with this technology as it becomes available. So when I went and looked up the top buyers of H100s or something from NVIDIA, you see kind of exact ones you would expect. You see, I think, Meta and Microsoft are the two top purchasers. Amazon's a big purchaser. So these hyper-scaling, hyper-large technology companies that are either providing cloud services in the case of Azure or AWS, or they're meta, training their own models, open source models, buying all these H100s.

45:59Who do you think are the customers in this different world where data centers are different, ASICs are the predominant chip, Fortune 500 companies are all trying to find their version of applied generative AI? do you think that the market map and market structure looks a lot different in terms of like, is it the same buyers buying these chips as buying the H100s? Again, there's a great analogy in the world of chip making. Well, the world of chip making used to be pretty cheap. Anyone designing their chips probably had their own little chip fab because it cost a couple million dollars. And as these things became more and more expensive, all but the biggest players became priced out.

46:38Now, only two or three firms, TSMC, Samsung, and maybe Intel, are able to make the world's state-of-the-art chips. I think the same thing will happen for a generative AI and these mega-scale data centers. Today, many companies are trying to explore probing the waters because it is so cheap to do so. But as having a state-of-the-art model becomes a$100 million, billion,$10 billion endeavor, However, it'll be much more common to outsource that to one of the few firms who will have one of these mega scale data centers. And don't get me wrong, you'll still be able to fine tune on top of these models these companies build.

47:16But they'll be much more like TSMC, the pure play AI generating house, and they will be some full stack integrator. I just think the world cannot support 100 different$10 billion models, but it can support three. So I see a world where you have these few enormous leading-edge state-of-the-art development houses, possibly at the current hyperscalers, possibly at some new company that specializes on just this kind of thing. And there'll be a tale, much like there are for semiconductors. Not every application needs leading-edge chips. For example, the U.S. government and U.S. military care much more about chips being produced in a reliable American factory than the chips being state-of-the-art.

48:01So fabs like Global Foundries have accepted they're not going to be as good as TSMC, but there's still a market for them to push by being American. I think we'll see a similar thing for these AIs. You'll have a few smaller data centers that don't try to compete on being as purely smart as the big megaliths, but they'll find some niche. Obviously, TSMC is one of the most interesting and intriguing geopolitical assets that we talk a lot about in the news. It's even a political issue. Do you think that these data centers and the control over the specific models becomes like the central political issue?

48:41Like there's a presidential campaign where the candidates on the stage are mostly just talking about who has the best models and where are they located and what's the security around them and nationalistic type issues. It's all seems almost like the start of a sci-fi novel or something. If I said that we were going to have threats of war over a chip fab 10 years ago, that sounded like a sci-fi novel too. This has to be the conclusion here. Some countries are going to have these hyperscale data centers. And if you don't have one, it's going to be really hard to catch up. China is famously trying to catch up on semiconductor development.

49:16They realize how critical that is for AI, but they did not get on the EUV train when it was leaving the station. They're now on this older immersion technology, and it's really holding them back. And while they have produced some great, quote unquote, seven nanometer chips, I do think they'll continue getting better technology. China is not state of the art for semiconductors today and does not seem like it's on a path to be either. And I think for data centers as well, the United States is very lucky to have all the big hyperscalers, very lucky to have the big AI companies too. But we cannot be passive about this.

49:55We're in a good state now, but that could change. So I think that export bans for state-of-the-art AI chips are really in the national interest here. Can you explain the essay, The Bitter Lesson, and then talk about the implications of the upper limit of how good these things could get based just on the available data in the world? These things have been trained on insane amounts of data, but there is a finite amount of data in the world. There's more being created every day. But talk about the bitter lesson and the importance of data and the sort of upper limits of how good these things could get.

50:29So the bitter lesson is an essay put out by Rich Sutton, great pioneer AI in 2019, before the whole transformer craze. Rich looked at the field and saw that what would happen all the time is somebody would take a model that worked, and they'd hard code some knowledge and do it. For example, somebody would say, hey, here's a great image recognition model, but it gets this one weird case wrong on occasion. I'm going to hard code a hacky fix on it. And inevitably, doing this whack-a-mole fix strategy works in the short term. You get a little bit better performance. You can go publish your paper all as well, but it never scales.

51:09The only techniques that have routinely worked are those that leverage the fact that compute is getting cheaper. For example, I mentioned chess engines before. In the very old days, people would write chess engines by saying, well, a rook is worth five points, a bishop is worth three. I'm going to add a rule to my machine that says, hey, if you can go trade a bishop for a rook, that's a good trade to make. So you should go make that move. And again, that works a little bit, but it doesn't really generalize. So the strategy that eventually worked for chess that enabled Deep Blue to become the world chess champion was not encoding human knowledge, not hard-coding rules, but by just leveraging a massive amount of computation.

51:54Deep Blue did search, much like this tree search we were talking about earlier. It would test hundreds of thousands of positions, find the one that it liked best, and then play moves to get there. And the same thing is happening for artificial intelligence. In the very old days, the image recognition, people would say, hey, there's a block of white pixels in this kind of shape. It might be a bird. It might be a light bulb. They had these big trees of fixes for it. And that didn't really work, didn't scale up. But the thing that did scale up is making neural networks very big and feeding them huge amounts of data.

52:29So the lesson of the bitter lesson is that the only techniques that will really advance the field of artificial intelligence are those that are able to leverage the cost of compute getting cheaper, that are able to leverage these enormous data centers that must be built. And you mentioned data as well. A key part of the learning algorithms is data. There's only so much data available in the world. Some people have raised the alarm that we're about to run out. The models aren't going to be able to go and get the next trillion tokens. It doesn't exist. And yes, I do think that'll be a problem eventually.

53:06But we haven't even begun to tap into the most valuable source of data, the video. I don't know about you, Patrick, but when I learned to walk, I learned to pick up objects. I didn't do it from a book. I did it by looking with my own eyes and seeing, hey, does this behave the way that I think it does? And we've begun to kind of dip our toes into it. People have built vision transformer models. And usually what they do is they train a model on text. And then at the very end, they add in the image part. It just adds a little side feature. And I think that's wrong. I think Transformers tomorrow will be trained from day one with enormous amounts of video data.

53:45So that solves the problem in the medium term. But eventually, these models will get 100 times bigger. They'll require 100 times more data. They will exhaust all that is available right now. But I think that self-play and generating your own training data is the final frontier here. One good analogy is the machines that played Go. They tried training AIs on human games to make them better than humans at the game of Go, and it worked, but you ran out of those games very fast. So what Google did with AlphaGo is they made the model play against itself and learned from those games. And the same trick has begun to show some promise in natural language processing, where you have an AI that writes out some output, then you ask it to analyze that output and say, was this good?

54:32Was this bad? As a human, you can do that. You can get better. That's why I revise the things that I write. And I think that technique of the models generating their own training data will allow us to scale even once we run out of all the images and video available on the web. When we get to that stage though, what's the objective function? In Go and chess, we know what winning is. We know what good and bad is. We can define it. I suppose we could define the rules of physics or something in an open-ended play-yourself game that these crazy models might do with themselves. These are goal-seeking things.

55:08How do we define good objective functions in that case? Well, one of the beauties of language models, I think, is that they learn these concepts themselves. You can test this. You can go ask GPT-4, here's a scenario. Is the human behaving ethically in this scenario? So the AI knows what is good, what is ethical, and it got the understanding by reading all the text we've generated as a species. So when we ask it to go say, hey, pick the response that is more ethical, pick the response that's better, it is using that concept that developed from reading all there is to read. Can you walk me through the most important big companies of today and just riff on each a little bit?

55:51There's the obvious ones. There's the hyperscalers. There's OpenAI. There's Anthropic. There's NVIDIA. There's all of these different key players. I'd love you to pick the ones that you think are most important, really just through the lens of your own strategic thinking. You're building a business. There's been enormous scale benefits in this last cycle of technology companies. It's staggering how big the big technology companies are. How do you think strategically about the role that they'll play? How, as a young upstart business, you might position yourself strategically or counter position yourself against these incumbents?

56:24Why won't those same five companies just win everything? Like this whole strategy game is really, really interesting to me. And I'm curious how you think about it or if you have certain lenses to approach it. I know that it lurks from afar, like Microsoft and Google and Amazon are dominating everything, and they do dominate, but at one layer of this enormously complex supply chain. Let's think about it from the top. At the very top here, we have the state-of-the-art AI models being trained by folks like OpenAI and Anthropic. These folks have enormously good talent. I think that's the big moat.

57:00But they don't really run the compute themselves. they get it from one of the hyperscalers. So we have the models at the top. Below this, we have the hyperscale tech giants. Those are the ones people are scared of. Google, Microsoft, and Amazon are the big three in this domain. Buying those chips, racking and stacking them, supplying data with power, and then selling that to these companies on top. Where do those companies get their chips from? Right now, it is almost entirely NVIDIA, but I think that's going to change. AMD is coming out with their new Mi300X next year. I think it's going to be a viable competitor to the H100.

57:42And companies like us are trying to replace this idea of general-purpose AI chips entirely. And they'll specialize ones just for the Transformers. So I think that us, along with some of the general-purpose guys, become this layer underneath it. But us, NVIDIA, and AMD cannot make the chips ourselves. We have to go to a fat house. TSMC is the one of choice. But the chip is just one piece here. On a chip like the MF300 or on the H100, the memory actually costs more than the silicon itself by more than a factor or two. Now, these memory modules are made by only two or three companies as well. Samsung, SK Hynix, and maybe Micron.

58:25Two of those are in South Korea, and Micron is American. So these people are other really, really critical folks in that supply chain. And folks talk all the time about NVIDIA, TSMC, ASML relationship. They don't talk enough about NVIDIA to Samsung, NVIDIA to Eskehynes relationship, because these memory modules are so critical for building the state-of-the-art GPUs. But then these companies themselves, where do they get their machines from? And now we get to a little bit of the diminishing returns. ASML is a much smaller company than TSMC. And there are so many others that help make these chips as well.

59:06But I think that this supply chain of model provider on top, then Hyperscale data center below that, then chip designer below that, and chip fab below this. This is the fundamental stack of the future. And now what I did not mention here is the end application. What does the user see? I don't think this will be built by OpenAI and by Anthropic and by these model companies. It'll be folks building on top here, this fifth level of a stack. And some of those folks will be incumbents, big established firms that want to build some new AI product. A lot of it will be startups who say, hey, chatbots are boring.

59:44I have this new interface, speech-to-speech. I have this new interface, enormous amounts of document ingestion and querying. I have this new interface, AI-powered search of the whole internet. So I think there's the most opportunity. Well, I think there's opportunity at all these levels, building new AI products, innovating AI models, building new kinds of data centers that are designed to operate these crazy high scales, building specialized chips and building them fast for running these models and fabricating those chips in a way that is much faster than today's processes. given that the application developers at the top of the stack have kind of an open green field which is really cool i'm sure we'll see amazing things the way they build competitive advantage and succeed seems kind of clear just like it has been in the past you're operating at a much deeper part of the stack how do you think about building a better product but also doing so in a way that three years from now nvidia doesn't just get one of them and look at it and say okay well we're the scale player here, we sort of have the right to win in ASICs, just like we did in GPUs.

1:00:50How do you think about that strategically? And I know the world needs all these things, and it's a big world, and there's a big and growing pie. I don't mean to suggest anything different, but how do you think about competing at a deeper part of the stack where there's more supply chain, more interdependency, huge giants that move fast, and so on? Let's be clear, NVIDIA is a fantastic company, and I believe they will eventually build specialized transformer chips, much like we're doing. It just makes sense. But right now, they're stuck in the innovator's dilemma. The H100 is selling hotcakes and has an enormous margin on it.

1:01:23And if they were to build, not even a transfer of specific ASIC, but say H100 for inference, or H100 with a little bit of the graphic stuff ripped out, that would reduce the margin they have on the H100 and would be bad for them entirely. They can't innovate or the advantage goes away. And this is especially for specialized chips. One of the biggest moats for NVIDIA is CUDA, their software stack that is very, very flexible. It's able to run graphics, able to run AI, crypto. That is a great moat in the world where general-purpose chips, general-purpose AI chips are dominant. But specialized chips like ours are much easier to program.

1:02:04In a world where you burn the transformer architecture into the silicon, you don't need CUDA in the same way. If the world goes to specialized chips, NVIDIA kind of loses that moat. So while I do believe they will compete, they're not going to be first. The only people who can really innovate are startups like ourselves. So I see a world where we get to market first, and NVIDIA gets to market a year later or so. Now, what happens in that year? I think, again, going to the crypto analogy makes a lot of sense. A lot of companies, 30 I think, tried to build Bitcoin mining ASICs. They're much simpler than GPUs.

1:02:41You can tape them out much faster. But the companies that got to the space first ended up winning. There's now this duopoly between MicroBT and Bitmain. Bitmain, when they were testing the waters for an IPO a couple years back, was valued at more than$50 billion. So because when a specialization moves into a space, general purpose devices become no longer competitive, the person who gets there first is almost guaranteed to win. I think that to go back to us, this means we have to move as fast as possible. Back when we began this company last year, before transformers were big, before this was obvious, then yes, we could afford to be a little more relaxed.

1:03:22But today, when people are saying, why is there no transformer ASIC? We must move as fast as possible, ship on time, and ship silicon that works for us to win. You mentioned the importance of CUDA for NVIDIA. we haven't talked about the software that you have to build at the same time as you're building hardware. What role does that software play for a company like yours? And the way that the chip itself can be programmed or tapped or accessed by other developers. Say a bit about those two parts of this story. Let's get some of the question of why did other AI companies fail? People have tried to build general purpose AI chips in the past and have generally lost out to GPUs.

1:04:03Graphcore, Sabinova, Cerebris, and Grok have had some modest success, have not taken over the world in the way that folks thought they would. And I think this is for two reasons. First, the software, and second, the chips generally weren't better. We'll do software first. It is extremely hard to do what NVIDIA does. CUDA has been developed over, I think it came out in 2007, more than 15 years. And because it is so old, it has become stable, reliable, the workhorse people can depend upon. A startup trying to get into the space won't have that. They'll be putting out something a little bit more raw that's much harder to use.

1:04:43And if you look at, say, the ease of compiling code for NVIDIA versus even, say, AMD, it's no contest. Huda is great because it is production great. And the only way to solve this problem, besides being old, I think, is to be less ambitious. If you're doing a simpler thing, you can get to real production, great quality, much faster. And luckily for us and for specialized chips, because there is so much less programmability on the thing, because you're only taking transformers to transformer compiled code, the software stack is much, much easier. Now, that's not to say I want to be complacent about it.

1:05:23We already have a prototype of this running internally, running with our chip and simulation. and because of this we can begin iterating on that software getting it stable reliable production grade so that when the chips come back the software works the way that you expect it to but the second piece of why these companies didn't succeed is that they weren't better if you look at say the amount of memory bandwidth that they have versus the gpus they computed against it's the same if not worse if you look at the amount of flops compute they have again It's the same, if not worse. And that, I think, is a much bigger problem than just the software itself.

1:06:04If you think about the performance improvement that you want to drive, you just said, look, these things didn't work because they weren't better. How much better do you want to be? Is it five times, 10 times? It's got to be more than 10 times. I don't want to share exact numbers right now. We got to have some secret sauce. But because we're able to specialize, you can get a huge amount more compute on these chips. And let me just give a concrete example here. It takes about 10 ,000 transistors to build a fuse multiply add unit. That's the building block of any matrix multiplier. And NVIDIA has not that many on their chip.

1:06:43Only about 4 % of the chip is the matrix multipliers because to feed them requires so much more circuitry. That's the cost of being so flexible. Whereas us, we will have more than an order of magnitude, more of these multipliers, the order of magnitude, more raw flops. And of course, turning that into a useful throughput, that is still not trivial. You need a very good memory bandwidth there as well. But the reason I can confidently say that we will not fall into this trap and that we will be better is that I already know the metrics on our chip. I already have PPA figures. we will just have so much more raw compute substantially more than an order of magnitude that's not just for compute but also for latency people talk a big game about hey if the cost of compute goes down you can do so much more stuff and I do think that's true but it's not what gets me excited with chips that have 20 times the better latencies than the state-of-the-art GPUs because they're able to run a huge prompt in total parallel that lets you build products you could not otherwise build.

1:07:49Let's you build this real speech-to-speech machine. Let's you build a machine that ingests the whole internet in a second instead of hours. And that, that is what gets me excited. One of the things that's so clear to me is the role of strategic leadership in making things like this happen. And by that, I mean, usually in a person, a single person. Obviously, like on the software side, Sam Altman seems to be a or the person that's really driven a lot of this forward. He clearly early recognized an opportunity and galvanized talent and a team around it and has managed them to build something really spectacular.

1:08:28I sort of think of you like the hardware person potentially that could do this. How do you think about strategic leadership? How do you lead a technical staff? I don't know how technical Sam is, but my sense is that he is more of a traditional leader, not in the deep, deep weeds. How would you characterize yourself? And what do you think are the key components of a good strategic leader in the hardware side of this equation, given that we're already familiar with some of the leaders in the software side? Let me tell you a bit of the story of how we got here. And then I'll bring that into what I think a good leader has to do.

1:09:05When I was on my gap year working for OctoML, living as a digital nomad, versus off in his dorm room, we had this idea for, hey, let's build a transformer specialized ASIC and that we get a huge amount of performance improvement. We went to a big industry vet who told us that, nope, not going to work. But also he said that despite being a veteran industry vet, he wasn't the guy to know for sure, but he knew just the guy to tell us why our chip would not work. Mark Ross, his resident dower. Mark Ross is a seriously legit guy, kind of person who makes you a little bit scared. He was a CTO of Cypress Semiconductor for a time, which sold for$9 billion in 2019.

1:09:49And Mark has shipped, I think, five chips that have done more than a billion in revenue. So we go to Mark. We show him our tech. And he says, well, it looks like this isn't going to work, but it's too early to tell. You guys should go build a functional simulation. You guys should go write a white paper. and then he'd come back to me and we can talk about, hey, here's the problem that you're going to hit. Here's why this isn't going to work and you guys are going to learn a lot by doing it. And then maybe from that we can figure out how to pivot and build a chip that is viable. So over a lot of long nights, Chris and I do just that.

1:10:22We write our technical white paper. We do the original architecture for the chip. We build our functional simulation and we go back to Mark. And Mark says, shit, this works. But you guys are college students, he says. This is semiconductors. Just to get started, you're going to need$2 to$3 million and a hell of a lot more after that. This isn't data caps. You guys have never built a chip before. And then we raised$5.5 million. We went back to Mark and asked if he knew anybody to be the chip architect. And after a series of conversations, he used to be a chip architect. And we talked him out of retirement.

1:10:57So Mark Ross is now our chief hardware architect. And from there, kind of the rest of the dominoes fell. we got ajat our vp of engineering fantastic chip guy intel vp before this who was known for building these very large chips on advanced nodes with teams it's like a dozen people that is not how that normally happens one of the few good men there who can still build the chip on a budget that we got a sap the deep co-founder oradine we got reynolds the first chip guy at cruise and dozens of other vets from Google, Microsoft, Amazon, you name it. What does a good leader do? Kind of tie this all back.

1:11:34It looks from afar like Sam Altman himself is pulling all the strings open at eye and masterminding the whole thing. But I think it's the folks under him who are doing the magic and the folks under them who are making that into a real product. The role of a good leader to set the vision, get the right people in those chairs, and not running out of money. That's it. I've heard stories, though, about you in meetings where you go to the one inch or one centimeter level on some very technical part of what's being designed. And I'm curious how much you'll miss that if you give it up and you just do those other things and or whether or not there might be a world where you being in the technical details, I think about like an Elon or something that apparently can walk the SpaceX floor, the factory floor and know every freaking detail somehow.

1:12:21Now, what do you think is the prospect of being a more technical, hands-on leader for you? I think there's a lot of value in a leader being technical. Understanding your team's problems, understanding why they're doing the things that they're doing is a requirement for bringing in the right people. There aren't that many levers you can push at the top. It's very coarse things. To figure out the right guy to bring in, you have to understand that problem. And that's what being technical gets you. So this is why Greg Brockman at OpenAI, despite being the CTO, you still see that guy coding. He can't understand how far along the model is unless he himself has seen the internals.

1:13:03Even though I love the folks who work here and I trust them completely, I can't understand how far along is design verification. How far along is this block on our chip as I'm talking to that guy and I understand why he's doing what he's doing. And also, to be quite frank, it's fun. Any closing thoughts about things in this world that would be most surprising to investors, to end customers out there, to CEOs of other businesses that aren't technology or aren't AI related? You've gotten so deep in the weeds on what's going on and trying to drive a certain future. Is there anything that would surprise people most that we haven't talked about yet?

1:13:44Let's take it back to this idea of transformer is becoming the way neural networks are, where it's not going to be replaced. It's going to be a building block. You're going to build, say, retrieval augmented generation on top of that to make the model smarter. If you read Microsoft paper where they have agents living in a simulation, you're going to build memories on top of these transformers to let them remember things. So a real key storage cache. I think that folks have talked and talked and talked about how the Transformers are going away. A much more productive line of reasoning is how do I build on top of this?

1:14:19And when I look out at what's happening in this landscape, it is the folks who are building on top of it who are building the most successful, coolest products, AI games based on having this memory, real conversational things, speech-to-speech, although with a higher latency than I would like today, or a primitive AI lawyers of the future. So again, the transformer is not just the next fad. This is a building block. What makes you most confident in that? When I asked around, I heard lots of people that agreed, but why that architecture, which obviously has led to all the stuff we're talking about.

1:14:57So obviously it's incredibly powerful, but why doesn't it tap out in two years, three years, four years, five years? Why might there not be a different architecture sure that just totally takes over if the world was fair there could be but the world is not fair the transformer by virtue of being the dominant architecture is suddenly the thing that has way way more humans thinking about trying to improve trying to optimize and after people hardware too nvidia is building a little bit of hardware support for the transformer to optimize certain kinds of memory loads into their next gpus that is going to preferentially favor the transformer over any replacement that comes out.

1:15:38For software, NVIDIA has three very highly optimized transformer libraries. That means if you want to run a transformer model, it'll be super easy. If you want to run it, say, some other kind of thing, state-space diffusion, you have to make the model so much better that it outweighs the fact of a 2-3 performance improvement you get by optimizing it so heavily. So for a new architecture to go and beat the transformer, it has to be so incredibly good. That it will outweigh the hardware disadvantage it will have because transformers are favored. It will outweigh the software advantage it has because transformer libraries are so common.

1:16:15It will outweigh the end-user application because end-users are used to working with these transformer models. That is just such a heavy lift. And again, to go back to this neural networks analogy, random forests or SVNs could have been the dominant AI paradigm, I think. But now to go cash that up to the state of neural networks, it's just totally infeasible. One thing we haven't talked about is in the world where there's a GPTX and maybe a Lama Y and an Anthropic Z, what about building models that are specialists that rely on access to a very proprietary data store? So there's companies out there or people out there that have closed access to really valuable data that could make these models better.

1:17:05What happens there? Do they train their own model? Do they just somehow add their data to one of the big existing models? What do you think about a proliferation of models beyond the top three, four of them? I think this data is very, very valuable, but it can be learned later. And the reason I'm so confident that it's okay to learn this stuff later, because humans do it all the time. If you go to work for, say, say, Bloomberg, a company that has valuable financial data, you come in there having a good understanding of the world around you, but not having the Bloomberg secrets, and you're able to get caught up pretty fast.

1:17:40I think the same thing will happen for these AI models. OpenAI will train GPT-X. It'll cost$10 billion. And then on top of that, Bloomberg will say, hey, I'm going to fine-tune that with my Bloomberg proprietary data. I'm going to pay OpenAI$10 million for this. and I'm going to use that fine-tuned version. Other companies, say a semiconductor company that wants to build a semiconductor-based AI for coding chips or some investment bank, you name it, they will fine-tune on top. And the second piece of why I believe this is going to be the future is that there's not a whole lot of advantage training your own thing from scratch.

1:18:19Remember how we talked earlier about the pre-training phase where you do things that are tangentially related but easy to grade? Then you do the RLHF on top of that to really make those capabilities come out. Pre-training phase is way more expensive than the RLHF stuff on top. The pre-training is going to look the same, regardless of whether you're an AI for finance, an AI for law, an AI for therapy. There's no sense in a large company doing their own thing from scratch, doing the expensive thing again when it's going to look the same. You just fine tune. Is there anything else to talk about in hardware, edge computing, or other things in the hardware world that we haven't talked about that you think are going to be important beyond just the chips and training and inference?

1:19:03Is there a market for smaller models? I think absolutely. I don't think there's a market for medium-sized models, though. I think there's a lot of demand for a thing that's super, super low latency. Going to the server and then going back is so slow that it causes problems. Much like, for example, speech recognition today. It used to be the case that when you said, hello, Google, and then asked a question, it would send the audio to the data center, it would transcribe it, and it would come back. And the latency there was annoying. So Google built a speech recognition model on the device for better latency.

1:19:37Now, that model is very small. And I do think we're going to see other very small transformers for tasks, like selecting the next autocomplete word, doing a little bit of drafting to help with your texts. These things are not going to be the crazy smart agents we're going to have in data centers. However, I don't think there's a future for medium-sized models, things in 100 billion parameter range. That's just too big to run on a device. And if you're going to run in a data center, you might as well use a smarter model. And you might say, well, 100 billion is going to be cheaper. But I think there are enormous economies of scale here.

1:20:11Because the cost of loading in those weights is so much higher than the cost of doing marginal extra computation. it is really easy to kind of get added to that batch. So I think we're going to see a promising future for Edge and a promising future for the biggest of the big and these hyperscale data centers and nothing in between. Kevin, I look forward to, I don't know, hopefully decades of learning from you about all the stuff that's happening in this world. It does seem to just be the most, certainly for me, the most interesting, entertaining, impactful, incredible explosion of technology that I've come across.

1:20:45And it's so cool that you're going to be driving the hardware side forward as folks like Sam and others drive the software side forward. Always ask the same traditional closing question at the end of my interviews. What is the kindest thing that anyone's ever done for you? The kindest thing anyone's ever done for me, this guy Gunnar Mein. When I was a kid in the middle of high school, I was very interested in engineering. But yeah, I'm young. It's hard to go turn that into impactful results. Gunnar dropped out of college in Germany, went to work for Microsoft for a couple of decades, and then decided that instead of working for Microsoft, he wanted to go teach high school, which is quite a pivot.

1:21:22He went and taught programming classes at my high school, which I tokened they were good. But really, he stayed very, very late to run the robotics program to run Xenia stuff. Gunnar thought that a bunch of high schoolers could build a nuclear fuser to go turn a deuterium into a helium, real nuclear fusion. It wouldn't generate power. These things have been built before, but the thought that much of high schoolers could do it was a little bit crazy. Gunnar Main decided that my school needed an electron microscope, and we didn't have the money to buy one. But he saw a$1 ,000 broken Joel 87 machine that was a user for parts being sold in Everett.

1:22:06he and some of my friends and myself drove over in a u-haul picked up the machine which weighed several thousand pounds drove it back to the school and spent the next year and a half fixing it we got it working actually it was a very old microscope it had a polarite camera if you can believe that it would flash pixels one at a time as it scanned across the image because each pixel would be exposed they'd slowly be turned into the full image and we ripped that out placed it with an Arduino. We tried to go and rip out the high-voltage box replaced with a newer one because it kind of broke sometimes.

1:22:38We got that working. We never figured out how to keep it from leaking, but that's what we had. The whole microscope would take about 30 minutes to get running. You'd have to go to the room, turn the machine on, wait for the vacuum to be drawn, turn the diffusion pump on, wait for that to be pumped out. It had to be done totally manually because the automated box had been broken. Gunnar would go sit there and supervise us when we messed around with that thing until like 9 p.m. at night, almost every night. And that got me interested in engineering, got me to Harvard in some ways, and got me to where I am right now.

1:23:14What an awesome story. So incredibly unique. What a cool guy. So many stories like this of someone taking an interest in teaching and sharing the world with someone so young. Amazing closing story. Gavin, thank you so much for your time. My pleasure. If you enjoyed this episode, check out joincolossus.com. There you'll find every episode of this podcast complete with transcripts, show notes, and resources to keep learning. You can also sign up for our newsletter, Colossus Weekly, where we condense episodes to the big ideas, quotations, and more, as well as share the best content we find on the internet every week.

1:24:02Thank you.