Andrew Feldman: Building the World’s Largest and Fastest Computer Chip for AI

0:04Hey everyone and welcome to Generative Now. I am Michael Magnano. I'm a partner at Lightspeed And this week I'm sitting down with Andrew Feldman. Andrew is the CEO of Cerebrus, the hardware company that builds AI supercomputers. He's a leader in the AI industry, creating the next generation of compute hardware. And this was a fascinating conversation. Just a few weeks ago, Cerebrus announced their latest and greatest innovation, the Wafer Scale Engine 3. This is the largest and fastest chip in the history of chips. It's about the size of a dinner plate and promises AI compute that is faster and easier to use than anything else on the market.

0:43And so without further ado, please enjoy my conversation with Andrew Feldman. Hey, Andrew. How you going, Michael? Good, good. Thanks so much for doing this. Happy to do it. Happy to do it. So you are the co-founder and the CEO of Subrebrus. And this is a very, very exciting company and very relevant, I think, to what's going on in the world right now with all the excitement around AI. You recently had a huge launch that, at least from my perspective, seems to change things a bit in the world of chips and AI. And I want to get into that story, but maybe just to level set for the audience. We'd love to hear a little bit about your story and sort of the background of Cerebris.

1:26Sure. Well, this is my fifth startup. An inveterate builder of hardware systems, of chips and printed circuit boards and cooling systems and embedded software and software operating systems. And so this is what I love to do. And 2007, we got together and we started a company and we ended up selling it in 2012 to AMD. and we worked there for a little while and some of its leaders got back together in 2015 and we wrote two things on the whiteboard we wanted to work together and we wanted to move an industry and we began thinking about what what we might want to build and as computer architects we we tend to ask the same question again and again it's it is there an interesting workload and if we built a machine, could we make it faster?

2:23There's one question in computer architecture, is how to make the particular software problem faster. And we saw on the horizon AI. And we didn't know it would be as big as it is today, but we knew it would be big and important. And that we thought, even in late 2015, that we could build a chip, could build a processor that would be vastly faster at doing AI work. And we had an idea that the hard part of AI isn't the actual calculation, it's the communication. And that it would take, as these problems got bigger, it would take more and more compute. That compute wouldn't sit on a standard chip.

3:12And more and more pressure would be on the interchip communication. And so we had an idea that if we built really big chip, if we went from the traditional limit that everybody had in their mind of how big a chip could be to 50 times that to the whole wafer, we might be able to radically improve performance, reduce power drill, and make it easier to build big systems. And in 2019, we yielded the largest chip ever made and uh not by little bit it was a a chip with more than a trillion transistors and it's 57 times larger than the largest gpu that was at 16 nanometer and two and a half years later we announced a seven nanometer version and a few weeks back we announced a five nanometer version uh it is the size of a dinner plate whereas most chips are the size of a postage stamp It has 4 trillion transistors and 900 ,000 cores.

4:17Those are little engines that drive your AI work. And so we built a whole system around these. And the system delivers power and cooling and the data. And then we started tying these systems together. As the problem got bigger, we built first clusters and then supercomputers worth of these machines. And we now build the largest AI supercomputers in the world. Then we started, we had so much compute capacity, we began training models like crazy. And have trained, you know, dozens of production models and thousands of test models along the way. So, I mean, this is fascinating because it seems like there is a big problem in the world today.

4:58There's so much demand for these chips. And you, as you mentioned, like in the founding of this company, you even said that you guys had the foresight to know that this was going to be a really, really big deal. But it seems like some of the biggest companies in the world that have been at sort of the forefront of chip making and, you know, some of the highest market cap companies in the world almost seems like they're playing catch up. So, like, how did you guys have the foresight to know this would happen? And maybe from the outside looking in, it almost feels like they didn't. In retrospect, we got it right.

5:28Trust me, made many bets of my career that turned out way less well than this one. And when you build a company, you make many bets that are wrong. As a CEO, you're making decisions every week. And if you get most of them right, you're doing really well. There were a collection of idiosyncratic elements in AI. First, in about 2015, as this was exciting, Intel decided that they could attack this problem with an x86 processor. And that was a big mistake. AMD was, under Lisa's leadership, under Lisa Su's leadership, building its processor business. They had come out of a period where they had made some really bad decisions.

6:14And Lisa had taken over in about 2013 or 14. And they were really focused on building a better processor. Not a better part for AI, but they're a core business of theirs. And they got that right. But what that meant is every ounce of their energy was really being focused on a different workload. And that left among the big three that left NVIDIA alone. You could see by AMD stock price over that time, they got the CPU part right. They built tremendous parts and were a customer of theirs. And now they've turned their attention to building GPUs for this market. And so I think that was the large company environment.

7:01I think there were a group of startups who saw some of the same things that we did. Intel bought one. They've bought actually several now. But we've done a little better than most of the others. Been able to differentiate ourselves, produce significant advantage, vastly better performance, easier to program. Things that we bet on turned out to be really important in the market, especially with the rise of large language models. Why does this matter right now? Maybe level setting for the audience, like what's happening in the market right now? Obviously, you know, if somebody is listening to this podcast, they're probably aware that there's a lot of demand for GPUs.

7:44Just don't swipe right. Yeah, exactly. Exactly. Maybe set the stage for why and how the wafer scale engine three could have such a dramatic effect on this market right now. So I think it's important to note two things. The first is that the type of AI we're doing right now, particularly language models and generative AI, is enormously computationally intensive. So that's the first part, that this is a brute force methodology. You and the amount of compute you need is approximately the product of the number of parameters in your model and the amount of data tokens you show for training. that is the defining characteristic and so as we've built bigger models and as we learn that showing the more data will improve the accuracy of the model we've ratcheted up the amount of compute we need and you know we now do ai with uh an amount of compute that previously was only in the nuclear physics domain, right?

8:51Previously, these supercomputers were at a small number of oil companies and at a small number of government labs. And now that's the basis for training models. On the inference side, the amount of compute you need is a function of several things, but primarily the number of parameters in the model. It is a linear function of the number of users. So as we began to train models, these models got bigger. They became more useful. We began to do inference on them, and more people used them. And so the amount of compute on the inference side went through the roof. And so the importance of compute is if you can train faster and use less energy, less electricity in your training, better for the society, you can test more hypotheses, you test more ideas in less time.

9:44and that's how these models get better. They aren't sort of born like Zeus's children, sort of fully formed from the heads of smart people. They are instead born of a set of iterations, tests of small models, hyperparameter tunings, and these go on. And in order to announce a big new model, you know, like Lama 3 or one of these, they've done hundreds, maybe thousands of small tests. and this takes vast amounts of compute and takes a long time. And if you can shrink the amount of time it takes, you get to get to good answers faster. Yeah, and so now the Wafer 3 exists and how do we go from where we are today to a point where developers can start leveraging this instead of having to use a broker to find whatever clusters they can of H100s?

10:39A GPU cluster in Botswana? Yeah. Yeah, exactly. We've stood up over the last nine months, eight exaflops of AI compute. We will stand up eight more in the next four or five months. We've stood it up for customers. We've stood it up for our own cloud, for our customers who are CSPs, who offer cloud services. It's available now. We have customers in North America, in non-China, Asia, in Europe, in the Middle East. The idea of going big was to solve two problems. The first is that when you break up a problem, when a problem doesn't fit on a GPU, you have to break it up into component elements. and you then spread that over many little GPUs, right?

11:38And you have to tie them together and you have to move information back and forth. And this is a process called distributed compute. And not only does it put tremendous pressure on the communication fabric because it is now central to the compute itself, but also it puts pressure on you to know how far away the slowest of, the furthest of the GPUs is because the slowest response determines when the answer is, right? You break up a problem, you send it out, the results are aggregated and you wait until the last result comes. Then you can find the answer and move on. And that's Amdahl's law. And that meant that you couldn't use traditional cloud technology.

12:31It meant that you needed to know, right, exactly how far away and have to be deterministic. And this is why distributed compute with GPUs is so challenging. I think Andre Iquipathy has this statement about just how hard it is in practice to do. Now, we went big so that you never had to do that. You never had to break up work. You could keep work on a giant chip. and when you move work around the chip, you move it at about the lowest power. We know how to move data and we move it about as fast as we know how to move data. And so by keeping it on the chip, you avoid the hard parts of distributed compute and you get to work at low cost and low power.

13:19Now, while we were going big to solve the communication problem, NVIDIA was aware of the communication problem. They bought Mellanox, right? So important was the control of the movement of information that they paid, what, six or seven billion dollars for a switching company. And so this is a fundamental problem. We solved it with innovation, making a giant chip, and they solve it with Ethernet or proprietary Infiniband. Yeah, I wonder if that's a business decision, right? It's like it enables them to bundle the solution to the problem with other products and services, right? That they can offer.

14:00If you're a switch maker right now, the worst thing on earth for you is if NVIDIA is selling your customers hosts because they want to rip your networking gear out and swap in theirs. Right. Right. And so this larger chip, you know, if I'm understanding it correctly, it basically reduces the need to cluster as much as you might normally have to, which probably means that you could build far more efficient configurations. That's exactly right. Okay. That's exactly right. The idea here is that the hard part of distributed compute is as you go from one or four or eight or 16 little nodes to 100 or 1 ,000, you have to keep track of where they all are.

14:43You have to divide your work up. And this isn't AI. You've made your model. This is now distributed compute. This is putting your model onto the compute substrate, onto the computational cluster. That often takes weeks or months. And with us, it's a keystroke. And so that whole amount of time that's called tensor model parallel or pipeline model parallel distribution goes to zero. Once you stand up, once you've done this, we're faster in the straight up training or lower latency in the straight up inference. So the WSE3, it seems to almost like blow Moore's law out of the water. It did. And I wonder, is that because of what you're talking about here and the increased, like the increased size, is that what enabled that to happen?

15:34Yeah, it is. And is this Feldman's law? This is not Feldman's law. Feldman doesn't need a law. We will leave lawmaking to others. To Gordon Moore. That's right. To far more important people. I think what happened was what we saw was you would need more silicon, right? And you can get more silicon one of two ways. You can make bigger chips, or you can take big whole wafers, dice them up, put them on expensive motherboards, put them in different systems, buy expensive switches, and tie them back together again. And all chips are made off a wafer. And so in the fabrication process, everything begins on a wafer.

16:23And then traditionally, these were chopped up. We call it dicing into 800 square millimeter chunks at the biggest. And these were then put on a motherboard and you might use a thousand. We're like, why are you cutting them up? What if it's, you know, why are you cracking hunt, dumpy, spreading it out all so that you can get them to be back together again? We're like, why don't we leave it as it is? Can we invent technology that allows us? And this was the first time in the 70-year history of the computing industry that you could do this to build a chip this big. And then you get all the benefits of communication instead of across switches and Ethernet and with Nix.

17:08You get it at the speed of a silicon wire. Yeah. Yeah. Assuming Moore's law does continue, even though I guess these clusters can now be built more efficiently because of the size of the wafer, we will still need as much compute as humanly possible. And so I wonder if this ends up, physically speaking, covering even more ground. Does that make sense? Because we're never going to run out of our hunger for compute, it seems like, right? I don't foresee, certainly in the next three to five years, us running out of our hunger for AI compute. This is trillions of dollars is going to go into AI compute.

17:53And I think one of the things that's different, that really, I was in the mid-90s, I was building the first generation of hardware-based switches. And that was extraordinary as we moved, sort of, we made IP networking. free. And there were about 20 companies and we were driving and we had no idea that later WhatsApp and all these other apps would sit up on top of this and change the world. We were just thought we could make switching really, really cheap. Here, nation states are playing, right? You have sovereign clouds being built. You have them built. One of our partners in the Middle East and the UAE is called G42, and they're building together with us in strategic partnership, massive clouds.

18:40You have in Singapore and Australia and the UK, you have nation states standing up giant super compute clusters for AI. And you have that. It used to be that if you wanted your compute, you went to the hyperscalers. And now you have dozens of choices who have huge amounts of compute for rent and for sale. um and so the whole landscape is changing and i i think the ability to do ai and do it at massive scale has become sort of a strategic imperative will our hunger subside um maybe when uh the scaling laws for llms ultimately asymptote or i mean i guess we don't even know if that's going to happen?

19:25I think that's the wrong question, Michael. I mean, we don't know. What we've seen is that, and we've seen it sort of behind the scenes with open AI as their models got bigger, but we're seeing it now with Lama 1 move to Lama 2, move to Lama 3. I think the models are getting better as they get bigger. Now they get more expensive to do inference on. And this is the really curious part, right? There's a cost to that twice. First, when you train it, and second, when you do inference on it. And so I think that the models that will go to production will be trained differently. They will be smaller, but trained with more data.

20:00And they will be domain-specific. And whether you do that as a mixture of experts or a standalone domain-specific models, it is difficult to, and extraordinarily expensive, to do inference on 100-plus-billion-parameter models. And I think if you can get most of the gain by going 10 times smaller, but with better data, I think most production users would take that in a heartbeat. Yeah. Yeah. And I guess the other thing that could slow it down is you mentioned data, if and when we run out of data, right? Or we can't, you know, we can't solve data through synthetic data or something like that. The use of synthetic data is hugely important.

20:44It sounds strange or scary, but if you think about the way we train airplane pilots, right? You train them in simulators, and that's a synthetic environment. And flying a plane most of the time is there's not a lot to learn, right? And so when you want to train a pilot, you want to train landings and takeoffs, you want to train when problems occur. and the simulator can do that, can create that environment extremely well. And so it's a really important part of a pilot training. And that's, you know, the environment there is generating data and the pilot is the thing being trained. And so to the extent we can do that in self-driving cars, it's the same.

21:29Driving on the freeway is not a very difficult challenge for a car. an unprotected left turn in the snow is a much harder problem for a car. And it's like that in 25 different domains where the data collection might be extraordinarily expensive. Human subject trials, for example, we do a lot of work with the Mayo Clinic. They have some of the great medical data in the world. unbelievably expensive to gather. Unbelievably expensive to gather. 30 years of patient records, right? To the extent that one can augment with synthetic data, we will be able to do even better things. Yeah, and I guess crossing over that synthetic data event horizon would probably be really good for your business because it means that we can keep training, right?

22:28The cost of data plummets. And right now, some people scrape the internet for their data. Maybe one way to think about the landscape of AI is there are three fundamental pillars. There's an algorithm, a training model. There's data and there's compute. And when the model hits the data on compute, it learns. And that's what's happening. And so you can find in that just about everybody's strategy. Some began writing algorithms. They scraped the internet. So they got their data for free and they partnered with hyperscalers for their compute. We came at it from the compute side. We built extraordinary compute.

23:14We had so much compute that we gained tremendous experience on the algorithm side. And many of our customers have unique data sets, right? And others, many of the pharma companies, many of the life sciences have extraordinary data. And they're looking to find insight in their data. And they're partnering companies like us to build models with them and for compute. And so this sort of dynamic. And so to the extent that good data for a particular problem becomes less expensive to gather and can still teach a model, it's enormously valuable to the system. Yeah. Going back to the WSE3, you know, maybe talk a little bit about some of the other challenges that come with building these chips.

24:01You know, for example, cooling. You know, how does this thing, you know, if it's so much bigger, how does that affect heat and the problem of having to cool these things as they overheat? I think your listeners might be interested. You know, we did fundamental innovation and that is expensive and time-consuming and you need a very patient board. You need to raise a great deal of money. And when you do that, even when you succeed, often you find yourselves in a position where nobody's ever got what you have. So there's no received wisdom. And there are no components. You can't go to a catalog and order a heat sink for a part 57 times larger than had ever been built before.

24:44Nobody's got that in their catalog. Right. Right? And so not only do you have to invent in your primary domain, you frequently have to invent in components, in test fixtures, in the software testing structures. All of those have to be invented as well. Now, for 70 years in the computer industry, nobody had been able to build a chip this big. And there were a litany of reasons why you couldn't do it. And in a grand irony, we solved all of those in about 10 months with$20 million. And none of them turned out to be the hardest problem. And it's very interesting. It's an analogy that, you know, if you were having coffee at the base of Everest with the first guys who failed to get up there, they came back down and they said, there's this really hard problem about halfway up.

25:35And I said, all right, you can make a note yourself. You get up, you get to the top, you come down, and you're having coffee with the same guys. You say, that wasn't the hard part, right? The hard part was near the top coming down. the hard part was different than because you knew because you knew that would be a problem or no because nobody knew nobody had even faced the problem everybody had been stopped two-thirds of the way there i see not to face the hard problems right right you never yeah no one has ever tried to cool uh 17 kilowatts to a chip right right i see never tried to deliver that across the motherboard nobody ever tried to cool a chip or package a chip of that size because nobody ever had a chip of that size.

26:17And so it turned out that these problems that hadn't been encountered before turned out to be easier than the problems that had broken every previous effort to solve this problem. And that was sort of interesting and non-intuitive. And so the problem of powering and cooling, we have dozens of patents around that. Our inventions are significant. Two interesting things happen. First, per square millimeter, we use about the same power as other people at five nanometer. We have about the same number of transistors per square millimeter as others. We just have more square millimeters. It's just bigger.

26:59It's just bigger. They have 800. We have 46 ,000. Yeah. Now, this does two things. First, when you cool a chip, generally you put a little fan on it called an impingement fan and you can only afford a very cheap cooling system because the chip's standing alone because we had consolidated all this silicon we could afford to use water and so we designed a water cooling system for it and so we use a cold plate on the back and we run cold water across the back of the cold plate And so the result was we could amortize a more expensive cooling system because we had so much compute to amortize it over.

27:41And the result is we run the wafer at a lower junction temperature. And that means in electronics, you have a longer lifetime. Got it. Got it. But all of these were really hard problems. And each time you thought you were in the clear, right, each time you thought you were through the tunnel, you know, you encountered another difficult problem. And that was sort of the fun part of this project is we were well beyond where anybody else had ever been. You know, on the old maps, the old cartographers used to write here the dragons where they didn't know, right? That's where we were. We were out there with the dragons.

28:22Well, what's cool is now when you build the next one, you already know that some of these problems are actually solvable. I mean, I'm sure there'll be new problems, right, on the next mountain. We ended up being among the world's leaders in a collection of areas that we didn't know we would need to be good at when we started. Right, right. And we did it through sort of relentless engineering discipline by carefully making mistakes and understanding which failure analysis, why we failed, not making the same mistake again and step by step. And this was true for not just the wafer scale design, but the packaging and the power delivery and the cooling and the compiler software.

29:02We made a ton of mistakes there. Over time, it just got better and better and better and better. Maybe talk a little bit about like the unique properties of the wafer scaler relative to your peers and competitors probably enables you to work with some interesting different types of customers, right? I understand you work with national laboratories and healthcare giants and obviously in the enterprise. Like maybe talk a little bit about the customers and some of the cool use cases you've seen as a result. There's some really extraordinary use cases. And we designed the chip and the system and the software that goes on it and the compiler chain that allows your ML practitioners to write PyTorch and run on our machine.

29:49And then what we sell is either a system for your premise or time on systems through our cloud or through our CSP partners' clouds. And so when you are this fast, we've been able to solve problems using generative AI models to predict mutations in viruses like COVID. Some of our work in partnership with Argonne National Lab was given the top award in the industry for that. We have a publication coming out shortly with another one of the national labs. We're doing work with, we published work with Gladys Smith-Klein, where we did AI work on epigenomic data. Working with our strategic partner, G42, we trained the largest Arabic model.

30:39GPT-4 is really weak in Arabic, and there are 400 million Arabic speakers worldwide. And so we trained an Arabic model. And then it's now, it's called JACE, which is one of the tallest mountains in the UAE. And it's now one of the foundations for Microsoft's Middle Eastern service, served by them. That's really cool. Yeah, we're working on extraordinary models with the Mayo Clinic. Models that allow you to look at an individual's genetics and make estimations about which rheumatoid arthritis drug will be most effective. How cool is that? This true personalized medicine. We worked with Totala Energies, one of the largest oil companies in the world, in reservoir simulation and using AI for the estimation of reservoir behavior.