The World’s Largest AI Processor with Cerebras CEO Andrew Feldman

0:05The shortage of Compute for AI, also known as a GPU crunch, is increasingly impacting AI companies big and small. It is delaying training runs and launches for multiple players in the generative AI world. One company coming to the rescue is Cerebra Systems, which is developing the largest computer chip and one of the fastest purpose-built AI processors ever. This week, Sarah and I are joined by Andrew Feldman, CEO of Cerebra Systems. Andrew is one of the few entrepreneurial veterans in the semiconductor world. He previously started Cmicro, a pioneer of energy-efficient, high-bandwidth microservers.

0:39C-Mirco was acquired by AMD in 2012. Andrew, thank you so much for joining us today. Ilan, Sarah, thank you so much for having me. I think you all recently announced that Cerebrus has closed a$100 million deal with G42 to develop one of the largest AI supercomputers in the world. We did announce a strategic partnership with a group called G42, and we announced that we were building nine supercomputers. Each supercomputer would be four exaflops of AI compute. So in total, 36 exaflops of AI compute. That was extraordinarily exciting. When you encounter a partner that shares your vision and wants to build with you, and you get to start building the biggest compute on earth.

1:25It doesn't get better than that. And I think in general, you all were very forward-thinking and early to identifying AI is a really important market for custom semiconductors. Could you tell us a little bit more about your thinking early on as you started Cerebris and why you focused on AI many years ago? In late 2015, five of us started meeting regularly. I think we were meeting in Sarah's offices, actually, at that point. All of us had worked together at a previous company, and we began working on ideas. And one day, our CTO, Gary, leaned back and said, why would a machine built for pushing pixels to a monitor be ideal for AI.

2:03Then he said, wouldn't it be serendipitous if 20 years of optimizing a part for one job left it really well suited for another? That got us excited. We began looking at GPUs, looking at AI work. And by early 2016, we decided that we could build a better part for this work. And our strategy was not to build a little bit better, but it was to try and do something vastly better. And we went to a technology that had never worked before called wafer scale. And we build a chip that's sort of the size of a dinner plate, whereas most chips are the size of a postage stamp. And we did that because we knew that this workload would go big and that the problems of memory bandwidth and problems of breaking up work and spreading it over lots of little machines would be daunting.

2:58This is my fifth startup. This is the first time I was wrong on the market size on the underside. I had no idea it was going to be this big. And I think very few people saw, even those of us who were in it, how big this was going to be. And so early 2016, we went out, we did eight pitches, we got eight term sheets, we raised money, and we're building multi-exaflop AI supercomputers for customers around the world. It's really incredible thinking about the foresight. I actually went back and looked at my notes from like our end of 2015 meeting. And if I'm sure you remember very well, but for our listeners, Andrew had a slide in there of how the top seven problems for deep learning were all long training times.

3:47And like the whole industry was bottlenecked on computing. This is even before the scale up of transformers that happened. It's kind of wild how correct you were, at least on the depth of this problem. I think what you need to do is save that snippet and send it to my wife. We got that right. I think also we got the fundamental architecture right in a large sense. We laid down the architecture before transformers exist, and we're the fastest at transformers by a lot. So when you break up a problem and it encounters things it has never seen before, and it's still really good at them, that's a sign in hardware architecture that you really got the architecture right.

4:27What are some of the benchmarks that you use in order to assess the performance of your chip versus others? And how have you all performed relative to GPUs and other sort of more standard? We are not big believers in canned benchmarks. When I was at AMD, we had a team of 30 whose job it was to gain benchmarks. When our CTO was at Sun, they had a team of 70 whose job it was to gain benchmarks. How long it takes your customer to train a model is the answer. and usually a collection of models. If you're trying to do a 13 billion parameter model, medium size, right? Almost always you begin at 100 million and you do 100 million, you do a series of sweeps, right?

5:04You're trying to do your hyperparameter tuning. So you do 100 to 300, 600, maybe a billion, and you're trying to find the right optimization points. And I think how you do on all of those is the answer. how long it takes you to move from customer model to distributed compute to finished run. And if you have to spend months doing distributed compute, doing tensor model parallel distributed compute, I mean, if you look at the back of some of these papers, they're crediting 20 or 30 people, sometimes more, who helped on the distributed compute. And if you don't need to do that, like you don't with our equipment because we run strictly data parallel.

5:51That's all time that you get back as a customer, right? That's all time that you get to train models and do interesting work rather than try and think about how to break up a large matrix, multiply and distribute it over a collection of GPUs. Can you actually talk a little bit about the architecture of your chip relative to others? All right. It's a data flow architecture. it's comprised of about 850 ,000 identical tiles each tile is a processor and memory and so it's a fully distributed memory machine and so you have huge amounts of memory bandwidth because the memory is speaking to a processor that's one clock cycle away so by data flow it means that the cores wait until they get a token or a uh a flit that arrives it tells them what to do do work on this and where to send it and then it sends it forward and this is a particularly good architecture for uh the type of work ai is um ai is a is a flow right It's a learning system.

7:04It's interesting. It was developed in the 80s at MIT by a guy named Arvind, and they didn't have a good workload for it, so it just sat there. Nobody, and then we got a good workload for it. It's phenomenal for this. That's a really interesting part of computer architecture, where a really cool architecture can have no interesting workloads for it, and then you get a good workload, and now it's ready. Now it's good at this particular problem. We keep a huge amount of SRAM on the wafer, all right? And so there are no memory bandwidth problems ever. That also allows us to harvest sparsity, which is something that others really struggle with.

7:44The cluster we build keeps the parameters off chip in a parameter store, and it streams them in. And the result of this architecture is that we run strictly data parallel, which means even in a 64-node cluster, you run the exact same configuration on each machine. Each machine sees the exact same weights, but they see a different portion of the data set. And it scales linearly, which is unheard of in compute. Can you, for a broader audience, describe how this addresses some of the challenges of using traditional GPUs for large-scale machine learning? Sure. One of the real challenges of a traditional GPU is they fix the amount of memory on the interposer.

8:32So you can buy a GPU with either 40 or 80 gig. What if you want 120? What if you want 20? What if you want a bigger parameter store? You've got to buy more GPUs even though you don't want those. And so what this strategy does is it allows you to support extraordinarily large models by disaggregating the memory from the compute. And in all GPUs, they're tightly coupled. And that means if you want more compute without memory, you still have to pay for the memory. If you want more memory without compute, you still have to pay for the compute. and that was an idea that came from supercomputing that we knew really well that we could organize this so you could run an arbitrarily large now a trillion parameter network on a single system now it would be slow that's a very big network but you could run it on one system you can identify layers you you can debug you can do all this and you can never do that on the gpu because there's nowhere to store the parameters.

9:33And so by separating parameter storage from compute, we allowed you to mix the various ratios at your will rather than at the GPU vendor's will. So I guess in terms of the relationship that you have with G42, there's a variety of different applications that you've looked at across healthcare, across a variety of different verticals. What do you view as some of the first use cases or best applications for your architecture? So the first thing that they allowed is a model called BTLM, which is the number one 3 billion parameter model on Hugging Face right now with more than a billion downloads. We have Fortune 100 companies running on it right now.

10:17We have them doing their own internal work on it. On Wednesday, we're announcing that we're open sourcing with G42's group called Inception and MBZ UAI University, the largest Arabic LLM. So we're putting that into the open source community. That was a collaborative effort with our strategic partners there. When you have a lot of compute, the mind boggles at all the cool stuff you can do. Why did you guys get into the training and open source model game? We had so much compute and that it was a way we could prove to the world how easy it was to train on us. We felt it was evidence that we could build and our systems could train the biggest and fastest models, the most accurate models in the space.

11:12In March, we put seven GPT models in the open source community. Everybody else was putting one. Why? Because it's really hard to redistribute work across a GPU cluster. For us, it's one keystroke. So we put seven. People are coming to us with extraordinary ideas. We had a customer who came and they said, look, we'd like you to design for us a model at$3 billion that we could prune and quantize such that it could be served off a cell phone. Really cool. What are the biggest milestones that you're thinking about going forward? There's been a lot of back and forth. Everybody thought we'd keep getting bigger.

11:53North of$175 billion is a giant network. It's expensive to train. And maybe even more important, it's really expensive to serve. 17.5 times more expensive to serve than a$10 billion parameter model. And so I think a lot of the models in production are$13 billion and smaller. as an effort to keep the cost of production down. We are training right now a whole collection in the 30 to 175 category. I think there's going to be a spike in multilingual models. Stability just put out one in Japanese. There was one in Spanish that came out. Inception, MBZUI, and Cerebris put out one in Arabic. I think these are underrepresented languages.

12:41and I think we have a challenge there in that most of the big data sets are giant internet scrapes and most of the internet's in English and so it's really hard to build big data sets in in languages that aren't English we don't have enough tokens and so techniques to carefully think about how you can do that I think will be very much in demand I think every every nation wants to feel like its language and culture is captured fairly in an LLM. That's interesting because a lot of the training also seems to capture social norms versus data. And one can imagine as well, as you think about different regional models or different language-specific models, people want their own set of cultural norms to be part of the model in some sense versus something that's imposed.

13:27That's exactly right. Language captures experience. I mean, there's words that are different and have very unique, I mean, it's just like the New York experience includes Yiddish, right? It will be important that we find ways to capture these cultural experiences. And we do that in language. And if you overwhelm the minority language with English, all that gets sort of sifted out. I think the Yiddish LLM would just complain all the time. That would be hilarious. That would be hilarious. Get a show out of it. It's like, what, your brothers are doctors? Every response is, why aren't you a doctor yet?

14:11Are you going to finish your PhD, Andrew? Oh, my God. No, don't worry. The Chinese LLM says that, too. Oh, it will. That's right. Those exact same mothers. I think it's like actually like kind of a deeper insight in the joke, which is there's cultural experience baked into both the actual tokens and pre-training curation and like what you're weighted toward as well as in the rhf layer right um because whatever you know human annotators are saying like this is this is what i want somebody wants to kvetch with their chatbot too that's exactly right one of the things we learned was that it's really important taking into account the the the language that has fewer tokens especially you know arabic hebrew hindi their characters sometimes require two bits rather than one bit.

14:59And so the tokenization scheme has to be really thoughtfully done, or you further bias the model in favor of the English. I think, you know, you put your finger on exactly why our strategic partners are so excited by this is they want their culture and their language properly represented. They want it reflected. They want the nuances in their culture reflected in an LLM and not sort of sanitized or washed with 99 % English tokens. Can we zoom out and just like talk a little bit about what it's like to start a chip company? It's a very uncommon experience, especially, you know, in as far deep as you guys are now with the money you've raised.

15:42Like what's the hardest part of bringing a new chip to market? I think the structure of the chip market is challenging. I think one of the things that VCs love about SaaS and about software companies is for not very much money, you can get a prototype and you can start seeing if customers like it. Chips are the opposite. We're going to spend two, two and a half, sometimes three years and 50 or 60 million dollars building our first part. And then we will learn if people like it or hate it. It has a huge gaping chasm between when you start and when you begin learning from customers. And that's one of the things that the software doesn't have.

16:25Now, it turns out that if you're building a SaaS company, you end up often using as much money. You get insight into what your customers like much, much earlier than in a chip company. And I think that's perhaps the biggest structural difference. The structure of engineering is different when you've got a two or three-year project. You can't do 150 one-week spreads. The way you manage QA versus development is different. Often in chip companies, for each developer, you have three or four QA people, DV people. And that's because a bug can cost you$20 or$30 million. It's not like, yeah, I'll fix that in 4.1.1.

17:05You're going to have to re-spin the chip if you have a big bug. And so the commitment to tooling, to QA, to simulation is profoundly different in the chip world than when you're building software. Now, we still have to build a huge amount of software, and we're about 75 % software. When you build a chip, the chip is the muscle and the software is the brain. You still need a huge amount of software. You need low-level software, software that runs right against the hardware. You need a compile stack that allows the user to write in PyTorch and just run as if it were a GPU. And you end up with a lot of different types of engineers in a system company.

17:48Andrew, you have such a unique expertise and depth and a question that everybody is thinking about right now. Can you give our listeners an overview of the AI accelerator market right now? Like who are the big buyers and then how might they decide to do anything but NVIDIA? NVIDIA's made hay when the sun shines. And they've done an extraordinary job. And you have to take your hat off to them, right? I think they're now in a situation where they're extorting customers. They're extremely expensive. They're unable to ship. And that has, among other things, opened the door for many of us who have alternatives.

18:23Nobody likes being dependent on their vendor. You can ask the guys at Google and Facebook who are dependent on Intel for years how much they hated that. They dislike that intensely. And so I think there's a battle between people's sort of dislike being dependent and the need just to keep running forward. I think large enterprises continue to be a good part of our business. I think often where you can provide a little consulting to help them accelerate their model deployment is something that provides an opportunity for smaller companies to get in the door. And when you show them how much easier it is, we had a situation where they were trying to train on a GPU cluster and they were at 60 days and it wasn't converging.

19:12And we stood it up and three and a half days later, their model converged. And they're like, holy cow, we have an internal cloud. So customers can just jump on your cloud. They can begin training right away. If they want to try before they buy, they can begin with a little training run and go from there. We have customers in large enterprises and generative AI startups. We have customers in the military and the intelligence communities. We have customers in the supercomputer market world. We sort of up and down organizations of different sizes. How much specialization do you think there'll be in the semiconductor layer going forward?

19:48In other words, do you think there'll be increasingly different chipsets for inference versus training, for transformer-based architectures versus diffusion models, for multimodal versus not? I'm just sort of curious how you view all that. The fundamental question in computer architecture is what do you make special and what do you make general? And the second fundamental question is what are you good at and what are you not good at? What do you choose not to be good at, right? And those things are really important. You know, if you put in circuitry for transformers, that has a cost. There's a penalty every time you do a convolutional network.

20:27And so these trade-offs aren't free. You put in circuitry for one thing. If you're not doing that thing, that circuitry just sits idle and you've wasted power and space. And so I think it is a TBD question on whether we will continue to see more specialization at the chip level in training. We do not do it. We cracked the problem in a different way. All of the problems are linear algebra and are predominantly sparse linear algebra. And so we focused on underneath the transformer. Now, whether you will have different silicon for training and for inference, I think you will. So your view on this, Andrew, is less a strong point of view that there will be something beyond transformers that is very important in the near term and more you've already solved in a more general way.

21:23That's right. The trick in architecture is to solve hard problems in a general way so you don't have to rely on product management to sort of guess what's the next cool layer type, right? If we go to a type of model that doesn't require very much data, if we go to single-shot learning, right, NVIDIA is totally out of luck, right? Us too, everybody. These models are designed for extremely data-intensive transformer-based. But whether you're doing convolutional neural networks or transformers or diffusion models, certainly new things will be invented. I mean, we haven't run out of ideas. We haven't even run through the ideas that the guys put together in the 80s.

22:08We still have plenty of work to do to improve our model making. And as the industry matures and we're still in the early innings, you might see more customization for a particular model type. But that's not my preferred approach. Are you making bets internally on separating training and inference? We make all sorts of bets, Sarah. We lose frequently and we win sometimes. And that's the real fun of building companies in a dynamic environment, right? I think inference, especially generative inference, is a very different problem than all other forms of inference. Basic classification is the exact same as the first pass in a training run.

22:59But when you're doing inference on generative AI, it's not, and you have to hold a ton more information at the inference point. And so I think we will discover, we will invent new and different ways to attack that problem. It's extremely expensive right now. I mean, people using 8H100s to do inference on a big model, I mean, that's half a million dollars. You know, there is this ongoing crunch to supply for GPU and other sort of AI-centric compute. And that's causing different people to delay launching their companies or training their models. and those are often the same thing for different types of AI companies.

23:41What do you view as the main cause of this supply crunch? And when do you think it loosens up? Or when do you think things loosen in the market? The chip market has a profoundly inflexible supply chain. And so TSMC, a fab is a pyramid in modern society. It is the absolute peak of manufacturing capability. It's a$20 billion building in which nobody works in it. and ideas come in and out come chips that cost a hundred dollars i mean these are unbelievably unbelievably amazing things but they can't turn on a dime and so when somebody misses their forecast by a lot and they come back to the fab and say i need twice as much or three times as much often that takes six or eight months to to sift through and that's the problem that's coming in right now.

24:35I mean, it's not just that Wall Street missed what NVIDIA would sell. NVIDIA missed it. They missed the forecast. And you can go to TSMC and you can ask them for wafers, but many of their wafers are already allocated to AMD. They're allocated to Qualcomm. They're allocated to other giant companies, right? And so you struggle forward. And so in addition to taking a long time to get to first customer learning, there's a huge premium placed on the ability to forecast accurately in the chip business. Given that we're going through this big discontinuity from a technology perspective, adoption perspective, et cetera, do you think we'll be accurate in forecast going forward anytime soon?

25:18Or do you think there's likely to be a crunch for the next couple of years? We're going to continue to get it wrong. We're going to continue to get it wrong for a while. Do you think we're largely going to underestimate it? Always. Yeah. So basically the market is accelerating rapidly. Everybody's underestimating it. And fundamentally, we all think that this is going to be many times bigger than anybody thinks two, three years from now. Yeah. And the dynamics of our supply chain are when we get it wrong and you order too much, you still have to take it. You don't get to say, oh, guess what, guys?

25:48I really don't want those wafers anymore. And they're like, see this$10 million worth of wafers? Yours. Yeah. That happened during crypto, right? So I think NVIDIA had a really big miss during crypto. It happened during crypto. It happened to Broadcom. I mean, Arista today is at 52-week lead times on switches. Given that you've been involved with training some incredibly performant open source models, and you mentioned that as you scale up a model, inference cost starts to really kick in as you get to bigger and bigger models from a parameter perspective. What do you view as the limits to scaling these models?

26:24Like, at what point do they get too big? or do you think we just keep scaling them until we have these sort of hyper models? It's a really interesting optimization problem. They get big, they get harder to work with. It's harder to retrain, right? When even GPT-4 came out, it didn't have any insight beyond 2021. They had to race. That was because it was so big, right? And so I think there are trade-offs. There are trade-offs between accuracy and size. There are trade-offs between size and cost to do inference. I mean, maybe I want a larger model for radiology files, right? Or for my doctors, maybe I'll take a smaller model for my chat or my customer service bot, right?

27:04We're going to have to think about these in terms of what they're delivering, what the cost of being wrong is, what the cost of serving is, and think about this as a business decision. And at first, everyone's just running trying to say, I can make a bigger model, I can make a bigger model. And then the guys who are trying to run businesses, you're like, well, I can't afford to give that away. And so we're going to be down at three and at 6 billion and at 13 billion, because that's, I can get pretty good, pretty darn good, and not break the bank with free inference. And so I think that there are these trade-offs that are really challenging and we're just beginning to grapple with now.

27:45Now, I guess OpenAI related to that recently announced a partnership with Scale around fine-tuning. How important do you think fine-tuning will be going forward? I think fine-tuning will be really important. We're still just learning about the capacity of these models to hold information and to hold insight. And fine-tuning is an extremely exciting approach. I think we've seen again and again that human feedback makes these models vastly better. And that's not surprising, right? I mean, that's how you learn to dance. That's how you learn to do everything, soccer, gymnastics, ballet, and somebody saying, no, that's wrong.

28:20Nope, do it again. That correction is a mechanism for improvement somehow doesn't seem very surprising to me. I think we don't yet know how far fine-tuning is going to take us and who will benefit from a model train from scratch versus continuous training versus taking a model off the shelf and adding to it their unique data. You know, there is this point of view from some of the large labs that very few people will do training and it will get quite concentrated. But I've heard you speak about how you think that there are really interesting data sets in different large enterprises. Tell us about that.

29:03You know, at first, everybody scraped the Internet, but then everybody had that. And now the companies that have extraordinary data sets, Reuters, Bloomberg, GlaxoSmithKline, AbbVie, these companies that have years of research and exceptional data are going to, I think, step into the fore. The AI has made progress to the point where people are looking around for exceptional data sets, unique data sets, the rise of data as an asset. I think it's the new gold. And I think that's something that isn't talked about enough and will be very much top of mind going forward.