The Biggest Chip Ever Built — Why OpenAI Runs On It | Cerebras CEO Andrew Feldman

35:38And nobody had done it before. And over that 18-month period, we became the best in the world at it from approximately zero. And we did that by sailing again and again and using good engineering methodology and doing a failure analysis, every single failure. So we failed differently again and again and again and again. And we told our board, you know, we met with our board over six weeks. And yeah, this is a strategy. Nobody's ever done this before. This is what we're going to do. And we had to invent new materials. We had to invent new techniques. We ended up building things that everybody else had partners who could do.

36:20But in all this to 2019, we announced we'd solved it. And, you know, my co-founders and I, the first time it worked, it was sitting in a tiny little lab. And we couldn't believe it. We were the first few things this tree gave me to make one work. And we just stared at it. You know, watching a server run is about as exciting as watching paint dry. And there we were, the five of us, just staring at this machine, not believing it might work. We made it work. Was it a bigger moment that actually rang the bell? or just completely different events? Like emotionally? Emotionally, it was a completely different thing.

37:00It was that we had made our idea work. And I think for deep tech founders in particular, there's always this little thing in the back of your mind that says, maybe it's shit. Right. Maybe we're actually crazy. That's right. Maybe it's going to fail. And maybe it's not going to work. And maybe I don't have time. Maybe we're going to run out of money. maybe, maybe, maybe. And the flip side of that is the joy that this was my co-founder's ideas. I mean, these are their ideas manifest in the world. And that is an extraordinary thing. And it's when someone's ideas take physical shape. And then the next step is when you watch other people's work sit on top of your idea, then you know you love making infrastructure.

37:50And so when we rang the bell and we went public on May 14th this year in the largest semiconductor IPO in history, and we did something unusual. We invited all our engineers who'd been with us to start. Everybody who'd been with us more than nine years and their families, and we all rang the bell together. We all got up on stage. which that was sort of a moment of, oh, a different pride that we had done this together. And that we should manage not to die. It's untied with a startup, you know, people aren't honest. They don't tell you that, but we'd avoided some, we'd made plenty of mistakes, but we've been avoided the sad ones.

38:42and we made it through to a level of success that gave us the opportunity to pursue a new level of success. That's what an IPO is. It's not the end. It's you've gotten to a plateau that you can chase a new level of success in the public market. And that felt pretty good, I'll admit it. Amazing. Thanks for sharing. So just to go a little deeper on some of the product and technical stuff. So one of the trade-offs of building a bigger chip, one that may come to mind is failure mode. Sure. If you have a lot of little chips on a big wafer, presumably you can isolate the problems. Yeah. If you have a big one, then everything won't fail at the same time.

39:37So you have to think very carefully about failure mode and we invented a technique that had about a million identical tiles. And if one fails, we can shut it down and we can use redundant ones, we keep going. And so if you're going to go big, you have to think about in the very architecture of the computer, how you're going to manage value. I mean, the JPs have a huge failure rate. So I'm sure you guys have spoken about this. Infant mortality is enormous, and they fail all the time. There's big data. Facebook put out a paper on the number of failures they get in a big cluster. Now, we can do some other things.

40:23Because we have all this compute in one spot, we can invest more to cool it. So we pioneered water cooling in AI systems, and we run these much colder. and GPMs. And the failure mode in electronics is temperature. And so by running them cleaner, we are more reliable. And so that was an advantage. But again, we had to invent the technique to cool off a chip this big. And so I think we have a system mentality, right? If you're going to do something big, you've made trade-offs. And you have to think about an architecture that allows for redundancy and repair you have to think about the pros and cons of every aspect of the architecture and that's how you do something different and new great and again just to drive the point home um to make sure this i guess clear takeaway uh you know for people from this conversation.

41:31So explain it to me, like I'm maybe not five, but 15. Why is this faster than a JPU? Like what fundamentally means it faster and what can't you be super fast with a JPU? How tokens are generated in inference is why it's faster. In an inference to generate a word and our answers are a whole stream of words. They can be code, they can be pixels, but call them words. The weights of the model are moved from memory to compute. Calculation occurs and that generates the word. Is it pre-fill versus decode? That is both steps. Okay. And do you want to maybe explain what pre-fill and decode are? Okay. There are two steps in the computation necessary to do inference.

42:34When you type in the chat GPT, explain to me the history of this village prior to World War II. And it can't see you. Two things have happened. The first thing is your prompt has been processed. That's step one. And step two, your answer has been generated. And the way your answer is generated is called decode. And decode is sequential. Processing your prompt, which we'd call pre-fill, can be paralyzed. So you can process many of them simultaneously. But the speed with which you get an answer is a function of the decode, and it is step-by-step-by-step in sequence, and that can't be changed. And how you do that step is you move weights, which are the intelligence from the model, to compute, to generate a word.

43:47You do a calculation, and you get a word. And then that word is used to generate the next word where weights are moved from compute. So the process is one of moving weights from memory to compute. So how big are the weights? In a little model, like a 70 billion parameter model, the weights are about the size of 100 HD movies. So to generate a single word, you move 100 HD movies from memory to compute. And then you have to move them again for the next word. And you want to do this a thousand times to get a good answer. A thousand words. This is where HBM is slow. This exact step is where HBM is slow.

44:34And that exact step, it is where by having all the SRAM here, we're blisteringly fast. And so the speed of moving waits to compute is about two and a half thousand times faster here than on a world in GP. And so that's the essence of what we've been able to do here and why it's so much faster. Yeah. Fascinating. Do you think that models need to really evolve as well in that fast AI world? Or is that just a problem? No, exactly. Because remember, two things are happening. One, we want the models to be smarter. And two, one of the ways models are getting smarter is with RL. And RL uses inference inside a training.

45:29And so the faster you can do the inference inside a training, the faster you're training. One, two things. Is that a market for you guys? Yeah, that's a market for us as well. So you're not just in France. We do RL and we do traditional trading too. Not for the largest models, for the largest lab, but for the next tier. For pre-trading, for pure pre-tuning. Pre-trading, fine-tuning, a full set. So in Chengdu, pre-trading, I mean, post-trading was RL, some pre-trading for the other labs. GPUs are still better than you for what job? So GPUs in training have some challenges that have been solved by very narrow selections of the community.

46:29GPU is a very small chip. And the calculations that we need to do in trading are very large. And one of the most complicated parts of trading is the breaking up the calculations and spreading them apart on multiple GPUs. And that's called distributed compute. That has historically been the domain of the supercompute world. and is very difficult, not just because cracking a problem and having lots of others work on it is hard, but they have to constantly share information. And that sharing information is why they needed to buy Mellanox. They needed to control a fabric over which all is sharing in order to get an answer.

47:20That breaking up a big matrix multiplier, that big calculation is called running tensor model parallel. You are breaking up the tensor and spreading it apart. And the best labs in the world are good at that. But nobody else is. When we run training, we don't have to run that way. We run what's called data parallel. And data parallel is very simple. And so it allows teams who are good and very good to quickly test ideas in training and so we are easier to use and faster because we allow them to use a technique which is much simple let's talk about the energetic world maybe starting with with um with reasoning um i think i saw a blog post where you guys have a given state of that reasoning was not always the right solution for all problems how do you think about this and where does that fit in the answer to it well reasoning is a technique a little bit like when you're in 8th grade and you wrote different drafts of the paper

48:44single shot in front of the seat you get an you write a query you get an answer it the easiest way to think about reasoning is you're going to do several drafts it's going to break the problem up it's going to solve in parts it's going to bring them together it's going to review the results it's going to improve the results and then it's going to give you an answer and so that is going to take more compute And if your computer is slow, that's going to be more than an irritant. That could be crippling speed. And so as the best models, all of them, whether they're domestic or Chinese, whether you're open AI or anthropic or Gemini, whether you're any of the Chinese models, they all move to a reasoning approach.

49:38But that meant more compute was being used during inference. That was a huge advantage for us because we were fast. And it made the GP slowness stack up. And it made our speed have an even bigger advantage. And so this was a huge building for us. We think this is going to stay as a fundamental essence of the way these models are run right now. And you also wrote about verification and whether what was the bottleneck in agents was whether the reasoning was good enough, whether they were smart enough or whether there was a verification problem. Sure. I think the verification problem is a little bit like the guardrail problem.

50:34And what you'd like to do is after you've written two or three drafts of your answer, you would like to be sure that it wasn't wrong. And that's your verification step. And you can do that maybe with a different model. You can do that by asking your existing model a similar question in a different way. All of these are ways you can pressure test your result. Guard rails can work the same way. You want to review either with a model or with another technique through a scoring mechanism that this question isn't out of line. that this answer isn't about how to make biological weapons or calling upon information that you are direct to the FBI, right?

51:35And all of that takes compute time. And so whether you're trying to improve through reasoning or whether you're running guardrails by being faster, you can get results in less time. So what do you think the world is evolving for agents? Is it a bunch of smaller models running faster, doing more verification versus a large model? But I think those work together. I think your big model produces an answer. Then you want to double, you just want to check your data. I mean, in the journalism industry, people would write papers that have data checkers, right? Somebody would go and make sure back in the world, journals check data.

52:19In fact, the data checker, right? That was a job. Each claim was checked independently. that's a different model right the main model wrote the piece and then a little model checks some of the answers and i think that's a very a very good way to go by where does a multi-modality for the larger models falling in your world we just announced uh sort of that we were fastest in the world on on one of google's multi-modal models i i think the truth is is that there's very little text that doesn't have charts and graphs right you you must to understand text uh be able to understand uh illustrations graphs uh charts and so that that's sort of the the first and easiest part and then you ought to be able to create both right and then you want to understand images and uh i think the the new models are very very good at that obviously what follows that is video because a video is just a collection of images um but that takes an enormous amount of compute right now and that's one of the resources in the sort of set aside by the leading labs and so unbelievably computation intensive Great.

53:51Let's talk about the super business a little bit, because you guys are obviously a chip maker and provider, as we discussed. You're also a cloud provider, data center provider. What's one of the different parts? we may compute. And that computer is optimized for AI. It is the fastest at AI in the world. If you have a data center, we will sell you hardware for deployment in your data center. If you don't have a data center and you'd like to rent it by the month or the year, we have data centers. So you can rent our equipment through our data centers and through our cloud. And so that allows us to get AI natives as well as large enterprises and governments.

54:44And what's the pressure? It was 50-50 last year. And I think this last quarter, it was maybe 75-25 in favor of hardware sales. I think this year it might be 50-50. And as our OpenAI deal continues to unfold, it will probably be 30-70 with 30 on-premise deployments of hardware, the 70 class. Great. Let's talk about that OpenAI deal. since it's such a major historical milestone record making. So it's providing up to 750 megawatts, which is interesting, by the way, as a metric, because we're in a chip provider, but there's this power. So is that shorthand for... It's a shorthand. It turns out right now, and we didn't talk about this because there's sort of in the adjacent supply chain, we went through the shortage of memory, We had to do the shortage of a process called COAS at three nanometer capacity.

55:50The other limitation in our industry right now is data center availability.

55:57And that is a limiting factor for everybody. And that's why Anthropic did a huge and sort of very expensive deal with Elon for data center capacity. Our deal with them, with OpenAI, was because data center capacity is a limiting constraint, measured the way data centers are measured in megawatts. The deal is 760 megawatts, 250 megawatts in 26 on a multi-year lease, an additional 250 megawatts in 27 on a multi-year lease, an additional in 28 on multi-year lease. And you're doing the data center for them, or are you providing the chips that are going to the data center for them? We're delivering a full cloud solution, so they connect to us via an API.

56:52Basically, you mentioned 2026, do I say it's immediately about general warning? I am looking for data centers. My next meeting is in fact with a data center provider. we're doing a lot in Europe right now a lot in the Nordics is that because it's closer to power sources? yes, it is because there is low cost power how about that? clean, low cost power and low cost doesn't matter in their time business whether data centers is located in terms of yeah, there is an additional latency called transport latency. And that's the speed of light through fiber to get from Helsinki to New York. And you have to account for that if your customer is in New York and your data center is in Helsinki.

57:50And it's usually about two-thirds the speed of light, in case how long it takes. You would like data centers on the same kind, both of them. That's the open AI deal. There was an exciting deal with AWS as well where you... It's a code chip solution? That's right. It's the disaggregated solution you mentioned before where their pranium part is doing the pre-fill and is doing the parallelizable step. So it's pranium. It's pranium. It's doing the pre-fill step and our chip is doing the decode. And so you get a lot more bolusierably fast tokens. It's a good deal for us. It uses their data centers.

58:38So these are deployments in the AWS data centers. Yeah, it's fascinating, right? Like talking about this industry is how it seems that flexibility is so important. There's just something to bring a solution like you need chips, you get chips, you need data centers. Everybody's buying from different suppliers to reduce dependency. I think that's one of the reasons why we went from being a traditional shipping system provider to also offering data centers is that what our customers want are fast tokens. And anything we can do to make the delivery of fast tokens easier. For some of them, that's in their data center.

59:20For some of them, it's with an API. Just point your traffic to us. which will point the fire hose to a fast token back. Yeah. And just to get a sense for where you start and where you stop, is you do not at least currently provide the cloud version. So if I want to run Kimi, like I know you have like incredible stats for Kiwi and Gemma and Chitospe, and I forgot to mention them, but you don't run those as a service or do you? We do. You do. Okay. So you have a service who is still competing with the base tents and fireworks? Yeah. I think we have an on-demand service where you can come to our site and book a month.

1:00:10I think you can even buy buckets of tokens for Kimi or GLM or some of these models. Many of our customers come there and get excited about it and then you move to a dedicated offering where they take hundreds of machines for a year or two or three or four once they've proven out that um the benefit for them in their work often they do a b tests not surprising people like faster you got so it's just more like a testing it's a full environment okay um you can you can go and use it it's at three births.ai you can play around fascinating but that could become like a yet another big clock at this time okay so you have uh ships you have data centers and you have a client business running in france on um thinking about modes so you know famously uh nvidia has uh kudos and well discussed mode what's your equivalent of you know well i don't think kudos are most correct korea we should talk about that yep I think two years ago, every state-of-the-art model was trained in a CUDA flow.

1:01:30And right now, Gemini is trained without CUDA. Anthropic Law is trained without CUDA. OpenAI is trained without CUDA. So in a one or two year period, they lost 70 % share of training models or is the state of the art. Because Gemini is trained on TPs. It's very good. Out of anthropic is trained oxygen. And so I think the storage of the mode is still present, where the data show the mode is clearly shrinking. There's no mode in inference. It takes you eight keystrokes to move from a GPU to us in the cloud. and it's for a seat oh it's each cheese cheese cheese for sure that's it to move your traffic from gpu api to us and so uh obviously gluda was sort of enormously important the creation of our industry and in allowing the the graphics processing unit to be more general than graphics its process.

1:02:51But since 2324 quick live, I think it's ability to serve as a durable moat, short, substantial. And you have a whole ecosystem strategy as you think about your moat, to the extent that any moat can be present in this industry, right? You build a whole ecosystem? Is that, is that? Yeah. Yeah. We built an ecosystem. I think, um, our moat comes from the fact that, uh, by virtue of our architecture, we are doing things no one else can do.

1:03:26And it's not that they can spend more money or they can't pay more for this. If you want fast, you can't have it. Well, I mean, Jake, it doesn't work. And so that's where we're building sort of our strength and that's how we're to deliberate value to customers. I do think about supply chain. We mentioned supply chain constraints for others, but what are your supply chain constraints? Are you in your old TSMC? We are TSMC. We have very close collaboration. They were investors of us. They've been exceptional partners. I'll tell you an unusual story. In 2017, we showed up and we were about 30 guys total.

1:04:21And we showed up in August. Horrible trend in Taiwan. I don't go to Taiwan in August. It's brutal. Not that it's so nice here today. It's only 90 years. You're embarrassed today. Yeah, but also the humidity. You got put on a suit. And we met with the leadership of TSMC. We said, we would like a little pipsqueak company. We believe we can solve a problem that nobody solved in history. And here's how we would modify the way you make chips to make this possible. They thought about it. And they said, we agree. Let's do it. In the meeting. In the meeting. In the meeting. It wasn't going for a month in the meeting.

1:05:11Was that a prepared mind or they were just exceptionally fast on their feet? First, the salesperson had gathered the decision-makers.

1:05:26Second, our proposal was sort of really good at allowing them to use what they were good at. And it didn't require them to change a huge amount, but it did require them to make real changes. And I think they saw this as sufficiently bold that they would learn as they did it. And they also knew that AI was better on big checks. And so the combination of fair of mind, willingness to take some risk, bold thinking from a very large company. Fascinating. It is fascinating. I mean, that's how big companies win. Isn't that right? And how rare is that? It was extraordinary. And what happened next? Like, how long does it take between a decision in a meeting like that?

1:06:22It sounds exceptionally fast to... Two years? Two years. chip making is a long hard process and and most of the time your first chip isn't a winner and there are lots of of startups now sell them with really smart guys like their first chip will not be order the tpu google had some of the best guys in the industry first chip wasn't a winner nor the second nor the third fourth was really good now they're on their eight, then it's a really good chip. The Anna Plena team at AWS, first chip wasn't great, second, third chip was really good. It takes time. And so we built a chip, we delivered it, and this gets to an earlier question you asked.

1:07:10We solved a problem that nobody in the history of computed solved, and we delivered it in 2020, and nobody cared. Nobody cared. Nobody bought it in, nobody cared. which you were like oh my god nobody everybody said we were crazy it would never work it out works and like nobody wants on it is so then we built the next one I mean the first one we probably sold quite a year of set and nobody wanted in because the marker was not ready because I'm product was not good enough um nobody wanted it because ai was a hobby it was right and who cares if your hobby's really fast so um you care about fast when it's in production you care about fast when you use it every day and so we built another one and you know that one we sold three or five hundred yeah hey we built the third one and we sold tens of thousands right yeah amazing yeah isn't that interest to aim it um and um so going back to supply chain uh do you need to think about uh on shoring diversification so it's very hard to diversify away from tsmc chips are so hard and you actually when you design a chip part of the design is for the rules of that factory right so you can't take your design from tsmc and go to somebody else because a huge amount of the work has been to be sure your design is within their rules.

1:08:49And so England, I think only with one or two exceptions in history, each chip generation goes to one fat. So we're going to be with the SMC for our next generation as well. I think we have a supply chain that is built in many parts, but we bring the chips back from TSMC to the U.S., we package in the U.S., and we assemble in the U.S. We do our manufacturing in the U.S., and then we ship from the U.S. I think when you're growing as fast as we are, there are a whole range of garden variety supply chain challenges. A vendor screws up a batch. It gets stuck in customs. The number of ways that things can go wrong, the supply chain is unbelievable but we manage these every day and we're increasing our manufacturing through quite exponentially and so it's really uh that part of the business in chrome really incredible so maybe to zoom out as a last question um what's your best guess about where all of this is going uh you know obviously who knows in the eye in the next like fears but like in the next year or two well we know some things we we know that the model we use it you're using today, Fable or Gini Petit Flash 6, will be the worst model you ever use.

1:10:19And whatever you think is cool about it right now is going to be boring and backwards in 6+. And that is so exciting. And I think I watched the way our young engineers use it, and it's very different depending the way I'm using it. And it's sort of a fun time where you can learn from your young team members that they're using AI very differently. I think the business of dashboarding and the business of AI doesn't, the damages to the SaaS is, I think, unrepairable. You could ask your AI, build me a tool like Salesforce. 30 seconds later, You have a working tool. It's just unbelievable. And all the things that were difficult because they cut across your inside organizational silos.

1:11:18One of the things that's really hard, if you want to know for your top performing people, when was the last time they got a stock option refresh and how much holding power is left? So how much uninvested stock they have left at today's price? It was like five systems. You're in work days. You're in your stock. You're in your car. None of them can. That's what a CEO wants. How much holding power is for my top guys? And I used to have little tools I wrote for this. And I've got a little app that I had. And Spark right. Well, what a story. It's just incredible to hear all of this from you. What a journey and what an exciting future.

1:12:13So thank you very much. I learned a lot. And this was terrific. Thank you, Andrew. Thank you for having me on your show. I really appreciate it. Hi, it's Matt Turk again. Thanks for listening to this episode of the Mad Podcast. If you enjoyed it, we'd be very grateful if you would consider subscribing if you haven't already or leaving a positive review or comment on whichever platform you're watching this or listening to this episode from. This really helps us build a podcast and get great guests. Thanks and see you at the next episode.