Reiner Pope of MatX on accelerating AI with transformer-optimized chips

13:18John:Speaking of metrics, you talked about tokens per second. We used to measure chips in flops, and I guess there's some kind of custom flop thing for AI chips, but is everyone just using tokens per second these days? Is the industry aligning on that as the chip metric?

13:31Reiner Pope:Yeah, so I mean, I guess it's sort of like an application metric versus the chip itself. Now, flops of the chip is the key chip metric. There's a little bit of like, if I go and say I've got like an exaflop chip to you, then sort of the appropriate suspicion is to say, OK, but can I actually use those flops effectively? I see. And so then you need to map the application to that.

13:56John:Yeah, yeah, yeah. So this is kind of telling you the usable flops, yeah, for your purposes. OK. As a consumer of AI, we have known for a long time that lower latency products succeed. Google talked about their internal testing where the differences were down to, was it 50 milliseconds? Something like that. Yeah, yeah. In result times where they noticed more Google engagement, the faster the results were. And you'd think that 50 milliseconds is imperceptible to a human. And it almost is, but turns out it's not. And I think Amazon has, I mean, certainly they've optimized the latency of the Amazon experience quite a lot.

14:35John:I don't know if they've talked about this stuff publicly, but you know that their internal metrics similarly show that the faster the product page loads, the more people buy it. And yet, in AI, Google has carved out a meaningful advantage via Gemini just being really fast for its level of intelligence. And as far as I can tell, ahead of most of the other labs, at a latency at a fixed high level of intelligence. Yeah, yeah. Why have you guys or Grok or kind of better chips not been adopted faster to give this product latency? It's just that like this will happen and you guys will be powering all the AI products.

15:18John:But I note that Google has an interesting lead there.

15:22Reiner Pope:I think there's ultimately like, for at least for existing chips in the market, there's a really uncomfortable trade-off between latency and throughput. The chips that are best at throughput have historically been the chips that are based on HBM as the memory. So that is Google, Amazon, NVIDIA. In order to have very large throughput, you need a lot of inferences in flight simultaneously. So that needs the large memory. But that hasn't been so good at latency. And then there's the Grok and Cerebras that are much better at latency because they've got the SRAM, weights are in SRAM, very low latency.

15:56Reiner Pope:The problem is, and the challenge when you go to a Grokoro Cerebra system is that the throughput you get there, it just is not very good. And so the fundamental dollars per token is just not competitive with Google or NVIDIA or Amazon. It is actually possible to do both in the same chip. It's kind of an obvious thing. You say you take the HPM, you take the SRAM, put them together in the same chip, you put the weights in SRAM and you put all of the inference data in HPM. That is what we are doing, in fact. And I think that actually hits a really nice sweet spot where you can get low latency and also be very cheap.

16:31Reiner Pope:So I think that's a really attractive point to be. It hasn't happened in the market yet just because of product decisions that have been made by the different chips.

16:38John:Got it. But we should expect it. Like, we should expect all the AIs we're using to get significantly faster over the coming three to five years. Or it might be faster, I'd say. Yeah.

16:47Reiner Pope:So, I mean, generally, HBM-based chips tend to be about 10 milliseconds or 20 milliseconds per...

16:52John:I'm sorry, HBM-based chips are things like TPUs. That's right. That's right. Yeah.

16:57Reiner Pope:There's just some simple math of, like, how long does it take you to read through all of HBM? It takes about 20 milliseconds, and so that's the amount of time per token it runs, whereas the amount of time to read through all of SRAM is much faster, and so you can typically get about one millisecond. So, I don't know if it might be faster.

17:11John:Famously, software used to be, like old-fashioned deterministic software, the kind that's now out of favor, used to be very easy and quick to scale. And you would have social networks that have some Southwest moment, and they can scale through 10, 100, 1 ,000 orders of magnitude of adding users because it's just a few rows in a database, and it's a very underutilized CPU. What's interesting about the AI world is there are very real bottlenecks. You spend lots of time talking about power, but it's not just bringing power online. You mentioned HBM is reminding me of, it seems like there's a view that maybe there's going to be some HBM supply chain crunch.

17:58John:And so where do you see, are we in for just a crunched world where some limiter is pacing the rate of AI build-out over the coming few years where the economics work of the products and everything like that, but ultimately we just can't bring the components online fast enough because we have to build up the factories and things like that. And what are those crunched components?

18:23Reiner Pope:Yeah, no, I mean, I think so. And I'll just comment, by the way, this is a great time to be a supplier in this place. Yeah. Or just really -

18:30John:You should have started an HBM company. I know, right?

18:32Reiner Pope:I think it's also just a fun time to be someone who optimizes software. That's always what I like doing. always the challenge is why am I optimizing this if no one cares? But finally there's a place where actually you can it's actually very meaningful in a very tangible sense. Like if I can make this 20 % more efficient then it can save that 20 % of the build out. The supply chain we're going to have crunches on all of the supply chain really. So if you look at the sort of the big components of what any company but like us for example build out, there's dependency see on logic dies from typically TSMC, maybe Samsung, or HBM, which are the big three HBM vendors, Hynix, Samsung, and Micron.

19:16Reiner Pope:And then there's also just the whole rack manufacturing, which includes, I mean, literally just sheet metal and so on that builds the rack, but also cables and connectors because of all the high-speed interconnect. That's what we - Racks don't sound hard. Are they sneaky hard? The big challenge is that you want to bring in a huge amount of power, get a huge amount of heat out, and also have phenomenal interconnect, which has very high signal integrity requirements. And so pack a lot of cables in with cables that don't bend too much. They have to have enough copper in them and so on that you don't lose data rate on the interconnect.

19:52John:Yes. So, yeah, if you push it to a limited time. Okay. Wafers, racks, HBM. What else?

19:59Reiner Pope:Data centers, which I think is power primarily, a little bit of build-out, but primarily power and grid infrastructure there.

20:07John:Okay. How do you then, as a startup that is looking to acquire all these components, elbow your way in amongst the giants of the Googles and the NVIDIAs and all these people who have long-running relationships and have been buying for much longer?

20:25Reiner Pope:Yeah. I mean, ultimately, what all of these suppliers care about, they do somewhat care about a diversity of their own customers. It's not a great position to be in. They don't want monopsony. That's right. Yeah. But then, you know, what is their hesitation or the calculus for one of these large suppliers is, if I reserve some of my capacity for you, a startup, are you going to be around in a year? Is anyone going to even buy your product? Our approach has been to just actually find buyers for the product, and then the buyers answer that question, ultimately.

21:03John:Got it. And so if you show up with a bunch of fairly ironclad contracts to a supplier, then that has happened.

21:09Reiner Pope:That's the nature of it, yeah.

21:11John:I presume also the round you just raised really helps there, where showing that you are incredibly well capitalized and not going anywhere also helps from a supplier point of view, a supplier validation point of view.

21:24Reiner Pope:Yeah, absolutely. Yeah. I mean, it helps just to say that we are around. We, in some cases, are actually, it depends on which part of the supply chain, but some parts of the supply chain, some are fungible. Logic Dyes are typically pretty fungible. But other parts of manufacturing are, you actually need something specifically set up for you. And so we're also able to cover the capital costs for that. Yeah, yeah. That makes sense.

21:47John:And coming back to the MATEX architecture. Okay, you want to build the best your problems. What is that? Yeah.

21:56Reiner Pope:Sounds great. Yeah. So, I mean, there's a few aspects to that. I think the first one is just pick your memory system right. And so I said, like, we've seen this HBM family. We've got the SRAM family. Put the most together is actually, I mean, most obvious idea, but, like, you can actually do it. There are a lot of details to make that work. Well, we've done that work. One of the things that shows up there is you've spent all of this area on your chip on SRAM. How do you fit in the matrix multipliers, which are the other big thing you really need to do? And so somehow create a much more efficient matrix multiply engine.

22:31Reiner Pope:There is a gold standard for that that is called the systolic array. Make a really large systolic array. You can't beat that in area or power efficiency.

22:37John:Like provably so? Practically.

22:39Reiner Pope:Practically. It is not known a better approach there. The main thing is, like, where are the inefficiencies typically? the inefficiencies show up when you leave the systolic array. So if you make a really big, if you make your systolic array really big, then you just don't leave it as often. So that's the idea. So make a really big systolic array. That is sort of the theme of several of the 2023 era startups, including us. But one of the challenges there is now, there is this part of the neural network as part of the transformer, which is this attention that doesn't map well onto a large systolic array.

23:15Reiner Pope:And so that's the tension. The mixture of expert layer maps really well, but the attention does not. And so what we came up with, which is quite different than some of the other startups in this space, is say, take a really large systolic array, but have a way to split it up into pieces without losing efficiency. So sort of that is the core of the design for us. And then there's sort of the third component. So first was HBM and SRAM. Second is the systolic array. Third component is just an interesting new approach on low-precision arithmetic. Low-precision arithmetic, in general, we've seen number formats get narrower and narrower.

23:53John:They get faster and faster as you make them less precise. Hey, number formats get narrower. What does that mean?

23:59Reiner Pope:Yeah, so float32 was how people used to train neural nets.

24:04John:I see, and that's just too much precision, like it's useful? Too much precision, yeah.

24:07Reiner Pope:It's like saying I've got an image with like a billion color bit depth. It's like too many colors. Like you'd rather have more pixels and fewer colors. And so that trend seems to go all the way, like almost all the way down to like one bit even, where just have very few colors, but a huge number of pixels. And that in net seems to be better, just more efficient way to train models.

24:35John:And so sorry, literally what precision precision are you dealing with in these plans?

24:41Reiner Pope:So we have a range.

24:47Reiner Pope:I mean, we actually have an ML team who we hired specifically to research different forms of numerics and how to make them all work together really well. We have a range of precisions. It's not just one precision. We think probably the main thing will be similar to where NVIDIA is at, which is 4-bit precision. But I think a mix of different precisions is useful for just when you look at the research, sometimes you want some layers in higher precision or lower precision and so on.

25:13John:Yeah, yeah. Okay, so four bits is 16. Yeah, you get 16 choices. That's it. Yeah, that's it. Yeah, it was pretty imprecise. Yeah, yeah. That's really interesting. I didn't know about that dynamic, but it makes sense. Yeah, and half of them are positive, half of them are negative. So like, it's even less precise than that. Yeah, yeah. How do you design a chip? Like, is that a whiteboard? like what software are you working in? I'd just love to know. I understand how you design software and what that process looks like. I've actually no sense for what chip design looks like.

25:42Reiner Pope:So the way that you actually type a chip into a computer is similar to software. So you write Verilog. Verilog is a programming language. It is a very parallel programming language, which makes it different than C or Python or something. But it is a programming language. So the mechanics of how you express the design are the same as software. And we have continuous integration, Git, all of those things.

26:05John:But like a program executes, like your Verilog program. We don't really run it, right? Yeah, exactly. We synthesize it, yeah.

26:13Reiner Pope:Okay. So Synopsys and Cadence provide EDA tools. So EDA, you have to remember, yeah. Electronic design automation. I don't even know what it means, really. I think it's electronic design automation. It takes the Verilog and says, first turns it into a description of what are the logic gates that are involved. ands, ors, nots, and then the wires between them. And then it runs for days doing some really difficult algorithms and then eventually produces, I mean, so gates are the first thing. And then even below that, it literally just produces polygons. It says like P-type semiconductor here, N-type semiconductor here, and polysilicon.

26:55John:Okay, so like you write Verilog, and then that compiles down into gates and ultimately like the Minecraft 3D, just this is where your elements should go. But like, then what is the iteration loop? Like when we write code at Stripe, we build a first version of something and then we try it out and then we refine it and we add more functionality over time. We're going to write some tests at some point. We'll ship that. We'll find product market fit and then we'll refine it in market. like do you just sit down and write the completed chip and it works really well?

27:35Reiner Pope:Yeah, like every year we tape out a chip and if there's a bug we just wait till next year. It's not really how we do it. Yeah, well, so what's the iterative? How do we actually do it? Yeah, it's much more waterfall than software is. So like waterfall is almost a bad word in software development. But it's just a fact of life in chip design. Yeah, so the waterfall goes from architects to logic designers who are writing Verilog and then there's this design verification and then physical design. So there's this really big architecture phase, which happens before even writing any Verilog, which is what do I want the organization of my chip to be?

28:12Reiner Pope:There's, in some sense, I mean, what I really, like I came to hardware after doing almost 10 years in software. I really like the blank slate you get in hardware. You've got like all of the raw materials you have are much more varied in what you have available. So what is the organization of your chip? Do I have 100 cores? Do I have one core? Do I have systolic arrays? Do I have vector units? All of those things. And then we spend a long time coming up with that general principle and then saying, okay, now I've got these applications I want to run. I want to run a transformer of a particular shape.

28:44Reiner Pope:I want to map that onto this architecture that I've got in my head. And so we do a lot of iteration. Well, I've got this architecture in my head. I write it down to communicate to other people that that's just like a markdown file. and then still actually a lot in my head, but maybe with Python simulation and so on, I'll see, do my applications map well to it? And so can I run LLM?

29:06John:This is where I was going to go. Okay, so you have a simulator where you write your chip. You can then simulate its performance and you have some battery of tests that you kind of see how this chip design works. Is it like an industry standard,

29:21Reiner Pope:you know is it the x-plane of uh chip testing yeah so i mean you there's an industry standard thing for the verilog once you've done the design um they're just verilog simulators that you can test against okay um that is but you've already invested a huge amount of work by the time you've got to that point and so you sure hope you haven't made a big mistake at that point yes um so the The thing that everyone does prior to that is we'll write our own performance simulator, which, I mean, it is very specific to your particular architecture, and you can write it quite concisely in just like a normal programming language.

29:58Reiner Pope:And so that is where most of the architecture work is done. And then the simulation on Verilog is more, I know what I'm doing. I just want to make sure I didn't have any bugs when I implemented it.

30:06John:Right. But I presume it's a game of inches where different people are trying different things, and then you do simulate it to see if it runs 1 % better across the battery of tests, or is that not how it works?

30:19Reiner Pope:In this space, not so much. So, I mean, just to sort of characterize what performance of an AI chip is, it is how many really, like, if you're just, like, first thing you care about is flops. How many flops have I got? That's a product of how many multiplies, like, I've got a grid of a certain size, like 1 ,000 by 1 ,000, And so that can do a million multipliers in a clock cycle. And then I have a certain clock frequency, like a gigahertz. And so I multiply them out. That is the speed of it. I don't even need to write that and test it to see how fast it is.

30:50John:It just is.

30:51Reiner Pope:Yeah. So what I plan in advance is it's going to be this fast. What I can then optimize on maybe a little bit is clock speed. There's not a lot I can do there. And then I can optimize a bit on area as well. So there is some room for optimization, but actually a lot of it gets set. Like, actually, just the speed of the chip gets set very much up front.

31:10John:Got it. And then how many chips do you fab? Is it only the ones going into production, or is it just build a few to throw away, or how does it work?

31:20Reiner Pope:Yeah. So the ideal, which companies tend to hit about 50 % of the time, is that your first tape-out, tape-out costs like$30 million. Tape-out is just production. That's right. It's the actual manual, like the first chip costs$30 million, the second chip costs$1 ,000.

31:37John:Yes, yes.

31:38Reiner Pope:So tape out is that first chip.

31:39John:Yeah, okay.

31:40Reiner Pope:The ideal is that your first tape out is actually is your production thing. So you do a tape out, you make maybe 1 ,000 chips and test them, and then you do production volume. In the unlucky 50 % of the time, you need to redo some or all of your tape out. So in good cases and in many cases, you can redo just the metal layers, which costs you only like$100 ,000. As opposed to the... Pay the$30 million again. But in bad cases, if you've made something serious and you can't fix the metal layers, you have to do the whole thing again.

32:14John:Why can't that be solved? Is that definitionally an error in simulation where it turns out these two gates were too close together and it just led to some reliability issues?

32:27Reiner Pope:Yeah. So, yeah, like what you're describing is like physical, like the physical implementation of the chip is wrong. That's one class. The other class is that the logical specification of the chip is wrong.

32:41John:But shouldn't that be... Shouldn't you have caught that before? Yeah. Before you spend$30 million on it. Yeah.

32:46Reiner Pope:So, I mean, yeah, we do a lot of testing. We try not to ship these things. I hear software companies also ship bugs to production as well. Fair. And sometimes things miss. There's a very good retort.

32:59John:Shouldn't you not be shipping bugs?

33:03Reiner Pope:But I mean, there is a real trade-off in you can spend more and more time on design verification. There's always this question of when do you stop? And so you stop when your coverage metrics have hit a certain point, but maybe not 100%.

33:18John:And then if, you know, Apple has to discretize the iPhone release cycle and they've settled on, you know, once per year. And so they'll decide, you know, we've got this better camera, but it's got to wait for the next version. Or, you know, we're going to improve the waterproofing, but, you know, that's got to wait for the iPhone 8 or whatever. And so they have taken a continuous process of like all this coming up with ways to make the iPhone better and discretized it into annual iPhone releases. what will your discrete cadence be?

33:52Reiner Pope:Many chip vendors have this sort of tick-tock model, which is you'll do on one generation, like maybe you're trying to release every year. On even numbered years, you'll do a physical technology upgrade. So new transistor technology, new memory technology, a new interconnect. And then on odd numbered years, you might do a architecture overall. I think that's a pretty good fit because you have different parts of your company that are skilled at different areas. And it allows you to keep sort of both of them occupied without like having instead every two years doing a massive risk release.

34:25John:Yeah, yeah, yeah. Okay, and so you think that's probably likely for you.

34:27Reiner Pope:Yeah, that's right.

34:29John:You mentioned interconnect. So there's an art about there that NVIDIA, a huge part of the defensibility comes not from the chips, which are good, but from the software layer and the ability for engineers to write these really parallel workloads and the fact that they've been refining CUDA for whatever number it was. Yeah, a decade or something. Exactly, yeah, a long time. Just how do you think about parallelization and is that narrative true?

34:56Reiner Pope:Yeah, it's true, for sure. It's true in many areas of the market. I think, and especially where you look at where NVIDIA entered the market, they're doing PC devices, lots of gaming and so on. there are thousands of games, maybe tens of thousands of games released, and they all need to be programmed against CUDA. And so there's such a huge investment in the software that this is really important, the compatibility. There are not thousands of LLMs. There's one LLM per Frontier Lab, and there's maybe five Frontier Labs or something like that. And so just the economics of that is different. The calculation for Frontier Lab roughly goes as I just bought a$10 billion compute cluster.

35:47Reiner Pope:I have hired 50 of the best people who can write optimized GPU or TPU or Tranium software. I pay them less than$10 billion, a lot less. And so let's put them to work optimizing the compute. And so they can, like, good work there can, I mean, depends on what your baseline is, but it can very easily double the performance of the software you write. And so there is a huge amount of custom software written for every generation of chip. When a new chip comes out, the software is, like, substantially rewritten to optimize for that specific chip. And that's just the right trade-off, given the relative costs of these things.

36:28Reiner Pope:What that means for us is that that ecosystem already exists. And that way of operating where you say, I'm just going to staff a 50-person team to write software for this chip works really well if you're trying to sell to Frontier Labs.

36:44John:Okay, so you're saying CUDA is way more important for the games environment where it just does a lot of games than this top-heavy AI market that we're in where if people say you need to then customize your workload for a Matex chip, it's like, well, fine.

37:05Reiner Pope:Fine, yeah, it's custom business. Yeah, yeah, yeah.

37:07John:That makes a lot of sense. Where will you fab the chips?

37:12Reiner Pope:TSMC.

37:13John:Okay. Yeah. Why is TSMC so durable? Yeah.

37:22Reiner Pope:I mean, it's interesting. They don't charge a lot as well. You'd think that if they're a monopoly provider, they should charge a lot of money. They don't. I think that is a big aspect of why they're so durable.

37:32John:It's like this cyclical conservatism crossed with Taiwanese business conservatism means you're at the most conservative part of the matrix. But, I mean, it does, I mean, like an American capitalist might say,

37:49Reiner Pope:well, they're just screwing up. They could have extracted more money from the market. But you could also say that there's actually this long-term sustaining advantage because they will just stay ahead for a really long time. They don't encourage the creation of competitors.

38:02John:Yeah. But isn't the creation of competitors kind of priced in because of geopolitical risk? And so it's not like everyone's fat, dumb, and happy with their TSMC dependence. They're actually thinking a lot about it.

38:14Reiner Pope:Yeah. I mean, so there is real technical advantage there as well. It's not just like the discouragement.

38:19John:But like, standing chips seems really hard. Building airplanes seems really hard. There are so many areas where competitive market forces create multiple options.

38:30Reiner Pope:Yeah.

38:31John:And yet, that has not occurred here.

38:33Reiner Pope:So, I mean, there are multiple options. You can buy from Intel or Samsung. But at leading edge nodes. Yeah, yeah. So, I mean, what do we even care about in leading edge nodes, I guess? The big advantage is on power. The advantage on area is smaller. The leading edge nodes, the density doesn't go up as much as it used to. So when you are really, really sensitive to power, it is a good idea to be on leading edge nodes. So that is AI chips and mobile phone chips. But there's a lot of the market where you don't, like devices in cars and so on.

39:03John:Yeah, car chips, yeah, that's fine. But you're kind of saying, like, if you exclude the two most interesting parts of the market. Yeah, that's true. just for this super high growth area of the market, it's interesting to me. Like, again, there's a lot of other really complex business problems out there that competition has solved. And yeah, chip design is like, why has someone not left TSMC and got and built a new fab?

39:28Reiner Pope:Yeah, I mean, I don't know. The cost of a lab, of a fab is extremely expensive. I mean, I recognize that also the cost of a lab is extremely expensive too.

39:41Reiner Pope:I don't really understand the technical details of why it's so hard. I mean, there is some amount of just a$10 billion fab versus a$100 million tape out and chip development. There's a huge difference there. But beyond that, I'm not sure. What's SharesMC you like to deal with? So they're very big. So as a startup, we tend to work with, not directly with SharesMC, but with an ASIC vendor who, I mean, firstly does a huge amount of the actual backend work for us, the change effects with them, but then also has existing relationships with them. Got it. TSMC cares a lot about diversity of their customer pool.

40:17Reiner Pope:And so...

40:18John:It gets back to that conservatism.

40:19Reiner Pope:Yeah. So they're great to work with from that perspective. They want to encourage startups.

40:25John:That's right. Yeah. That's very cool.

40:27Reiner Pope:Why don't the labs design their own chips? I mean, Google does, but... Google does. OpenAI is starting. It's really a trade-off of how much advantage do you get from vertical integration versus how much advantage do you get by concentration of R &D work. So you take the five labs and if they all buy from one player, then you can put like five times as much R &D into that chip. And does that beat the advantage you get from saying, I know exactly what my model is? Because of the like several years delay from designing a chip to being in production, you can't actually say, I know exactly what my model is because models change like much faster than that.

41:04Reiner Pope:So even the labs are forced into this position where they have to make predictions and they have to hedge against what they might do two years from now. The calculus is sort of like, what is the probability distribution of what a model might look like? And then sort of design a chip that gets like 90 % of that probability distribution or something.

41:22John:Yeah, yeah. Elon is excited about data centers in space. Yeah. The two criticisms I've heard are that cooling is very hard and then just repairing the chips is hard. But I know nothing about chips. You do.

41:39Reiner Pope:Yeah. So, I mean, the repair, I think, is really interesting. When you look at how NVIDIA deploys their X, how we do something pretty similar to what NVIDIA does, I mean, in general, you always need to design for the fact that some of your chips are going to be down. Like, mean time between failure of chips is not that large. And so in a cluster of 100 ,000 chips, there's going to be chips that are down all the time. One way you can do that is you can make a rack where one rack has some spare chips in it. NVIDIA has eight spare chips and a rack of 64. That's pretty good. Like, the common on Torx works really well for you there.

42:17Reiner Pope:That's the sort of, you can actually, like, because you can pick which ones to avoid. you can, with very high probability, tolerate a lot of failures. And then the other family of things is to say, my rack has to work, but I have some spare racks as well. So you can math that out with the tax of reliability here is only like 10%. It's pretty good. But that relies on someone coming and servicing the part within a day or something like that. If you say they're going to service it never, then I think you actually can get where you want to be but maybe with 100 % tax on reliability rather than 10%.

42:55Reiner Pope:So for example, if you think the average lifetime of a chip is in the range of three to five years so that means if I deploy twice as many chips then three to five years from now half of them will still work.

43:06John:Yeah, and also the burn-in is particularly failure-y. How about the cooling?

43:11Reiner Pope:So most of the challenge... I mean, I guess there's actually really a data center design aspect then at the rack level the challenge of cooling is just getting the heat out as quickly as possible out of the rack into the cooling network. How you get it out of the spaceship other people would know that better than I do.

43:31John:Okay, yeah, yeah, yeah. Again, that seems to be the main objection but I don't know.

43:37Reiner Pope:Yeah, I mean, I think it's sort of like if you think the cost of repair is that you need to have deployed twice as many chips then like it's a trade-off of the capital of the chips versus the power saving. Exactly.

43:47John:The repair thing, it feels like, can be solved. Because also, I think part of the best, you know, probably one's claim, is that we will just be so power limited that, you know, you have no option but to go to space. And, you know, people can argue about that. But were that to be the case, then yes, it's like, well, you can get power in space and you cannot on Earth. And so you might as well go there. Whereas, like, the cooling is a more fundamental, does the product actually work at all?

44:17John:Rainer thinks about AI the unglamorous way, compute, systems architecture, and what it takes to run models reliably at scale. And if you're building an AI product, the business model similarly has a ton of unglamorous complexity. You're not just selling AI, you're monetizing consumption across API calls, tokens processed, GPU hours. Stripe Billing is a scalable system for usage-based billing. It lets you launch token-based pricing, subscriptions, credits, hybrid models, whatever you want. So you can create revenue models based on usage without rebuilding your pricing system every six months. If you're building an AI product, Stripe Billing is worth a look.

44:55Reiner Pope:What are your AI predictions for Twin26? I mean, what I'm really excited about is just being able to... I mean, I'm still excited about the coding. This is what we do as a company. It's what many others do as a company as well. The one aspect of this is expanding into more domains. So, for example, where we spend our time, as a company, we write Rust, we write Verilog, we write Python. No Haskell. Yeah, no, there's a story there. I used to love Haskell. Rust is my current favorite. Mutation is good.

45:38Reiner Pope:The models are extremely good at Rust and Python. They've done a lot of RL on them. They have not done as much RL on Verilog. They've done almost none on, okay, write me a markdown file that describes a chip architecture. And then how do you even RL on that? You have to say like, what is a good chip architecture? I have to somehow say whether that's a good result or not. I think one of the things the labs are doing is trying to broaden what they've done RL on, source it from customers and so on in order to sort of fill out the knots, make it less spiky, fill out the gaps between the spikes.

46:17John:I presume the labs would love to work with you on improving the models by doing RL on this specific task. However, it's also... That makes sense for us. Yeah, you're a special sauce. So do you want to come up with some AI approaches but keep them proprietary? Is that...

46:39Reiner Pope:Yeah, so, I mean, we've looked at a few different aspects here. There's the, I mean, what we're able to do by ourselves, our business is not training models. We do it in order to do the research on numerics, but like actual production models we don't do. So it's like the biggest mileage, I think, is on the RL, and it's not something we can really do ourselves. We'd love it if we could have a custom model just for us, but that doesn't seem to be a product. Well, you could, right? The terms we've been offered by labs so far have not been on those terms.

47:08John:Because you have to share the IP back.

47:10Reiner Pope:The way they prefer to do it is that they put it into their mainstream model. Because it's good for them.

47:17John:Yeah, yeah, yeah. Which obviously you don't want to do. Yeah. I mean, how do you think you, what does you using AI to design a model do you think look like? Because this is actually, I think, an interesting sight glass into the, you know, a weak version of recursive self-improvement where, you know, we're using the AIs to develop better AIs. And so I'm curious, yeah, what you think that looks like? Is it your own proprietary recursive models? What else? Is there kind of day-to-day AI usage that's load-bearing?

47:44Reiner Pope:Yeah, I mean, so the stuff that is available today and I think will become even better very quickly is just the stuff that looks most like software. So writing Verilog, running tests, running continuous integration, and so on. And that is a big fraction of the development time in the chip. It's probably 9, 12, 15 months or so. there's some stuff that's downstream of that which is physical design which is you take that mirror log and you generate the gates and the polygons we don't have a clear path for like it's not at least the most obvious thing is not clear for how to compress that like the goal can you tape out a chip in one month one month would be the goal in theory you could compress all of the logic design and design verification down to a short amount of time if the just by continuing on the same path we're doing now.

48:36Reiner Pope:But if you wanted to take the physical design down, that has to leave code. You're now doing like graphical interfaces and saying, well, I want to play stuff and so on. Actually, there has been work on this even prior to LLMs, which is like a specific model trained for that particular problem. And I think the vendors, which is like Synoptys and Cadence, probably should move in that direction. Most of the focus has not been do it faster. It's been do it with higher quality. But that is a big bottleneck on, like, can I have a new chip every month? And then there's just the practical thing of, like, a new chip every month doesn't really make sense because then if I'm deploying, like, if it takes me a year to populate a data center, that means I'm going to have different chips in different corners of the data center.

49:23Reiner Pope:Yes, yes.

49:24John:Sir, when you talk about one month to tape out, So you do all this work to ultimately produce a file.

49:34John:Everything TSMC then does, it's not entirely in software. Like, is there some typesetting that has to happen of moving stuff around? But yeah, what happens when you send your file to TSMC? Then what?

49:45Reiner Pope:So they create a mask. So that is where the ASML tools come in. And a mask is really just a stencil. You shoot the lasers through the mask or the x-rays through the mask, and then that produces the different p-type and n-type semiconductors. So they produce the mask. That is the expensive part. And then they're building up these 15 or so metal layers. So they place it on the silicon, and then there are different layers of metals, which connect all the transistors together. They do that on a wafer. it happens on a stepping basis. So there's sort of a maximum size of chip you can build, which is constrained by this machinery.

50:30Reiner Pope:The wafer stepper is part of the ASML special sauce, right? Yeah, I guess there's probably some important alignment requirement. Yeah, I think I remember that being quite like the,

50:40John:you know, it's a classic manufacturing throughput problem. And I think they've done a lot of work on optimizing it.

50:44Reiner Pope:Yeah, yeah. So they take that. So then you just produce hundreds of copies of your chip. you have to test it because there's defects. You typically, I think the average rates really depends on process and so on, but small single digit number of defects per chip. So you test the chip and see whether it has any defects in it. Many chips are designed to be able to tolerate a few defects. And so you need to configure it to tolerate the defects. And now you have a die that by itself works. And then you need to package it. So you put it in a package together with memories. Typically, that's the HPM.

51:21Reiner Pope:And maybe you escape the wires to connect to other chips.

51:26John:How long does it take to make a mask?

51:29Reiner Pope:So, I mean, what we see is time from, like, tape out to chips back. Again, depends on Node, but it's ballpark four or five months.

51:37John:Oh, so tape out is just, like, sending the file?

51:40Reiner Pope:Yeah, well, I mean, we consider tape out to send the file. And then there's a whole process. So you make the masks for all the layers and then actually just producing the chips. Got it.

51:48John:So producing the masks and producing the chips happens after tape out. That's right. I see. Okay. So is the term tape out from you send a magnetic tape with the instructions or something?

51:57Reiner Pope:It could be.

51:58John:I was in software when so was created. I'm curious what the tape actually means. It feels like, you know, everything about AI predictions. one thing I'm really struck by is how still in 2026 every time you open a chat window it's contextless it's got no memory and now to be fair it's like guys it's been four years not even four years it's been three and a half years just calm down we'll get there but I also interpret a lot of the current enthusiasm for OpenClaw and all that stuff as it's like this super hacky backdoor into state management where your little claw will write a markdown file of what it's doing and then look at that markdown file the next time and things like that.

52:48John:But it just feels like state management and memory is going to be a huge deal and that will really change the character of AI products.

52:57Reiner Pope:Yeah, it's really interesting. So, I mean, long context is the reason, is one of the biggest bottlenecks on speed of the model. Yes. Every single token you generate, it reads through all of the previous tokens, or maybe it reads through a subset of them, but reads through a lot of the previous tokens you've written. And so memory bandwidth for that is really constraining. You can think of model-level ways to solve that problem, which is to say, maybe I can compress it into fewer bytes or something like that. But it's interesting that the most effective way to solve it has been, I mean, it's really a combination of everything, but the most effective way to solve it has been once you hit your 300 ,000 token limit, have the model go back through it and compact.

53:42Reiner Pope:Yes, yes.

53:44John:I mean, it's kind of what OpenClaw is doing. It's like compacting everything you've done. But it's funny that it's so manual.

53:52Reiner Pope:Yeah, I mean, I think... Manual is the wrong word. I mean, it's so primitive. It's maybe because it's so controllable. You can, like, if you want to iterate on how you compact, you give a different prompt and you say, compact this way, compact that way. You can iterate that on that in seconds or minutes. Whereas if you're trying to do some iteration on the model level where you say, now I've got a different model architecture,

54:13John:it's going to take months to try and launch something. Yes, yes. Any other AI predictions?

54:18Reiner Pope:I'm generally just interested in what makes models cheaper and faster. So that's just at the model architecture level. Really tied into this context thing. I think the context size will stay ballpark the same way it is, maybe a few times larger. But the parameter count will go up. Like parameter counts will grow much faster than context length, actually, just because of the underlying physics of what's available. So has that been the story?

54:45John:Like, would that be a reacceleration of parameter count? Because it feels like we've leveled off slightly in the last year or two, and instead we've been focusing on more and better RL. Yeah, okay.

54:56Reiner Pope:Parameter count or thinking tokens, I guess. those are available but the context length I think is sort of struggling to grow yeah

55:06John:yeah okay but you think we we say context length is struggling to grow but you're saying we keep context the same length keep context the same length but we're better at working with large context is that what you're saying because

55:16Reiner Pope:yeah I mean just have application level interventions to manage large context like compacting yeah yeah

55:22John:because I think everyone has the experience you know currently of like the chat conversation and the further down in the chat you get It just gets looser. Yeah, it's sloppy. It's just like really sloppy by the end. And it's like making mistakes. So you're saying we start to do better with large contacts. Okay, I buy that. When will I be typing into a chat window and it is a MATAX chip underneath it, powering it?

55:45Reiner Pope:Tape out in under a year. And then that means chips available in sort of... Okay, that's okay.

55:52John:Okay, so in 2027, I will be seeing very high-performing chats as a result of...

55:57Reiner Pope:In the 1 % experiment of the users or something like that. Yeah, exactly.

56:00John:I need to find a way to fine-angle myself into the A-B test. MadX is 100 people? That's right. How have you gone about building the team, the culture?

56:14Reiner Pope:Yeah, so what we have on the team is hardware, mostly hardware, but a big software team and also a big ML team. I think the ML team is quite unusual in what we ask them to do. When you look at a typical ML team in an AI chip company, it will be what I might say ML engineering or ML performance. They're writing kernels that actually just use your hardware as well on a given model. There's sort of a missed opportunity there if you're saying, all we do is we take other people's models and we write kernels for them. You're optimizing this, but you can optimize this at the same time. And so we want to optimize the whole thing at the same time.

56:55Reiner Pope:So like real co-design. So our ML team is actual, real ML research. What they do every day is they train small LLMs from scratch, focusing on numerics and attention. And this has really, really helped us make an interesting product. It's showed up most strong in our numerics. we often what you see when people design numerics is they say well back in when when flow 32 was popular it would be i'm going to follow the ieee standard now it is like follow the open compute standard um and there's lots of little details where you say things like um maybe what's the rounding mode i'm going to use like round to nearest even or something like that which is like uh the the best known standard for how to round we want to cut corners anywhere we can um and so Maybe don't do the best rounding.

57:51Reiner Pope:Maybe don't get all the corner cases correctly. That's a very scary proposition if you're just making those choices blind. But if you have the benefit of a research team who can sort of back you up as you do that, it's really powerful. And it's really interesting that we can make some sloppy choices.

58:11John:I feel like often technical advances come through better iteration loops. A favorite example of this I found recently was that the Wright brothers actually had a failed season before first flight. So I guess first flight was 1904 and they were down in Kitty Hawk in 1903 and not making that much progress. And they went back to Ohio and they had a wind tunnel. And they were like testing their design in the wind tunnel. You can imagine not a lot of wind tunnels in 1904. And they did a lot of wind tunnel testing and their successful flight was after that. Is this something you're focused on where, to get better chips, you allow for a better testing and iteration loop.

58:53John:And what does that look like?

58:55Reiner Pope:Yeah. I think this mostly happens in the architecture and product definition stage. Maybe even more generally, I think AI chips seem to live or die by product definition and architecture. What is the most extreme form of fast iteration? It's doing it in your head. And so can you map a model to hardware in your head? Can you estimate the performance of what it is in your head, you're not going to be 100 % perfect, but maybe you can prove some kind of lower bound on performance. And so, I mean, the simplest possible thing is my model has a trillion parameters. My device can do a billion multipliers per second, so it takes 1 ,000 seconds to run or something like that.

59:35Reiner Pope:Just do that simple division. But then there are much more complicated things. We tend to look at resource balances.

59:42John:And so, how many memory fetches do I need to do per multiply or something like that?

59:45Reiner Pope:So we do, I mean, at least the way I like to do design and architecture and optimization is to be able to sort of estimate the performance to within about 30%, 40 % before even typing anything in at all. And so we've tried to do that a lot. A lot of our architecture comes from there. Then sort of the next stage of iteration is, oh, that's kind of on the performance side. And this also happens on the circuit design side as well. Can you take a circuit and say, what is the gate count on that? So like a 16-bit multiplier has approximately 16 squared many gates, and you can do that for more complicated things by sorting networks and so on.

1:00:28Reiner Pope:So we already have a pretty good idea of the costs and speeds of things at that point after doing these calculations. Then what we tend to do as sort of the next step of iteration is on the ML side, we run model experiments. You get iteration speed just by having small models, mostly. And then on the hardware side, we use simulators, performance simulators, to do the next level of detail to make sure we're seeing all the things we want to see. Yeah, yeah.

1:00:56John:This idea that the best iteration is in your head is kind of reminding me of Jeff Dean's numbers. Yeah, yeah. Do you have your equivalent of that, numbers every MATX engineer should know?

1:01:06Reiner Pope:Yeah, we have go slash gates in our company, which says, what is the cost of an XOR gate, an AND gate, full adder, SRAM bit cell, and so on.

1:01:15John:And you want people to be working with that stuff in their head and have an intuitive sense for it because it leads to better iteration. What is the pitch to someone joining MATX?

1:01:25Reiner Pope:I mean, I think if you are someone who likes optimizing, just optimize something. Software, hardware, factorial, whatever, if you're trying to fit something into the smallest budget possible, I think it's a pretty exciting place to be. I think hardware companies in general are really exciting because you have such a broad range of skills of people on the team. You have software people, you have hardware people, you've got physical design, you've got people who are just looking at the insertion force of a card into a rack. and so there's like so much discussion and learning you can do. I think Maddox in particular, we really care about this and I think we extend it all the way up into the application and the machine learning as well.

1:02:15Reiner Pope:And so really, I mean, really, really, really interesting tackling the problems and I think just generally, like there's lots of interesting people to talk to. Yes.

1:02:27John:And presumably, in terms of impact, if you can design a meaningfully higher throughput chip, a 20 % higher throughput chip means 20 % more AI is happening. You know, if the bottleneck is elsewhere, like power or something like that, or cost, you actually just are meaningfully increasing the amount of intelligence in the world, which is presumably exciting to people.

1:02:48Reiner Pope:Yeah, yeah. I mean, I think this shows up both as just kind of applying more applications as well as just how smart is the model. Yes, yes. Quiet Rust. So a previous project I worked on at Google, we did a lot of Haskell. I did Haskell when I was at school. I loved it, like very principled, very interesting. I like Haskell, but I also like making stuff fast. And then the question is, what is the first thing you want to do? You want to be able to modify your memory. Haskell, you jump through hoops to do that. Maybe I just want a language that is like functional programming that lets me modify my memory.

1:03:26Reiner Pope:So I think Rust has a lot of the nice things, which are like type classes or traits and a rich type system. One of the things that we have done, like interesting ways we use it at Maddox, are the range of sort of data types that you express on software. Like, what are the integer types? int 32, int 64, int 8, maybe that's all you care about. But it turns out in hardware, you care about every single bit, and so you want to use like 17, 18, 19-bit integers. That is quite natural to express, and we've built up sort of a whole ecosystem of rich hardware data types in Rust as well.

1:04:10John:Has Rust beaten Go for the position of sort of performant type programming language with modern features, or do they actually address different markets?

1:04:20Reiner Pope:Yeah, I mean, so there's what the Rust marketing will say, which is safe without a garbage collection, which I think is a real, I mean, is the objective thing that you can say is different, but sort of barriers the lead, which is it's also just like it's got nice type system features

1:04:39John:that Go doesn't have.

1:04:41Reiner Pope:And then, like, why is garbage collection, why does it matter at all? Like, it's not, I mean, people often focus on the time it takes to run a garbage collector. But the other thing is that every time you allocate an object, you've got the object, and then you've got the garbage collector header at the beginning. And so it uses a lot more memory as well. And so if you want to design some, I don't know, data structure that, like, uses the right amount of memory rather than, like, a bit more, then...

1:05:04John:Oh, sorry, sorry. I hadn't realized that in Rust, you're allocating your memory manually versus in Go, you have a garbage collector? Yeah, yeah, that's right. I hadn't realized that. Okay. And you prefer that for what you're doing.

1:05:13Reiner Pope:I just really like dealing with the details. Like you give me a puzzle and I'll be like, let me solve every single piece of it. So that tickles that part of my mind with Rust.

1:05:21John:It seems like you're a fan of optimization generally. Is that a fair characterization?

1:05:25Reiner Pope:Yeah.

1:05:26John:Where else have you, so chip optimization is one domain. Where else?

1:05:31Reiner Pope:Yeah. So, I mean, I started, I mean, one of the really exciting things I found about working at Google is The whole Google code base is available, and you can look at how does a memory allocator work? How does a mutex work? How does a hash map work? Any of those things. And you can go and look inside the implementations. And Google has excellent implementations of those, some of the best you could write. So one of the things I did on my nights and weekends when I was at Google was just go find those implementations, write a benchmark. How many nanoseconds does it take to allocate eight bytes of memory?

1:06:10Reiner Pope:And then can I make that faster? Can I, maybe I inline this function. Maybe I look at the assembly and say, looks like there's a few memory moves here or there's some registers that are being used that I don't need in the fast path, I only need in the slow path. Can I do something there? So I don't know, that was always my, like, just fun and learning activity. Being outside of Google, I feel, I mean, I probably could have done this inside of Google as well, but outside of Google, I felt the luxury to be able to talk about these results as well. One of the things I've looked at recently is just hash tables are used so much.

1:06:51One prompt for me was like, what would, if I wanted to design custom CPU instructions for accelerating hash tables,

1:06:58Reiner Pope:like hash tables are one of the most common things. I'm looking at them up and writing them all the time. What would the optimal CPU be for that? And so then following down that chain is like, what is the best hash table implementation in the first place? And so I spent some time looking at different SIMD implementations. And there's this really cool technique called cuckoo hashing, where you hash into two different locations. And then you use the bucket, which is less full. it's been in the literature for decades and yet like the best hash table implementations don't use it because it's somehow like not practical and so practical hash tables are these days considered to be ones that use SIMD vector instructions to scan like eight buckets at a time and the way cuckoo hashing is normally described is I look up one bucket here and one bucket there.

1:07:59Reiner Pope:And so I'm not using the vector instructions. Vector instructions are much faster than scalar instructions. And so there's kind of a missed opportunity. Again, just like take the two good ideas and stick them together. Do vector instructions on Goku hashing. You have to be careful to get the details right. But if you get it right, you can actually just win.

1:08:16John:And Cesare, is your claim that one could design a custom CPU that has way better hash table performance or even on current chips, you could get way better hash table performance.

1:08:27Reiner Pope:So both. I mean, I'm interested in what you can do in designing custom hardware, but Maddox doesn't make CPUs. They're not going to make CPUs. You could. New line of business. I mean, we just want to focus on shipping one product well

1:08:42John:for the time of year.

1:08:44Reiner Pope:Fair. Good answer. So, I mean, I think it's an interesting exercise, but I don't get to feel the endorphins of seeing the number going down. So I first did this on just Intel CPUs. And you can get better performance than some of the best hash table implementations available using cuckoo hashing on Intel CPUs.

1:09:04John:And what are examples of workloads that are really hash table reads intensive? I mean, I know kind of everything.

1:09:12Reiner Pope:I mean, JavaScript, I guess. But, yeah, I mean, it's sort of a tricky exercise because, like, when you really think about it, you're like, did I really need a hash table there? I probably didn't, but you just reach for it all the time.

1:09:25John:Okay, but you could go to the Google JavaScript team and probably help them eke out better performance in the Chrome JavaScript engine.

1:09:31Reiner Pope:Yeah, I mean, potentially. Like, it's, I mean, I'm not going to spend my time on that. Well, if you're listening to this podcast,

1:09:36John:here's a free idea from Reiner. And then explain the dragon.

1:09:41Reiner Pope:Yeah, this is from a book that, when I was working on the JAX team, So the JAX team is one of the ML infrastructure teams at Google. I was there as the most recent team before I left.

1:09:50John:I'm sorry, what does the JAX team do?

1:09:52Reiner Pope:Oh, yeah. So the JAX team develops, this is sort of Google's new, more modern version of TensorFlow or competitive PyTorch. It's how you write models in Python to run on TPUs. A big part of the JAX team, however, is to say, okay, we have JAX the technical artifact. Can we help enable users to actually use it really well and get high performance? And so ultimately that became, well, who are the users? It's people writing LLMs. How do you get good performance on LLMs? And so really, really strong team, the JAX team at Google, although as with a lot of brain people are now elsewhere as well. And so we developed a lot of the different techniques for how to lay out models efficiently on many chips.

1:10:33Reiner Pope:And so ultimately some people at Google, and I contributed after I left Google, wrote this guide called How to Scale Your Model, how to run an LLM as fast as possible. It is sort of the main reference for how to get high performance on TPUs. There is now also a GPU version of this as well. It's a dragon because it's how to train your dragon.

1:10:52John:I see. Okay. Last question. People might not have thought that there's room for new chip companies. It might have seemed unusual or very hard. And you guys, it seems like a very good approach with that. But where do you think are other opportunities for companies to be started here in 2026? Where do you think people should be looking for entrepreneurial opportunities or just technical challenges that haven't been properly addressed?

1:11:21Reiner Pope:More labs, I think, is still interesting. Can we do more on model architecture is always interesting.

1:11:27John:You think we have not fully explored model architecture space?

1:11:30Reiner Pope:Yeah, I mean, the front-tail labs have done a pretty good job of exploring it. But I think, I mean, as the hardware changes, the shape of the model should change for sure.

1:11:40John:Yeah, okay. And presumably you're not thinking like yet another frontier lab pursuing the same architecture. You think there's probably off-the-wall looking architectures that will actually make a lot of sense.

1:11:50Reiner Pope:Yeah, I think there's a little bit off the wall.

1:11:52John:Okay. For sure. Do you have a specific architecture in mind?

1:11:55Reiner Pope:My mentality is always sticking within the transformer family, but what are the constraints that are currently available, like currently imposed that you could lift. So for example, one of the things is there's this idea when you're doing transformer inference, you do pre-fill, that is sort of processing what the user said to you. And then there's decode, which is generating the response to that. And those are totally different in pretty much every aspect of how they actually run. One runs a step at a time, the other one runs really in parallel. So there is this somewhat artificial constraint today that those are the same model that's doing both.

1:12:31Reiner Pope:maybe lift that constraint. Another example would be, there's this idea that the model that you, I mean, this is more fundamental constraint, that you have to train the same model as you serve. But again, training is very different from serving. At training, it's very compute intensive. At serving, it's more memory bandwidth intensive. And so maybe, is there a way you can make a model that when you use it at inference time, it increases the amount of compute it does to use some of the available resources. Yeah, makes sense.

1:13:00John:Well, Warnir, thank you. Pleasure.