How Ricursive Intelligence’s Founders are Using AI to Shape The Future of Chip Design

0:00Anna Goldie:Right now, we can't have much of co-design between chips and models because of this asymmetric design cycle for chips. Because it takes so long, it takes so much time to design chips, the cycle is there's a mismatch between how fast we can create the next generation AI methods and how fast we can build the next generation chips. But if we can make our chips much faster, then we can enable this co-design and co-evolution of workloads, applications, and chips all together.

0:49Azalia Mirhoseini:This week on Training Data, we explore the biggest bottleneck holding back AI, compute, and the design of chips themselves. Our guests are the founders of Recursive Intelligence, Anna Goldie and Azalia Meherseini, the team behind Google's pioneering alpha chip project that helped design four generations of TPUs. They're now applying AI to the entire chip design process, transforming the industry from fabless to designless. Anne and Azalia paint a picture of what a Cambrian explosion of custom silicon means. We'll explore the holy grail topic of recursive self-improvement, how AI-designed chips unlock radically creative new chip designs, and why democratizing chip design could accelerate the path to superintelligence.

1:31Azalia Mirhoseini:Enjoy the show. Thank you so much for joining us. We're so excited to celebrate the announcement of recursive intelligence, and congratulations on embarking on this new adventure. Thank you.

1:44Anna Goldie:Thank you so much.

1:45Azalia Mirhoseini:To kick off, you've said that chip design is the compute bottleneck to advancing AI. Can you share and paint a picture of what's happening now? What are all the bottlenecks to chip design today? I mean, I think when we were saying that, we were kind of alluding to this motivation that we had originally with our moonshot, which was this observation that, you know, neural networks, these concepts that have been around for decades. But the AI that came out of it wasn't that effective until we had like more powerful computer systems and chips. And we thought that now that we had very powerful AI systems, we could use those to kind of tackle the bottlenecks in chip design and, you know, create more effective compute.

2:25Azalia Mirhoseini:So if you see these like scaling laws, basically the more compute you apply to training a model or inferring with a model, like the more intelligence you get out. And we're seeing that there is this mismatch. Like we're using, say, GPUs are like originally designed for graphics processing, but we're somehow repurposing them for crypto and then for training neural networks models. I guess they're good at like large matrix multiplies. But if we could better co-optimize the models and the hardware, we could get more effective compute, and then we could push ourselves out on those effective scaling laws.

3:00Azalia Mirhoseini:You were the co-creators of AlphaChip at Google, which I think is used in four successive generations now of TPUs. Maybe take us back to that project. How did it get started? What were the key results you drove?

3:12Anna Goldie:Yeah, so we started the project in 2018. and we had some earlier version of placement that placement work that we were interested in solving and that was not about chip placement it was about compiler and basically mapping neural networks to chips so we did that project we got great results we published and we were like what are the like highest impact projects that now we can kind of push this further and have real world impact and that's where the chip placement uh came in in in the picture and we started the project working very closely with the tpu team at google because back then we didn't know much at all or at all about how chip design is uh how the process is so we work very closely with them from very early on and at the time it's interesting because at the time ai was like was still like very talked about, but nothing like the scale was nowhere near what it is today.

4:17Anna Goldie:So we had to really work through things and showing them data and like constantly like iterating over our approach and hearing them what they needed from us, listening to them, iterate through the process. And eventually we went from a research concept on chip placement all the way to something that was actually used in product and we could tape out and all that.

4:45Azalia Mirhoseini:I remember one of our very first meals together when it hit me that you were so customer obsessed and had so much empathy for the customer. And I asked you how and why. And you said, well, I've been working with the TPU team as our customer internally for so many years. So I think that really kind of gave you that other perspective. What were the TPU team's early reactions to what you had created? I mean, they were like super skeptical. For example, like Azali and I, we were researchers, like we had read a bunch of research papers and we're like, okay, there's this half perimeter wire lanes, that's what academics report results on.

5:23Azalia Mirhoseini:So like, we made this RL agent that could optimize half perimeter wire lanes for a placement. And then we were kind of excited about sharing that with them and we showed these results and they were like actually kind of like angry at us like why are you showing us these results like we don't care about half perimeter wire length like we want like routed wire length congestion like horizontal and vertical congestion timing violations power consumption area like and so you know we we just kept listening to them and we were like okay so what is the thing that matters i think another part is just to make your customer feel like they're part of it too.

5:59Azalia Mirhoseini:Yes. So for example, we worked with them to create the cost functions that they cared about, approximations of those. So like Mustafa, who was on the TPU team, developed these very fast congestion cost function. And then we could optimize against the end density and then wire lengths as well. And then we show results on those. And then we run the commercial tool and we show that this actually correlates to good results on the metrics that they do care about. Yeah. So I want to talk about floor planning. And I come from an EDA family. You know, floor planning is the crown jewel of EDA. Can you say, just for our listeners that don't come from the chip design industry, what exactly that is and how your technical methods help solve it?

6:40Anna Goldie:so floor planning is a process where we place the components of a chip onto the silicon and the complexity of this process comes from the large size of the input problems like for a block of a chip not the entire chip like a single block in a chip the graph that we are optimizing could have millions of nodes, and all of these nodes need to be placed and routed, and we need to make sure that all these constraints that earlier we were talking about, like PPA, power area performance, are optimized, and all the physical constraints are met. So the placement problem has to do with how we can do this such that all those constraints given by these very very small technology node sizes given by TSMC and others are met.

7:35Anna Goldie:So we are dealing with a combinatorial optimization problem that's large scale, but also all the metrics are hard to evaluate and measure. Got it.

7:46Azalia Mirhoseini:And this traditionally was done together with EDA software, like a cadence or a synopsis, right? How does your method differ from the classical methods?

7:53Anna Goldie:So what we developed here was a learning-based approach to the problem where we would train a reinforcement learning agent that would try out different ways to approach the placement problem. And it would learn through interactions with this environment that we built to learn from the positive placement and also the negative examples and iteratively improve itself. So one of the main differences between our approach and all prior approaches were this ability to learn from experience, which would enable the model to self-improve. just like a human expert that becomes better as they solve more instances of the problem and harder problems, our agent was able to kind of exhibit similar behavior.

8:48Anna Goldie:And that was a very major kind of delta and change between what we had and prior approaches. Yeah.

8:57Azalia Mirhoseini:And I remember with, I mean, if you take AlphaGo as an analogy, move 37 was kind of shocking because it was just so different from how a human player would move. Are you seeing something similar happen with floor planning and chip design? Yeah, we saw these very strange, like curved placements. So there are donut shapes as well. I think humans would tend to make the macros very, so macros are memory components that are larger and they make them very aligned. And then they would put logic maybe in the middle and then the wires would connect all of these components. But if you make the shapes curved, you can reduce the wire lanes, which can reduce power consumption and timing violations.

9:33Azalia Mirhoseini:But it's just the complexity of making this curve placement was like beyond what humans would have wanted to take on or it felt risky to them. So yeah, these aliens. What was the moment you realized that this had the potential to transform the entire end-to-end process, not just the floor planning stage, and when you decided that it had the potential to really be a company?

9:55Anna Goldie:There were a few milestones that first was showing that it works, that it gets the superhuman results on some blocks. And then later on, the chip was actually taped out and it came back and it was actually working, which is a big kind of milestone because it can be really, really costly if for some reason our AI missed something. So those were big moments, the real impact. And also on the algorithmic side, when we saw this kind of self-improvement properties and the fact that our models were becoming better as they solved more problems. That is very... Because when we get to that point, then there is no stopping for the AI.

10:45Anna Goldie:They can see more data points and solve more instances of the chip optimization problem than any single human can ever do. So that's the moment that you're like, yes. So there's a lot more potential here. But AlphaChip was targeting a module, a part of physical design. But chip design, as we know, there are many more stages to it and many more components. and right now we think that everything from the state of AI and the capability of AI, both on LLM side and other like graph optimizations and other approaches, and the way that we can really scale up these algorithms and enable synthetic data, very large-scale distributed computing, everything is ready to tackle the end-to-end chip design optimization problem.

11:42Anna Goldie:So that's why we're excited about doing the company right now.

11:45Azalia Mirhoseini:Super cool. How do you think about getting the right training data for this market? I think it's, you know, it is kind of a binding constraint to AI in many ways. And is this a market where you can have, you know, enough synthetic data or self-play to be able to bootstrap? We're excited about synthetic data. So obviously there's some open source data, but it's not that interesting or meaningful. And, you know, customers are willing to share data with us, but we don't want to train our models on that because we want to keep their data private and siloed from each other. But synthetic data is where we think there's the most promise.

12:18Azalia Mirhoseini:Like Azali and I, we've been working on synthetic data approaches for LLMs in various code domain tasks across like Claude and Gemini. And we see a big opportunity to like get a scale of data that would go far beyond what any customer could ever share with us, like orders and orders of magnitude more. With the TPU program, I guess, at what moment did the chip experts stop doubting you? And as each successive generation came out, what was the progressive impact of what RL was able to drive in those designs? I mean, I think that our approach with the TPU team is like every week we would show them data over and over again.

12:55Azalia Mirhoseini:Hmm. Every week. Yeah. Every week for like a couple of years. Yeah. Because, I mean, the stakes are very high here. like if there's something wrong with this layout, like the setup on the TPU team is like this. They're a human, the TPU is quite large and complex. So they divide it up into dozens of blocks. And then each block is owned by a human or a team of humans. And that's their responsibility to get this block right. If anything goes wrong, like it's their fault, right? And so they would generate their own layouts in collaboration with commercial tools. And then we would show them our layout, some AI generated layout, it would look weird, it would be curved.

13:32Azalia Mirhoseini:And they would, you know, they would have to say, okay, like I'm picking this AI generated layout over my own layout. And I'm taking responsibility for that. I think, yeah, it requires a lot of trust. We have to be better in every single metric for them to choose that. And we saw across each successive generation of TPU that we were being adopted in more and more of the chip block and more of the area. and also that we were getting increasingly superhuman performance. We published an AlphaChip blog post, I think, September last year. We showed this curve. Can I say a word on that? Yeah, on the blog post and the superhuman performance.

14:08Azalia Mirhoseini:Yeah, I think just that across every single generation that we were used, which I think there were three generations shown in the blog post, but actually were used in another generation after that was published. Every single one we were seeing more of the area and more like increasingly superhuman performance.

14:27Anna Goldie:Yeah, so basically the delta between alpha chip layouts and the baseline layouts are also growing, which is like it's a property of AI and how it scales with data.

14:38Azalia Mirhoseini:It makes sense, right? Alpha chip was trained on more and more TVU blocks, so it gets better. Why is the company called Recursive? Because we're recursive. We're AI for chip design and chip design for AI. so recursive self-improvement in terms of like the name the spelling of the name so the name itself is recursive so the initials ri recursive intelligence are the first two letters of the company's name what does recursive self-improvement like like why why does that matter for you know the broader ai race like just say a word on i think this is a no problem just say a word on it

15:15Anna Goldie:Yeah, so chips are the fuel for AI, and scaling laws are driving much of the progress in AI, whether it's on pre-training, post-training, test time, and all that. And so the faster we can make chips that are more custom or better designed for the AIs that we run, the faster we enable this more efficient kind of compute. And that bends the curve for our scaling laws. So that means we get to the next generation of AI faster and we can create design better AIs faster. And so this is a type of recursive self-improvement that we want to see because our AIs then can help our chips become better and we can design them faster and so on.

16:04Anna Goldie:So that is the recursive self-improvement loop that we are going after.

16:09Azalia Mirhoseini:One of the visions that you have is to transform the industry from just fabless to designless. What does that mean? So fabless was basically this concept of, it used to be that people thought that no serious chip maker could exist without their own fab. This was like obviously before like this multi-trillion dollar companies like NVIDIA. You know, TSMC basically created this whole new world where we could have these incredibly valuable fabless companies. So we think that there's an opportunity to create incredibly valuable companies that don't need to have their own in-house design teams. So many companies, they serve these models at massive scale.

16:48Azalia Mirhoseini:They're spending like, I think$100 billion plus on AI inference alone this year, and it's growing rapidly. So companies would benefit from maybe custom chips to serve their models or train them, but that requires enormous teams of hundreds or thousands of human experts in-house. And we don't think that's necessary. So we want to move towards a designless paradigm. It's fascinating. We're seeing it come true with, obviously, Google and TPUs, Amazon with Terranium, also OpenAI and Broadcom, and maybe even Tesla. Do you think the future is then, you know, even today, there's almost like model application co-design.

17:31Azalia Mirhoseini:Do you think there's going to be chip application co-design and even individual companies will have multiple chip architectures as a result?

17:38Anna Goldie:Yes. We think that we are going to see more and more co-design across the deep learning or AI stack from modeled and data to software and all the way to the chips. and we are going to enable that. And co-design is really the secret or the path to more efficiency and more performance going forward. But right now we can't have much of co-design between chips and models because of this asymmetric kind of design cycle for chips because it takes so long, it takes so much time to design chips. The cycle is there's a mismatch between how fast we can create the next generation AI methods and how fast we can build the next generation chips.

18:31Anna Goldie:But if we can make our chips much faster, then we can enable this co-design and co-evolution of workloads, applications, and chips altogether.

18:41Azalia Mirhoseini:One of the things that I loved that we talked about early on was that the value you bring isn't just on reducing the cost it takes to actually design a chip or the speed at which you can accelerate the chip design process, though that is transformative in itself. It's actually to unlock completely new potential applications and maybe custom silicon as well alongside it. Can you share a little bit about that? What is the Cambrian explosion that you might expect? I mean, we think computing is going to be increasingly ubiquitous in every aspect of our lives. So like, obviously, there are things like AR, VR, there's like maybe chips in space, even like hearing aids.

19:21Azalia Mirhoseini:Like there are experiences that aren't possible unless you can serve them at like sufficiently low inference, like latency or at like low enough power. And we think that like custom silicon can enable these applications.

19:35Anna Goldie:Yeah, we think that AI is going to be everywhere in every kind of experience, every aspect of industry and life going forward. And there are chips that would enable these AIs to run. And given the scale, we would want them to run as efficient, very efficiently and low power, high speed, all of that. And custom silicon is really going to enable that. And we want to enable custom silicon for basically any workload that is being run at sufficient scale.

20:12Azalia Mirhoseini:Since we have the ear of whoever's listening on the podcast, who might some of your ideal customers be?