EP 192: Talking Apple Silicon with Apple's Tom Boger, Kaiann Drance, and Sri Santhanam

0:05Ben Bajarin: Hello, welcome to another episode of The Circuit. I am Ben Bajarin. My co-host, Jay Goldberg, is in the wind taking a well-needed vacation in all over Italy. But I am joined by some very special guests here at Apple Park on the back of a whole slew of very interesting announcements. So I'm going to let my guests introduce themselves, and we'll go right into it. So Tom, let's start with you. Tom Boger, I lead product marketing for the Mac and the iPad. I'm Kai Andrantz, product marketing for iPhone and watch. I'm Sri Sainthanam, I head up the Silicon Engineering Group. So this will be fun. I've spent time with most of you individually, but not necessarily as a collective whole.

0:51Ben Bajarin: And there's all sorts of sort of goodness, right, that came out of there. There's technical goodness, silicon architecturally. There's a whole slew of very interesting product announcements. So I kind of want to get into the interweaving of all of those, right? And I think a good way to kind of frame this, right, both for our audience who enjoys some technical, you know, nuggets, but also, right, the practical is to weave in anything that's been done architecturally, right, to silicon choices to the end goal, right? Because technology is great. We can do all of these things. We can move to new process technology, but it needs to have an end user benefit.

1:23Ben Bajarin: And so I want to weave those into the products as well, because I think, you know, as we've sort of largely explained on this podcast, right, Apple's vertical integration is one of those differentiators. And you see hardware, software, and silicon, like, coming together for these product goals with product vision. And so I think having that kind of overall perspective is great to talk technology but land, right, sort of the use cases. So let's talk, right, there's the whole jump within S11. to A20 Pro and M6 to 2 nanometer. There's been lots of speculation. Everybody loves to figure out when Apple's going to go to next process technology.

2:04Ben Bajarin: There's obviously good reasons to do this, but I'd love to just know, you know, the whole product line is on kind of 2 nanometer. What does that mean? You know, why is it significant? What are some of the things that that enables in terms of performance per watt and any of those things around, you know, product benefits to that new process technology? Yeah, sure. I'm glad you brought up the point about vertical integration because it's, sometimes when we build a new SOC, people talk about the process technology node, but it's more than just that. We optimize, obviously, with process in mind, but architecture, design, implementation, packaging, the entire system, the vertical stack, really is what makes our SOC shine.

2:47And there are many years where we actually don't get a process technology bump, and we still innovate across these other dimensions. But you're right, this year we had some tailwind going into 2 nanometer. And with that higher transistor density, we were able to do things in like A20 Pro. We doubled the neural engine. We added extra GPU core. We added extra memory channels for more memory bandwidth into the chip. And also, you know, we were able to increase the performance of many of the compute blocks on the chip with the two nanometer transistors we had a 20 % faster CPU relative to the predecessor we 40 % faster GPU and and we want to do this in an efficient way we want to deliver performance efficiently and not all of that comes from the process technology a lot of design effort went into reducing the power of all of the compute blocks in where that's increasing IPC or or lowering power, it all came together.

3:51It's because we want to deliver performance efficiency and features to drive the product, so it was a good tailwind. Yeah, and two things come to mind based on what you said. Having the two nanometer across the different chips from S11 all the way to the M series is the scalable architecture that we have. So in fact, you know, I were talking the other day, the S11 chip actually inherits the CPU, GPU, neural engine architecture, and then as you go up, you know, we're still using that two nanometer process node and being able to have a lot of things that we end up learning and sharing across the chips, such as the packaging from the M series chips coming down to the A20 Pro this year.

4:30And when we design it in that vertically integrated way, we have a chance to look at not just the rote performance that you're going to get out of the great chip design, but also how do we get it to be its best within the system, right? So within both the iPhone 18 Pro, Pro Max, and the iPhone Duo this year, we had custom thermal solutions designed in. And using that custom package, we were able then to put the chip right on the vapor chamber to allow us to get the best sustained performance throughout. So running the strong AI workloads and gaming, not just having a peak performance, but in big improvements, the sustained improvements as well.

5:04So that kind of allowed us to look at it across the device to the end user.

5:10Ben Bajarin: So from a, you know, there's a lot that goes into, obviously, thermals, right? You make a node jump sometimes that you hope you get better efficiency, right, better performance per watt. And, you know, you talked about that for iPhone, and I want you to talk about this for Mac too. But I think sometimes people get lost, like, why does thermals really matter? Like, I get that devices can get hotter, right? But really just the efficiency, because it's not always just battery life. You guys are able to run at better sustained performance, which leads to other things, whether that's AI or gaming or things like that.

5:44Ben Bajarin: So maybe just talk about why thermal gains, why that matters when you think about a product or a software roadmap. Well, one of the things that has always been a relentless focus for us when it comes to Apple Silicon is performance per watt and power efficiency. and it manifests itself in many different ways. On the Mac side, one of the things that we really enjoy is the power efficiency of the Mac Mini. And now with M6, the Mac Mini has been taken to a whole new level from a performance per watt standpoint. And one of the reasons it's so important these days is just the rise of agentic AI and the use of a Mac Mini where it's sitting there on your desk, maybe to the side of a Mac you're also using as a primary device and just running constantly.

6:38And so being incredibly power efficient as it's doing that is something that is really part and parcel to the use of that product and makes it quite unique. And then you scale that all the way up to the Mac Studio where now people are beginning to cluster multiple Mac Studios together, and you can use four of them to run Frontier models right on your desk, but yet plug it into a common wall outlet. That makes that product very unique. So power efficiency manifests itself in different ways across our product line, and from a Mac standpoint, it makes the Mac Mini and the Mac Studio incredibly unique in the industry.

7:22And from an iPhone standpoint, right, it manifests more in how the users are going to be doing more with AI workloads. If you look at multitasking on the iPhone Duo, you can imagine like even, you know, more usage of the performance benefits of the device, whether it's you're playing a game, whether you're doing some more CRE AI processing in the background and things that are going on. These are all things that are going to benefit not just from a peak performance, but like over time. You know, if you're, the easiest example is the game, high frame rates in the game, giving you that great quality experience.

7:55You don't want the frame rates to drop. You want your main, and you also want the device to feel comfortable in your hand. And we don't have as big of a thermal envelope as the Mac side. So the team has to very specifically go look at the system design of the iPhone forms every year and think about, well, how are we going to optimize for the thermal performance within that wonderful design? And how do we make it the very best it can be? And I think that's where a lot of the collaboration between the product design, the systems engineering, and the silicon team come together for us to be able to do that.

8:27Yeah, and our goal is to maximize performance with efficiency in mind. And maximizing within the thermal constraints and thermal envelope of the platform. So each of our chips, depending on the platform it goes to, our goal is to fill that thermal envelope with the best performance we can and do it as efficiently as possible. And so some of the techniques we applied on A20 Pro, like moving the DRAM off to the side from top of the SOC to the side of the SOC, allows us to extract heat from the SOC and unlock the performance of the SOC and the compute blocks like the CPU and GPU by connecting it more efficiently to the vapor chamber.

9:07So you get better burst performance as well as better sustained performance because a vapor chamber has helped spread the heat across the platform. So thermal constraints are really important.

9:19Ben Bajarin: Yeah, so maybe let's dive into the packaging a little bit more. I think if I could be wrong, but the last time I remember Apple talking of packaging was with M1 Ultra, the first time, right, that chips perfused together on substrate. If Greg, if I'm wrong, I think that's the last time. So here again, we had, hey, we did something new in packaging, right? It's an approach that we took. because we saw a problem, in this case thermal, and we wanted to extend that, which I think also led to some benefits in unified memory, which I think would be interesting to talk about, too, that you could do design-wise.

9:52Ben Bajarin: So maybe just go a little bit more into the packaging side, and then we can talk about how that's led to very specific product experiences as well. Yeah, I mean, and actually, A20 Pro, the packaging change we made, was actually inspired by what we were doing on M1 and all the M series chips where we actually had an MCM packaging. And we wanted to, we saw an opportunity to bring that to iPhone with A20 Pro as well as address thermals, as well as address the opportunity to get more bandwidth into the chip. And so, you know, that's a technique that also helps improve the performance of AI applications, general graphics compute across the board, so better graphics, better thermals.

10:41And so it does increase the XY footprint of the device because now the DRAM is off to the side. But we work closely with our system partners to make sure that we could accommodate that in the main logic board design and attach it to the vapor chamber in an efficient way. So a lot of great innovation came in that. And despite that, the vapor chamber is like a lot bigger this year than the prior year. So that's what's leading to that 40 % gain in sustained performance. And then we use some, you know, graphite material. We use this new nanotwin copper material throughout to really sort of accelerate how much we can get out of that vapor chamber cooling system.

11:20So, you know, despite having, you know, changes in dimensions on the, you know, with the MCM packaging and being able to navigate that and still having a larger battery in it. So I talk a lot about when we make a change, there's like this delicate orchestration of like several things down the line. It's like a domino effect of everything else that needs to be reconsidered and moved or components sometimes redesigned to be smaller and more efficient. But that's I mean, that's the magic of it. And hopefully a consumer never has to know that. Right. They just get a better battery, better performance, better sustained performance so they can run eight workloads, get camera features that are even better.

11:57And our packaging technologies scale all the way up to the M5 Ultra in an amazing way with a tremendous amount of bandwidth between the quad-die architecture going from one die to the next and enabling a solution that's, quite honestly, there's nothing like it in the entire industry. Yeah.

12:20Ben Bajarin: And I think it's interesting, just coming back to what does a new thermal profile allow in these sorts of things, like even the example you gave, which is we've got an additional opportunity to design hardware because we've got better thermals, which then leads to we can now do these things with software that we couldn't before. And so I think that's, again, just like the vertical integration that comes back in because, you know, as you guys design silicon, you know, you're on an SOC. You're limited by space, right? So it comes to what's architectural choice designate. But similar, right, when you guys design an iPhone, you're like, it's only so big.

12:56Ben Bajarin: There's only so many things I can do, right, to get to these use cases. So I think that kind of lockstep of a silicon choice leads to better thermals, leads to a design decision, right, leads to a software. It's just such an interesting thing, right, that keeps coming back to an advantage for Apple, which is so interesting. Yeah, we've talked about this before. We are designing the silicon and the products in concert. Our silicon team has the advantage of knowing what our roadmap is and designing to the objectives and goals of that roadmap as opposed to vice versa where in the rest of the industry, typically you have to basically pick chips from a menu and try to design a product to a chip that already has been designed.

13:41Right. And this goes back to, have you heard Johnny say, we're not a merchant silicon vendor. We're really building the solutions for our own product experiences. And sometimes it's a very specific thing. Like we have to decide a couple of years ahead, hey, do we put down these extra lanes in the chip? Because we know in X amount of time, there's going to be this new capability, whether it's in camera or AI, that we're going to need this extra bandwidth to be able to do that. And so we do have to be a little bit organized in thinking ahead and envisioning those experiences and predicting what we're going to need.

14:14But also building in headroom, right? Because one important thing is that our customers can hold on to our phones and keep using them, getting software updates, getting new intelligent features. And so having the latest chip with these faster components, double the neural engine, is going to allow them to run everything right now. but also give them that runway over time because we want our phones to be durable.

14:35Ben Bajarin: Yeah. So let's talk about the now dual Apple neural engine. You guys never were the, you know, we're very early to what we all, the industry calls NPUs, you call Apple neural engine. But for like every year, I kept waiting. I'm like, when are they going to go from 16 when the whole rest of the industry is like, we can do 200 tops. I mean, nobody can do that, but I'm making that up. But you know what I mean? Like the whole, so, but now you have, right? You have doubled that. It's dual. And so, like, what drove that decision? Sort of, you could say, why did we take our time, right, doubling Apple Neural Engine when the rest of the industry wanted to talk up their NPU tops.

15:14Ben Bajarin: But really, like, now that you've made that decision, what does that enable? Like, why is now the right time? And what did we see in, again, this vertical approach that we can now, you know, bring to software and hardware? Yeah, I mean, like I mentioned, 2nm also gave us the opportunity to put a dual neural engine. And just to describe it, these are two discrete 16 core neural engines, but they're tied together at the hip. So you could actually run different models or workloads simultaneously in parallel, two in parallel across the two A &Es, or one workload split across both. And the frameworks with core AI will decide what's appropriate across the two A &Es.

15:55And, yeah, we felt like we had a huge opportunity here. Siri AI, for example, runs on the dual A &E. A lot of camera processing runs on the dual A &E. For example, one of the largest neural networks running on the phone, Learn Fusion, runs on the dual A &E and helps the low-light photography. And there's opportunities for performance as well as efficiency. if there's something that can be done with different processing on the A &Es, we can actually, and doesn't need the full potential of both the A &Es, the workload can split across the A &Es and run at lower frequency, lower voltage, more efficiently.

16:35So we can use these dual A &Es for performance, parallelism, as well as efficiency. So there was a lot of foresight putting that into the chip. Yeah, and just to expand a bit on the photography capabilities, Obviously, we added variable aperture this year to the 18 Pro models. But it's, like you said, it's a sophisticated neural network. It allows us to do even better, sharper detail when you get to those lower levels of lighting. And I think, you know, we've been continuing to try to make improvements in those difficult low-light situations. Low-light video is also improved shot-to-shot times.

17:09So shot-to-shot times every day, shot-to-shot times even in night mode photography as well. So translating that power to that. And the second point I'll make is, Sri mentioned the core AI framework. This is great because developers don't have to go in and make a decision about which part of the neural engine are we going to run on. The framework itself will figure that out on behalf of the developer app, whether it's best to run across both of the neural engine core sets, one of them, and figuring out what's the most efficient way to do that. I think it's also important to point out that the neural engine and now the dual neural engines, It's part of a balanced approach to AI workloads, right?

17:49So the neural engine is the most power-efficient way to run a workload, but now it's complemented by having these powerful neural accelerators in the GPU for other tasks that you might not want to be as or need to be as power-efficient. You just want the most performance as possible. We also have neural accelerators in the CPU for certain workloads, and we feed all of that with tremendously high bandwidth for memory and capacities. And so the entire system has been architected for AI workloads and now with the emergence of orchestration being incredibly important too, we have the world's fastest CPU cores that run orchestration better than any other chip.

18:37So all of that works together with the A &E in concert with each other to give, you know, just a tremendous platform for AI.

18:46Ben Bajarin: With all the kind of increased capabilities, like I'd love to think about like or talk about how like developers would think about this. I get they don't need to decide, right? No one should have to say like, oh, this is the workload I want to run on A &E because it's the best tops per watt or whatever. Like no one should have to, but I imagine that it enables new things that they couldn't do before. And so kind of across the board, right, whether that's, again, the thermals or more in memory. How best should developers think about taking advantage of these things, meaning that they can do things with software that they couldn't before now that you've got some of these increased capabilities?

19:26Certainly, I think there's a bunch of things. So we have these increased intelligent capabilities with the A &E across GPUs, the accelerators, as well as the dual A &E itself. We have just rote performance gains as well. We also have better sustained performance. So it enables developers to be able to push things, I think, to a level that wasn't necessarily possible. And that's a progress that we hope to make every year, to encourage them to push the envelope on their apps. And there's a lot of creativity in the developer community. I mean, I would say, you know, some of the things that I've been really impressed with already, you know, looking at the form factor of iPhone Duo.

20:06I'm sure you saw the Netflix app, being able to think about, well, how do we push performance now running across two displays? How do we, you know, show the reflections even on the bottom half of the display? I'm sure that's quite intense from a processing perspective. But that's not something that I think would have been possible before, both from perhaps a thermal performance form factor as well as just rote performance of the chip. And so it's these kinds of experiences that get us really excited because we can see that creativity in a developer community. And holding that up as an early example, I think there's even more possibilities that can do.

20:44And also to augment our own intelligence capabilities, what the integration of apps with Siri AI could be another great opportunity as well. Right.

20:53Ben Bajarin: So, you know, it's interesting, the architectural decisions, right, that you guys have made around, you know, unified memory and stuff for Mac, where like, back in the day, like when you first started showing that, nobody was like, we're going to run a bunch of AI on device. like maybe you saw it coming and you were more prescient than everybody but it just turned out it was a fairly good decision but now that this is sort of right the hot topic that we want to do more on device we've talked about this about the opportunity to do more token generation at the edge right with max both for consumer and enterprise use cases i think this will be true both right for iphone and you know things where obviously duo can become maybe some more enterprise right use cases too and that becomes a reasonable right place to do more more edge ai but now that you know we could look ahead and talk about right edge ai running local devices even hybrid as a kind of platform strategy what this now looks like right from a vertical we plan silicon to we plan devices you know because before it was largely you know you talk about what camera benefits were you talk about gaming and now there's ai and so i'm just curious like that from a kind of unified approach how you all think about that relative to edge AI now across the BUs?

22:10Yeah, well, I think we talked about the scalable architecture. I think private cloud compute is a similar umbrella architecture there too, where depending on the models that are being run in the consumer use case on the enterprise use case, we have the flexibility to run as much as possible on device, given the great work on the silicon and systems teams. But when there is a more complex model, we have the ability to move it to private cloud compute in a way that we've been emphasizing the extension of the protections on your device and privacy to private cloud compute as well. And that can scale whether you're a user sitting here, or you have a Mac on your desk, or if you're using the Mac professionally for some enterprise-level models that can then easily seamlessly move between the device and the cloud.

22:55We're seeing tremendous interest from customers to obviously do as much on-device AI as possible. One, for the fact that you brought up that, you know, their consumption of tokens is skyrocketing and they're, you know, looking to lower the spend on token costs. But also privacy and security, right? To keep your data local, not have it go somewhere in the cloud where now it's no longer under your control. Of course, if, as Kayan pointed out, with us, private cloud compute is great for that. But in other instances, you want to keep that data local. And so that is really a tremendous ask from our customers.

23:42And I also think what you're seeing in the industry is that models are starting to specialize. And so the ability to decide which model to do what, which model is best for a certain task, is kind of the new trend that's happening in the industry. Like Tom and Kayan are saying, we strongly care about AI. And what do you need for AI? You need energy-efficient compute. You want access to memory, unified memory, and you want high bandwidth. And I think the chips that we just recently announced address all of that. We've got fastest general-purpose compute in M6, fastest CPU there, 4X AI compute in M5 Ultra relative to its predecessor, and access to large memory, high bandwidth.

24:31And so we're strongly focused on AI for now and in the future.

24:38Ben Bajarin: Yeah. Yeah, and I think it's interesting. You have to make some bets, right, architecturally too, just about where models go, right? That's one bit that's sometimes in your control, right, with stuff that you're doing, then sometimes out of your control that people want to run. But we know that that is still a lockstep, right, in terms of how you think about products. But I want to just close on security and privacy because there's a couple things I think that are interesting that don't get talked about a lot. And one thing we've been looking at, like, largely to enterprise, but I think there's some consumer access, is just what's going on with cybersecurity in the age of agentic AI.

25:16Ben Bajarin: But this is not a, like, this is a computationally expensive problem, right? You say we do security and privacy, but I don't think people know there's a compute tax to do that, to run in the secure enclave or the secure ex-slave. That's not trivial. It requires some innovation in silicon, in hardware and software to do this. So maybe just talk about how that approach impacts both product but also the architectural decisions. Because, again, it's not trivial. That is a computationally complex problem that I just don't think people give enough credit to how much silicon power and design goes into just security and privacy.

25:57Yeah, I think that's a great point that you raise, Ben, because we're able to look at it across multiple facets, right? So you've got software protections, you've got server protections, but we're able to start from the foundation of it is what do we build in the device on the hardware side? And even down to the chip itself, as you know, years ago we developed a secure enclave. I believe that happened when we first introduced Touch ID. And that was to protect your biometric information for Touch ID to Face ID later on to other sensitive information like that. And now we've developed a secure exclave, right, that protects a lot of the sensitive sensor-based collection data, such as audio and mics.

26:36It's used for some of the features on watch this year. But we have that secure X-Lave both on the S11 chip, and we've also had it in iPhone for a couple years, and now really making use of that even more with the audio intelligent features on watch. But it's looking at, hey, there's some cool user benefit things that we want to build, but if privacy and security are a fundamental priority for us, where do you backtrack to that foundation that's going to give you the strongest ability to protect that for the user? And at that level, on the silicon level, it started with, hey, we need to have the secure X-Slave component.

27:14And maybe, Sri, you can talk about the cost involved in us doing that. But it was an important decision. Certainly effort went into that. But even last year, we introduced a new technique of tagging memory from a security perspective, which took a lot of effort and plumbing inside the SOC. And there was a lot of debate about how to do that in an efficient way and not impact the performance of our CPUs. So we invest heavily in security and privacy across the board. And maybe for the regular users, the secure X-layer, it has a separate component hardened off from the rest of the chip. The application processor cannot access it.

27:49Other apps cannot access it. It's in its own operating environment. So that's why we call this like a secure compartment within the chip. And we take a very holistic approach to security, right? Even in the operating system, features like FileVault in the operating system, to the fact that we locked down our boot ROMs to the fact that we got rid of kernel extensions years ago. We take a very holistic approach all the way up in the upper layers of the software, all the way down through the system and into the processor.

28:22Ben Bajarin: Yeah. I just think it's so interesting, right, because it's one of those things that architectural decisions have to get made to do. It's not easy. I think you guys say, we want to protect security and privacy. And you do. And I think connecting that to, again, how deeply architecturally that has to be. And then, again, like why that yields, right, the cross product benefit for that conviction. And again, an era that I think will be increasingly needed with agents and things that can, you know, cybersecurity that can happen out there. That unified approach, right, just makes a lot of sense. And it's, again, it's a differentiator because you can plan, right, in lockstep.

29:03Ben Bajarin: Well, we're out of time. I greatly appreciate you all. It's very awesome to get a kind of under the hood kind of scene on the engine across the BUs and Silicon. So we appreciate your time, and you're always welcome back on the circuit anytime. Thanks for having us.

29:32Thank you.

30:02Thank you.