973: AI Systems Performance Engineering, with Chris Fregly

28:27Jon Krohn:Right. And so earlier in this episode, I said how it would be very difficult in 60 minutes in an hour-long podcast episode to convey all the things that somebody needs to know in order to optimize their AI system. Your book actually ends with a 175-item optimization checklist, which kind of goes to prove my point that if we took a 60-minute episode, it would be able to spend 20 seconds on each of those items in the checklist, which obviously is not going to convey any meaningful information. But if a listener had to take a handful of those, maybe like three of those as the kinds of optimizations that would give them the best return on their time, investing their time in that, is it easy to kind of say that these handful are the kinds of key things that they need to be doing?

29:19I'll give you a little tip, which is if you have the PDF of the book, which obviously I do, I give that to Cursor. I give that to CloudCode. And when I'm analyzing a system, right? Because these models don't have all of these things at their fingertips. There's a lack of training data, for example. So one thing I do is I actually put the book in context and it's a thousand pages. So you need a big model that has a big context window and ask it, what are the top 10 things I should be looking at? And it'll start to run its checks and everything, assuming that you have the right tools installed, which is the NVIDIA Insight, and the NVIDIA Compute.

30:18So that's kind of longer term what you would do, which is try to give enough context. Now, I think this will change because I know that the labs, I know firsthand that the labs are training more and more on hardware specific for AMD, because they themselves have this interest in their models getting better at like optimizing kernels. But to be very systematic about these things, I'm certainly going to talk through a few of these, but make sure there's some sort of like before and after profiling best practices, things like locking the GPU clocks and making sure it doesn't hit peak. But then there's really having like a methodical way of going through.

31:11So in my GitHub repo, for example, I've got this harness that will do things consistently before and then after. And if there's any inconsistencies, it'll fail the comparison and say, hey, this is not apples to apples. So it's extremely important that you have a very, very methodical way of comparing changes. But yeah, things to keep an eye on, Well, yeah, I'll start small and then kind of go bigger. On a single GPU, obviously you really want to understand if you're on Blackwell, has extremely different characteristics than Hopper, then we'll have in the future Vera, or I think it's Ruben is the next generation, and then Feynman is the next generation after that.

32:03But separate from the GPU, there's a lot of things that can go wrong at the actual operating system level. And with the GPU driver, for example, there's the famous CUDA toolkit and then the NVIDIA driver. And yes, anyone that's tried to enter this world has spent, I'm sure, countless hours trying to get these things synced up and the versions all set up properly. Once you get past that, and I do want to emphasize that you should always be on the latest. You know, I don't always upgrade for the sake of like upgrading in my life, just sort of generally. But I've noticed in the GPU world, it's extremely critical to stay up to date.

32:48That does cause problems because sometimes the, you know, PyTorch doesn't keep up with the latest CUDA 13.1, you know, or, but you really, really have to be aware of the versions in this world. But yeah, I would keep an eye on memory fragmentation, extremely important. You want to disable swap, for example. These are things that if you just get a basic Linux system, these things are all turned on because they're optimized for CPUs and just sort of general purpose workloads and a lot of different tuning that goes in. This was actually one of my favorite things about the SageMaker

33:32sort of ecosystem is that there was a lot of work put in. Yeah, so SageMaker is the Amazon, or it's a service within AWS that offers you GPUs. And specifically, there's a service called SageMaker HyperPod, which is pretty bare bones in a good way. It's got Kubernetes installed. It's got all of these things tuned. You've got a lot of people focusing on this. There's a lot of customers that are tuning, that are helping Amazon with the right configuration for all of these things. You almost always want to occupy the entire GPU. So you don't want to share like you would in a CPU land where you can split it up and virtualize it.

34:19You want to get as close to bare metal as possible. Containers have gotten a lot better throughout the years with GPUs. So we're seeing a lot more folks use Kubernetes and Docker for these kinds of things. But I would almost encourage to stick with bare metal. IO optimization, I think is the number one killer here. This is the lowest hanging fruit. Use the fastest disks that you can get. Use local disk. Don't try to use any network file system. It's convenient to use a network file system, but it will almost always cause you problems. I had said earlier that Jensen, the CEO of NVIDIA, has been focusing a lot on storage recently.

35:06Probably not as much as I think he should be going forward, but he does have a lot of partners that are working with, and there's something called GPU direct storage, which can load data from disk right into the GPU without going through CPUs, without creating copy buffers, and those kinds of things. So that would impact the sort of data pipeline. And when you scale, of course, to multiple nodes, multiple GPUs, which is these days just a given, you really can't do anything that complex on a single GPU anymore. So then you start to optimize the network and then you have to make sure that you're using direct GPU to GPU communication and use things like NVLink, right?

35:55The NVIDIA interconnect that's been optimized for the... So one piece of advice is when you're looking around for a GPU cloud provider, I guess these days people call them NeoClouds, like CoreWeave and stuff, try to find one that has bought into the entire reference architecture, you know, like by NVIDIA. If you, you know, there's a group of, I guess, cloud incumbents that have their own networking, that have their own stuff that's been built up, you know, throughout the years. And they are basically patching and bypassing the sort of natural way that the NVIDIA GPUs were designed. And that's actually, in my opinion, the highest value proposition for these Neo clouds is they're not tied to all of those sort of CPU legacy ways of doing things.

36:55They have bought into the NVIDIA ecosystem. In some senses, they're favored by NVIDIA, I would posit, which seems natural, right? because you're going to favor customers that buy a lot more from you. Because keep in mind, one of the most pivotal aspects or the most pivotal acquisition by NVIDIA was this company Mellanox. So they bought this company Mellanox back in, I think, 2019. I think it closed in maybe 2020. They're purely networking. And so now like NVIDIA since 2019 has gone from just a GPU company to an AI systems company. And so if you buy into that whole thing, there's a lot of efficiencies that happen, even in the network hardware can actually do calculations with these like Mellanox routers that, yeah, that like NVIDIA purchase.

37:53So, yep.

37:54Jon Krohn:Fantastic. Thank you for all of those tips, kind of giving us, distilling for us the most valuable of the 175 items provided in your book. I think that was like four of them. Yeah. And that's great. I mean, what a nice thing to be able to do in this episode. And yeah, of course, check out Chris's book, AI Systems Performance Engineering, to get the full list of 175, the full thousand pages on how you could be optimizing your systems. um let's let's shift gears now a little bit chris to talking about inference specifically uh you know we talked earlier in this episode about how inference is most of what's happening at scales with ai models training is an important part but inference is most of what we're doing with gpus these days and so you talk in the book about multi-node inference and disaggregated architectures.

38:46Jon Krohn:What the heck are those? Yeah. Welcome to another chapter that was supposed to be a single chapter or a topic that was supposed to be a single chapter. Yeah. So this is one of, this is one of the three books, one of the three sub books. Exactly. Inference, there's a lot going on in inference that folks probably on the surface don't realize. So stated like simply with inference, we're doing just a forward pass, where we've got the weights, the weights, which are the parameters in this, let's say it's a trillion parameter model. We just need to compute the answer, boom, just go forward. So on paper, this seems extremely simple.

39:29In reality, transformer-based inference, there's a lot of places for things to slow down. And so there's a lot of caching that needs to happen so that we're not recomputing things on every, you know. So think of when you go to ChatGPT or to Cloud, you type in a query. It's like, it is going to look at all of the context that you've passed in. It's going to compare everything to like everything else that's in there. it's going to find things that are relevant. And it would have to do that the same way every single time on every single query, for example. And yes, even within the exact same query, it's doing that many times because it's generating the next token and the next token.

40:18And so there's something called a KVCache. Funny enough, I was going to put KVCache on my license plate because I just shipped my car from another place. And somebody in California has already taken the word KVCache on their license plate. So I'm like, that's how popular KVCache is.

40:36Jon Krohn:You can probably in any other state in the US still get that. Totally, right? So someone beat me to it. But think of it just like any sort of cache system where you're pre-computing. But on a single query, these caches can be humongous. I mean, especially with people that are uploading documents, that are trying to scan things, there's a lot of computations happening. So, now you've got a case. So, there's building the cache, which is step one, and then there's actually generating the next token, which is step two. And so, and, you know, yeah, getting, right, like starting to actually generate the response.

41:27So those are called pre-fill is the first step where it's like generating all of this, these computations that's extremely, extremely compute heavy. And so that's where you potentially want a node or a set of nodes in your cluster that are configured differently. So memory bandwidth, it's still important, but it's not as big of a deal as the second step, which is actually generating those tokens. Because that second step has to move all the weights back and forth from your GPU RAM, sometimes CPU RAM, based on how big the GPU RAM is, sometimes on disk. So these are very different configurations.

42:13And so you will often hear people talk about disaggregated PD, which is disaggregated pre-filled decode. I would guarantee you if you are going for like an interview in 2026 and need to study something for your interview, it is going to be about this pre -filled decode. It's going to be about KVCache. So if you're going to talk to any of these big labs in any sort of engineering capacity study, I can't remember what chapter it is in my book because it was supposed to be a half a chapter actually, and it ended up being, I think, three or four. So there's trying to understand it, and then there's trying to optimize it.

43:01Jon Krohn:And pre-fill, decode, disaggregation, and KV cache. There you go. There are your interview tips of the episode. Speaking of tips and ways that people can be getting ahead more quickly, you mentioned to me prior to us recording that you've been loving using AI coding assistants. And so, you know, tools like Cursor, Cloud Code, Codex. How has your workflow changed as a result of these kinds of tools? How does that impact in particular AI systems performance, optimization? and yeah, what do our listeners need to know about where this is going? For sure. I tweeted about this a couple of weeks ago.

43:44If you're manually writing code in this year, 2026, you are way behind. And I would not have said that if I didn't spend all of last year watching this sort of evolution because I started off in my sort of legacy ways, right, where I was writing every line of code. And I vowed for 2026 to only use these coding assistants and to see how far I could get. Now, I've done a couple of projects for some friends and for some portfolio companies. And just to make sure that, you know. So the short of it is you can fire off about 10 to 15 different things, like different aspects of either features that you're trying to build or bugs that you're trying to find.

44:41I'm personally using these tools right now to do a lot of optimizations and to look at different aspects. And so I'll say, take a look at the occupancy percentage for this particular kernel that I'm trying to optimize. At the same time, I'm having another GPU that has the same code that's analyzing a different part of it. And so think of a four GPU system and you can run separate experiments, each one running on a different GPU or even four separate GPUs, like separate nodes that have their own memory and stuff. But yeah, my personal workflow, I'm going to be a little controversial and say I prefer codecs.

45:28Yeah, I prefer the open AI. One thing, and again, very, very workload specific, I would say. I think if I was doing UI stuff, I would probably maybe prefer Claude right now. Claude's great with the UI. It's kind of my little dirty secret that I use Codex. And so while all the other developers are flooding, right, the anthropic GPUs and the inference stacks, I've got kind of my little group of people that just use Codex and I still have really good performance until people realize that, you know, Codex is actually better for other stuff. But for right now, I'm enjoying good performance and a fair amount of, you know, token.

46:08But the one thing I would recommend for AI systems, performance would be be on the machine. Don't try to do this stuff on your MacBook and hope that it works or even on a personal, like NVIDIA GPU, like an RTX. Yeah, I don't know, 50. Like, I've literally never used any of the personal GPUs because to me, the profile is so different. The hardware is so different. The memory bandwidth completely different. So if you are working on an on an AI project and you are ultimately going to deploy on a Blackwell or on a hopper, you have to be on that machine during development. Don't try to take any shortcuts.

46:50I recently tried the DGX Spark, which is the little mini one that got a lot of buzz, you know, the end of last year. There's so many things disabled on that, that none of my benchmarking was even working because it's just completely missing a lot of, you know, core hardware components and then, you know, software components on top. The driver isn't the same. It's, you know, a lot of different things.

47:14Jon Krohn:Nice. Really appreciate your insights on what you're doing with the coding assistance today. When you're working on those, Chris, do you still review all of the code before it goes to production? I did. I was once, yeah, yeah. I was once like you or once like the question that you're asking. In fact, I was just working with, with yeah, I was just working with someone this weekend, trying to get them a little bit up to speed. It's a good friend of mine. And I'm like, look, you're doing things like you're going to be outdated within the, you know, by St. Paddy's Day. Yeah, it was the joke specifically.

47:53And I was watching their workflow. And, you know, these assistants, they like to write little snippets of Python code or, you know, any or bash scripts to get stuff done. And this person was literally reading every single line of the bash script. Like it's happening during the thinking process and they would go up and they would expand it and they would be looking and they try to find the script on disc. And I'm like, dude, you have to let go. So the short answer is I did and it's exhausting and I've learned to step back and there's like temporary scripts that these things write to write the final code.

48:41And you could do that and you could be very disciplined and review all the lines of code. You could write tests. You can have the LLM actually write the tests as well too. I've actually even sort of let go of unit tests. And this is very controversial for all of my test-driven friends and folks that are listening. I focus on the evals. So I clearly set up and provide examples that I know are correct. Those are sort of the collaboration set. And then I use the LLM to actually judge the quality. I have it constantly running. So I've got evals that are always running in the background in a separate tab while I'm, you know, building the software.

49:34So short answer is no, I stopped doing that. And it really slows things down. And it's hard for people to do. But I'm shipping code a lot faster. I'm, you know, with kernel optimization, you have to be very careful. these models want to hack the reward, or it's called reward hacking, and just get the fastest thing possible. And if that means zeroing out everything to make the computations a lot faster, it's going to do it. And so one thing I spent a lot of time on with my GitHub repo is these correctness checks and correctness verifications. And these pop up every single month, someone like release some small little startup releases something that they have achieved, you know, 50x speed up.

50:30And the first thing I do is take that code and I put it into my harness and boom, I see that they're not using CUDA streams properly or that they have that it's not their fault. It's just that they've given the LLM too much freedom. And so we have to get better about what it means to review. And I don't think even going through all the code that you would be able to catch these things, because I assume that they went through and looked at the code, but not unless you're actually running the code on the machine, and you've got all of the profiling, you know, set up where it can show you how many streams are being used and, you know, all the different aspects.

51:11If you don't have correctness checks, if that's in the form of evals for your sort of end user application or, you know, performance metrics and like, real, you know, correctness checks within your, your performance harness, then you're not doing it the right way.

51:26Jon Krohn:Yeah, really cool. I guess I need to stop looking at the code. Yeah, do you still do that? I'm living in the stone age. So my last semi technical question for you, before we start wrapping up, there's so many interesting things about you. We actually, we've really just scratched the surface here. We focused on your latest book, which again, you know, if people like this episode, you've got to get Chris's latest book, AI systems, performance engineering. Um, but in addition to that, you've done tons of other interesting things. So your previous two books were both, uh, they're both on, they were like on AWS books.

52:04Jon Krohn:It was like data science on AWS. Uh, I think that might've been with Ante Bart. that was the one that was with her. And I think you ended up at AWS because you were the chief product officer at a company called Pipeline AI that was acquired. And you've also, you've worked in principal engineering roles at Databricks and Netflix, including getting an Emmy at Netflix for the work that you've done. But the thing that I want to talk about, the only thing that we have time to talk about now is your role as an investor. So you've been an investor or advisor in companies like XAI, which is now at the time of recording, it was just acquired by SpaceX.

52:43Jon Krohn:You invested in Grok, G-R-O-Q, which makes LPUs, language processing units. And that firm was acquired by NVIDIA. And so, yeah, I'd love to understand your thinking on what's going on with hardware. Do you believe, for example, that the future of AI performance lies in specialized LPU hardware like Grok builds or in better software orchestration on top of general purpose GPUs? And you can also answer this question more broadly with just generally where you think Harper is going and where people should be putting their money? I'm in a fortunate position where I do have extra cash and I can take some chances.

53:32These are all secondary market shares. They're not public companies or some are about to IPO and things. On the investor side of it, I look for, this is very trite these days, but you have to look at the founders. Yes, Elon, he's an absolute nut job, of course, and he's very controversial. And my sister hates Teslas because of him and all this kind of stuff. So I can't even talk about some of these investments when I'm home for Christmas and things. But you have to look at their ability to fundraise. Right now, the secondary market is where all the action is. And so if you're still investing in Apple and things like you're, you know, you're like behind the times like this is not, not where the money where the money is currently being made, especially because there's been such a dry spell with these IPOs.

54:35So, you know, hopefully that changes. Yeah, I'd say my best investments so far have been Databricks. Of course, I was an early employee there back in the day, and I know those founders very well. You have to look at folks that can raise money, that can convince people that they're on the right path, that have their stuff together. Where are things going? There's so many different types of chips coming out. In fact, I'm about to go to lunch with some folks at Etched, which is another little player and similar to Grok with a Q. It's good to keep an eye on these folks to see sort of generally where things are moving.

55:19I mentioned before that the biggest bottleneck is this memory bandwidth. This is something that's that like this is the main reason why the NVIDIA folks purchased with a Q or did their little weird acquisition, you know, licensing deal is the, like LPU is, has all of that memory, like on the same chip, right? Like right next to the actual, right? Like the accelerator itself. And so that then decrease and that's really good for like inference. Cause like I said, when you're doing inference, there's that a decode step that has to move a lot of weights around that's very, very memory bandwidth bound.

56:01And so it's a really great complement to the Nvidia chip, which suffers from this, you know, memory bandwidth limitation. And it's just a fundamental architecture design that dates back to their graphics days. And so instead of building a cycle of, you know, five or six years out where where they have the chip they ended up. So, and also with like XAI, it's interesting because I passed up on SpaceX a few years ago, as I didn't see the leap that Elon is currently talking to. I couldn't, I didn't have enough vision for that. I saw that I liked XAI in that, you know, Grok with a K, which I actually personally don't really use that much.

56:50I should because I'm an investor. But I liked the work ethic of those guys. I actually met with them quite a bit last year while I was writing the book. They're big backers of SGLang, which is like VLLM, which is an inference engine, but it's SGLang. It's another project that kind of came out of Berkeley around the same time, VLLM, that does things a little bit differently. So we had some conversations. Some of those conversations actually are part of the book now. What's the specific question again? I talked my way out of that. Yeah, I had a question around, it's basically, where is this going?

57:35Jon Krohn:Where are the opportunities for investing? And of course, we've also got to give the disclaimer that we are not investment advisors. But I think it's just something that people are interested in. You've got a really interesting being in the Bay Area with the kinds of companies you've invested in, you have kind of an inside baseball perspective on where things could be going. You've already given us some of the tips around things like, you know, your advice is that the secondary market is where there's a lot more money to be made than in the public markets, your apples, for example. But yeah, and you also kind of already answered the question about whether the future lies in specialized LPU hardware or more general purpose GPUs.

58:13Jon Krohn:So I think you've covered that, Chris. The folks over at Together AI, the person, I think he's the chief scientist, one of the co-founders of this company called Together AI, is a big fan of general purpose GPUs in that it gives you more flexibility to support different types of algorithms. Not all the world is going to be transformers. I mean, this is going to change, it just has to change because we need a better. And all of these chips that are specialized for transformers may not be the best chips going forward, which was quite honestly a tiny bit of concern when I made the Grok with the Q investment is, oh crap, if something changes and another type of model starts to dominate, like another type of algorithm, you know, where is that going to leave folks like Grok with a Q?

59:16But so, you know, fortunately I've got shares on, you know, both sides of it. I've got public NVIDIA and I've got, you know, private things as well too. So you have to kind of have a balance there. But yeah, and I'm the worst investor in the world. I sold Netflix when it was low and I bought it when it was high and then it went low again and I lost a bunch of money and, you know, things like that. But so don't take advice from me. Great, great. I just have to be getting lucky.

59:43Jon Krohn:Buy high, sell low is the takeaway advice from Chris Fregler today on investing. Great. Yeah, certainly look at the addressable market. If you're familiar with Sequoia, it has a very famous pitch deck. And I'm very familiar with this pitch deck because I spent many years fundraising for my startups. And when I go into even a job, if I start talking to folks about potentially joining, even as an advisor or any type of role, if I decide to go back into engineering full time or whatnot, you want to stick. there's a famous, I think it's 12 or 13 slides. And it seems so, so silly to only have 13 slides to make or break, you know, millions of dollars, but you know, think of like, yeah, try to find that slide deck.

1:00:37It's, it's out on, um, yeah.

1:00:40Jon Krohn:And it's, it's the subject of countless blog posts out there having the experience of building these decks myself. It's, you're not going to have a hard time finding, you know, guides to, to your, you know, dozen page deck. And I think the key takeaway, maybe this is exactly where you're going. I don't mean to take the words out of your mouth, but if you can't explain in that short number of slides, why this is a compelling investment, it's probably not a good investment. Yep. And value proposition to me, that is the most important slide. I mean, yeah, obviously team and, you know, yeah, like assuming team and all that stuff's there, but if you can't clearly explain why now and why, you know, yeah, why am I the right fit for this?

1:01:22And, and, you know, what's the real value? Like, that's always like, to me, the hardest slide is like, yeah, how do I quantify this? And, but yeah, I mean, always look at, you know, the addressable market. And also, you know, these days, it comes down to the, like, business model and the pricing, gone are the days of this SaaS-based seat pricing. Now it's all about value, right? So like value-based pricing and yeah, they call it outcome-based pricing, right? Which is very obvious for things like customer service and stuff, but yeah, trying to think about that. So you really want to look for founders that are in tune with that and especially repeat founders that might not have that have traditional have been part of traditional uh you know pricing models business models yeah you really gotta suss that out great thanks for

1:02:15Jon Krohn:all of those inside baseball tips chris on investing and yeah so now we'll start wrapping up the episode you already know you've been warned by antir barth that i always ask for a book recommendation at the end and don't let you use your own book and so i think you have come prepared Yeah, one of my favorite books is called Founding Sales. And these days, it's a relatively old book. It's from August of 2020. Founding Sales, the early stage go-to market handbook. And I'll tell you why this is an important book and was so critical to me. So, the author goes through his evolution in learning about go-to-market GTM.

1:02:59I used to laugh when my friends that came out of business school would talk about go-to-market. It was so far from the things that I would think about on a day-to-day basis. And then I became a founder. And then I was like, oh crap, I have to figure this out. So there was a point in time and I still, I'm fortunate that I live somewhat near Stanford. So I take a lot of weekend classes and I don't have any kind of executive degree or anything. I just have an undergrad from Northwestern, but I can soak up like what is valuable to me. So really thinking through what does go-to market mean specifically in a SaaS environment And not all the things will be relevant to LLMs, of course.

1:03:44This is an older book. But it really gets you to see the pain points that he had gone through and the different pivots that he made and how his competition really affected his route through this. So I point this out because I think this audience is mostly technical, right? And if you ever do think about becoming a founder, yeah, check out Founding Sales because it's a nice sort of engineering friendly way, like logical way to think through how to come up with your real true value proposition for your ideas. And you'll read something and say, oh, that's exactly what I was thinking. And now I see how it's bad, how it's not really going to convince anyone to use this product or to give me money.

1:04:37Jon Krohn:Nice. That's a really great tip. And I like how it follows along nicely with actually the arc of this episode as well, where we started off really technical, really heavy in the weeds on AI systems optimization. And then we kind of ended with commercial discussions and a commercial book recommendation. So perfectly on theme, Chris, in the end. You didn't know that's where I was going with questions. So pretty cool. And yeah, then my final, final question before I let you go is how can people listen to you after this episode? Obviously, your book, AI Systems Performance Engineering is the way to go.

1:05:10Jon Krohn:I'll have a link to buy that book as well as to the GitHub repo for the book. Beyond that, how should people be following your thoughts after this episode? Yeah, I'm C. Fregly, C-F-R-E-G-L-Y, pretty much everywhere. I recently discovered I have a second cousin named Chris Fregley who does lots of van conversions. He's in tech. He's a Kubernetes guy and we actually live pretty close to each other. So we've been connecting. He's half my age, which is real fun to get his perspective on the tech industry. But we're both in San Francisco. So, but yeah, I'm the original C. Fregley. Yes. Okay. I see you saying like, watch out for that guy.

1:05:56Jon Krohn:Don't follow him. If there's van conversions you've gone, it's the wrong guy. Totally. That's the most like unique thing is I don't do van conversions. I prefer to live in a house, you know, condo. Yeah. So see Fregley on Twitter, see Fregley GitHub. See, or yeah, Chris Fregley LinkedIn. Perfect. Yeah. We'll of course have links to all of those in the show notes. Chris, such an honor to have you on the show after the kind of decade of being aware of you. You're such a rock star. Thank you for taking the time. And yeah, next book comes out, you've got an open invitation. I am swearing off book writing.

1:06:34Yeah, which I said after the first one, after the second one, but yeah, we'll see. So, hey, John. Yeah, man, great chatting.

1:06:42Jon Krohn:What a great episode today with Chris Fregley in it. He covered how DeepSeek R1 achieved 10 to 20X training cost reduction over comparable Western models by co-designing hardware, software, and algorithms together, a recurring topic throughout the episode. He talked about how memory bandwidth, not flops or tensor cores, is the single most critical metric to pay attention to when evaluating GPU performance from generation to generation. He talked about mechanical sympathy, a term borrowed from Formula One that means understanding the full hardware-software stack, and Chris argues it's the most valuable skill for anyone studying computer science or related jobs today.

1:07:19Jon Krohn:He talked about how the PyTorch profiler only shows the tip of the iceberg. There are 50 to 60 additional GPU level metrics involving streaming multiprocessors, occupancy, specialized function units, and instruction pipelines that are essential for real optimization. And he spilled the beans that he's abandoned line-by-line code review in favor of continuous evals and correctness harnesses and says that if you're still manually writing code in 2026, you're way behind. As always, you can get all the show notes, including the transcript for this episode, the video recording, any materials mentioned on the show, the URLs for Chris Fregley's social media profiles, as well as my own at superdatascience.com slash 973.

1:08:01Jon Krohn:All right. Thanks to everyone on the Super Data Science podcast team, our podcast manager, Sonja Breivich, media editor, Mario Pombo, partnerships manager, Natalie Zajski, researcher, Serge Masise, writer, Dr. Zara Karche, and our founder, Kirill Aramanko. Thanks to them all for producing another super episode for us today, for enabling that super team to create this free podcast for you. We're deeply grateful to our sponsors. You can support the show by checking out our sponsors links, which are in the show notes. And if you'd ever like to sponsor the show yourself, you can get the details on how at johnkrone.com slash podcast.

1:08:35Jon Krohn:Otherwise, share this episode with people who would want to listen to it, review it on your favorite podcasting platform or on YouTube. Subscribe, obviously, if you're not a subscriber, but most importantly, just keep on tuning in. I'm so grateful to have you listening, and I hope I can continue to make episodes you love for years and years to come. Until next time, keep on rocking it out there, and I'm looking forward to enjoying another round of the Super Data Science Podcast with you very soon.