NVIDIA's Josh Parker on How AI and Accelerated Computing Drive Sustainability - Ep. 234

0:10Hello, and welcome to the NVIDIA AI Podcast. I'm your host, Noah Kravitz. The emergence of generative AI into our collective consciousness has led to increased scrutiny around how AI works, particularly when it comes to energy consumption. The interest and focus on how much energy AI actually uses is, of course, important. Our planet faces energy-related challenges ranging from grid infrastructure needs to the impact of climate change. But AI and accelerated computing have a big part to play in helping to solve these challenges and others related to sustainability and energy efficiency. Joining us today to talk about all of this is Joshua Parker, the Senior Director of Corporate Sustainability at NVIDIA.

0:52Josh brings a wealth of experience as a sustainability professional and engineer. Before his current role, he led Western Digital's corporate sustainability function and managed ethics and compliance across the Asia-Pacific region. At NVIDIA, Josh is at the forefront of driving sustainable practices and leveraging AI to enhance energy efficiency and reduce environmental impact. Josh, welcome and thanks so much for taking the time to join the AI podcast. Thanks, Noah. Long time listener, first time caller. It's happy to be here. Love it. I always dreamt of hosting an AM radio call-in show, so you're inspiring me.

1:29So I'm going to just kind of open this up broadly to you to get us started. I kind of alluded a little bit to it in the intro, but computing uses energy. everybody is talking about AI, obviously. And there's, you know, with good reason, interest, scrutiny, focus on, well, how much energy is AI using? And if we start using more and more AI going forward, what's the impact going to be on, you know, all of these energy related things that we deal with on our planet? So let me ask you to start. How much energy does it really use? Is this a warranted discussion? What are the things that we should be thinking about and talking about and working on when it comes to energy and sustainability and not just AI, but accelerated computing and the other advanced technology that goes with it?

2:16It's definitely a reasonable question. And as someone who's been in sustainability for a while, it's something that we always talk about, climate and energy and emissions. Those are very big, urgent topics that we're all thinking about in every context. So when you see something like AI that really bursts onto the scene, especially so rapidly, it's a very legitimate question to ask, okay, what is this going to do to energy? And what is this going to do to emissions associated with that energy? So it's the right question to ask. The answer though, turns out to be pretty complicated because number one, we're in a period of rapid, rapid growth.

2:55And it's hard to predict where we're going to be in just a couple of years in terms of the expansion of AI. Where is it going to be used? How is it going to be used? What benefits do we get from it? And there are lots of nuances to that as well, including things like the hardware that it's being built on. This accelerated computing platform itself is rapidly, rapidly evolving in ways that actually support sustainability. So it's the energy efficiency gains that are being developed in that accelerated computing platform are really, really dramatic. So if you want to paint a really accurate picture, as accurate as we can get in terms of where we're going with AI energy consumption and the emissions associated with that, you need to have a really complex, nuanced analysis to avoid coming to very inaccurate and potentially alarming conclusions.

3:48So let's dig into that a little bit within the context of a half-hour podcast. Let's talk about some of those nuances. And you mentioned the hardware. And so obviously GPUs, a big part of that. How is accelerated computing sustainable? Accelerated computing is a very tailored form of computing for the type of work required for AI. The accelerated computing platform on which modern AI is built takes the math that was previously being done on CPUs in a sequential order and basically uses these very, very efficient, very purpose-built GPUs to do them in parallel. So you do many, many more operations, and these GPUs are optimized to do that math, the matrix math that's required for AI, really, really effectively and efficiently.

4:42And that's what's driven both the huge gains in performance and also the huge gains in efficiency in AI. And it's really what has enabled AI to boom the way it has. The traditional CPU paradigm, CPU only paradigm for trying to run this math just wasn't scaling. And so we really need GPUs to unlock this exponential growth really in performance and efficiency. So if you compare CPU-only systems to accelerated computing systems, which have a mix of GPUs and CPUs, we're seeing roughly a 20 times improvement in energy efficiency between CPU -only and accelerated computing platforms. So very dramatic.

5:28And that's across a mix of workloads. So it's a very dramatic improvement in efficiency. And if you look just over time at accelerated computing itself, so compare accelerated computing platforms from a few years ago to ones that we have today, that change in efficiency is even more dramatic. So just eight years ago, if you compare the energy efficiency for AI inference from eight years ago until today, we're 45 ,000 times more energy efficient for that inference step of AI where you're actually engaging with the models, right? And it's really hard to understand that type of figure. One 45 ,000th of the energy required just eight years ago is what we're using today.

6:13So building in that type of energy efficiency gain into your models for how much energy will we be using for AI in a couple of years is really, really critical because it's such a dramatic change. Yeah, that's a huge number. I don't mean this as a joke, but the best way I can think of to ask it is, are the workloads now 45 ,000 times bigger or more energy intensive than they were eight years ago? Or is really the efficiency outpacing all of this new attention on AI? So that ends up being a very complex question as well, because you have to get into the realm of figuring out how many times do I need to train a model versus how many times can I reuse it with inference?

6:57So, you know, big models like CLOD 3.5, CHAT GPT 4.0, and so forth, they're trained, takes a lot of time to train them. But the inferencing, when you're actually engaging with the model, if it ends up being durable, then that inference step is very, very efficient. So because we're still in this inflection point where things are moving very rapidly, it's hard to see how the compute requirements are scaling versus the energy efficiency. Certainly, they're scaling. We continue to see bigger and bigger models being used and trained because companies are seeing huge benefits in doing that. But yeah, this is what makes it complicated is that the energy efficiency is rampant up very dramatically at the same time.

7:40Right. So along those lines, there's been a tension. Well, there's been a tension on the stability and durability of power grids, national, regional, local, as long as they've existed, but certainly over the past, you know, five years, 10 years or so. But since AI has come into the public consciousness, there have been news stories and what have you about kind of the localized effects of, oh, this data center was built in wherever it was and it had this huge impact on the local power grid or people are concerned it might. Can you talk a little bit about the common concerns around AI's energy consumption, particularly when it comes to the impact on local power grids, whether it's in the area where a data center might be or other places where people are concerned that AI is impacting the local energy situation?

8:33The first thing to look at when you're trying to put this in context and figure out what the local constraints might be on the grid is the fact that AI still accounts for a tiny, tiny fraction of overall energy consumption generally. If you look at it globally first, and then we'll get to the local issue, look at it globally, the International Energy Agency estimates that all of data centers, so not just AI, all of data centers account for about 2 % of global energy consumption. And AI is a small fraction of that 2 % so far. So we're looking at much less than 1 % of total energy consumption currently used for AI-focused data centers.

9:15Now, that is absolutely growing. We expect that to grow. But ultimately, this is a very small piece of the pie still compared to everything else that we're looking at. The second thing to consider is the fact that AI is mobile, especially when you think about AI training. So when you're working to train these models for months at a time, potentially very large models, you need a lot of compute power. That training doesn't have to happen near the internet edge. It doesn't have to happen in a particular location. So there is more mobility built in to how AI works than in traditional data centers, because you could potentially train your model in Siberia, you know, if it were more efficient to do that, or in Norway or Dubai, wherever you have access to reliable, clean energy, it would be easy to do your training there.

10:07And some companies, including some of our partners, have built business models around that, locating AI data centers and accelerated computing data centers close to where there is excess energy and renewable energy. So to get back to your original question, will we see kind of local constraints and problems with the grid? I think for the most part, we're able to avoid that because of those issues. AI is still relatively small, and the companies who are deploying large AI data centers already know where there is excess energy, where there are potentially constraints. And, of course, they're trying to find locations for the AI data centers where it's not going to become a problem, and they're going to be able to have good access to clean, reliable energy.

10:53So is the sort of pessimistic or it sounds like overblown if data centers only comprise 2 % of global energy usage and AI-specific data centers are only 1%, is the sort of proliferance of pessimistic stories around AI's impact on the energy grid, is that just kind of the dark side of a hype cycle that we're sort of used to and this is how it's coming up with AI? I don't want to say that those concerns are misplaced. Certainly, if you're living in a community and you see an AI data center going up, you may have questions about what this is going to do to your local grid. And we are, because we're in this period of very, very rapid and to some extent unexpected deployment of AI, because ChantGPT really took the world by storm and by surprise two years ago.

11:42There is some churn right now, which you would expect when you have a new technology, a new industrial revolution that's bursting on the world. There's going to be a little bit of time where resources are not perfectly allocated. But what we're seeing is we're already working through that phase. And the companies who are deploying the big AI data centers are finding ways to do that that are sustainable and that won't threaten local grids. even if in the near term there are some constraints that we all need to work through. In the long term, even in the medium term, we're very optimistic that these are solvable issues.

12:22So sort of to look at the bright side of things relative to AI, as with so many industries and so many problems that people are trying to solve in all walks of life, AI can be a help when it comes to optimizing grids and energy use and even perhaps trying to solve some of these climate challenges that we're all facing. Can you talk a little bit about that and about how AI can or perhaps already is making a positive impact on our energy situation? Sure. There are two examples that I'll focus on that speak to different aspects of sustainability. The first one is helping us adapt and mitigate the worst impacts of climate change.

13:07AI and accelerated computing in general are game changers when it comes to weather and climate modeling and simulation. uh nvidia has a platform called earth 2 and we partner closely with national labs and non-governmental organizations and other organizations to develop uh systems where we can much much more accurately forecast weather model weather and help mitigate the the worst impacts of near-term weather and also look longer term at climate so that we've got a better understanding of where we're going with climate and can better prepare for and plan for that. And then the other piece of the puzzle is that accelerated computing and AI, both of them, have real-world applications that directly reduce energy and emissions.

13:57And one example of that is we've transitioned the Pandas library in Python. It's one of the most well-used libraries for simulation and for high-performance computing. We've taken that basically and written libraries that will translate that code, code that's written for that library, onto the accelerated computing GPU platform. And doing that, we've basically opened up for the world of researchers a way to run their simulations in that library without any code changes at 150 times speed and many, many times more energy efficiently. So the application of this itself is going to end up reducing energy consumption and also reducing the associated emissions.

14:47Right. No, it sounds like a virtuous cycle. So to kind of dig into that for a second, you know, people familiar with NVIDIA, longtime listeners to the podcast, perhaps understand that NVIDIA is not just a hardware company. It's hardware, it's software, it's all of the tools and everything in the stack to leverage the GPUs in all of these different systems, accelerated computing, AI, and what have you. Can you talk a little bit more, you mentioned earlier some of the efficiency gains, but a little bit more about some of the efficiency improvements in AI training and inference, and then also NVIDIA's role in developing more efficient models.

15:25You mentioned Earth 2 just a second ago, but some of NVIDIA's other work in increasing the overall efficiency of the hardware the software and then these models themselves sure if you start with inference uh one data point that i like to share is that just in one generation of improvement so if you look at one generation of nvidia hardware um our ampere platform or sorry our hopper platform which is the um the one that we're shipping in the highest volume right now right and you compare that to the blackwell platform which we're releasing next um we'll come out within the next several months the blackwell platform is 25 times more energy efficient for ai inference than hopper was wow so just in the space of one change one generation of nvidia hardware and software it's using 125th of the energy that's a 96 percent reduction in energy used and there are performance gains associated as well that's right yeah yeah Significant performance gains.

16:28Yeah. Yeah, no, I understated it. But performance gains are amazing. But that's incredible. So a 96 % efficiency gain while also getting these enormous, you know, next generation performance gains as well. That's right. And you asked a little bit about how we're getting there. Yeah. That 25x improvement is through innovation in many spaces. It includes things like quantization, where we're using lower precision map, basically finding ways to optimize the model on training and inference in ways that allow us to do it even more in parallel and do it more efficiently. It includes things like water cooling our data centers so that we're using less water and significantly less energy to keep the data centers cool.

17:16Of course, it includes things like better GPU design and all of these levers we're pulling at the same time to drive those energy efficiency improvements. And we expect that to continue because energy efficiency is something that we care about and our customers care about. And it really helps enable more performance when we're able to take out waste and to be able to do things more efficiently. It enhances our ability to drive more performance and to make the AI even more valuable than it was. Our guest today is Joshua Parker. Josh is the Senior Director of Corporate Sustainability at NVIDIA. And we've been talking a little bit about energy and energy in the AI area, climate change, sustainability, all these incredibly important things that kind of form the basis of our ability to live on Earth.

18:04and how these things are being affected by all of these rapid advances in technology and obviously being fueled by the interest in AI, which is, you know, AI has been around for a while, as you well know, listening to the show. But over the past couple of years, it's really ramped up in intensity. Josh, you mentioned just a second ago customers. Maybe we can dig in a little bit to some case studies, customer examples, real-world applications of AI improving energy efficiency. Sure. One that I love to talk about is with a partner of ours called Wistron. It's a Taiwan-based electronics company. It has a lot of manufacturing that many people may not have heard of, but it's a large, sophisticated company.

18:46They took our Omniverse platform, which is a 3D modeling platform, and they modeled one of their buildings in Omniverse. And then they used AI to run emulations on that digital twin that they created in our Omniverse platform. And we're looking for ways to improve energy efficiency. And after doing that, after using that digital twin, applying AI to run some emulations, they were able to increase the energy efficiency of that facility by 10%, which is a dramatic change just based on the digital twin. And in this case, it resulted in savings of 120 ,000 kilowatt hours per year. Fantastic. A word that I hear a lot that I'm not really sure what it means, I've got kind of a working understanding is decarbonization.

19:38And my understanding is that NVIDIA has been involved in some work optimizing processes for decarbonization in industry. I think you know a little bit about that, and I think it's relevant to what we're talking about. So could you dig into that a little bit as well? Decarbonization is a really broad term, and it makes sense to have a nuanced appreciation for everything that it encompasses. But basically, I think best understanding is that And it describes our efforts to reduce greenhouse gas emissions and an effort to try to mitigate the climate change that we're seeing. So it can apply to a lot of things, including things like carbon capture and storage, where we're actually pulling carbon out of processes or out of the atmosphere and finding ways to store it.

20:25That's an area, actually, where we have some good work being done. and some partnerships with NVIDIA and Shell, for example, where we're finding ways to use AI to greatly enhance carbon capture and storage technologies. So that's one example. Other example of decarbonization, and this goes directly to emissions, is we've also partnered with California, I believe in the San Diego area, to help firefighters there use AI to monitor weather risks that could lead to wildfires. And in doing so, we've been able to expand their responsiveness, to improve the responsiveness of their firefighting efforts, and not only to potentially save lives and property, but also to significantly reduce emissions associated with those wildfires.

21:20So that's another example of decarbonization that we're seeing. And then the third example I'll give is NVIDIA itself. We are trying to decarbonize our own operations by transitioning the energy that we use from standard energy to renewable. And this year we're going to be 100 % renewable for our own operations. So I'm very excited to be transitioning over there. Quick show note to listeners if you're interested. We did an episode previously about the use of AI in firefighting and combating wildfires in California. I would imagine it's the same organization. Forgive me, it might not be, but definitely worth a listen.

21:59It was a great episode. Josh, you mentioned a little while ago data centers. We talked about, you know, being a citizen and seeing a data center come up. And of course, it's good to have questions and concerns about how's that going to impact things. The design of the data centers themselves obviously plays a big part in how efficiently they do or don't operate. You mentioned a little bit earlier water cooling as a technique that's been effective in reducing or increasing energy efficiency, I should say. Can you talk a little bit more about the data center, how data centers relate to sustainability kind of broadly and some of the innovations that have helped in that regard?

22:41Yes. The first thing, again, to mention is to put data centers in context because it's easy to think about data centers being really impactful in terms of sustainability. You see them, they're large, you hear about them using all this energy, all this water and so forth. So it's a legitimate question to ask. But ultimately, again, the IEA estimates that all of data centers only account for 2 % of global energy consumption right now. So it's much, much smaller than most of the other sectors. Not to interrupt you, but I, you know, So I've obviously been working in this arena for a while now, but that figure really kind of blew my mind.

23:19I was expecting something a little bit bigger than 2%. Yeah, and that makes sense because there's so much attention on this. AI is very much in the zeitgeist right now. We're talking about it. We see the rapid expansion. And yeah, so that's one of the things where it's important to put it in context. But yeah, the innovation in data center design is one of those levers that I mentioned that we're all pursuing to try to improve energy efficiency. And as we're transitioning to this new generation of products at NVIDIA to Blackwell, our reference design, our recommended design for the data centers for our new V200 chip is focused all on direct-to-chip liquid cooling, which is much more efficient, really unlocks better energy efficiency, of course, but also unlocks better performance because the cooling is more effective.

24:13So we're able to run the chips more optimally in ways that lead to better performance as well as to better energy efficiency. Just to paint the picture, sorry to interrupt you again. When you talk about direct-to-chip cooling, what is that replacing? What's the thing that it's more efficient than? So that's in comparison to air cooling where you have heat sinks and you're using airflow. Direct-to-chip liquid cooling, we're able to get liquid in closer to the silicon and to more effectively get heat away from that. And one of the reasons why this is so effective and helpful with accelerated computing is that the compute density is so high.

24:54If you look at a modern AI data center and you see a rack, for example, of a modern AI data center, there's as much compute power in that rack as there was in several racks, many racks of a traditional computing data center. So the compute density is so high that it makes more sense to invest in the cooling because you're getting so much more compute for that same single direct-to-chip cooling element that you're using. So obviously, all of this is a work in progress, so to speak, that advances in AI aren't slowing down anytime soon. And you're talking about, you know, the increases in efficiency and the increases in performance and all of that from generation to generation.

25:39As you said, you know, this is early days of the world leveraging AI, to put it that way. So as we like to do it on the podcast, as we move towards wrapping up, let's talk about the future a little bit. And, you know, not to put you on the spot to make crystal ball predictions, but what are some of the things that are being explored as future directions for AI and energy efficiency and really, you know, kind of supporting this growing energy demand, not just from AI and data centers, as you rightfully pointed out, but just in general? You know, how is AI being explored to meet future energy demands kind of across the globe?

26:19There are many ways and most of them are yet to be discovered and talked about because we're in such early days that it's hard to know what opportunities are yet in front of us. But some examples, again, are, for example, grid update and updates to the grid in ways that will enable more renewable energy. So when you have more and more renewable energy coming online, that is more cyclical than traditional energy. If you're burning coal 24-7, it's a steady stream of energy. If you've got wind or solar, it's more variable. And also with things like residential solar, you have times when you may be wanting to allow those residential solar panels to send energy onto the grid instead of pulling data for the house off the grid.

27:11All of those types of things benefit from modernization and modernization in a way where AI can play a significant role in helping to avoid waste and to create ways for that energy flow to be optimized. So that's one very near-term area where we're seeing progress. And a partner of ours, Portland General Electric, is using AI and using some of our products to do just that, to put smart meters on, to help them manage the growth in renewable energy. And there certainly is a perfect opportunity right now for us to do this, to engage in grid modernization, because we have so much value to potentially unlock with AI.

27:52And we've got these big companies who are really good at developing infrastructure, who are motivated to help us modernize the grid and introduce more renewable energy and do that in a responsible way. So it's a perfect time for us to be focusing on this. There are also fantastic other sustainability-related benefits from AI in terms of drug discovery for human welfare and materials discovery for things like electric vehicle batteries and batteries more generally. We've heard reports from Microsoft as well as from Google about discoveries they've made in material science that could potentially lead to much more efficient batteries in the future, which, of course, would not only save resources, but also save energy as well.

28:39Right. It's funny listening to you talk about sustainability, and it's obviously related, but it makes me think about, or when you mentioned about residential solar and being able to send solar power back to the grid and, you know, thinking about the virtuous implications of that, it made me think about recycling for whatever reason. And recycling being one of those things that, you know, individually, if we all do it, it sort of forms a collective. And then obviously, if we're moving from residential to industry and large corporations and, you know, factories and what have you, the importance of recycling, you know, is sort of bigger, obviously, in these spots in a factory than in an individual house.

29:20I'm wondering about the importance of collective action when it comes to all of these things that you've been talking about with AI and sustainability and energy demand. And then sort of dovetailing from that, what about the role of industry or even of governments in, you know, driving these kind of new and emerging best practices that will support sustainability for all of us? I think there's a great role for governments, policymakers to play here in terms of, number one, setting an example of how accelerated computing and AI can be used as tools for good. And we're seeing some great work by regulators, especially in the United States, but also in Europe and elsewhere, where there's a real appreciation of the potential benefits to society and to sustainability in adopting both accelerated computing and AI and using that for the public good.

30:17So I think that's the first way in which I think there's an opportunity here, accelerating all these workloads, transitioning them over to a sustainable platform, and then using AI to try to benefit society in an environmental way and in a social way as well. And then also to encourage that type of sustainable deployment in industry as well and make sure that we're, again, modernizing the grid and creating the environment where we have a clear path towards sustainable deployment of AI. because ultimately, you know, this is moving very, very quickly. This fourth industrial revolution is we're living through it.