ITIF's Daniel Castro on Energy-Efficient AI and Climate Change - Ep. 215

0:10Hello, and welcome to the NVIDIA AI Podcast. I'm your host, Noah Kravitz. Before we get into today's episode, a quick note. If you're interested in attending NVIDIA's GPU Technology Conference, GTC, this March in San Jose, California, and exploring the cutting-edge technology shaping our future, we have a special offer. Use the discount code AIPODCAST when you register for the conference at nvidia.com slash GTC, and you'll receive 20 % off your registration. That code again is AIPODCAST. Today, we're talking about the big stuff. AI is reshaping, well, pretty much everything. It's why we do this podcast.

0:51Climate change is also reshaping a lot of things. It's defining our era as potentially impacting every living creature on our planet. So today, we're joined by Daniel Castro. Daniel is vice president at the Information Technology and Innovation Foundation, ITIF, and director of ITIF's Center for Data Innovation. and he's here to talk about the intersection of AI and climate change. With his extensive background in IT security, management, and policy, Daniel has been a pivotal figure in discussions of technology's societal impacts. Today, he'll discuss the findings of his AI Energy Use report, focusing on the role of AI and GPU acceleration in enhancing energy efficiency and sustainability.

1:34Daniel, welcome, and thanks so much for taking the time to join the AI podcast. Noah, thanks for having me on. So could you start by telling us a little bit about the motivation behind the AI Energy Use Report and what you aim to uncover through your research? Thanks for that question. You know, I've seen, I think, probably the same headlines most people have seen over the past year, which suggests that as we're seeing the deployment of AI, the increased use of AI, maybe we should be very worried about its energy consumption. And so what's interesting to me is, you know, we've heard this type of concern before in the digital space.

2:14We heard this concern when Amazon was taking off and there were headlines about, you know, every time you buy a book, a lump of coal is being burned. There are concerns about, you know, everyone streaming online and what that meant for the amount of energy that would be consumed. And so in each of those cases in the past, it turned out the energy consumption figures early on were misleading and they were ultimately wrong. And I wanted to know with AI, you know, we know this is a fundamentally, you know, revolutionary transformational technology, what's actually happening. And so, you know, that's what I set out to uncover.

2:53And, you know, it's really a fascinating look at how conversations around energy use of digital technologies often get, you know, a little hazy when you don't follow the science. And it's interesting how it shapes where policy debates end up. So what did you find out? So what I found out was that there really were one or two early studies that estimated that future energy use from training AI systems could be enormous. And that those early studies is what has shaped so much of what people think about AI's energy use today. But it turns out those studies were wrong. In particular, this first study that so many people attended the site that came out in 2019, it estimated that training an AI model had the equivalent carbon emissions of 300 round-trip flights from East Coast to West Coast.

3:53And it turned out that that estimate was simply wrong. And it was wrong for a number of reasons. It was wrong, first of all, by a factor of about 80. But it was wrong because, you know, they hadn't had the right numbers. They had to make a lot of assumptions that turned out to be incorrect. It was misleading because it was also kind of not the AI models that were predominantly using. So, you know, the study looked at three different AI systems. One system, you know, very similar to the large language models we're, you know, using today that are very popular. Another that is not at all like what we're using today.

4:30And it was the one that is not at all like the ones we're using today that had this massive energy use. Gotcha. And there were just a number of other factors as well. And, you know, it turns out, of course, that what we've seen over time is that, yes, you know, there are some AI models that use a lot of energy. But what we also see is that over time, there's continued investment in how do we make these more efficient, whether it's through improving the chips, improving the algorithms, optimizing models in novel ways, or sometimes just moving to different types of algorithmic types of systems. And so what we see is that over time, performance improves and energy use either stays the same or sometimes even declines.

5:17And, you know, again, the question, of course, is where are we going with this? You know, what will this end up looking like in a few years? And while we don't know for sure what that looks like, what we do know is that the early estimates are entirely misleading. And so you said that that early study you mentioned was from 2019. so roughly three to four years ago now, I know as sort of an end user and observer of AI that we've seen tremendous acceleration in AI's capabilities. And as you alluded to, the use of LLMs and other generative AI technologies in particular have really exploded in the mainstream, the public consciousness.

5:58What's been happening on the energy consumption and energy conservation side? Has it been the case where, as you mentioned, sometimes happens, the performance has improved, but the energy consumption has stayed the same or even gone down a little bit? Or are both things sort of trending up over the past few years? Yeah, well, there's definitely multiple factors at play here. So, you know, the energy used to train a system, I mean, some of that has gone up as we're talking about, you know, training for weeks or months instead of days, right? So certainly energy costs have gone up there. But what we've also seen, though, is that, you know, sometimes emissions have gone down or significantly lower for certain systems.

6:46So, for example, there's one system, Bloom, that was trained in France primarily on energy that came from a nuclear facility. Very different energy profile, carbon emissions profile than one that's trained on a coal-burning power plant. And so these are the factors that I think are often left out of these conversations. And so, you know, for example, when you look at kind of where the trends are, well, the trends are that most of this, you know, these large models are being trained in data centers run by some of the largest tech companies. These are the same tech companies that have made incredibly bold commitments to clean energy use and to carbon offsets.

7:27And so in terms of, you know, the net impact, it's actually, you know, pretty good in terms of where this is happening. And so when you look at some of the reporting, you know, the macro trend lines, for example, with a company like Google, their energy use, it's gone up somewhat over time. You know, it continues to grow as they are a larger company that's grown over time. But the amount of energy they're using for machine learning as a percentage of their total energy footprint, it really hasn't changed significantly. It's been in this 15 to 20 percent for the last three years where we have data.

8:01And so, you know, that shows that even as machine learning is growing and it's, you know, kind of eating the world, well, yes, but we're seeing optimization, of course, happen as well. And why does that happen? Well, there's economic factors as well, right? It can't grow so fast and so big because even the largest companies simply can't afford to do that. So that's where we have to remember there are these, you know, balancing factors here. This might be a slight tangent, but you mentioned the importance of where the energy is coming from and the difference between training a large model using energy coming from nuclear power as opposed to coal burning power, for instance.

8:40I've noticed as a consumer, I've noticed, you know, over the past six months to a year, more and more on my phone in certain apps, I'll see something I think labeled a grid forecast and kind of mentioning, you know, when the power is kind of more likely to be clean versus not clean and that kind of thing. This feels sort of like a common sense question, but I want to ask it anyway, because you're here, you're the expert. How big of a factor and how much consideration or how important is it that consideration is being given to where the power is coming from and kind of shifting these things along with just those larger, you know, sort of headline grabbing numbers of, you know, the raw amount of energy being used to train a model?

9:24Yeah, it's a huge factor. And it's probably one of the most important things we can be thinking about, you know, as we're talking about, you know, growing data centers and where we're training and also, you know, using the models, one of the things that, you know, I didn't know, again, before I set out and really dug into this, was how much energy use was on the inference side, not on the training side. You know, based on the, you know, again, many of the headlines, people say, oh, training the model is so big, but really, you know, it's inference that's going to be anywhere between 60 to 80 percent of the energy consumption over time.

9:58I mean, it's huge. And so, yeah, I mean, where you're training the models, but also where you're using the models has a huge impact. And then the energy source And I think that's why, you know, as we think about where AI is going, there are some really interesting questions about how will it be deployed? Because deploying it on, you know, a battery powered, you know, endpoint device is very different than using it in a highly efficient data center. And so there's lots of, I think, opportunities to figure out how we optimize different deployments of AI in ways that are serving the broad public interest in the best ways.

10:36Right. GPUs obviously have been a huge factor in this AI explosion in recent times and the current way that AI is used, to put it that way. How has GPU acceleration transformed the energy efficiency of AI tech? And particularly, if you could talk about the impact in weather and climate forecasting. Yeah. Well, I mean, obviously, we wouldn't have the current generation of AI systems without the advancements we've seen in GPUs, right? They wouldn't even be here. And what we've also seen is that when you look at some of these various AI models, different types of classifiers, for example, the efficiency significantly improves over time.

11:22And that's generally because of two factors. One, because of improvements in the hardware, and two, because of improvements in how they're optimizing these AI models based on the improvements in the hardware. And so, you know, that's where, you know, when we look at over time, significant efficiency growth. But there's also a question of how are they going to use this technology to, you know, address the overall efficiency of the grid and overall efficiency of, you know, the data centers as well. That's where, you know, for the data centers, we've seen, I don't know that I'd say it's peaked, but it's definitely, you know, we've seen some really solid numbers in terms of, you know, the power efficiency coming out of these data centers, in terms of cooling, in terms of everything.

12:12I mean, they're highly optimized. In terms of the grid, I mean, that's where I think, you know, there's still a lot of opportunities to start using AI to, you know, increase grid resilience, stability, you know, all of the factors that are going to be key towards, you know, turning this technology and using it to actually have an impact in people's daily lives. Are there any specific case studies or examples you've come across talking about GPU technology, NVIDIA's GPU technology has really improved the energy efficiency of, you know, data centers or other AI operations? What I was looking at in this paper was really, you know, what's the net impact on the net environmental impact, right?

12:54And so when we're talking about energy consumption, we're talking about energy consumption because we're concerned about, obviously the carbon emissions there. And so we're also concerned, well, what's all this AI being used for? And how are these uses going to have a potentially positive or negative impact in different areas? And so one thing that we looked at is the substitution effect, right? So are we using AI for things that, you know, activities that would otherwise be generating more carbon emissions. And so, you know, you think about the classic things that people are using Catch-Ept for, right?

13:31They're using it for writing text, and they're using systems like Dolly to produce images. So the comparison, you know, you're not going to get rid of a human. You know, you might substitute a human that would be working in that, but the human's still there. Human's still kind of living and breathing. So, you know, the human's still doing the carbon emissions. But you can say, well, if a human was actually completing this task, what other energy would they be using? So if I'm, you know, writing a, you know, a one-page document, you know, I'm going to be spending an hour at my desktop or laptop, you know, typing at the keyboard.

14:02And we know what the energy profile is for that computer, or we know what the energy profile is for, you know, using Adobe Illustrator and, you know, drawing something there. So that's where we can see, okay, you know, that's a factor of anything from 30 to 80 more carbon emissions by having a human do this manually on a computer versus having AI generate it in a couple of seconds. So there is one trade-offs. Sorry, just to be clear, that the human is generating 30 to 80 times more emissions than the AI system would. Okay, got it. Yeah, the human's generating 30 to 80 times more. So using AI in this way, positive substitute.

14:41But then there's also the bigger question of, okay, well, what are all the industrial uses, right? What are all the ways that companies are using this technology to, you know, just optimize their operations, right? And that's where, yeah, I mean, there's these, you know, big opportunities, for example, in California, the government figured out, you know, they have all these fire watch stations and you used to have to have someone out there, you know, looking to see where is, you know, a forest fire, where are they starting and how can we intervene sooner? Well, you know, using cameras and using real-time surveillance of this, you can detect it much faster and obviously have full 24-7 monitoring.

15:18So that's the type of opportunity, well, that's all carbon that would be burned up and released, right? Where there's a huge positive potential impact. And I think we see many scenarios like that, right? Whether you're talking about optimizing transportation fleets and traffic optimization, as I mentioned, grid efficiency, government operations, There's a lot of opportunities like that to start using AI in ways that will pay a positive dividend in terms of carbon emissions. And obviously, if we want to address climate change, we need to start doing more of that. Sure, absolutely. This may be too kind of abstract or I'm trying to ask too much with a single question.

15:58So, you know, redirect me if that's the case. But when we're talking about AI and, you know, on both sides, training and data centers and everything. And then obviously inference and comparing, you know, an AI doing a task to a human doing a task and that kind of thing. And then we're talking about sustainability, which is, you know, kind of a big word and maybe in the public consciousness, a little bit of a buzzword that, you know, you kind of understand, oh, you know, solar power and not using fossil fuels and that kind of thing. But actually, I don't really know what it means at the detailed level.

16:31How do you think about sustainability and AI? And, you know, is it a matter of kind of these sort of base computations you're talking about with comparing, you know, the carbon emissions of an AI system versus a human doing the same task? Are you able to sort of factor in things like, well, if we use AI to help us figure out how to make better renewable energy sources and better battery storage technology or whatever things go into sustainable, renewable energy? Like, can you factor in AI's, you know, impact and accelerating these advancements? Like, how do you kind of make sense of all of that in the context of sustainability?

17:14Yeah, I mean, the simple answer is it's incredibly complex. So we can, you know, you can't draw a straight line from one thing to the other. I mean, we know, for example, that when you make things easier, you do more of it. When you make things cheaper, you do more of it, right? So, I mean, AI makes certain things easier. We're going to use it more. And that's not necessarily, though, a bad thing, right? I mean, the question is, what's kind of the net impact? And so I think in all of these cases, what we want to see is, you know, are we kind of putting the foot on the accelerator towards using AI for sustainability?

17:49So, you know, are we using, you know, more of those smart thermostats? Are we using, you know, AI more for integrating, you know, distributed energy sources? I was just seeing something recently about, you know, recycling and how hard it is to get, you know, the human workers that need to go through these, you know, recycling centers and sort all these different things. And, you know, that's a great example where AI is really good at it, right? Using, you know, computer vision, it can identify which thing needs to go in which area and using robotics, you know, process this much faster. So that's where I think we need to be kind of leaning into how AI can be part of the solution to sustainability.

18:30And then, you know, at the higher level, we always will have this question about, you know, as we are using data centers, as we're building more data centers, how do we make sure we're doing that the most efficiently and using, you know, clean energy sources? And again, that's where I say, you know, I'm glad it's, you know, big tech that's leading in this space because they're the ones that are making these bold commitments to, you know, by 2030, by 2050, you know, not just carbon zero. Some of them, some of them are, you know, lifetime carbon zero. So they've been going back and retroactively erasing their carbon footprint.

19:05I'm highly optimistic that, you know, they've made these commitments. They're going to stick with them even as AI increases. And so I think the real question is, as so many other sectors of the economy start using all this technology, let's make sure they are also making these commitments and they're using it in ways that have this positive impact. I feel like I'd be remiss not to mention that, Dan, you mentioned both wildfire prevention and recycling as two areas where AI is having a positive impact. And if you're listening and you're interested, we've done podcasts on both of those subjects.

19:42So I encourage listeners to check out the archives. They're both great episodes. And you can learn more about how AI is helping on both of these fronts. But right now we're talking to Daniel Castro. Daniel is vice president at the Information Technology and Innovation Foundation, and he's also the director of ITIF's Center for Data Innovation. So you mentioned the role of big tech in kind of taking the lead and committing to renewable energy and reducing carbon emissions and carbon offsets and all of these good things. But sort of on a broader level or maybe kind of shifting from industry to sort of the government level, what policies or frameworks do you think might be necessary to encourage the development of AI-efficient tech?

20:24Is it, you know, obviously the private sector and the public sector don't exist in their own bubbles. There's all kinds of, you know, work together that happens and battling over things and cooperation and all that good stuff. But do you see anything emerging as, you know, sort of the right way to approach policies and frameworks to address energy efficiency? Absolutely. I mean, there's a lot of talk right now about how do we regulate AI? Should we regulate AI? What's the best way to do that? And one thing that... If you have ideas on that, you know, go ahead. Any ideas there, but on the energy side specifically, I mean, one of the calls is, can we have more transparency in models?

21:06You know, can we know things about the models, whether it's, you know, the type of data they've trained on to, you know, how they should be used, you know, questions about bias, all of these things. Well, one thing we can also do is have energy transparency standards. Simply put, you know, this is, you know, the estimated amount of carbon emissions in training this model. And this is the estimated amount of carbon emissions in using the AI model for a, you know, kind of given scenario. So that it's comparable, right? So that when different potential users are making their decision about which AI model they want to apply for a given problem, they can consider more factors than just accuracy and speed.

21:46They can also take in consideration energy profiles. Making me think of those stickers when you buy like a hot water heater or something and you've got the estimated cost and estimated electricity use for a year, that kind of thing. You know, I think a lot of companies care about it at the end of the day because they care about, one, you know, some of their net zero costs. And if they're paying for offsets, they're going to care about that way. They care about their energy costs. And it's just something that I think a lot of responsible businesses are paying attention to. The second thing I'd say is, you know, this doesn't have to be new laws and regulations.

22:19Some of this can be voluntary agreements between government and the private sector. The White House recently got a number of companies that were doing that produced foundation models to agree to certain, you know, red teaming privacy standards. It can get them to agree to certain energy, you know, again, transparency around here. One of the things that I think we have to remember, though, is that some of the other types of policies and requirements the government has thought about, you know, government regulators have thought about imposing on AI systems, whether it be privacy or certain kind of safety requirements, all of those have energy costs too.

22:57And they aren't thinking about them. And so we have to make sure that we don't get in a world where the policies that are coming down actually have a high energy cost that nobody's thought about. And that's where companies are kind of forced into using, doubling their energy costs when maybe there was a better way to optimize that. Yeah. One step forward, two steps back. What about the public's role in all of this? When you mentioned recycling, it made me think about, I live in California now. I used to live and work in New York. And I remember back in my first days of working in an office back in the late 1990s, hearing, and it was a rumor, but this horror story that I was like, well, which recycling bin do I put the Xerox paper in?

23:42And somebody said, it doesn't matter. None of it gets recycled. They just throw it all out at night anyway. And I say that just to say that, you know, your example of recycling facilities and how it's much more complicated and difficult than people might think about to actually go through and sort the different types of recycling into the proper places. If we're talking about energy consumption and using AI and to your point, you know, I do a lot of writing and I found generative AI tools to be super helpful in my writing process. But I'm not thinking about, you know, the energy costs and the carbon emissions of doing these things.

24:19What should the public's role be in this? And maybe more importantly, what kinds of initiatives might be important for educating the public and kind of forming a public industry partnership, if you will, when it comes to the potential of AI for energy efficiency alongside of productivity? Well, I think in many of these cases, it always comes back to, you know, costs. Consumers are motivated by costs and they're responsive to costs. And you want to make sure that, you know, the energy costs are not an externality in any of these decisions. You know, if somebody, you know, the reason somebody isn't going to, you know, leave their computer generating, you know, thousands upon thousands of new images is going to be because of the cost, not because of their, you know, their kind of turning the lights off to be responsible.

25:08It's because they have to pay for it and, you know, and they should have to pay for it. And, you know, that should drive efficiency and responsibility. and i think the the only area where we've ever seen digital technology and the energy use kind of deviate from maybe a responsible or desirable outcome has been in you know cryptocurrencies and that was driven by speculation right and that you know so it was rational actors kind of acting irrationally because of speculation yeah and you know we're not going to have that same thing in AI because there's no speculation around, you know, the randomly generated text or randomly generated images.

25:47People are going to be motivated by the economic factors. And I think here is where, you know, we just, you know, part of digital literacy is going to need to include AI literacy. We have to have people using the technology responsibly, understanding how it works, understanding, you know, how everything's interconnected, but also hopefully being optimistic and, you know, willing to use this technology when it does have a positive impact. I mean, I ideally want to see more people saying, you know, why don't I have a smart thermostat? You know, why don't I have, why aren't I using, you know, AI to improve navigation and reduce the amount of traffic on the roads?

26:23You know, we want to see people embracing this technology. So I'd say, you know, to the extent that education can also help, you know, just lower those, barriers, the resistance to adopting new technology, that's probably going to do the most towards having a positive long-term impact, AI's overall impact on our climate future. So you kind of touched on this just now. And so in some ways, this question is, how do you envision the future in terms of the interplay between advancing AI technology, energy sustainability, and climate change? So I think AI is going to be a core part of how we address climate change and how we build a more sustainable future.