The hidden environmental costs of a single AI prompt, with Dr. Sasha Luccioni

0:00Starting a business comes with its share of ups and downs, which is why staying true to your vision is essential. a non-negotiable for Romeo and Milka Bregali, Capital One business customers and co-owners of Ra's plant-based restaurant in New York. Romeo and Milka took a leap of faith when starting their own restaurant, gutting an empty space and building it from the ground up, every pipe, every wall, every detail. But building from scratch came with a heavy financial burden, which is when they turned to their Capital One business card. With the flexibility of the card's no preset spending limit, they were able to spend more and earn more rewards while bringing their vision to life.

0:36Today, Raz's success is proof that with passion and the right support, it's possible to make your dreams a reality. Learn more at CapitalOne.com slash business cards.

0:53What I've noticed is that people have this like disconnect. They'll be like, oh, I care about climate change. I try not to take the plane. I try to take the train, whatever. And then when you talk about machine learning, people don't necessarily have this reflection about their own work. That's Dr. Sasha Luccioni. She's making the point that even the most eco-conscious people may not be aware of the environmental footprint of AI. So in this episode, we're turning our lens to the climate impact of AI with Sasha's help. I think people in general can be like super dedicated to fighting climate change.

1:28But then in their professional job, it's like, well, that's my job. And kind of like, I don't really think about it from that perspective. And so I do think that awareness is a huge part of it. Sasha is the climate lead at Hugging Face, a hub for all things machine learning. She's made it her mission to grow this awareness. Maybe you're like me. I'm generally an environmentally conscious consumer. And I use AI a lot. And I know my use of AI has an impact. but so far this hasn't affected how I use it. Then there are people so concerned about AI, they won't use it at all. Like these folks posting on TikTok.

2:07I don't mess with generative AI as a scientist, as an activist, or as a creative. And it's because I understand the environmental impacts of it. Stop using AI. It's a climate disaster. Just stop using it, which is easier than you think. Sasha is here to help with some nuance. We asked her on to do something pretty cool and super useful. She's helping us unpack the entire energy footprint of a prompt, step by step. Starting with all the infrastructure and compute power that's needed to build these AI models to the moment when you enter a prompt and get an AI-generated answer.

2:45I'm Rana El-Khalyubi, and this is Pioneers of AI, a podcast taking you behind the scenes of the AI revolution.

3:02Welcome to Pioneers of AI, Sasha. It's so great that you're joining us today. Very thrilled to be here. So we're going to get into the nitty gritty of the energy consumption behind generative AI. But before we get into all of that, I would love to hear more about your work. So you're the climate lead at Hugging Face, which just got listed on the Forbes top AI50 companies. So congratulations on that. Thank you. Yeah. So tell us more about what Hugging Face does and what your role there entails. Yeah. So Hugging Face is essentially the biggest platform for sharing AI models and data sets. And what's really exciting about it is that it's used as much by big tech companies as by kind of academics and nonprofits to share models and to build upon each other's innovation.

3:50So for example, you take like a big model trained by Meta and then you tweak it to your context, your use case, and then you reshare it with the community. And my job and all that is to evaluate the environmental impact. So energy, for example, a project that I've been leading recently is creating AI energy score ratings to help people pick, like if they want to do a given task, whether it's image generation or I don't know, whatever, object detection to pick a model that's the most efficient for that task based on essentially how much energy they use. So essentially to guide people in making sustainable decisions.

4:21So I've been thinking a lot about AI benchmarks. And today, a lot of the benchmarks are focused on how smart, how accurate these models are. So for example, you know, all the various GPT models have already passed the bar exam, which is the, of course, exam for lawyers. It's also passed the medical exams. And I think it's really interesting because nobody's paying attention to other aspects of AI, but of course you are, and you kind of referenced the AI energy score. Tell us more about what that is, how you built it, and maybe some of the results so far. Sure. People essentially, when they're picking AI models or even training AI models, they tend to focus on these technical benchmarks, but there's also so many other aspects to that because, for example, if you have a model that's, yeah, sure, like 99 % accurate, but it uses 10 times more energy than this other model that's maybe like 98 or 97 percent accurate, but it's a lot more efficient.

5:14And also we did give like absolute measurements, like kilowatt hours, but people don't really know what that means when you give them the actual measurements. So that's why we started doing relative comparisons. And so essentially we tested hundreds of models and we're ranking them instead of giving kind of the absolute value per query. That's very cool. What are some of the models that are most energy efficient out there, if you can share some. What's interesting is that, so there's really a trade-off, like for example, recently we, I mean Hugging Face trained a couple of small LM, like small with an O.

5:48Small LM as in S-M-O-L are a collection of small language models that Hugging Face developed. They're really energy efficient, but like training took a little bit longer, but actually like it It enabled converging upon like an end model that was super efficient. And like for a smaller model, you can get a lot more performance, which is really cool. Compared to large language models, these models can be run locally and reduce inference cost. They also outperform other models in their size category. And conversely, everyone was like, oh, DeepSeq is like it's actually a win for the environment, whatever.

6:27but and and sure like the training of the model must might have i mean supposedly took took less time but the um the ensuing model like the result of that training is massive like it takes like i don't know four gpus just to load it into memory and so you know there's a trade-off here because like sometimes you need to kind of train a little bit longer use more resources up front but then you have models that are that can be used by the community or vice versa yeah you have these huge like reasoning models that are like essentially unattainable for the average person without access to like a supercomputer.

6:59Basically, there are lots of trade-offs to the AI models out there. Some take so much power to train but are lower on inference costs. And then other models could never run locally because they're just too big. It's not always clear-cut what is the most energy-efficient model because it all depends on what you measure. The training, the deployment once it's in use. But we definitely do know that some consume a lot more energy than others. For example, Hugging Faces LLM, Bloom, is trained on less data than, say, GPT-3. And in this case, it needs less compute and consumes less energy. But why exactly does AI, and specifically generative AI, take so much energy?

7:45After a short break, we're digging into the energy life cycle of an AI prompt. From before you even write your prompt, to when you get your AI-generated answer. Stay with us.

8:26Atlassian's teamwork collection. It has planning in Jira, documentation in Confluence, video updates in Loom, and now AI agents in Rovo, which connects the dots across your work so nothing gets lost. It's one AI-powered teamwork platform designed for how modern teams actually build. Learn more at Atlassian.com slash TeamChanger. That's A-T-L-A-S-S-I-A-N dot com slash TeamChanger.

9:01Let me set the scene for you. You open an AI chatbot of your choice and you enter a prompt. Something like, write a thank you note for my doc sitter. You hit enter and that starts a process that uses a lot of energy. We're talking 20 to 30 times more energy than a regular internet search. And we're going to get into why. I think when the average user prompts an LLM, it's like a black box, right? It's like disconnected from what actually happens when it spits out an answer. And I think a lot of people don't realize that AI exists in the material world, right? It's not just ethereal. So I want to spend the bulk of our conversation kind of looking at this lifecycle from start to finish.

9:46And I want to start with AI infrastructure and particularly step one, which is we need AI chips to both train and deploy these models. What are some of the energy and environmental considerations when we are manufacturing chips? So chip manufacturing is actually also a black box. We have very, very little information. So like NVIDIA, the main maker of GPUs, hasn't yet shared any like specific numbers. But we do know that there's a lot of rare metals that get used in that process, like gallium and germanium and cobalt. Just like mining those metals, like extracting them from the earth. It's a lot of pollution and it's just like resource intensive.

10:22So there's that. There's also the energy it takes for the actual like connecting all the different components. And it's very, very precise work. And you need like really high powered tools. And all of the fabs, as they call them, are actually based in Taiwan. FAPS, as in the specialized factories where chips are produced. Not all of them are based in Taiwan, but for such a small country, it definitely houses a significant portion of these facilities. Most of Taiwan's energy comes from fossil fuels. So you know that the energy being used is also pretty carbon intensive. And then the water, there's a lot of water that has to be used for purifying every little layer of silicon that gets connected into the chip.

11:05And so every like essentially it's like millions of liters of water that get used by these fabs as well. And like a couple of years ago, there was like a drought in Taiwan and the government actually had to make a choice between like farmers planting crops and chips in general being produced. And then they chose the fabs and then the farmers couldn't plant their crops. Yeah. What about e-waste? All these chips that are becoming obsolete or outdated, where are they going and how do you think about the consequences of that? Yeah, there was a recent study that came out that said that in the next couple of years, we're going to see like tons and tons of e-waste, specifically from AI.

11:43Part of that is because people want the latest and greatest. So essentially, it's like, whereas before you could use a computer for 10 years, now it's like you always want the newest GPU, the most powerful. And so you'll switch it out every three years or two years. And so that really adds up. And also, we don't have very good ways of recycling all of those electronic components like cell phones, computers, servers, what have you that we're just not very good at. It just takes a lot of effort and human time in order to kind of get all the different components. It's definitely an imperfect process.

12:13And a lot of that just goes to waste. Goes to waste because old chips are being replaced by new, more powerful chips. So, assume your prompt asking for a thank you note to your dock sitter is processed by new shiny chips that took a lot of power to make, from mining to manufacturing. Now, we're ready for our next steps in the AI lifecycle, training and deployment of these AI models, which happens in data centers. These data centers aren't only energy hungry. Sasha says they also need... Millions of liters of water. We don't have any specific numbers per data centers, but for example, Microsoft did provide some numbers like in general for Azure.

12:56And we're really talking about the equivalent of like a small town. And I mean, data centers are actually huge. It's like a football field situation filled with computers. It's so loud. You can't talk. You have to wear earplugs. It's hot. Like if you put your hand near the actual servers, they get they're like super hot. They're like you can burn yourself hot. And then they have these pipes running through between the servers and then cold water gets brought in and then it goes through these circuits. It's actually like it's overwhelming. Like visiting a data center is massively, it's a very massively overwhelming experience.

13:32Yeah, that's crazy. You know, are there like creative innovations happening around, you know, alternative cooling mechanisms? So the problem is that water is a really good coolant. That's why it's being used. But of course, it's scarce and there's all sorts of issues, especially if you're building data centers in Arizona or Texas. Like there's not a lot of water to begin with. But yeah, people are doing all sorts of like improving the efficiency. Like sometimes they use other liquids, liquids other than water. Sometimes it's air. And essentially, yeah, there's a lot of like innovation happening.

14:03But data centers are such huge investments and they take a lot of time to build. And like you're not just going to like randomly, it's not like a pair of shoes. It's like you have to change. the whole like circuitry. And like, for example, you can't use seawater because the salt like corrodes the pipes or whatever. And so it's like, they are working on things, but it's definitely going to take a while. So in the meantime, for the most part, it's water cooling these data centers. Before you can even enter your prompt asking for a thank you note to your doc sitter, the AI model needs to be trained on lots and lots of data.

14:35For example, Bloom, one of Hugging Face's models, was trained on 46 natural languages and 13 programming languages. In total, 1.6 terabytes of data. Okay, so now let's move on to the model training step. So we've got the GPUs, we've got the AI chips, and we're going to put them to work. What's the data on emissions at this stage? So essentially, that's kind of the bulk of where we have the information. We have numbers about training AI models. So we know that it goes anywhere from a couple of tons of CO2 for kind of smaller, large language models to, you know, 25 tons of CO2 for a model like, for example, Bloom, which I worked on, and up to like 500 tons of CO2 for models that are bigger, trained for longer with like non-renewable energy, essentially.

15:28Hmm. So I read in one of your reports that training an LLM with 213 million parameters is responsible for CO2 emissions. It's roughly equivalent to the lifetime emissions of five cars. And then just for reference, GPT-3 has 175 billion parameters. That's a lot of cars. Yeah. So that was like the initial number that was provided in a study by Emma Struppel and her colleagues. And since then, we've been kind of getting more information about what are the factors that influence that number. And, for example, so a couple of years ago, we looked at what part of the overall footprint is like the actual GPUs doing the training and what part of it is like the overhead, like the heating and the cooling and the data transfer.

16:14Because all of that actually also plays a role. Like we don't think about it because we seem to focus on like the actual active training part. But there's like this huge data center that has like all these heating and you have all the storage and the Internet and all of that that also adds up. And so we kind of did a lifecycle assessment, kind of like what we're doing now. And we found that if we try to kind of count all of this, then the resulting numbers will double from what you thought initially. Wow. OK, let's unpack that a bit. So, yeah, it's one thing to measure like the CO2 emissions of the model as it's being trained.

16:48but you're saying there's also additional considerations where if you're like drawing on data that's stored in a different server or yeah, I don't know, like what else? Like what are the other things that are happening while you're training a model? There is the storage, the input output. Now people will try to train across different cloud compute servers. And so there's a lot of data transfer and logistics going on. And actually, if you're looking at one model trained like on a specific cluster, for a specific time, like you can get an idea of these things. But nowadays, like so much of it is distributed, so much of it is dynamic too, right?

17:25Yeah, exactly. And so it's so hard to get like exact numbers for most of this. Yeah. Where do you think the innovation is headed? Is it in building these like smaller language models that require less compute and less data? Or, I mean, I don't really know if there's a lot of innovation happening in these like ancillary parts of the problem? I think the innovations are coming from different parts. Like people are working on smaller models and like kind of like the small LMs I mentioned. People are working on distillation on different techniques that kind of make the model smaller for when you're deploying them.

18:01People are working on, like the thing is, for example, GPUs, which are the ones that we use for most of AI training, weren't actually made for AI. They were made for like graphics and video games. And so there's all sorts of people who are working on like specific hardware to be more like customized for AI for both training and deployment, which I think is interesting as well. There's all sorts of really interesting innovations going on, actually. All right. I think also a lot of people think about training as a one-and-done thing. Like you train the model and then you're done. But people don't realize that once you ship a model, you're on to the next.

18:38You're continuously training the next version of this foundation model. Um, so do you have any additional data on the emissions of this continuous training? It's really hard to get any numbers. So the thing is like for training, you can kind of, it's relatively tractable because it's like, there's only so many labs. There's only so many people training models from scratch. Right. But when it goes to like fine tuning or, or adapting, there's so much going on and some people will, you know, make a major overhaul. Some people will just change the model slightly. And so there's just like a lot, a lot happening there.

19:07And I think that, like, so for example, Allen AI recently did Almo, their model. And I think that there they talked about, like, the different steps. So I think, like, it's starting to be a thing where you don't only talk about training. You also talk about fine-tuning and you try to have more granular numbers. But overall, what we've seen so far is really, like, a huge focus on training only. Yeah. Do you get a lot of pushback from companies sharing their – because I imagine with your energy AI scorecard, You can do that for inference, but not for training, right? Because training is kind of a, you know, unless the company decides to share data with you, you can't get access to that, correct?

19:50Exactly, yeah. And you can't run trainings from scratch because it's so, yeah, it takes too much resources. Now, we've added in the energy used for how an AI model is trained and fine-tuned. and it's ready to power up and give you its best effort at writing that thank you note. This is the final stage of the lifecycle, deployment or inference. Training is one thing, but now we've deployed these models. So every time you prompt a model, the question actually for most models, a lot of the models are not local models. A lot of the models run in the cloud. So the question travels all the way to some data center somewhere and it gets cranking at the answer.

20:32And I guess you're saying the energy cost there is not always obvious. No, not at all. We don't have any information about that. AI models, especially large language models, are seen as like products or commodities. And so people have given less and less information. And nowadays it's like they don't even want to say how many GPUs they used or how long the training time was. It's crazy because it's like for me, it's not a secret. Like, why would it be so secretive? But yeah, companies are really cracking down, sadly. And that creates a lot of like urban legends and kind of a lot of like, I don't know.

21:09Yeah, like people do all sorts of like back of the napkin calculations. And then those kind of start having a life of their own. Yeah, life of their own. Exactly. Like I try to avoid that because like the question that haunts my nightmares is like, how much energy does each chat GPT query use? And the thing is, we don't know. They've never given us a number and I don't think they will. And most people use like a proxy number that somebody estimated. And now people are like, oh, that's the actual number. And then it becomes like this whole like snowball effect of like, oh, let's compare ChatGPT to Google.

21:36It's 10 times more. But both of those numbers, people just invented them. So like, what are we even talking about? Yeah, so interesting. Different prompts and different AI tasks have different energy costs. So, for example, if you're prompting ChatGPT for, I don't know, a dinner suggestion, that's going to be very different from an energy perspective than if you're generating an image or a video or even like my son's been using all these AI tools, he's 16, to generate like research, right? So he uses OpenAI's Deep Research Agent. What kinds of deployment tasks cause the most emissions? And again, like, how do we even know?

22:15Well, so in the work that I've done, we found that image generation tasks, like generation tasks in general, like generating new content is more energy intensive, uses more compute. image generation is more energy intensive than text generation, which makes sense because for an image, you've got like pixels, you've got essentially whereas a text is kind of like more 2D. Of course, the length definitely plays a role as much of the length of the input as the output. But it's interesting, sometimes it's like, well, for a reasoning model, right, you'll ask a question and then the question is short, the input is short, but the output is super long.

22:48And then in case of deep research, it's like the input could be really long. And then the output is like a sentence. But we don't have enough numbers. Like I'm actually working on a follow up to fire hungry processing that's going to look specifically on input and output lengths. And I want to figure it out because what we did in the first study was kind of like to establish ranges for different tasks. But now we can go deeper and figure out like what the variance is. So let's say the prompt is complete and so is the energy cycle that goes with it. You have your thank you note ready to send to your dog sitter.

23:20Mission accomplished. We wish we could give you some neat and tidy answer, like, I don't know, like the thank you note equaled one light bulb running for 15 minutes and three bottles of water. But the reality is, we just don't have the data we need to make those calculations. If this is important to you, let the companies behind your favorite AI tools know and ask for answers. After a short break, Sasha gives practical advice on how we can be more savvy users and looks towards a more hopeful future. Stay with us.

24:08Meet Nicole Nicholas, Capital One business customer and co-owner of Ansett Uncles, a plant-based restaurant and community space in Brooklyn, New York, that got its start from a need for unity. The inspiration, it was born from the desire to create a space that felt like home, where we can connect community culture, good food, and come together with family and friends. That's how we birthed aunts and uncles. Nicole and her husband, Mike, were fulfilling their dream of bringing people together out of their home kitchen. But they soon learned that the demand for community was greater than they knew.