AI2’s Christopher Bretherton Discusses Using Machine Learning for Climate Modeling - Ep. 220

0:10Hello, and welcome to the NVIDIA AI Podcast. I'm your host, Noah Kravitz. Climate change is a defining issue of the 21st century. Can machine learning help us model climate even though the future will not be like the past? Can it help us plan for coming new extremes of heat, flood, drought, and rising sea levels? My guest today is here to help us break it all down. Chris Brotherton is the Senior Director of Climate Modeling at the Allen Institute for Artificial Intelligence, also known as AI2. And Chris is here at GTC 2024 in San Jose, California, to talk about machine learning and the future of climate modeling.

0:46Chris, thanks so much for joining the NVIDIA AI podcast. Thanks, Noah. So let's start with the basics. Before we get into your work specifically, can you tell us a bit about AI2? AI2 is a group of about 200 people now who are primarily supported philanthropically through the Paul Allen estate, working on a diverse range of AI tasks. Some of the most important tasks include open source, large language modeling, machine reasoning, and knowledge distillation. And environmental AI is a newer focus for AI2, but now in addition to climate modeling, AI2 is using artificial intelligence to detect illegal fishing, track wildlife, and other really interesting and diverse AI challenges.

1:43So we're here to talk about climate change, and everybody's aware of it. We're in California right now. You traveled down from the Pacific Northwest. West. So we're both pretty aware of droughts, wildfires, record-breaking snowfall and rainfall. Just these extreme weather events seem to be coming one after another. Can you tell the audience what climate modeling is, how it relates to climate change, and how it's helping us all prepare for whatever's to come in the future? In the Western United States, there's always been extreme weather. We've always had floods and droughts. We've always had fires.

2:21But as I think everyone who lives here realizes, something is changing. We're getting a lot more smoke. It seems like the floods are stronger. Of course, snowpack is going down on average, but the extremes are getting more severe. And we wonder what's going to happen in the future. Well, a lot of what is going to happen is a consequence of basic physics, which we can encode into mathematics. And at the most basic level, we can understand it on the back of a napkin. But if we're interested in understanding what's going to happen in places like California, for instance, we need to go a lot further.

3:03And 60 years ago, Suki Manabi at a NOAA lab called the Geophysical Fluid Dynamics Lab built the first climate model with that in mind. And what a climate model is, a physics-based climate model is, is it's a representation of the Earth, of the atmosphere, of the ocean, of other land surface components like land and ice and so on, on a grid encoding mathematical algorithms that represent our best guess at how different kinds of processes work. atmospheric winds, ocean currents, more complicated things like how clouds are formed, even how soil decomposes vegetation by microbes into CO2 and so on.

3:50So there are all these things going on in a climate modeling model. And there's the 60 years of evolution of complexity and using finer and finer grids that have made them more sophisticated. But they are both expensive to run. They can only be run and developed at major climate modeling centers. And there's still uncertainties. They don't tell us the whole answer. So there's a lot further to go. And a few years ago, I got interested in trying to think about how machine learning might help with that. And so climate modeling used to be done with Fortran? Climate modeling is still done with Fortran.

4:32Okay. As soon as I said used to, I was like, oh no, I'm setting myself up. Right. No. In fact, the heart of many climate models still dates back to the 1970s. And in fact, it's hard to hire people to do climate model development because they have to learn Fortran. Nobody does Fortran, right? Although the Fortran of today is not the Fortran of the 1970s. It is a much more modern language. But that is a hindrance. Because, for instance, now we have computer languages like JAX, which naturally produce code, which is friendly to using with machine learning, but no climate model, no operationally used climate model is written with that friendliness in mind.

5:14And so that means that the field is going to have to be transformed in a somewhat revolutionary as opposed to evolutionary way because of this sort of legacy technology that it's based on. And so you said a couple of years ago, you started thinking about how to use machine learning in climate modeling. And so what's that been like? What have you been able to do or sort of started thinking about doing? And you said it's going to be more of a revolutionary than evolutionary process. Where, you know, both you specifically and the field in general, where are things at with bringing machine learning into climate modeling?

5:53Yeah, well, let's start five years ago. Well, well, actually a little over five years ago, I think in 2017, when I was actually standing in front of Paul Allen and proposing a project on using machine learning for climate modeling to him and a committee of experts in both climate and actually in computational science. At that time, there was an extreme degree of skepticism that machine learning could be helpful at all in the near future. In fact, so much so that our project was originally rejected and only later got revitalized after Paul Allen's passing and turned into a small venture with initially a two-year pilot period only to demonstrate some success.

6:45Wow. Things have changed quickly. Things have changed pretty quickly. And they haven't just changed in our field. There have actually been a lot of other groups that have suddenly got interest in this too. Sure. For the same reason that basically the public and almost every scientist or technically minded person is now thinking about ways they could possibly apply AI to their work. Right, right. The tools are just so much more powerful. The computational technology is powerful. And I think we've all seen how transformative it can be in some examples like large language models. So what were some of the limits, or I guess I should say are, some of the limits of Fortran-based modeling?

7:27And how are you able to overcome them or might be able in the future to overcome them using machine learning? Yeah, well, many of those limits don't actually have to do with Fortran, but they have to do with the algorithms that we use to, for instance, represent the flow of air in the atmosphere, the wind, basically. They basically have to do fundamentally with how you numerically solve equations on grids. And without describing any details, basically they require a limit on how fast a model like that can be stepped forward and what size steps it can be stepped. And for a climate model, that means that climate models written and phrased in the traditional way require time steps of a few minutes.

8:14With machine learning, we are able to actually sidestep that limitation because our algorithms are not formulated in the same way. And so we can take much longer steps of many hours. And that by itself allows the machine learning to accelerate the model by a factor of 20 to 50. Wow, okay. Even with no improvement in computation. Right. Furthermore, machine learning runs very efficiently on GPUs. And so we're able to much more efficiently use the computational resources that are available. Right. And so that actually adds to the efficiency yet further. I don't know if this is a good question to ask per se, so tell me if I'm off base here, but can you compare the size of a climate model to something like a large language model just to kind of give the, well, for me, but also to give the audience a sense of, you know, In my head, describing the wind in an algorithm seems much more complicated than describing a sentence of text.

9:18But I really have no idea. Yeah, well, of course, we're not comparing describing the globe to describing a single word or a single sentence. We're sort of thinking about it, describing all the sentences that you might make. And so it's more like, you know, the Encyclopedia Britannica versus representing the earth on a global grid. And of course, an issue is that it depends on how detailed the grid is. If you represent the Earth on a 100-kilometer grid, you might be talking about hundreds of thousands of different grid points and parameters that you're trying to advance in your model at each time step.

9:58If you're talking about a kilometer-scale model of the kind that we're actually hoping to train our machine learning on, you're talking about hundreds of millions to a billion different points. But that's still small compared to the number of parameters in a very large language model where there might be a trillion parameters and where you're definitely training on trillions of tokens of data. And so when we're talking about a climate model, how does it work kind of for the layperson to understand? Is it a single model of the Earth? Are there different models for different geographical regions?

10:37Is it something totally different? How do these models represent what someone like me might think of, right, like the planet and like climate change over time? Right. So we modularize the model into different components representing parts of the climate system. And in fact, the part of the climate system we're working to improve by machine learning in our project is the atmosphere. So we represent the entire three-dimensional structure of the global atmosphere by breaking it into about 100 layers and representing each of those layers on a horizontal grid. There are also somewhat similarly constructed models of the ocean for representing ocean currents, which are much slower but are complicated, too, and have a lot of detail.

11:29And those models interact with each other through what's called a coupler, which allows information to be exchanged between these models. Similarly, there are models of ice, of sea ice, for instance, of chemical processes, such as create the little particles that actually cause clouds to nucleate when water vapor condenses, and a whole variety of other processes. So a climate model consists of all of these different components interacting through a coupler of some kind that's kind of the, if you want, the conductor of the orchestra. Right, right. And then what are you able to see from the model?

12:14What kind of information? Is it forecasting? Is it current conditions? What kind of information are you getting back from the model? So think about a traditional climate model and actually what we're trying to achieve with machine learning as being an Earth system simulator. So it's trying to predict climate. And climate is, by definition, how do the long-term conditions look like? How are things like extremes of precipitation changing? But it's statistics of climate. The way it's getting at that is it's actually simulating the system. It's basically making artificial weather and the ocean equivalent in a changing environment that allows the different model components to talk to each other, the atmosphere, to talk to the ocean, to talk to the biology and all the chemistry and so on.

13:09But it's a simulator. And so as a result, when we do machine learning, we actually retain that spirit of trying to make a simulator, which is just predicting, you know, an artificial or pretend likely evolution of how the atmosphere and the ocean might go. And I'm a little hesitant or perhaps fearful to ask, but what are the models saying? What does the future look like for all of us on Earth? Right. Well, I think conventional climate models have told us everything we need to know, to know that the Earth is getting warmer. That has many consequences, which are very robust. It warms more land. It warms especially in the polar regions.

13:51Basically, in order to turn that around, we're going to have to get to a carbon neutral power system and economy as fast as we can. The effects of emitting carbon dioxide are cumulative. And so we know that, and we have all the information in which we need to act. And so you might ask, okay, well, why do we need climate models that are any better than that? And the reason is that, you know, if you think about society and how we generate our power, it's kind of like a, it's a huge battleship. It has many components. And it's also like a battleship without one captain. It has hundreds of captains trying to steer the ship.

14:27They have a lot of other priorities on their mind other than climate change, no matter how much they're aware that that's an issue. And so we are going to be dealing with a warming climate and all the extreme heat and more extreme precipitation and so on that it's going to create. We are going to have to plan for that. We're going to have to adapt to that as well as try to mitigate it by switching away from greenhouse-dependent fuels. And so it's that adaptation that we may be able to inform better with better climate models. Because, for instance, if you, say, are a homeowner in the suburbs of L.A., you know, an area which was recently affected by an atmospheric river that was quite destructive.

15:16You are wondering, OK, well, in my watershed, are we likely to see, you know, rainfall events that are twice this strong? 30 years from now? And if so, what do I have to do for it? You know, how do I build, or how do we as a society build our infrastructure or keep people from living in certain places or plant different crops? You know, all of those things are very local decisions. And so for many of those decisions, we need very local information because climate change might be, you know, it might affect the windward slopes of Hawaii different than the leeward slopes, for instance. And so that's where we really can hope to do better.

15:57So is localized modeling one of the things at kind of the top of the list that machine learning is able to help with? Yeah. So that's certainly part of our vision. So the idea really is twofold with what we can do with machine learning. The first part of it has to do with localized modeling, but rather indirectly. And that is that climate models are typically run on a grid of 50 to 100 kilometers or so. And the reason for that is that you have to be able to compute for hundreds of years. So you need to be able to simulate the Earth for hundreds or thousands of years with a model that you can computationally afford to do that.

16:38On the other hand, we can simulate the Earth much better with a model with a much finer grid, say two or three kilometers, so two miles or something like that. That kind of is neighborhood scale practically. And so it represents all the details of the Earth's surface and the atmosphere and individual storm systems. And so in some sense, it's a fundamentally better model. So if we can train our machine learning on models like this, which we can't afford to run for hundreds of years, but the machine learning version or emulator of the model can run for hundreds of years, many times, we can get the information, we can leverage the information out of that fine grid model to help make better predictions.

17:21So that's one of the goals. But the other thing is that nevertheless, when we train the machine learning model, we don't train it on a very fine grid. We train it actually typically on a 50 or 100 kilometer grid. But then another machine learning tool called generative AI, in particular something called diffusion modeling, can be used to take the information we have on that 100-kilometer scale and downscale it to the 3-kilometer model to the scale with the help of this fine-grid model that we use for training it. And so these two parts kind of fit together like hand in glove. The existence of these fine-grid models both allows us to develop ML to emulate them, And then also allows us to downscale that information back to that very fine scale, which people wanted at and help people in figuring out what to do with that bridge in the neighborhood that they're afraid is going to wash out.

18:17You took me back to my own neighborhood. They're doing work on the bridge right now, in fact. Yeah, right. So I guess two questions. One is sort of about your work specifically. But before that, since you mentioned talking about or we're talking about localized modeling and things that people and neighborhoods and local governments might be able to do with some of the forecasting information. What, if anything, can individuals, can small communities be doing now to prepare for, you know, what climate change has in store for us in the near and kind of medium future? Well, I mean, I think the first thing to realize is every culture, every region, everywhere is going to have to change the way that it lives in many ways.

19:02You know, we have to move to EVs, but we have to generate concrete. We have to make concrete differently. We have to farm in ways that release less nitrogen oxides to the atmosphere. It's another greenhouse gas. We have to release less methane from fracking. And so this is a problem where we have a death by a thousand cuts. And so don't pretend that someone else's technology is going to solve the problem for you. You know, think about this as a problem where we have to consistently support policies that take a holistic view of the problem. Carbon taxes, policies that basically say, you know, we will emit 50 % less CO2 by 2035 than today.

19:46if we don't do that we're not going to get to the end game and so this is an urgent problem what we do now matters a huge amount for the future so that's one thing for people to realize and we know everything about that right now and uh and i can't stress more how important it is uh to to work on the prevention uh because that is in the end the way we'll get our get to the end of this problem. But then, you know, thinking more about, okay, well, and what will more AI do for people's view of this problem? As I said, we are going to have to adapt. And the role of AI here is to basically make the amount of computational power and the technological prowess you need to have in order to get a climate prediction, climate change prediction for your backyard.

20:36It's going to bring it down to the scale where individual communities potentially can, I won't say they can run their own model, but they can find someone who can do it for a reasonable price using cloud resources, using things that are within their capability to go after. And so it will be the case that no one will be able to afford or want to make planning decisions without accounting for climate information. And right now there's a lot of friction in that process because there's a long way from the huge climate modeling centers somewhere else to the decision you have to make next week. But that doesn't have to be the case.

21:15And really, ML is a way to get there. And a feature of ML that's particularly important there is that I think we'll be able to combine these kinds of models that will tell you about climate with language models so that you can even query them with the questions you want to ask and the way you want to ask them, and they'll be able to give you sensible answers. That's really the sort of biggest vision here. Yeah. And so kind of along those lines, I was going to ask what's next for the work that you're specifically doing, or maybe kind of folded into that. Are there problems right now from kind of a science and technology standpoint that, you know, you're working on that, not that you're waiting for someone else's advancement to help you get over the hump, but are there kind of hurdles right now that you're trying to get past that, you know, can kind of pave the way for the next part of the vision?

22:12Right. So there are sort of two different kinds of hurdles. One of them is I mentioned that the Earth system is complex and the way that we model it is modularized. And And in machine learning, we find that you can get a much better answer if you can put your entire model under the machine learning hood and sort of optimize everything at once. Sure, right. But when you have a very modular starting model and some of the components are not available in some kind of machine learning form, that gets to be difficult. So one thing we're trying to do is work with our other colleagues doing machine learning, say, of the ocean, to try and at least build emulators of the ocean, couple them to emulators of the atmosphere, try to have them emulate sort of coupled atmosphere-ocean models.

23:06Because that's the minimal configuration that I think we need in order to talk about 21st century climate change. So this coupling of the atmosphere and the ocean is one big challenge that we're working on right now with colleagues like Laura Zana, who's talking later in this GTC. So, and forgive my ignorance in asking this, but is that a software problem? Is it a sort of infrastructure and hardware problem? Is it a, you know, humans kind of getting together and figuring out how to best work together problem? It's a people power problem in that machine learning software doesn't design itself from scratch.

23:48And an emulator of either the atmosphere or the ocean is a piece of software which starts with something you might be able to take from another field, like a vision model or a video model or something like that. but it has to be adapted to your particular situation and that takes experts and it takes a lot of decisions and it actually takes a lot of playing around with the model and so it's really people power so in this particular case we have an atmospheric emulator from ai2 that i think is an appropriate atmospheric component for such a model but the oceanic equivalent doesn't really exist yet.

24:31An ocean that you can play for a long time in this emulator mode and have it stay stable and accurate and represent meaningfully a climate. And so that's a real, it's a development challenge. It's sort of a mixture of science and technology. It's not really being limited by the hardware that we have right now. But once we start running this model, will get limited by the hardware because we'll be wanting to run the emulator for hundreds and thousands of years. Right. And I should mention that actually our climate model is based on the SFNO architecture of a product called ForecastNet, which is an open source piece of software that was developed in NVIDIA.

25:18And SFNO, developed by Boris Bonov here, has been totally instrumental in making our project successful. Excellent. But another thing that we're also trying to start collaborating with NVIDIA on is this downscaling issue. How do you take information from an emulator that might work natively at a 100 or 25 kilometer scale and really reduce it accurately to a local scale using machine learning based statistical techniques that are very efficient and that we can sort of apply on as little as a single GPU? Right, right. And so that's also a challenge that we're currently working on. How do you measure the, I don't know, accuracy is the right word, but kind of the performance of an emulator that's doing something like climate modeling out into the future?

26:10Yeah, that's a great question. So when people have developed weather emulators, which have actually—weather forecast emulators have been developed, and at this point, they're actually more skillful than our most conventional weather forecast model in the world after 10 days, which is remarkable since these emulators have only been developed for the last few years. So the emulators are better at forecasting? They make better forecasts than the original models. And the way that they do it is they're trained on the last 50 years of historical data blended together using these physically-based forecast models.

26:46So they still are actually still relying on the physically-based model for their initial guess. But from then on, they can make forecasts that are more skillful simply by training on observed data. Right. Now, the problem is if you take these same models and you try to run them out for longer, a lot of them drift away or they go unstable. They're not actually suitable for climate directly. But nevertheless, so one obvious way to train a climate model would be try and train it the same way. But the only problem is the future is not the present, is not the past. And one of the adages of machine learning is you should never expect your model to generalize too much out of sample.

27:30So we don't do that. And instead, our approach to emulation is we are just going to try and emulate physically-based climate models. So right now, the physically-based climate model is one that a major climate modeling center could affordably run for themselves. So we're doing nothing that is of interest to them, except that we can do it many, many times faster. But in future, our goal then, as I mentioned before, is to emulate these three-kilometer global models, which are inherently more accurate and believable, not just in the present climate, but also in future climates. And at that point, we're doing something which you can't do right now.

28:10Right. And so that's sort of the aim. But nevertheless, it is important to recognize we're just trying to emulate another model. So we inherit all of the pros and cons of the other model. You're still extrapolating, but now you're taking advantage of the physically based model to use the physical reasoning that we can and physical laws that we can build into that. and we're using that then to train our machine learning in multiple climates so that it can also do that same job of being able to predict across changing climates. Are there certain regions where it's harder or easier to forecast the climate and climate change than others?

28:56Yeah, actually, regions like California are quite difficult. I mean, I think you could ask hard versus, you know, well, what's the problem? Right. So in some sense, the problem in places like the southwest U.S. is there's a lot of variability, but it seems like the climate is getting drier. And is it getting drier? And it turns out if you query something like 20 or 30 of the world's leading climate models today about that, they don't all agree. And so some of them say, okay, the Southwest is drying out. Others of them say, well, it's not really drying out. It's not getting any wetter, but it's not getting.

29:37Right, right. And so there are regions of the world, and especially these are kind of in the edge of the tropics, in regions sort of at the latitude of California, where it's actually hard to predict what some of the trends are going to be like. Yeah. And which are also subject to a lot of extreme events. And so those actually are complicated regions. Other issues, though, are things like there's some areas which are very susceptible to, say, tropical cyclones, hurricanes and typhoons, where one extreme event can wipe the place out. And so they care a lot about something that's very rare and not so easy to simulate.