Controlling Fusion Reactor Instability with Deep Reinforcement Learning with Aza Jalalvand - #682

0:02All right, everyone. Welcome to another episode of the TwiML AI podcast. I am your host Sam Charrington. And today I'm joined by Aza Jalalvand. Aza is a researcher at Princeton University. Before we get going, be sure to take a moment to hit that subscribe button wherever you're listening to today's show. Aza, welcome to the podcast. I'm happy to be here. I'm happy that you're here and I'm looking forward to digging into our conversation. We're going to be talking about your research into applying deep reinforcement learning in the context of Fusion. Why don't you tell us a little bit about your background and how you came to work in that field?

0:41My background is basically AI and machine learning. I did my bachelor and master in Iran on artificial intelligence and robotics. And then I went to Belgium, Ghent University, and Ghent, in this lovely city, I really miss it. I studied machine learning neural network with application and speech recognition and image analysis. And then I became a postdoc there at Ghent University, applying machine learning on different topics from speech, audio, radar, health data. And well, it was totally by accident that I met a professor working on fusion and applied physics. And he told me, you know, we have a lot of data for fusion and we need someone to process it for us.

1:31And that's how I got into the fusion community. It was three, four years ago. And honestly, before that, I didn't know much about fusion. Well, but that's how life goes. And then I joined Princeton University in a plasma control group. And I've been here for two years. Maybe you can start by giving us an overview of fusion. It sounds like plasma control is one of the big challenges there, but maybe give us a bit of a backgrounder on the way fusion works and some of the big challenges. Well, you know, fusion, producing energy from fusion is not that new. It's like around 70 years old, 80 years old.

2:16And the idea is that, you know, the main source of energy that we are using right now these days come from the sun. So sun is the main resource of energy. And it works by fusion. So fusing atoms and releasing energy. And then the idea was that, okay, can we have a small sun here on Earth so that we can produce energy for ourselves, like basically unlimited energy. And that's the main idea behind fusion. The challenge is that you know how hot the sun is. So it's that big and that hot. We want to have a very small version of that. So it should be even hotter than sun to produce energy. And that means we're talking about like 150 million degrees, which is, I think, 10 times hotter than sun.

3:09So having that environment confined, controlled, and running 24-7 is a challenge. So basically what we do is that we have a donut-shaped vessel where plasma is developed and runs this donut and it gets hot enough so that the fusion can happen. And when the fusion happens, then some energy is released. The whole idea is to run this plasma stable enough so that the energy that is released by this fusion is more than the energy that we injected to the vessel. So if we get more energy compared to the energy that we injected, then we're actually producing energy. Are the reactions that you're describing the ones that are used in kind of conventional nuclear reactors, or is this required for a different approach than what we've used historically?

4:14Historically, we have been using fission reactors in which the atoms get split and then they release energy. Fusion is completely the other way around. The particles fuse together and they release energy. So it's a totally different scenario. And the advantage is that, well, it's way less dangerous than fission reactors. So in terms of the radiation and if things get out of control, basically the plasma dies. So it cools down. That's it. Worst thing that can happen is that maybe the wall gets damaged, but it's just hardware problem. There is no chaos there. Where would you say we are on kind of the path to fusion energy production?

5:10I think fusion will be the future way of producing energy. When it happens, maybe in two or three decades, we will have a reactor that really produces energy for us. But these are just speculation. Talk a little bit about where ML and AI come into play in helping to address this plasma stability problem. Well, that gets me back to the previous question. As I said, the physics knowledge is very limited. So if we want to wait until we have all the physics knowledge to control and understand plasma, it might take decades. I don't know. But on the other hand, we really need to have clean energy as soon as possible.

6:04So we need help. And this is where machine learning comes into play. Although we do not have all the physics knowledge, but we have collected tons of data from all these experimental machines that have been running so far. So the plan is maybe by processing that data, that historical data, the machine learning can learn some patterns, can learn something that helps the physicist to control the plasma in a more stable way. So, for example, well, we are talking about, I mean, there is one of these machines which have been running here in the US. It's called D3D. And it's almost 40 years old, and it has been running for experiments from time to time.

7:00For every experiment, lots of data have been stored. And we are harvesting that data to train the machine learning models to understand the pattern of the instabilities and the plasma environment, to predict instabilities and control them using AI. D3D, is it kind of a model of a fusion reactor? Yes. Okay. And what are some of the data that it collects? D3D is, I think for now, the main functional reactor for experimental use. You can imagine that there are many groups in the universities and research institutes who are working on this area. and when they want to run this experiment, they run it on D3D.

7:53So you can imagine how busy their schedule are. And what's happening is that whenever we run one experiment on D3D, we have data from the sensors that collect what's happening inside plasma in real time and store it somewhere. We have the data from the actuators that are being controlled by the physicists. For example, the physicists say, you know, I want to heat up a little bit. So change the knob and then the plasma gets heat up. That data is also stored. And before and after experiments, there are physicists, physics operators, engineers who also write down their notes. What they learned, what they observed, what were the challenges.

8:44So we have different source of data. It can be text, it can be video, it can be time series, signals, and, well, basically plans and conclusions. So these days we are just using all these types of data to have a better understanding of what's happening there. Talk a little bit about the way you set up the problem and applied reinforcement learning to the broader problem that you described. There are many types of instabilities that happen when the plasma is running. Some of them are minor, some of them are major, some of them are fatal. Basically, we would like to focus, the most important one are the fatal ones.

9:29Tearing mode is one of those instabilities, which if happens and if it gets out of control, it just, it results in collapsed plasma. So we will not have plasma anymore. That means we will not have that environment to run the fusion. So what we thought, let's focus on one instability and try to solve that instead of having a giant model that solves all the instabilities. Because, you know, we should start from small steps. Is tearing mode one of, you know, 10 or hundreds? One of hundreds, I would say. Okay. Every kind of instability has different classes. So if we talk about main instabilities and subclasses, I think we are talking about hundreds.

10:19And well, Turing mode is one of the most fatal ones. And we thought, OK, what do we know about Turing mode? what are the hypothesis behind that that we should look at which sensors to capture tiering mode? And then we thought, okay, if we can capture it, what are the sensors? What are the signals that we should look at to be able to predict tiering mode? What makes it so that classical control doesn't apply in this problem? Classical control doesn't apply effectively. It's not like they don't work. They don't work effectively because the conventional prediction models don't work very well enough, very accurate enough.

11:08So if we cannot predict something accurately, we cannot control it.

11:16And for the control, we did not invent a new actuator. or we did not invent a new hardware. We just made a better use of the current controlling knobs by using reinforcement learning. And since we were able to predict the happening of tiering mode well in advance, we were able to avoid it sufficiently soon enough to avoid passing a threshold of instability, basically. When we're talking about reinforcement learning, we basically have two main components, a predictor and a controller. The predictor is a simulator that looks at the environment. It observes the environment at this time. And it looks at the actuators that we have designed, that we have tuned, and predicts what would happen in the future.

12:15That's the simulator. If we have a good simulator, we can have a good controller. A bad predictor is just, a bad simulator is just not working. And well, it doesn't lead to a good controller. Basically, whenever I want to talk about that, I can just use this analogy when, how the pilots learn how to control the airplane. When they are learning, we do not just put them behind the wheel and let them, okay, try an error because it's, well, it's dangerous, right? It's expensive. Yeah. That's a good way of putting that. Basically, we have really good simulators, the aviation simulators. And the pilots learn by working with these simulators, what are the consequences of changing this knob or moving the joystick, what happens.

13:15And this is how the pilot learns. And that's what we did for this example as well. First, we developed a reliable and accurate predictor, a simulator, and then connect our controller, which is a deep neural network-based model, deep reinforcement learning-based model. We connected to that simulator and that simulator was trained based on all, not all, but the most important historical data that we had collected from D3D from the previous experiments. Because as I said, we already had the diagnostics, we had the actuators, and we had the result of those experiments. So we had a good simulator, and then the reinforcement learning interacted with the simulator and learned what are the consequences of changing the knobs and actuators.

14:10And then we put that into action in a real experiment on D3D. And that was the base of that paper, basically. Were the simulator and the controller trained jointly or independently? We developed the simulator independently, and then we connected that reinforcement learning to the simulator, and it started to learn about how to control the stuff. but of course well when we had the first version of the simulator and we had the first version of the reinforcement learning model you know there are bugs here and there and then we see that okay the reinforcement learning model doesn't learn well what's happening?

14:59Is there a problem with the simulator or we have to redesign our controller? So it was like a back and forth procedure we had the first version of these two and then in time we improved them And then when we were confident that, you know, this is the best that we can get so far, we applied to run the actual experiment. You mentioned that the simulator is based on the data that you observed. Is it itself a learned system in the sense of machine learning or is it more like, you know, fitting parameters to a physical model? That was basically a neural network based model. but we trained it by the physics knowledge that we knew.

15:46For example, these are the most important diagnostics that we should look at. These are the most important actuators that we need to control tearing mode. So there is some physics knowledge injection to the model, but the model is basically a neural network-based model. Is it a tearing mode simulator or is it a simulator of some low-level state of the fusion reactor? Well, we had to narrow down the problem to one problem that we want to solve. So the problem of all the instabilities, we narrow down to one, which is tearing mode. Well, the assumption that I'm making is simulating the entire system at the molecular levels, an intractable problem, and it's too difficult.

16:36So you're like raising the level of abstraction to focus on this tearing mode instability. And I'm wondering, you know, what that means and like what are the inputs and outputs of the simulator? Let me put it that way. When we were kids, you know, we learned how to ride a bike. And back then, of course, none of us had, you know, it's a complex physics dynamics, but we didn't have to learn that. We just learned it by practice, right? And we had two objectives when we were riding the bike. First of all, stay safe and have fun or stay safe and run as fast as possible. So if we wanted to stay safe only, we could have gone very slowly with the bike, but then there is no fun, right?

17:30For this controller, we also defined two goals. First, to avoid the tearing body instability and to keep the performance of the plasma as high as possible. So to get close to producing the energy. So we had these two goals, and the model has learned that, first of all, it should avoid the Turing mode, but not with a very low level, with a very basic plasma. It should keep the plasma performance as high as possible. But I would say that, again, this model that we designed was only for achieving these two goals only, avoiding Turing mode and high performance. It didn't learn anything about the other instabilities.

18:23So it doesn't know how to control them. Then this was a proof of concept. The next step would be, okay, how about the other instabilities? Do we want to have one giant controller that controls everything or we want to have several microcontrollers which are working in a bigger framework with like a hypercontroller on top of them. These are all the questions that we still need to answer, I would say. And you mentioned that one of the factors is the performance of the plasma. Is that taking into account both the input and the output? So there are several metrics to measure the plasma performance.

19:13First of all, so far there have been only a few experiments around the world, again to my understanding, in which they achieved the equal energy of larger than one. That means they produced more energy than they consumed. And they were in a very controlled environment, in very controlled experiments. so in our experiments we didn't want to our goal was not to achieve the energy of higher than one it was just to make sure that the plasma is in its highest stable performance so that it's ready to produce the energy and there are different metrics for that in the paper there are some plots that shows how the performance goes up and it stays up.

20:08And if there is an instability, it tries to reduce the performance a little bit so that it avoids that instability and then starts to increase the performance again. Just like you were saying earlier, very focused on this one problem, getting the energy out greater than one, that's another problem. Maybe another model, maybe another single, a different single model or some ensemble of models to be determined. Exactly, yes. The main point is that the whole idea of using AI infusion is very new compared to the other fields. And it's also way more, I wouldn't say more complex than any other problem, but it's way more complex than many other problems.

21:00so we are we are still learning and how we can efficiently use ai for a stable fusion plasma is the complexity in understanding enough about what's happening in the physical systems to be able to create the simulator and and have that give enough signal that you can then train a controller? Or did you need to do new and different things in the controller to apply it to this particular use case? Where does the complexity live? Like many other problems, there are different levels of complexity. In its lowest level, we do not know much about the plasma behavior in general. We know a lot, but not enough.

21:48So sometimes we even cannot reproduce one of our previous experiments because there are a lot of external factors here. Just for one example, when we are talking about this fusion device, we are talking about a toroidal shape, a donut-shaped thingy, and we want to have plasma there in 150 million degrees of siliceus and there is no material that can tolerate that heat, right? So we have to keep this plasma confined and keep it away from the wall around it, inside the vessel, right? This is called confinement. We are using magnetics to confine the plasma so that it doesn't touch the wall. But the wall is a material, I think sometimes it's carbon or tungsten.

22:44And if this plasma touches that wall, it makes some impurities. And that impurity highly affects the performance of the plasma. So even if you run one experiment two times in the same day, you might get different results. We know a little bit about what would cause this difference, but we don't know much about it. That's the lack of physics knowledge. When we do not have the physics knowledge, how can we confidently develop a predictor? We just think that these are the most relevant signals that we should look at. How do we know that based on the physics knowledge? But honestly, maybe we should look at some other signals, but the physicist didn't think about it.

23:37And this is where actually AI can help physicists to get better understanding of what to look at and how to learn from the historical data. In creating the controller, talk a little bit about that process. Was it a relatively straightforward application of reinforcement learning or did a lot of innovation take place in the controller itself? The main challenge that we had was not to design the neural network itself, but how to collect the data, how to preprocess the data, and which data we should use for effective prediction and control. That was one of the main challenges. So we didn't invent a new type of neural network.

24:36We used, well, the known deep neural network-based models. Of course, we had to optimize it. How many layers do we want? How many filters do we want, et cetera? But there was no breakthrough in designing a new model. The breakthrough was how we should utilize the signals that we have. For example, we have diagnostics inside plasma, which measure different aspects of plasma behavior. Some of them, just for some examples, some measure the temperature, some measure the density, the pressure. Should we just collect all of them and give them to one model? Or we should have some kind of ensemble model that are trained for each specific signal independently and we have a hyper model on top of that.

25:31Most of the time that we spent was on preparing the data and defining the problem correctly so that we can deal with it using neural network. For example, we want to avoid the tierability. but what are the signs of tierability and which signals should we look at to predict them well in advance? We had to look at the previous data, the historical data to see, okay, if a tiny tiering mode has happened, which diagnostics saw that coming well in advance and then use them as the input to our model? Some of those parameters were kind of control parameters and others of those, the input data were just internal state representations of some sort.

26:27And so your controller would tweak the tunable parameters to optimize those two outputs that you were looking at. Yes, that's actually exactly correct. Based on my experience, because I've been applying AI and ML to many different models, many different problems, I would say the majority of effort is spent on making the data and loading the data, exploring the data, and knowing how to extract the features, how to pre-process it to get it ready for training a neural network. That data engineering part and data understanding part is very difficult and it needs a lot of heuristics. Well, these days, with the help of all these GitHub pre-written scripts and ChatGPT, if you have a table or data for input and output, well, you can try many different types of neural networks, very complex ones, and see how it works.

27:39but interpreting of how the model is working and if it is not working, why doesn't it work? And am I using the correct data? Am I using enough data? Did I pre-process the data properly? These are the main challenges in data science, in my opinion, other than developing and optimizing a neural network. So when I say it was the challenge for us to understand how to use the data, it's not only limited to fusion. I would say in any use cases, processing the data is the difficult part. It's, I would say, more difficult than generating the model itself. Historically, deep reinforcement learning is challenging because of its relative data inefficiency or sample inefficiency.

28:37Did that apply in your case or is it once you have a simulator, you can just let the model go to town and collect as much data as it wants? That's a good question. The point is that based on our historical data, we knew, well, the model can learn what happens when some, let's say, if the environment is changing inside the plasma, what happens next and how should we control it? But these are only based on the historical data. And that's maybe one of the challenges when we are talking about data-driven models. It learns from the data. If there is a phenomenon, if there is a situation which has not been experienced before, the model doesn't learn it.

29:39and you never know during the experiment day, is the fusion reactor in the same condition as all the historical data that we trained it on or not? For example, on that day, one of the tunable parameters, one of the actuators didn't work just because of some hardware failure. So we had to deal with that. And basically we had only half a day to run as many experiments as we want and tune the data, tune the model as much as we can. That was actually, I think that that was the first time that I really, I mean, I was in our group, but for my experience, that was the first time that I developed a model based on the historical data, trained the model and thought, okay, this is a good model.

30:35Let's put it in action and see the actual result in the real world. Before that, it was mainly about simulations and using the training validation test and publishing the paper. But that was actually the first time that I really experienced what are the challenges that we don't think about. Like something goes wrong. How can we deal with that? And how do you overcome a challenge like, you know, an actuator breaking? Well, fortunately, when we run experiments, at least in D3D, we are the session leaders. So because that's the experiment that we design. But there are tens of people around us who are experts in controlling the fusion device.

31:22They are physicists, they are diagnosticians. and when one of the actuators didn't work, one of the physics operators said, you know, based on my experience, if we tune the coefficient of those two other actuators, it might compensate for the failure of this one. And we tried that and it worked. So this is actually how the physicists and data engineer and data science collaborate together to make a model working with, well, unexpected conditions, in unexpected conditions. How is that deployed to D3D? Is there like an API for this fusion reactor? Like, how does that all work? The first part is exactly as you said.

32:12D3D has a database of all the historical data that we can access and fetch them. and then we go offline, train our models, do whatever we can, and then we say, okay, this is the controller that we think is ready to be tested. It's a piece of code in Python, for example. The controller in D3D is a C-based controller. So first thing is to convert our giant model in Python, converted to C so that it's readable by the controller. And what we are testing is just one new algorithm among hundreds of other algorithms which run in parallel when D3D is running. So it's very important to make sure that this algorithm does not have any conflict with the others, right?

33:10So there is a lot of offline tests, hardware tests before the experiment day to run our algorithm, to simulate our algorithm and see if it runs well, it doesn't harm the other algorithms, it doesn't make the controller collapse or not. Does that mean that there's a whole system simulation with all of the algorithms that are running that you kind of like an integration test or something like that? Exactly. So we have several levels of test. The basic one is just testing our algorithm inside the D3D controller environment in offline mode. That shows us if our algorithm is even compatible with that controller.

34:01And then we run it along with the other algorithms inside PCS in offline mode to see if they collaborate with each other well enough. and then the last test which we called hardware test is basically we put the whole D3D algorithm PCS means precise control system so that's like the controller the core of the controller of D3D we put it in a simulation of the real plasma and then all the algorithms run there and then we will see if there is any problem or not When our algorithm passes that test, we merge it to the actual controller framework, and then we get ready to run the experiment. When I say we get ready to run the experiment, that means we should apply for getting time in D3D.