Phaidra’s Jim Gao on Building the Fourth Industrial Revolution with Reinforcement Learning

0:00A lot of times when we talk about AI, both in the valley and elsewhere, I think there's a conflation between AI and automation. AI can absolutely automate things. There's no doubt about that, especially routine things. I think that, honestly, undersells the real promise of AI. I think the real promise of AI is what Demis Aceo of DeepMine calls AI creativity. right? It's the ability to acquire knowledge that did not exist for it, right? And I of course experienced this firsthand. The reason why I'm such a true believer in the technologies because, again, I was the expert who helped design the system, but this very AI agent that we created is telling me new things about the system that I didn't know about before, right?

0:47And that's a very, very powerful feeling.

1:06Hi and welcome to Training Data. Please welcome Jim Gow, founder and CEO of FADRA. Jim was previously the leader of DeepMine Energy, one of the first and only AlphaGo style reinforcement learning applications in the wild. DeepMine Energy used reinforcement learning to manage Google's data centers and drove some staggering metrics, including 40 % energy savings. We're excited to ask Jam about reinforcement learning in the industrial world and learn more from him about what other real world applications are poised to be transformed next by deep reinforcement learning. Thank you so much for joining us.

1:40Maybe before we get started, we're going to spend a lot of time today talking about your deep mind energy journey. But maybe can you give everyone one or two sentences on your background and what you're building? Yeah, of course. So, Fajr is an AI company, of course, fundamentally we are an AI automation company. So what we do is we use a type of AI known as reinforcement learning to directly control and operate our customers very large mission critical industrial facilities. So in practice, these AI agents, these AI agents, they act as virtual plan operators, virtual members of the plan operations team.

2:13Let's go back in time and talk about the journey that led to this journey. and I believe that you once said an email with the subject line, reinforcement learning plus data centers equals awesome. Can you, question mark, yes awesome. Sorry, sorry, they're sitting for me. Reinforce with learning plus data centers equals awesome. Can you, can you tell us, who did you send that email to? Why did you send that email? What was on your mind at the time and then of course, What did that lead to? Yeah, of course. So the reason why there was a question mark is because it was generally an unknown if the combination of reinforcement learning with industrial facilities would actually be awesome.

2:56So that was an email that I had sent to a person named Mustafa Suleiman who would later become my boss at DeepMine and really the impetus was something called AlphaGo. So to set the stage properly, I had been experimenting as part of my I famed the 20 % time at Google with machine learning technologies. And it was actually a very specific course, introduction to machine learning by Andrew Ingan Coursera that had just come out, this is back in 2013, I think I was like the second cohort or something. And that class had completely changed my life. I taught myself how to program and just started tinkering around with machine learning on the side, right?

3:36It was just, it was very interesting technology. And your background was mechanical engineers, That's right, an environmental system. Yes, that's absolutely right. So my responsibility at the time was to one, help Google design and operate the very large data centers. And once these very large data centers, which consume enormous amounts of energy were built, we've of course shifted our focus to operating these complex industrial systems in the most energy efficient way possible because they use billions of dollars in electricity. So that was kind of the background. I was already tinkering around with machine learning technologies on the side to analyze the enormous amounts of data that Google's data centers were generating.

4:11In 2016, AlphaGo came out and I was one of hundreds of millions of people around the world watching. It was like 3am in the Bay Area or something and I found it absolutely captivating. And to the point where I sent an email to Moose, describing this idea that if DeepMind could be the smartest, most intelligent people in the world at complex games like Go, then surely we can train the same AI agents to play a very different game that I'm familiar with called Let's Optimize the P .E. The Power Use of Repetiveness at Google's Data Centers. So that was the context for that email. And I remember internally the way I pitched it to Google's leadership, so specifically Joe Kaba, who leads Google's Data Centers and Ours, was I showed a picture of a Go board on one side and a video game control and like an Xbox controller and the other and I'm like look there are objective functions I were trying to minimize or maximize there are concrete to like knobs 11 so actions that we can control there are constraints that we have to stay within and all of this happens within a very measurable environment I think you know reinforcement learning and operating large complex industrial systems are actually one and the same thing right so that was the the original kernel of inside, I guess, that is spider.

5:32And I know Sonya has accused me of going rogue with some of the questions we ask here. I'm gonna go ahead and go rogue for a minute. Already, it's been like one minute. We're gonna come back. I wanna skip the story. This is using a brief diversion bear with me. The three things you mentioned that allowed you to see the parallel between reinforcement learning and control systems or control theory, objective function, actions, constraints. Yeah. Are those the three key ingredients for where reinforcement learning can be applied to real -world systems? Yes, absolutely. That is 100 % how we think of it, right?

6:05You know the Reinforcement learning systems they need like KPIs to optimize for they need to know how good or bad an action is, right? They obviously need things to control and they need to know what are the constraints they have to stay within so Really what we're saying is as long as we can map the problem we're trying to solve into a reinforcement learning framework, which really from a mathematical perspective, what we're saying is we're solving a constraint optimization problem. If you can map the constraint optimization problem, if you can define it and map it to the underlying data, then it should be able to be solved using reinforcement learning.

6:44So that's very much the lens through which we look at things at Phagear as well. And to take it one step further, we often talk about how reinforcement learning and controls and optimization are like two wildly different fields historically that have somehow independently converged to the same area. They're two very similar concepts. Well, we've been calling them by different names this whole time. So you've had almost these independent evolutions, a different ways of tackling the same problem. and the major is really kind of the intersection of both of these. Okay, let's get you back onto the story.

7:19So you said the email to Mustafa and then what happened? Yeah, so he said the email to Mustafa two weeks later, Moose had actually flown out to Mountiveview, whereas working at the time on Google campus with a team of D -Mind folks, and we actually started mapping out exactly how reinforcement learning could be used to control and optimize Google's data centers. So that actually kicked off the original partnership between Google and DeepMind around the application reinforcement learning for the data center work. It was very, very fascinating, but most importantly, it's actually also how I met one of my two other co -founders.

8:01right. So Veda was one of the original engineers on the AlphaGo project. So he had gone to go to, you know, he went to South Korea, right, and you know, he actually got to meet Lisa Do and later page and all, you know, all those fun stuff. And after AlphaGo, he came back to the season, you know, or rather to the UK and he was wondering, well, what is my next big thing going to be, right? And I managed to convince Veda like, hey, what if we applied self -learning frameworks like AlphaGo to control and optimize Google's data centers. So that's actually how I started working with my co -founder beta.

8:36Did people think he was going to work? Or was it like, this is a crazy moonshot? Let's just try. But I mean, I don't even know if it was going to work. Like, can this actually, it made sense to my mind, right? I'm like, hey, you know, this is, it's a operating a data center is just a different game to play, right? And there's all kinds of different games in the industrial world, right? Maybe the game is maximized energy efficiency. say maybe the game is minimized water consumption, maybe the game is maximized the yield of a factor, right? But there's all these games that were constantly playing, right?

9:07So in my mind, it made sense, but to ask your question directly, no, I had no idea if it was going to work. I still vividly remember to this day when we, you know, turn on the AI system and we watched the energy just drop. And it was so surprising for two reasons. Number one, well, we had designed a system. I played a role in designing that very mechanical system, right? That the AI was now controlling optimizing. So in theory, I'm literally supposed to be the subject matter expert who knows everything about these systems. But the AI is teaching me things that I didn't know about the system. I helped design in the first place, right?

9:45And to the moves that the AI was making, we're just very counter -tuitive. When we looked at the decisions that were coming out, we looked at the plan operators and we were sitting in a giant cornfield in Iowa where we were like to put its data centers. And we were looking at the decisions that we thought to ourselves, there's no way this is right. This AI sucks, they learned the wrong thing. But we're here anyway, so let's try what the AI is saying. And we tried it, and it worked. And we saw the energy plummet. So I think that was kind of when I became a believer in this technology that fundamentally this technology Is creative it helps us discover new knowledge that didn't exist before from raw data Was there performance trade -offers this to straight up Pareto game like performance hell then that's exactly No, it was it was it was expected exactly the same constraints and the plan operators and engineers had already put in place.

10:45So this is pure gain, right? Respecting exactly the same temperature profile is exactly the same constraints around how quickly you can turn on and off a chiller, minimum pump VFD speeds, all that sort of stuff. So this is pure optimization, pure gain, which I think is one of those crazy things like we don't really expect, like usually when you think about energy efficiency, for example, right? Like in the world that I come from, people usually think about expensive catbacks. Like, oh, we got to rip out the chillers. We got to buy a bunch of new chillers from Johnson Controls and trains or whatever, and then we have to install them.

11:17So they're like hardware efficiency gains, right? But you don't really think about like pure software, like data driven efficiency gains, right? And I think this part of all was surprising for us. Can you let this through the before and after? Maybe before what you all implemented was this industrial control systems? Was this manual plant operator is turning knobs? Like, how did this work before and then after? Yeah, it's a great question. So let me set the stage for, you know, for folks who are not as familiar with like large industrial facilities, right? So the very modern industrial facilities are very, very complex, right?

11:53There's all kinds of machines that people are operating and controlling, right? So, you know, I often, you know, tell folks to do like a simple thought experiment. So imagine you have just 10 machines you're controlling. So say they're like pumps, right? And each one of those machines has 10 possible set point values. So 10 modes associated with it. So think something like 10 % pump speed, 20 % pump speed, 30 % pump speed, et cetera. Then in this very simple toy example, you have 10 raised to the 10 or 10 billion different permutations for how you can operate your toy system. So then the question becomes, well, at any given point, what is the most optimal way of operating your toy system?

12:34And by the way, these are dynamic systems. So the IT load is changing, the weather is fluctuating, the people operating, these systems are changing, the pipes are crowding, the heat changes are fouling. So the point is these are very complex dynamic systems. Railroad systems have a lot more than 10 machines, and each machine has a lot more than 10 set points. So you can start seeing why technologies like AlphaGo, which manage to navigate MX complexity, are helpful over here. It also helps explain why there's often so much room for optimization in the first place. Because there's so much complexity, right?

13:09Like if you think about the total action space, right? Like all the possible actions within a modern data center, for example, right? Because of risk of verseness, but also because of hardcoded rules and heuristics, right? We've only ever explored like 0 .0001 % of all the possible ways that you could operate that system. So then the question becomes, what is in this 99 .999 % of the atro space we've never explored. Surely there are more optimal ways of operating the system than what we've done historically. So it's kind of an intuitive explanation, hopefully, of why there can be such large efficiency improvements in the first place that are undiscovered.

13:50And the way that we operate these facilities is constrained by a mixture of hard -coded controls logic. So don't give me around. These are automated systems today, I read it, right? They're just not opt - you know, automated intelligently. I would argue. Right? And you know, there is a healthy mixture of human intuition as well, right? Where we have people like myself or plant operators who are constantly monitoring the system who are like nudging the system by adjusting things or setting, adjusting the rules, right? For that system, the constraints of the system has to operate within, but fundamentally, human intuition plus hard coded controls logic is still limited when you talk about this degree of complexity, right?

14:36Can you talk to us about the key results? So you saw the energy levels drop immediately. But what results were you able to drive for Google? Initially, so there's two types of results for Google in particular. There was a results from the pilot. So in 2016 we released, we announced like the results of the pilot, right? Now the pilot was done on a couple of data centers, but fundamentally it was not an autonomous control system, right? So what I mean by this is it was the AI generating recommendations, right? Which for for human experts like myself to manually review and implement. And of course, you know, we didn't want to jump straight to taking our hands off the steering wheel, right?

15:18because it's a new novel technology, right? But also like, no one knew at the time like is it even possible to use AI from the cloud to control big -ass infrastructure, right? So step number one was do the pilot, right? The agenated recommendations, that's where we saw like really steep like 40 % energy savings, right? Now that experience taught us like, hey, we think there's something real over here, we should actually just let the AI control things directly to get the value automatically. And also quite frankly, the plan operators were getting tired of checking their email like every 15 minutes, waiting for the AI to tell them what to do, rather than manually from that thing, they had better things to do.

16:01So we actually decided, and rather, Ores and Joe decided like, hey, it's time to go to a fully automated system, right? This was total uncharted territory. At that point, like forget about can AI control things, we didn't even know, is it possible to control machines from the cloud, like huge industrial infrastructure in the cloud? Because to our knowledge, no one had done it before. Is it fair to assume that a lot of the hardware, a lot of those machines are things that Google built for scratch, or does Google use a decent amount of commercially available data or stuff? It's a mixture of built.

16:32So, you know, obviously, Google does a lot of things in houseware, but it doesn't manufacture shillers and that sort of hardware. So, Google does buy off -the -shelf hardware, But there's a lot of modifications on Google specific things that we did. For example, programming some of our own PLCs or making modifications to the building management system. The software control layer looked quite different. That was done in -house. But I still remember very vividly actually to this day, Vita and I, we were standing in a large 90 megawatt data center. It was a fairly large data center. And you know, Vita is like typing away in his MacBook, right?

17:13He, uh, he submits the PR, right? It's merged. And all of a sudden, this huge, honking huge chiller that is a size of a bus that we're there were sanding right next to, Roars to life. And as it's coming to life, right? Like the ground is shaking vigorously. I were like, oh my God. Like, with a few keystrokes on his MacBook, like we just turned on this enormous chiller. And that was like the very first data point to us, like yes, it is possible to control things from the cloud. So now the next question is how do we control things intelligently from the cloud, right? You know, where all the compute resources.

17:50What were your biggest takeaways from that experience? You mentioned the creativity of the machine. Any other big takeaways or learnings? Yeah, so the creativity is absolutely a big one. I think, you know, just a library of that briefly, you know, a lot of times like when we talk about AI, right? And it built in the valley and elsewhere. where I think there's a conflation between AI and automation. Like AI can absolutely automate things. There's no doubt about that, right? Especially like routine things, right? But I think that honestly undersells the real promise of AI, right? I think the real promise of AI is what Demis Aceo of DeepMine calls, you know, like AI creativity, right?

18:29It's the ability to acquire knowledge that did not exist for it, right? And I, of course, experienced this firsthand. The reason why I'm such a true believer in the technologies, because, again, I was the expert who helped design the system, but this very AI agent that we created is telling me new things about the system that I didn't know about before, right? And that's a very, very powerful feeling. It's kind of like when, you know, if you think back to AlphaGo, right? Like, Lisa Dole was the best in his field at Go. He was the world champion for a decade, right? He was at the top and his elo rating was just something outrageous.

19:07It was like 20 hundreds or something. It was outrageously high. But it had flat, you know, flatline, right? So for a full decade, his elo rating was the same. And there was no one to challenge him because he was at the top. So once he hit the top, he just kind of plateaued. And then after AlphaGo happened, and he actually got to play against AlphaGo, you know, privately a few more times because DeepMind, you know, had, you know, had let him continue reacting with the system. what happened for the first time in a decade his e -librating started climbing. And so this is what I mean when I say that, I think the real power of AI is helping us discover knowledge that we didn't didn't necessarily know about before.

19:44And where you're going to see the most gain from that, it's not going to be in routine automation things, like call centers, whatever, right. It's going to be, I think, in very, very complex areas, areas where human intuition is insufficient because of immense complexity, but that is yet underpinned by data. So that's why you're seeing such things like protein folding, for example. Maybe that's fucking extraordinary, right? And it's those areas are just massive permutational complexity underpinned by data. That's where I think we're going to see some of the most interesting companies and products.

20:25So that was a rather long tangent. But But so one creativity is something that I learned. The other one lesson that my co -founders and I learned is really around, you know, if you want real impact, you got to turn the technology into a product. And this is actually the more reason why we decided to leave D -Mine and Train Technologies to start a feature, right? Like over and over again, we were seeing the technologies that we were helping to develop that D -Mine were just extraordinary, right? I mean, they were achieving crazy things like with protein folding. But the problem is, in order for the technology to make the most impact, you have to get into the real world.

21:03People have to actually use it. That fundamentally means we're talking about a product. Turning a technology into product is like, you guys would know much better than myself. It's like 100 fold, 1000 fold more work. That for us, let us to the conclusion that it's time to leave. right is time to actually start a company that creates these intelligent virtual plan operators. These intelligent AI agents has a real product. Let's talk more about that. For what you're building now, how much of what you learned at Google D -Mind sort of translates directly into what you're doing now? How much is new because the environments are different, the customers are different, or something different about it.

21:51I think the most important thing that we learned from our Google D -Mine experience is that it's possible. Like this is not a crazy, and that isn't to like, you know, like downplay like what we learn. Like it's actually a huge thing, right? We learned that it is in fact possible to use, you know, closed loop learning systems like reinforcement learning, right? To drive very large improvements and complex industrial facilities. They hadn't been done before to our knowledge, right? And that was a massive proof point. I think the problem though is that like the real world is quite diverse. Every single customer is diverse and especially when you talk about industrial facilities, like every industrial facility is a snowflake, right?

22:34So for us, I mean, the learnings have just been like massive since we left Google and DeepMind, right? Because every time we onboard a new customer, we're learning something new about like how equipment are connected or some product gap that we didn't know about before that needs to be fixed, right? Or new ways that data can break at this point, I can tell you like a hundred different ways that data associated with mission critical cooling systems can break. Probably not the most interesting party topic for most folks, but I personally find it quite interesting. But yeah, data certainly being quite a lot of learnings in that regard.

23:09Are the folks you're talking to? Are they ready to let the technology take over the system and let the cooling system just start going? Yeah, I mean, yes and no, right? And that actually gets back, you know, to your early question pad as well, right, about like the specific learnings from Google. I mean, when I look back, you know, I think what we helped pioneer at Google and DeepMind could only have been done, right, at a company like Google. The reason why I say that is because Google is a very forward leaning company. Yeah, but also like one of the things I've learned right is that like Google is absolutely in anomaly when it comes to like how much data it has and the pristine quality of the data and the ease of access of the data right like Google is fundamentally a data analytics company, right and You know as such it invested all this like infrastructure in high quality high availability data on which you can do things like real -time intelligence applications like what we were doing and there are many other examples of this within Google and DeepMind.

24:17Having left the nest, one of our rude awakenings was Google is definitely anomaly. And I mean, gosh, like everyone is in various stages of their AI journey, right? Like Google is certainly on one extreme. We have customers who have encountered where like, you know, forget about real -time intelligence, they're like, they're not capturing the data in the first place, right? Or, you know, they may be sensorizing, in the industries we work like pharmaceuticals and distracooling and especially data centers, almost always the customer is sensorized, right? Because these are billing dollar facilities.

24:52Of course, it makes sense to throw a million dollars with the sensors on it. But that doesn't just because you sensorize, it doesn't mean that you're storing the data, right? A lot of customers of ours aren't necessarily storing the data beyond like 90 days or six months or a year, or whatever, right? And they might cite some reasons like, well, it's costly to store the data, right? Or like, well, we're the more commonly, we're not using the data for anything, which is a true statement, right? A lot of our industrial customers, they aren't using the data, right? It's more like a forensics thing where if something goes wrong, then we go back and we look at the logs to see what happened, right?

25:30Yeah. And then, you know, so if we think about it like Maslow's higher key of data needs is something, right? you got your sensitization, you got your storage, then you have to invest in making sure that the data is cleaned. Right. There's a lot of effort as we all know here, you know, around making sure the data is actually cleaned and usable, right? And you know, that requires you to know what bad data looks like, what good data looks like, and how to convert bad data into good data. So it's actually useful. And then once you have clean data, you also need to make it accessible in a streaming and like batch historical manner, right?

26:05So there's different gradients, I guess, is what I'm trying to say of AR, right? And it's the customers whom we work with are all over the spectrum. But, you know, like Fager today is at the point where we are autonomously controlling data centers for our customers, right? So I was gonna ask you, if the basic workflow or the basic loop is data goes in, which is a lot of what you just talked about, getting the data into the system step one. Step two, decision is made. Step three, action is taken as a result of the decision that was made. Step four, action is evaluated against the objective function of the system, and then the loop continues.