#241 Patrick M. Pilarski: The Alberta Plan’s Roadmap to AI and AGI

30:24Part of that is saying like the agent learning from as raw a sensory motor signal stream as possible allows it to sort of undo some of the assumptions I might be making as a designer, which means it might be able to find just a much better solution, maybe even faster than the other way around. Now, I think, you know, building up human language, it takes a while for a human to learn human language through trial and error, through motor babbling. It's going to take a while to learn language. Would we rather just like, again, juice the internet, make some amazing lemonade and use that as our foundation model?

30:54Like, like, yum, large language model, pre-trained, it's great. I think, well, of course we would. This makes a lot of sense. Everything everyone is doing, I think, does make a lot of sense. But there's this other path, and that's why, as again, going back to the Alberta plan, why I think we want to make sure this stayed in the public mind's eye is that there are alternate and compatible views, which allow us to maybe fill in some of the gaps, or maybe to start from a very different foundation point and get to something that is like, maybe our current curve goes up and then it sort of starts to taper off.

31:24Maybe there's another approach where you do pursue this raw sensory motor learning about the stream experience, but then that actually has like a higher final peak. So let's keep, again, we don't want to have a monoculture. So it's keeping that diversity in our mind stream. And then also being really like not being dogmatic or like siloed about it and saying, well, where is the opportunity to do something like we saw in DeepSeek where they're like gonna throw just like raw reinforcement learning to help the model get to the next step of its chain of reasoning. I think it's really important that we aren't, again, too siloed or too cliquey in our communities.

31:59Be like, no, I'm going to sit over here and do raw sensory motor learning and ignore everyone else. I have the luxury. Yeah. Are there projects to do just that? Start with raw sensory motor learning. I presume that the learning then is encoded in the weights of a neural network, right? I mean, when there is a learning, but is there a project to start with, you know, whether it be YouTube videos or a robot arm or something and see how much learning can accumulate and whether it scales? Yeah, I think there's there's like a happily a pretty spectacular library of examples globally. Now I think we're starting to see, and like, I think where we, where we see them most is in specialized targeted domains.

33:00So like the, the work from deep mind on stabilizing plasma in a tokamak reactor, decision-making system acting thousands of times a second on a very diverse stream of data to literally control a lobe of plasma hotter than the heart of the sun. That's really neat. That's reinforcement. It's learn from reward. And then we see systems that can do elite coding tasks that were also driven directly by reward. And so as we start to see all of these different domains, like, well, one of those is very sensory motor. I would argue that a stabilizing plasma in a tokamak fusion reactor is, in fact, very much a sensory motor experience kind of thing.

33:38Same thing with cooling a data center or even compressing bits in YouTube using MuZero. things like that are much more sensory motor they're very much like again the the nerves firing down the spinal cord and letting my arms do all sorts of cool things and then something like well i mean doing really well at like elite coding tasks and like programming competitions well reward driving that feels more like the languagey thing like that that feels a lot more cerebral it feels like you know that like the deep thinking not just like the reactive i'm a tiger running through the woods kind of thinking but what we're seeing is that reinforcement learning applied in its sort of raw sense is being able to generate insight in both of those domains.

34:19So we'd be here probably till, I don't know, next week if I were to start the list off. I usually, if I give talks, I actually have this best of hits collection at the beginning of my talk. I'm like, hey, what's been happening in reinforcement learning? Well, have you met Gran Turismo Sophie? Okay, cool. You have? That's neat. Our colleagues at Sony AI did a really cool job of building a racing system which could do high performance superhuman driving while also learn the implicit rules of driving in racing, not cutting people off. You don't sideswipe people. There's rules of racing that I was able to begin to build up.

34:53And that was also, again, through process of reward-based learning. So going back to reward is enough in this case, you need some social interaction skills. Well, you could, if the task requires it, you can get that through reward. Do you want to be able to solve a programming task? Well, yeah. I mean, you, if you need to be able to think about data structures and the way you chain together methods and, and, uh, different computational artifacts, well, yeah, you could drive that with reward. There's a whole bunch of them thinking, well, what's the, like, is it just bits on a wire? Like Patrick proposes.

35:24Okay. Probably not. Like, this is a very bold, like if we were to go all the way to sort of like Joseph O'Dale is another person who really enjoys sort of the, the bits on the wires, Rich does too. But like thinking about an agent's experience just in terms of raw ones and zeros without any kind of labeling on them there's a there's a like there's a quick shortcut you can take which maybe you shouldn't take again if you believe my argument about like design biases but there's a quick shortcut say well let's just actually use floating point numbers and integers like patrick said we shouldn't and then there's another which like well let's just use like you know we'll use more more advanced data structures as the representational elements the foundational elements the machine is starting to use the actions it might take.

36:05And then you might say, well, wouldn't it be faster if we just use some of those that were already learned through processes of supervised learning? I mean, I like to think about supervised learning, and again, a lot of our colleagues would say that it's like learning through labeled examples. So as opposed to supervised, well, that was like learning through labeled examples. Do we learn through trial and error? Do we learn from the structure of the data? There's cases where each of those might be a great way to start to build the fundamental units of experience that an agent is operating upon.

36:33And constructing those, you could say, yes, there's like we back in 2011, a large number has put out a paper on generalized value functions. And one of the reasons we were excited about that, one of the reasons that we thought about it's like generalizing the idea of learning a value function about reward, like expectations of how much reward will I accumulate in the future, assuming I care about the future in this way, to do that for any signal of interest. The reason we've been passionate about this and continue to pursue this for well over decade now is because we viewed those predictions, those forecasts, those expectations of future signals to be those foundational elements of building up other more advanced behaviors for an agent.

37:12So that again goes the choice of well what is the fundamental units of an experience stream. Yeah but the examples you're giving are sort of narrow domains. How does that generalize into uh or how how do you foresee that generalizing into general intelligence i like when i even think about the like attention is all you need transformers what are we getting from some of these systems like i i view them as like the multimodal versions of themselves not just like a stream that's predicting language tokens but to think about like well what about a vision transformer What about these systems that are truly multimodal?

37:54When you extend that one step further and just think about what the operations are happening in these systems, they're creating associations across space and time. Like many different pieces of information without any of this like sort of a bit like staged convolutional foveation on a specific part of the data. Like these, these systems are able to consider the flow of time, spans of time, which are getting increasingly vast and then creating associations across that like large span of time, but also the different, the different channels of information could be anything from like the stream of tokens in a sentence alongside the audio data coming in alongside the time series of the vision pixels to the random bits on a wire that I was proposing to you earlier that the associations across space and time are very, very powerful.

38:40I mean, we like Dale Sherman's and colleagues have the, have an awesome paper where they're like, yeah, if you like give transformers redire operations, they're turning complete. Grossly paraphrasing Dale and colleagues on that paper, but it's really cool. Cause this is like, oh, well, yeah, this is, and I'll sort of paraphrase Dale as well here and say that this is another way of thinking about doing computation. It's a fundamentally different one. And so what's happening with all this, we have, we have cross attention. We have the ability to chain together facets of information in the beautiful gemstone of the past experience of an agent to create predictions about the future.

39:16Those predictions are acted upon. They're either going to move a joint or more likely they're going to be able to spit out a new word token or to spit out a new pixel in a generative video. The same thing happens in the base process of learning a generalized value function. you're thinking about based on a representation of some kind, which we hope is able to form, again, associations across both space and time, that a system is able to generate a prediction about the future. And these predictions for generalized value function, for instance, are temporally extended. They might be thinking about the expected sum of a signal conditioned on a policy that assumes some kind of preference over the time window that it's being integrated.

39:55but either way this is you could definitely do token prediction with that if you wanted to you can also predict the future current on a robot arm and start it moving early to be able to like catch a ball mid-flight so the associations across different modalities or channels of information and across time and using that to create a prediction like these are different ways of achieving the same thing you could imagine using them flexibly together you could imagine and committing hard to one of them and working on that. So I do, I feel like that's the potential here is to see it as, oh, wow, this system is able to put together the fact that I had coffee last week, plus the fact that I'm going to buy milk and then get like, oh, you're probably gonna make coffee today.

40:37Like that's a very sort of high level human thing or like it's grading my cashmere or whatever. Like the systems are going to be able to put together these pieces of information to make predictions about the future. And then how you think about the form of those is maybe where I see people are starting to disagree, but sort of zoomed out a little bit to bring together the cross-attention idea with some of those token prediction in terms of raw sense remote experience. There's a lot of unity there that I think that we're just not celebrating, quite frankly. And I really like it because it does, like some of the tricks that we're starting to see, and the tricks is not to trivialize the elegance of the computation for like taking a vast history and giving the access, like a large language model, having access to a vast history of tokens.

41:25Like me sitting there with like Google Gemini and typing and having this lovely conversation about like structural neuroscience. Like there's a lot of tokens that are going in. Like we chatted for an hour. And like the tokens, like getting these long histories and then still thinking about the relationships across these vast spans, I think is something that is exciting and new. Like history is hard. And often we try to make it, Like a lot of the work we do for the Albertan line, we're thinking about how to make learning incremental by making sure that every new sample that comes in is sort of bolted out the old so we're not keeping around histories.

41:58But there's new ways that we can start to think about this maybe close to Markov level unfolding of experience based on these technologies that can integrate long, vast histories. Context. Context is critical. I think we're getting closer to context. This is the thing that warms my heart. Reward is all you need. I mean, the idea is that reinforcement learning can build up this world knowledge or this representation of reality through which it then can make an agent can make predictions and act in the world. But, as you said, there are quicker ways to get to a baseline and maybe reinforcement learning.

42:45You start with that baseline and go from there. I mean, I've been talking to Pedro Domingo. I don't know if you know him. He had that book, The Master Algorithm. and he's saying that, you know, it's not going to be one of these schools of thought. It's going to be a blend. And that's why I was interested in DeepSeq's reinforcement learning training. And I met recently Mark Raybert from Boston Dynamics or originally from Boston Dynamics. And he does not believe that end-to-end neural networks can, that robots can operate on an end-to-end neural network control. That you need a traditional control theory.

43:44you need very exact equations for the precision that's required. And Boston Dynamics, for all they do, that's not machine learning. So how do you feel about that? I mean, you're working on prosthetic arms or legs. How do you use control theory and then build up knowledge through reinforcement learning starting with control theory? Or is it raw reinforcement learning and, you know, you can eventually get to the precision required? I mean, yeah, if you could talk about that. We've tried a number of different approaches over the years. And maybe I'll say one that continues to shine for me. And it works really well in a clinical setting.

44:40Works well in an environment where you're trying to bring together technology, people who build technology with clinical practitioners who are experts in their clinical fields. And that's that I love starting with an approach that the system will begin at the gold standard, whatever it is. And this is especially for human facing technologies. That's where I mostly work. Human facing technologies. Like I would very much like a system that starts at the gold standard and then gets better with experience. It does learn through interactions with its raw stream experience. It is able to integrate those other facets of experience into its learning and improve or optimize its behavior from the gold standard.

45:16But that then fails back to the gold standard. If and when it makes a mistake, then it still defaults back to the like whatever that gold standard operation would be. So this actually says that if we are, of course, like there are cases, especially human facing cases, where do we want to appeal to control theory? We sure do. Does control theory currently allow us to think hard about soft robotics, deformable systems? Imagine a liquid metal robot that's able to change its form completely and has very strange dynamics like jamming systems, anything else that you might see in like any of these sort of reconfigurable robotic systems.

45:52Well, there's branches of control theory which are attacking those problems, but these are hard problems. So there's case where, yeah, we can have this piece is definitely going to be nailed by control theory. Can we improve upon it to make it more resilient, more agile, more able to adapt to the world around it through learning about its stream of experience? I think the answer is absolutely. Like some of our colleagues here are, RL Core Technologies is a company from Edmonton and some of our colleagues from the University of Alberta are leading that company and they do like reinforcement learning technologies for one of their examples is for water treatment.

46:24You can take a water treatment system and start to think about how RL can be used to improve the ability of a community to treat the water and make sure their drinking water is safe and reliable. There's going to be some parts of that. And I think you want to interview like Adam White or Martha White about this. I think that would be a great guest to have on the show to talk about how they're pursuing this. But again, to paraphrase what they've been working on, I'm like, yeah, you're going to have some parts of this like water treatment plant, which are very much like solid gold standard control engineering, but then you're going to be using this learning through experience, learning through trial and error process to improve the pieces that can be approved upon that gold standard, such that it can adapt to new circumstances to be more adaptable or to be able to operate for long periods of time without constant human intervention.

47:09So I think these are really the space explorations of the great space where you're like, yeah, do I, do I want some kind of control system probably governing my like plasma thrusters? I sure do. But do I want, the spaceship to like steer itself like maybe learn to avoid that yeah yeah that would be great in new environments but there's like certain low-level processes like our biological processes in our own bodies like the the protein machinery or our cells is going about doing its business um and yeah there's some adaptation going on there but a lot of what we think about is intelligence is is the the the thing that's moving around all those little protein machines right Yeah.

47:48So the reinforcement learning is learning, right? It's trial and error, trial and error. And it's that learning then, and I'm sorry, I'll cut this out if I'm revealing too much ignorance or if I'm off base. But, you know, I understand neural networks and backpropagation and how the network learns. But with large language models, once a model is pre-trained, its learning is kind of fixed. I mean, you can fine tune it, but if you start, I mean, there's this problem of catastrophic forgetting. If you run it long enough, then it's going to start forgetting or the weights are going to start changing in the network that encoded previous knowledge.

48:52How do you deal with that with reinforcement learning? Is it, what happens to the learning? Does it, is it encoded in the weights of a network? And then can you just keep expanding that network so that you never lose any of that learning? This is an excellent question. I think I'll take this question to the domain of continual learning. So this is really exciting. Like I think a lot of us in the community are passionate about continual learning. And that I mean, a system that is able to continue to learn, adapt and change. It's very constructivist in nature. Like you would hope that a system is able to continue to integrate new information into what it knows, not overwrite the old stuff, unless the old stuff is supposed to be overwritten to be able to flexibly compose skills and behaviors it's learned in the past in a new way and new situations and to continue to do so forever, literally forever.

49:46So this would be continual learning. And there's some good review papers. Kimmy Kirtapala and Joanna Precup have this awesome review paper on continual learning, which is definitely worth going through. But so continual learning, the way the field, like there's a new conference on continual learning colas, which is really cool. And there you see in that community, a variety of approaches and that you called them both out. So I like this. Like one is, well, okay, I have a giant, let's imagine I have a giant large language model. Okay, cool. It's got all of these factoids baked into it in whatever way, whether there's a grandmother neuron or whether it's a collection of activations that allow us to have the concept of a grandmother.

50:23You'd want maybe that system to be able to be incrementally improved without having to go back and spend how many billions of dollars, or I guess now millions of dollars if you take the deep seek approach, how many billions or millions of dollars to retrain the whole thing from scratch on the data. Well, maybe you don't want to do that. Maybe you just want to take new information and have it be able to be integrated into that system. Like my brain, I don't want to retrain my brain every single time I learn something new by reading a paper or like talking to you today. I don't want to have to retrain my whole brain.

50:53So there's a whole community of people that are in fact looking at that problem. And there's been some really good advances there, which is again, taking a piece of information and being able to use that to upgrade an already in place system. And again, like if we go down to the brass tacks of it, like there's, I don't think we have any analog to like glial cells in the neuroscience of our digital system so for the most part we're looking at weights and you are saying oh like can i modify either the weights or some other holding memory variables that are impacting the operation of the network if there's other shenanigans going on then can you go and take this new piece of information actually modify those one those ones and zeros and the the other numbers that are comprised that comprise those ones and zeros um so that's one approach is to say yeah i would love my continual learning to be a thing i would I'd love my large language model to continue to learn.

51:39I would love to be able to take whatever news article is popped online, which hopefully isn't written also by a large language model, because then, as we saw from that recent paper, you start to get these collapse iterations happening. So, okay, that's one branch of how we might think about continual learning. And the other is that, well, we actually don't want just be sort of modifying like a large language model. So we're actually thinking about the process again, those ways as being constantly updated from the raw experience in a very incremental way. And that's saying, I'm not taking new factoid and just jamming it in.

52:11It's saying that every single moment of interaction is already changing the system. And both of those communities, like taking both of those views, you're modifying weights, you're modifying numbers that are stored in some kind of digital mind of the machine. But you're hoping that they don't overwrite things that shouldn't be overwritten. They respect context. Like if I'm upstairs making tea or I'm drinking tea down here, Or maybe that I want to be acting differently. I want to be thinking about my predictions about likelihood of spilling hot water myself differently. So I think those are really important.

52:44And you do want to see that continual learning can happen for long periods of time. There's been like some of the there was a various paper that by some of our university of our colleagues that actually showed ways you might think about stable continual learning. This came out came out recently. And so that paper as well is starting to say, we have new ways to think about allowing a system to continually keep changing its weights without having catastrophic, like forgetting or catastrophic failure after like hundreds of millions of epochs of learning or time steps of learning. Like you want systems that, that should stay stable for very long periods of time.

53:18If I send out a continually learning system on a spacecraft, it's going to be Alpha Centauri. I sure hope that like we don't find out, oh, shoot, our learning approach after 100 million time steps, everything just flatlines. And it's like it predicts zero for every single outcome. Like, boom, there goes your spaceship. So I think we like as we imagine long spans of experience, machines that learn and act over long spans that we do definitely want to consider both of those styles. One, which is very strategically, surgically adding new information into an existing system in a way that retains old content without getting rid of the essence of the new content.

53:53Or systems that can continually update everything in the network in a flexible way that is more of an incremental hands-off way. I think we want to consider both of those. And luckily, the community is doing both of those. Yeah, and just intuitively, and maybe it's the first, can you just keep expanding the network, adding layers or adding nodes per layer or something as it learns? Well, this is a very fun topic of conversation. You should have Rich back on the show to talk about his current thoughts on this. I won't want to take any of his thunder and share it. But the two examples I was just pitching are both sort of assuming that the network architecture itself stays the same, that the com nets connected to the fully connected network, all that stuff.

54:46You have all of the pieces of the network that are more or less the same shape and size throughout the course of learning. And then you're thinking about how do you modify those? There's a very reasonable, natural approach that you might consider, which is, well, how do we have the architecture of the system? So this is a separate piece of continual learning. Like one is, well, we've got a set sort of skeleton. And now we're just thinking about how beefy the muscles are in different parts of the skeleton. And the other is, well, let's just add more bones onto the skeleton and grow muscles around them.

55:13The weights being the muscles and the skeleton being the architecture connections. So I feel like that's a really very interesting. And it's not like there's deep lines of thinking on this. This is not just like, oh, shoot, we just figured this out this year. We should think about it. But respecting all of those old lineages of thought, like how we might grow a network from scratch is very constructivist. It's like you start with a single weight and a single input and output and a single weight. And well, clearly you're not solving an XOR problem with that. Like, sure, sure not. Okay, well now we have like, oh, we got like a bunch of inputs and you're still not solving an XOR problem.

55:46Oh, but if you just add this and then it crosses, oh, cool, you can solve, you can solve a different kind of problem now. You're like your perceptron goes into a multi-layer perceptron. So thinking about how a system can use its own learning experience to grow and change its architecture in addition to its weights in a continual sense is a very interesting, I think, a very exciting open problem. And luckily, people are thinking about it. Again, go talk to Rich sometime and see what he's been up to. In your work applying reinforcement learning to prosthetics, can you just talk a little bit about that?

56:17And maybe we'll do a separate episode at some point. but uh uh how how how you're approaching that yeah i think uh just as sort of a reprise the like we're really thinking about the interaction between a human and artificial limb and how artificial intelligence and machine learning technologies can allow someone to use an artificial limb to do all the things they want to do in their daily life that they're not limited by the technology that they want to express themselves they want to engage in their communities they can do that so how do we use ai machine learning to sort of power that interaction and the view that we take and the view that i've sort of based my entire research line on is in fact a very controversial one i'd say which is that to take the to consider the relationship between a person and their artificial limb as a multi-agent relationship now this is like this is actually very contemporary because previously the sensors the actuators the computation available sort of prohibited us thinking about the machine part of that relationship as an agent.

57:21Like it wasn't able to build models of the person in any appreciable way. It wasn't able to consider the environmental factors. And so I think now what we're seeing is that with these surges in computational capacity and the ability for systems to be more like agents, we can now actually think about what would it mean to have a person and their bionic limb jointly acting in the world, working together to try to affect change in their shared environment in both space and time. And I think that is a really neat and different perspective. And so, yeah, we've been considering that quite actively. That's the unique sort of line into it that we've taken.

57:56Okay. Well, I don't want to take up too much of your time. As you can see, I'm operating kind of at the outer edge of my knowledge. But that's what I like about these conversations. I'm sort of expanding my network. I mean, to that point, that's what the brain does, right? It grows new neurons. You're absolutely right. And we have left behind the thinking of the brain as this crystalline entity that slowly decays over time. We know there's new neurons created. We now know that it is, in fact, a system that is continually refreshing itself. That's a big change in the neuroscience community over the last 50 years.

58:37And I think that is, again, a nice way for us to consider how we might think about our digital systems as well. The Alberta Plan is an approach, is a point of view that sees reinforcement learning as the key to building intelligence. And there are various research projects under the Alberta Plan that are either domain-specific or more theoretical. what's the end game of the Alberta plan if there is one if you look at the like so we have there's 12 steps to the Alberta plan in addition to those sort of those commitments we have I think it is like if you look at them all it's it thinks about prediction about being able to have control behaviors about being able to planning with learned models and if you put those all together you get a system and this is the end game it's a very simple one it's right and I think in the first paragraph of our manuscript which is that you get a system that can truly learn from its raw sensory motor experience and continue to do so over whatever its lifespan might be in any environment, any environment, not even a human environment.

59:48And I think that's a really beautiful thing to aspire to and something that will help us more fundamentally understand the phenomena of intelligence itself. What does the future hold for business? Ask nine experts and get 10 answers. Bull market, bear market, rates rising or falling, inflation going up or down. Can somebody please invent a crystal ball? Until then, over 40 ,000 enterprises have future-proofed their business with NetSuite by Oracle, the number one cloud ERP, bringing accounting, financial management, inventory, HR into one fluid platform. With one unified business management suite, there's one source of truth, giving you the visibility and control you need to make quick decisions.

1:00:42With real-time insights and forecasting, you're peering into the future with actionable data. If I were a larger organization, this is the product I'd use. Whether your company is earning millions or even hundreds of millions, NetSuite helps you respond to immediate challenges and seize your biggest opportunities. Speaking of opportunities, download the CFO's Guide to AI and Machine Learning at netsuite.com slash ionai. That's netsuite, N-E-T-S-U-I-T-E dot com slash ionai, E-Y-E-O-N-A-I, all run together to get the CFO's Guide to AI and Machine Learning. The guide is free to you at netsuite.com slash ionai.

1:01:39netsuite.com slash ionai.