In short
The TWIML AI Podcast Episode #670 Summary
Episode Overview
- Podcast Title: The TWIML AI Podcast
- Episode Title: AI Trends 2024: Reinforcement Learning in the Age of LLMs with Kamyar Azizzadenesheli
- Host: Sam Charrington
- Guest: Kamyar Azizzadenesheli, Staff Researcher at Nvidia
- Focus: Discussion on advancements in reinforcement learning (RL) and the integration of large language models (LLMs) in RL applications and strategies.
Key Themes and Insights
- Integration of LLMs in Reinforcement Learning
- LLMs are transforming RL by providing knowledge abstraction, allowing RL agents to leverage world knowledge rather than relying solely on trial-and-error learning.
- Examples of RL applications enhanced by LLMs:
- ALOHA: A robot capable of learning tasks like folding clothes with the help of LLMs.
- Voyager: An RL agent utilizing GPT-4 to outperform previous systems in games like Minecraft.
- Advancements in RL Research in 2023
- Notable excitement in the RL community, with significant breakthroughs attributed to the capabilities of LLMs.
- Traditional methods of RL, like trial-and-error exploration, are being supplemented by LLMs' ability to instruct and guide actions based on context and knowledge.
- Theoretical and Practical Perspectives
- Theoretical Perspectives: The introduction of LLMs necessitates a reevaluation and reformulation of RL theories and methodologies.
- Practical Applications: New algorithms are being developed to tackle complex tasks more efficiently, utilizing LLMs for:
- Interactive learning with agents.
- Designing rewards for RL tasks.
- Examples of RL Applications and Their Impact
- Cooking Pasta: An illustration of how LLMs can guide RL agents through complex, multi-step tasks by breaking them down into manageable subtasks.
- Robotics: LLMs enable interactive communication with robots, allowing for real-time instruction and feedback, thus enhancing the learning process.
- Risk Assessment in RL
- Increasing sophistication in assessing risks associated with RL applications, particularly in sensitive areas such as finance and healthcare.
- Importance of designing RL algorithms that optimize for risk as well as reward, highlighting the need for fairness and accountability.
- Future Directions for RL
- Emphasis on the need for more foundational work in integrating LLMs with RL, including:
- Developing algorithms that utilize LLMs to optimize exploration and exploitation strategies.
- Addressing the balance between theoretical rigor and practical applications.
- Discussions on the importance of compute resources in pushing the boundaries of RL capabilities.
Conclusion
- The conversation with Kamyar Azizzadenesheli highlights the dynamic landscape of reinforcement learning as it adapts to incorporate advancements in large language models. The integration of these technologies is poised to propel RL applications into more practical, real-world scenarios while addressing challenges related to risk, fairness, and computational efficiency. The future of RL appears promising, with ongoing research and collaboration expected to yield significant advancements in the field.
Additional Resources
- For complete show notes and more information about this episode, visit [TWIML AI](https://twimlai.com/go/670).
Written by AI. May contain mistakes. Listen to the episode to check what was said.
Transcript
Automatic transcript. May contain errors.0:09All right, everyone. Welcome to our AI Trends 2024 series. As you know, each year we invite friends of the show to join us to recap key developments of the prior year and anticipate future advancements in several of the most interesting subfields in AI. Today, we're joined by Kamyar, Aziza Denisheli, research staff at NVIDIA to talk about all things reinforcement learning. Kamyar and I last spoke a couple of years ago for our Trends 2022 series. Of course, before we get going, be sure to take a moment to hit that subscribe button wherever you're listening to today's show. Kamyar, welcome back to the TwiML AI podcast.
0:49Yeah, thank you so much. Thanks for having me back, Sam. And it's a pleasure to be here talking about reinforcement learning AI and new tools and new technologies that are pouring into reinforcement learning that are going to shape the future of reinforcement learning. I'm glad to be talking about these subjects with you. All right. So, Kamiar, let's maybe get started with your general take about 2023 as a reinforcement learning researcher. I think clearly a lot of excitement in 2023 has been focused on LLMs and generative AI. And you kind of get a couple of reactions, at least on Twitter. there's the one, hey, there's other research happening in AI, but then there's another reaction, which is really LLMs have implications broadly beyond just chatbots.
1:45Let's embrace this and see what we can do with it. I think I know what your answer is based on some of the themes that you prepared for our conversation, but I'd love to hear you just kind of riff on how the year felt as an RL researcher. Yeah, that's a very fascinating question. and also a fascinating time to be working in reinforcement learning while having ideas of classical reinforcement learning, the topics we have been developing since like 40, 50 years ago in this field. One of the major theme of this year and also the future theme of reinforcement learning is upon this creation of LLMs and generative AI tools in the sense that we now can use these tools as a means of knowledge abstraction, I would say.
2:40These models, they are chatbots, but at the same time, they are extremely flexible. They are multimodal and they are able to generate not only text, but images and videos and create imagination for AI agents. And when you enable such an imagination for AI agents and provide layers of abstractions for AI models, we can use these to frame novel and new foundational reinforcement learning principles basically it means that if i want to train a reinforcement learning agent these days i don't need to start from scratch we can actually use the power of this llm and generative ai models to instruct the ai agent so ai agent doesn't need to or reinforcement learning agent doesn't need to go all the way down and do all the exploration and exploitation and learning and try an error all by itself without having much abstraction in the knowledge related to the task at hand.
3:49Given these LLMs these days, we can actually use these LLMs to instruct these reinforcement learning agents to tackle hard tasks. It means that we can actually go after many, many tasks that before were out of reach because we didn't have right level of abstraction. This is going to unleash so much research. I'm very excited about it. It had started. There are so many works these days that researchers and colleagues that develop methods, which uses these large language models and generative AI models to reward design and break down the tasks, navigate robots, navigate AI agents or reinforcement learning agents in complex domains.
4:31And this actually enables us to take leaps that I would say we thought for a while were like way out of reach. But nowadays we are getting there and I'm very excited about that. Awesome. I'd love to have you talk a little bit broadly about 2023 and kind of how that shaped up as a year for our research. like just in broad generalities? Was it an exciting year with lots of breakthroughs or was it a boring year? What was your general take on the year? Okay, that's a great question. I want to answer this question from two perspectives. One of them is from a theory perspective. One is from practical, empirical study perspective.
5:17From theoretical standpoint, we have been extremely excited because having LLM in our formulation of reinforcement learning enables us to go and develop foundations given LLMs. So is it kind of rewriting all of the theory to be old stuff plus LLMs? Exactly. So we want to reshape and reformulate reinforcement learning. I can tell you, in 1995, we were talking about hierarchical reinforcement learning a lot. Like every day. I mean, I was not there, but my colleagues and friends were there. And thankfully I had enough time during my PhD to read so many of the old literature. We were talking about hierarchical reinforcement learning, abstraction in reinforcement learning.
6:03All of them were like hypothetical and all of them were based on these assumptions that there exists in hierarchy that we can extract. Okay. But we didn't have any of those things at the time. We actually have a tool that lets us do that. Exactly. Now we have a tool that literally allows us to do that. And this is from a theoretical standpoint. We will reformulate reinforcement learning condition to the fact that we have access to LLMs. This has great implication. I can give you examples what reinforcement learning agents would have done before LLM and now with LLM, how they would tackle problems.
6:44Before we do the example, you said there was the theoretical perspective and then the practical perspective. So for the practical perspective, now that we have this LLM, we can do so much. We can develop so many new deep RL agents, basically, that they can tackle extremely hard and challenging problems. and another aspect that it actually unleashed is the interactive learning of reinforcement learning agents and and robotics now before in the past if you wanted to train a robot we had a design task and the design the goal plus the cost functions and the agent we would use to like optimize the agent to tackle that task now we can interactively talk to to the robot helping the robot to explore the world.
7:34We can actually tell the robot, hey, first, in order to accomplish this task, you need to first go there. And from there, you need to go to other place and sequence all of these tasks, all of these steps while just talking to the robot. We don't need to go down. The abstraction is like, we can actually have a communication with the robot to get to the goal of training. That's the second thing in reinforcement learning that we can have interactive interaction with the RL agents. and bring LLMs to design reward. This is very important. We now can have LLMs to design rewards for reinforcement learning agents.
8:11So not only they can instruct the reinforcement learning agents, they also can design the reward, both LLMs and generative AI tools, design reward for RL agents to accomplish very hard tasks. So that's from a practical standpoint that many, many of colleagues and friends are working on this. And I think 2024, we will get it, But for the future steps of reinforcement learning, we will have a lot of these things in our new technologies. If I can try to paraphrase those two aspects, and this isn't necessarily the theoretical versus the practical, but I heard kind of two important things in there.
8:48One is that LLMs kind of bring to the table a world knowledge, if you will, that RL agents didn't previously have or they had to build up from scratch from nothing. We remember seeing the videos of like RL agents kind of banging their heads against the wall, so to speak, in games, trying to figure out like some basic things. And now, as we'll talk about in some of the examples that you curated for us, they're able to start from a higher level and access this essentially a database of the world or a reasoning engine about the world that's provided by LLM. So in a sense, it's the advantage of or the power of pre-training that we've seen in so many places in other aspects of AI.
9:33The second thing that I heard you say is that LLMs can be integrated into reinforcement learning such that they, I guess, provide more intelligence to the way we direct the agents kind of internally, more of a structural impact or structural influence on the way the agents behave through manipulating the reward or the objectives based on their knowledge of the world. Is that a good summary of some of the key ways that LLMs have impacted RL? Yeah, these are basically the two main things, the global knowledge abstraction that they have about the world and the fact that we can use them to instruct RL agents.
10:18Awesome. So you mentioned that you had a couple of examples to give to illustrate this for us. Yeah, the first point that you made, the fact that they have knowledge of the world and the fact that all the RL agents, they were banging their head against the wall and they were doing random stuff. So one example of this one in reinforcement learning realm is consider a kitchen and a robot or an RL agent there. And the task is making pasta. So that's the whole thing. And all of this, I could communicate with words with you. In the past, I'm going to be a little bit exaggerating here to make the point.
10:55Fair enough. But in past, the RL agents, when we were putting it in the kitchen to accomplish such a task, and we had a reward function there, which is like, if you cook a pasta, you get a reward one. The RL agent, both in theory and also in practice, could have thought of making an airplane in the kitchen as a possibility of the action space. The RL agent had no idea about abstraction of the word. It was a plain plate starting from scratch, not knowing anything about the world or any understanding of what is even pasta. It was mainly through trial and error. Agents we used to make for Atari games had no idea when they were starting.
11:40They had no idea about the game or what is the thing they are dealing with. They had no understanding or abstraction of the environment. So at this point, now that we have these LLMs, they can literally break down the task of cooking pasta in the kitchen. They would not instruct the RL agent to consider making or creating, building an airplane in the kitchen as a possibility of potential outcome. This makes me think back to one of our earliest conversations about RL. I think the way I remember you describing this is that one of the key challenges in RL is making the problem tractable by kind of carving up the space of possible things that are limiting or constraining the space of possible things that the agent can do so that you don't have to try everything.
12:34And what LLMs do is give the agent reasonable things to do as opposed to having to try every possible action to achieve the goal. Yes, exactly. That's actually the case. The point you made was literally what is happening in the past. We knew we need to have narrowed down this space. We knew that we need to narrow down this space of possibilities. We have so much theory and philosophical thoughts about it. But we didn't have that abstraction to do it. Now LLMs would give us. Now LLM says, you want to make pasta? Explore everything. Just go, go to the cabinet, take pasta out, boil water, breaks down the task for the agent, and agent is going to solve the subtasks.
13:18And even for the solving a subtask, the LLM or the generative AI models can go and instruct the agent. So this is fascinating. Let me tell you what I mean by instructing. The LLM can ask or RL agent accompanied by LLM model can go and break down the task of making pasta to many, many sub tasks. And for each task, let's say the first task is going to the door of the fridge. So that's the first task. So this task of going to the fridge door, I can write in words. And Genitive AI can think about what that would look like, right? And the RL agent goes and does something. And then we can use LLM to see, to describe what the agent has done.
14:05And if the thing that agent has done, which is like a video, RL agent goes and does something, we can see it, record it, give it to the Gen AI. Gen AI model or LLM would give a caption. And if the caption doesn't align with what has been asked, you're going to lose points. So this way, you can actually use these models to instruct RL agents. So you break down the task first using the abstraction or knowledge of the world that LLMs have. And then after that, for each piece, or when you break down the problem to smaller pieces, for each piece, you can actually use the LLMs to instruct directly the RL agents to accomplish the task.
14:45So this is basically one of the intuitive examples, but theoretically we may be given some other examples. An example is like, let's say your task is to get to the door of my apartment and there's a key on the table. RL agent before LLM would have tried all possible ways to get to the door. Let me try to open the door with the chair. Yeah, literally, that would be one of the ideas. And but for millions of time steps, they just sprint to the door. Right. The task is to get into the door, they just sprint to the door and they don't even look at the key. But if they know, if LLM tells them this knowledge of or concept of the key, the problem is going to be solved in two steps.
15:26So this is what makes us about bringing this to reinforcement learning. You identified a couple of papers that were good examples of this kind of integration of LLMs into the field. And your examples really remind me of the Voyager paper most directly, which I thought was super interesting. Can you talk a little bit about, give us an overview of that paper? The field is full of smart people. They come up with, the researchers come up with many great ideas to develop models, reinforcement learning models that utilize these LLMs. And in this work and a few other works that are doing similar approach, you can actually use LLM to generate codes that when you compile, your agent does the job you want.
16:13Your LLM generates a code, your agent does something, and then you ask whether that thing was right or not. If it was not right, you actually tell LLM to tweak the code. So it's like LLM is actually going directly, not only instruction by language, it actually goes and code up the RL agent what to do. So that's very fascinating. What's the role of this code in the Voyager paper? The code can tell you that if you hit a wall, what to do. If you have a hammer in your hand, this is for Minecraft. You can say, if you have a hammer and in front of you is like a gem, go with it and just break it down.
16:52If you're seeing a beast and you need to switch your tools to something else and go to a like a fight with the beast or take the beast down or if you see some i'm not an expert in minecraft but if you see something cool that needs to be done to gain something the agent would just go and do that so that's basically the code level so you don't train let's say dqn model for the rl agent you'd literally go and code it up and say if you see this thing go up or if you see this thing go down or break it down or change your tools or gain new tools or put something in your bag or something? I didn't fully get the role of the code.
17:32I thought it was super interesting that code was being generated. I didn't realize that it was replacing the traditional RL algorithms. There are some great examples of similar to the ones that you gave, but we'll take the inventory of you have a bunch of tools like a planks, a stick crafting table and a stone. and that's given to GPT-4 in this case. And then GPT-4 is providing a task that moves the agent towards a broader goal. So incorporating the idea of curriculum learning. So as opposed to giving it this end goal, like just do this next task. One of the things that really struck me about this paper, which is I guess I'm trying to get to, is that it incorporated not just LLMs, but a lot of disparate ideas that we kind of talk about in the context of RL, like curriculum learning and interactive prompting and a bunch of different things to create this model that ultimately seemed to do better in terms of collecting different things in Minecraft.
18:41Again, I don't know Minecraft very well, but that seems to be the metric they were demising for. Yeah, I can imagine this curriculum learning, basically means like you break the task to smaller tasks through having the knowledge of the world. That is actually what fuels all of these methods to accomplish tasks that are very challenging, basically. There was another paper that you identified called Mastering Diverse Domains Through World Models. Tell us a little bit about that one. That paper and also in that direction generally is, I would describe it in a sense that how you can bring more of LLM to this picture.
19:20You can think about having generative AI to imagine things for you before going to accomplish a task. So let's say I have a robot and the robot's task is to go from the point A to the fridge or to the kitchen. And the robot can actually build an imagination of this prescription and consider that one as its goal. The idea basically, I mean, generic idea is if I'm looking at you and at the same time, just think about going to the fridge, I can imagine it. And then that would be my goal. And then when I get up, I can actually, based on that imagination, considering that imagination as my goal state, I can aim at going there.
20:06Okay, so that's one of the new concepts in reinforcement learning that LLMs can provide. previously i said that the agent does something we have a video of it we can caption that video and see whether the caption is in line with the prescription here's is other way you give the prescription the model actually imagines it given the context of the world and consider that one as a gold state or gold trajectory and then it comes up with a with an algorithm to follow it that's basically one thing that's exciting these days. You can train the model of this sort on many, many, many multi-modeled scenarios and many domains and get a versatile model, which is able to tackle extremely challenging tasks through imagination and setting goal for yourself.
20:54I mean, the AI agent can have a goal for itself and just follow that goal that is imagined. And over time, the model is going to get better and better. That's a philosophy of using Gen.ai from other perspective, going from description to imagination. You imagine what you want to do and set that one as your goal and try to accomplish it. Another aspect is like you do something and then someone describes it for you and see whether the thing you did actually was right, according to the description. So these are the two directions that researchers are working on. Depending on the task, One of them is better than the other one philosophically.
21:32But at the end, when the models get bigger and the knowledge gets deeper, they can become almost identical, except the computation side that we need to balance. Got it. In your examples, you've mentioned robotics a few times. That's another area where there's been a lot of interesting work this year. How are you thinking about that part of the space? in terms of robotics as this interactive use of llms are heavily related to robotics we can literally tell the agent that we wanted to go to the kitchen and literally ask the agent to consider goal state and the agent goes there these are actually recent works people in the area have been doing will imagine what would be the the outcome that you're aiming for and tries and goes and accomplish that or your robot is supposed to pick up a cup and you literally can say hey go a little bit to left you're holding in the bad way or after you lift it up you need to go to left or right and put it somewhere else and these are things that we can teach the agent in basically robot in an interactive fashion so this is like similar to the previous discussion we have had there are works that people do use llms and gen ais in general for both imagination and also the description there is another area of area that sometimes people refer to as robotics but sometimes as control this year and also a few years back have seen a lot of advances i would say let me put it this way so in control that is like the place that we again, develop reinforcement learning agent to solve control problems.
23:17For those settings, many of the problems are not that flashy, so they don't make it to the media that much, but they have been adopted a lot in industry and also in academic institutions. There are so much research in those areas. Is it that they're not flashy or that there are traditional control methods that are proven and work well? I always had the impression that it was more that than And I mean, I guess those two things are intertwined, but maybe the question for you is, given that for the class of control problems, we have solutions that are well understood and based on heuristics, physics, those kinds of things.
24:01Why are you seeing more reinforcement learning solutions being deployed? What are the advantages of those solutions? That's a great question. Let me give the example that is actually currently being used. We have, these are not jets, but these are small planes, but they're fast planes and they have wings. Okay. And when we deal with controlling the wing, so you need to put a lifting force to the wing to turn it in order to stabilize the plane. Traditional methods have been working very well. There's so much approximation. mission. When you have a wing which goes through the fluid, we have a ton of turbulent interaction between the wing and the air around it.
24:43So in order to solve the turbulent flow around it to see what is the optimal behavior of the wing, for each second you might need like a month of computation to come up with. So if you want to really see how the air around the wing of such small fast planes evolves we need basically a ton of compute and we don't have it because we need to make a decision and that fraction of a second we don't have a month to design and understand what is optimal control so what have been people doing they came up with a lot like very nice and great working and practice approximation to approximate such dynamical systems but all of them are based on approximation and they're like years of fears of engineering needs to go there to design or tune these approximation methods.
25:28And in the end, you're not getting... For new planes or new types of vehicles. Yeah, for new planes and new types of vehicles, you need to do the tuning again. But even for this given ring, if you do all the tuning, one AI agent easily can go and outperform it by like 30, 40%. We are not talking about Boeing. We are talking about smaller businesses. And the new industry people and stuff spending like five, six years tuned those parameters, they just use rl methods get the really good performance and move on the same thing is is holding for imu for like sensor localization that we're using for almost everything on the planet your phone your watch everything has its imu which is localization engine all of these are based on approximation of traditional methods but now you use reinforcement learning models and AI models in general to navigate those.
26:22Or for drones, I remember we could fly them in turbulent wind when the wind was like five meters per second. With RL Asian, you can take it to 20 easily, five minutes of training. So these are things that are making big improvement in this area. But yeah, they are not LLM flashy, but they are immediately deployed in practice by industry. And RL is a broad field. Are these examples here, like online RL, offline RL? Is it traditional agent-based RL or is it other types of RL? No, these are traditional concepts of reinforcement learning in RL. For the drone setting, these are all online. So you do offline pre-training, basically off policy setting.
27:11You do pre-training, but there's another thing. for this wind tunnel settings that there is a wind blowing to the drone the wind is constantly changing and it's turbulent it's it's not a constant wind is hitting it's just adversarially changing it goes up down change the direction and the wind is not just flat thing coming to the drone you might have higher wind on the top lower wind and the lower wind speed the bottom and there like many many weird pattern that blow at this uh this drone and drone is supposed to go and do a very detailed maneuver and this adaptation happens all online so online basic reinforcement learning and it's policy gradient style methods for the smaller jet and fast planes that i was talking about there are some model free methods but this one is i had in mind that is actually does adaptation very, very fast and it gives like huge performance boost is based on model-based reinforcement learning.
28:11Actually, fun fact, model-based reinforcement learning for such problems is extremely fast and accurate. You don't need to always go with model-free for so many of these problems. If you have some ideas about turbulent flow, you can go with model-based. So these are literally traditional reinforcement learning methods, but adapted and developed specifically for a problem at hand. So one thing in machine learning, and all this I tell my colleagues across other fields who are not in machine learning is in machine learning, we develop concepts and algorithms, but for each domain, we need to invent new ML foundation.
28:47If you're working computer vision, we develop many, many ML foundations, architectures, problem setups. If you go to language, we do other stuff. If you go to voice and sound, we developed other methods and or music we did develop other methods if you go to jets or if you go to airplanes if you go to drones if you go to like walking robots we need new algorithms and new foundations these are different domains so if you design something very specifically dedicated for drones you will get an algorithm which is extremely efficient these areas are moving fast but yeah they don't make it that much to the news but i'm excited about those directions a lot.
29:29Interesting. You identified the Aloha work as one of the examples of some of the progress that we've made in robotics this past year. And that's probably best thought of as like a series of work. This is Chelsea Finn and others at Stanford and maybe some other places. I guess to start things off, like when I've read up on Aloha and it's been in the news recently with Mobile Aloha. It's like they're doing a lot of things. There's one aspect of it that's like a systems integration kind of challenge and like creating these low-cost robots. At least for me, like the RL innovations kind of got lost in the mix of all the various things that they've done.
30:12There's like a telecontrol piece. What's the core RL thing that they've accomplished with that paper that prompted you to bring it up? Yeah, that's a good point. In robotic settings, building the robot and getting it up and running is a huge task, as which they did a great job on that. From RL's side, the part that is fascinating is this interaction. And maybe let me interrupt you and have you explain a little bit about the project and the robot that they created and what it's able to do. And then with that as context, you can talk about the RL magic. They have this robot, the new robot that can interact with the world they created around it.
30:51There are many tasks like folding stuff on the table, opening a door, walking around, moving around in open areas. So that's basically the setup. Robot is extremely agile, meaning that it can do things fast. It can do various stuff. It's not just one thing, folding shirts and stuff. It can do many things. So that's the setup. But the RL part that is exciting and I think is a direction that's going to help develop the future of robotics in many aspects is the interactive part that you can interact with with a robot using recent technologies in generative AI. That aspect that you can interact with.
Read the full transcript
31:29The earlier topic we discussed that this new foundation or new novelty of using LLM in reinforcement learning and robotics is interaction. And in this work, they actually do interaction with a robot in a language sense. That's the part that is novel. The rest is developments that have been in the field in the last many years. The fact that they could have this agile robot that can take on many, many tasks and deploy LLMs and being interactive with the user or the instructor, the person, that was the novelty that they have had. And we will see more of it. What's an example of that interaction? The examples that I've seen, like you have a person.
32:10So a big part of what they've done is like imitation learning. so someone is essentially controlling the robot to do the task that you want the robot to do and then after a few times much fewer than previously like the robot knows how to do it but they're using like controls as opposed to i didn't see like the language part how does language come in by language i'm referring to all the gen ai here as llm one thing that's very important here is that they have been doing. And also it will be the direction that would, in terms of prediction, I think is going to be the way that we will develop our robotics agents is you do a lot of, similar to actually the way we train large language models is we train them on text, on passive text for like gazillions of data points.
32:58And then in the end, we do our LHF. we tune them according to human feedback. One thing here, this work is on that path is that you can do a lot of pre-training on human interaction or even on video in general, and many of the past interactions that robots have had. And then in the end, you can do our LHF style approach to just fine tune the model. Their steps are in this direction. And it will be like the way we will be doing many of this robotics training, I would say. Got it. So a big element of it is not necessarily that a human is interacting with the robot using language, but the robot has this linguistic resource that it can use for description of what it's seeing and for hints as to what to do next and how to interact with the environment.
33:51So in a sense, it's not necessarily seen in the interaction between the user of the robot or the robot and the environment. It's just it's a resource that the agent can take advantage of. And the API for that resource is language, among other things. So that's basically the foundation that it will be used in the core of the algorithms for robotics that you can basically interact with the robot with using like basic language. So that's one of the things that is going to enable robotics in the years to come. Got it. Awesome. Another theme that you identified is the idea of risk assessment. And the last time we talked about, hey, is RL ever going to be used in production because it's so hard?
34:37And now you're like, RL is being used in all of these devices, planes and all these production and commercial settings. things. And that raises some interesting questions about how do we assess the risk that the RL doesn't do what we want it to do or does bad things. And it sounds like over the past year, we've gotten a lot more sophisticated in how we're able to think about and analyze that risk. Can you talk a little bit about what's happening there? Yeah, you brought up a great point. So there are two points in risk assessment and reinforcement learning. One of them is that you have this RL agent, let's say in small planes flying, and you care about the risk.
35:19How risky are they? Are they going to fail? Are they safe? That direction is mainly about safety and constraint satisfaction. There's another which the field has been working on it. Community has been working on it and we have so much improvement and the methods are so good that they barely violate these constraints. Another setting is you actually have a lot of risk in your outcomes. It's not about just a RL agent. Let me give you an example. In the stock market, if you have a policy which does the trading or in healthcare that you have a policy which does the measurements, there are risks in the outcome itself.
35:59In the stock market, you might actually sell things or buy things, but on average, you might do well with some risk level. And the question is, can you develop a model or policy which gains a lot of money, let's say in a stock market or hedge funds, that gains a lot of money, but at the same time, making sure that your risk is not high. So basically you do reinforcement learning with the objective of minimizing the risk. And risk is like a variance of your reward or variance of dollar value you make per day. To restate that, historically, RL is about optimizing an objective function or a loss function.
36:41And that loss function is usually expressed in terms of some average and expectation. And what that doesn't then capture is the associated risk in meeting that expectation. and you're saying now we're getting more sophisticated into incorporating those risk factors into the reward function itself. Yeah, one of the traditional example a friend of mine used to give was, let's say we come up with a policy which increased as an RL policy, which is optimized and trained to maximize basic wealth of United States people. And after deploying it, we were saying, hey, we increased it by 5%. And then you go and look at it as like top 1%, got a lot richer than bottom 99 okay so that was not outcome you wanted you just said you wanted to be like you want to increase the wealth of the nation but the risk was that if you look at the distribution of reward there are small tiny teeny number of people they got a lot of money and the rest of the population didn't get that much money so that's like basically a risk and we got yeah sophisticated enough that in banking, in healthcare, we actually go after optimizing for risks directly and considering those in our algorithms because it just matters.
38:03In healthcare, you don't want to improve the outcome of your policy or prescription method that on average increased the health of the people by 5%. You don't want that because if that happens, there's a possibility that you just killed so many few, but you just gained too many more. So it's just not the right metric to look at the average reward. So taking a look at the risk is now bread and butter in these domains, specifically in insurance companies and stock market and healthcare. They care a lot about these things. And speaking of which, people in insurance companies were the people actually started writing mathematical books about risks in the 60s, 50s.
38:49So the whole idea of risk did not emerge from there, but they were pioneers of it. It's foundational to insurance. Absolutely. Yeah. So many risk functionals that we have in reinforcement learning that are named after insurance policies. That's how influential they were. So they care a lot about risk everywhere and they know the value and importance of AI and reinforcement learning. They are bringing these ideas to their settings. Okay. When I say they bring these ideas to these settings, I don't mean all the insurance companies are like that. They are starting. That needle is tilted. They are not doing it.
39:24Now they're doing it. Same as banking and healthcare. Part of what's interesting here for me is that you're speaking about this in the language of risk, but it also has clear implications to fairness and responsible deployment of RL algorithms. Yeah, that part specifically in healthcare and loan allocation, it becomes extremely important. These are areas that you will not be able to mathematically model. So when we're talking about these concepts, we need to read the book. We need to read the law and see based on law, what are the things that are actually compliant with the law. And for that, the law is not based on mathematical equations.
40:08It's just a very weird thing from people like me, from our perspective. It's just so much details in it that lawyers need to parse it for us. And when we parse those, then we need to turn those to mathematical formulas. And that's the part that is extremely hard. But if you do that, you would get something that is functional and useful in practice. I would say the models are getting to be really good. People in this industry that are working towards this concepts of fairness and understanding and interpretability moving in these directions. But we expect to have more and more of these in the future because these companies start deploying RL agents everywhere.
40:52we need to have more of the congress forcing to go there and make sure that these things are complied at the moment we are a little slow because in the country we have so many issues that people are taking care of but we should start allocating resources to do these cases where reinforcement learning is used in practice specifically in healthcare and banking systems and we need to have clear-cut laws to navigate these companies. What I hear you saying is that there's broader societal work that needs to happen to align on the right way to think about these rewards. And in parallel, the RL community is trying to get more sophisticated in the way that factors like risk and responsibility can be expressed in rewards functions.
41:45So hopefully when the society gets there, the technology will be better equipped to be able to implement it. The field or the people working in this area would get much more attention when these companies start making a lot of impact to the society. And then we need to have immediate response to come up with new laws to navigate these companies who are trying to improve human life status. Okay. Awesome. So let's maybe shift gears and talk a little bit about the future and how you see things evolving. I imagine that getting started in this conversation about the future, we're just getting started in integrating LLMs into RL.
42:29You've said as much. I imagine that that's one of the big things you see moving forward. Yeah, certainly. The use of basically two paradigms that we already talked about, but we can visit one more time. One of them is the fact that we use LLMs a lot in our Gen AI in general, a lot in RL formulation and our algorithms, both in terms of imagination and also description, instructing the RL agents to work in practice. So that's one general theme that we are going in those directions. And I'm excited about seeing new works and capabilities coming out of it. Yeah, I'm not going to predict in two years we're going to have robots at home, but it feels much closer than I used to feel like a year or two ago.
43:19When you look at a system like Mobile Aloha, and we'll put videos, some maybe demo videos or something like in the show notes page. When you think about a system like that, do you think that the kind of core remaining challenges are RL related or are they systems integration related or mechanical? Help us understand in the context of robotics, how much of the delta between, you know, we're living in the Jetsons and where we are today is RL and getting that part right? I might be a little biased. As an RL researcher? Yeah, but if you ask a person from mechanical perspective or like building robot perspective, they might say, yeah, there's a lot of work needs to be done to make the robots better.
44:07But for those, I think the roadmap, at least from my understanding, the roadmap is clear. If you want to make a really crazy, agile robot, we have the physics and you can build it. Okay, it's just a matter of building it. We have open questions, but they're not open questions that have not been solved in the sense that the variations have been solved. Another way to maybe say that is we've got the various Boston Dynamics robots, but we're not setting them out loose to do things because the intelligence part isn't there. The mechanical part is. Yeah, if you want to build it, we can build it. But yeah, of course, it takes extremely talented people in that area.
44:48Not to diminish all the other things, absolutely. Yeah. But if you have enough money and resources, can you imagine how to build a really fascinating robot? The answer is usually yes, they know how to. But for us, if we still don't know, we're still doing actively basic research to see what is the right way of designing intelligence of the robot. And we are not there yet. And these are open questions. Getting things working together is very important. And so nowadays that we have new Gen AI technologies that are going to be used more and more in robotics, getting those and seeing what is the right way of using those is an open question.
45:28We don't know how to do it. And as you know, in machine learning, we have a task. We try many, many ideas and we see which one works and we communicate with the rest of the field, which is the beauty of open research in machine learning that I should use this time and praise it. We should keep doing open research. It would help to basically many people would start working in this area and develop methods to see what is the best way of developing this Gen AI accompanied robotics models to tackle like hard tasks. So that part is not obvious to us at the moment. The similar version of it has not been solved.
46:06So we don't know. If we get a really good RL algorithm that can be used in robotics and the robot can achieve, there will be so many effort to build more advanced robots. The problem we have at the moment is the intelligence part of it is very primitive. And so if you were to drill into that, you're essentially talking about reformulating RL, having LLMs as a tool. Can you identify specific directions that you think need to be explored or specific kind of threads of research that you think are promising that you expect to yield fruit over the next one to five years? From theoretical perspective of reinforcement learning, we have so many algorithms that are provably optimal.
46:57Optimal in the sense that given the assumption that we have, they're optimal. So now we need to go and design algorithms, the optimal algorithms that when you give them the power of LLM, the power of this abstraction that we have, which also we need to mathematically define what these powers actually mean in practice. sorry, in theory, we need to go and design algorithms that say, hey, given the fact that you have access to LLM or Gen AI tool, what is an optimal algorithm to be used for exploration and exploitation in the reinforcement learning setting? That's from the theory standpoint. Yeah.
47:33So to restate that in traditional RL, we're at the point where we can analytically quantify the expected performance of these algorithms. Like there's an equation that you can write that says, that can say if you have a good algorithm or not. In terms of incorporating our LLMs and Gen AI, we're at the stage of kind of getting it to work and showing examples, but not really saying how good it is beyond benchmarking against other things. and you're saying that from a theoretical perspective, the next thing is trying to be more rigorous around describing the performance of these LLM-based RL systems.
48:19Do you expect to be able to do that? Do we even have good theoretical foundations of LLMs independently? And do you need that in order to get to the level of rigor you're describing with RL thrown into the mix as well? Well, that's an open problem. We don't know yet how to formulate it. There's an open question that using this, given the presence of LLM, how to formulate it. And that's the question that I've been talking to some of the old colleagues who are like classical reinforcement learning researchers, who are still classical reinforcement learning researchers. The only thing we converse is what is the way to do it?
48:58I don't know. But one thing I know is traditionally, when we were not making such an assumption of having access to LLMs, we developed so many algorithms that are optimal and blah, blah. But in the end, there are recent works that actually show that just stochastic gradient descent that we have been using before is as optimal as all of them. So meaning that all these algorithms that we have developed, because we were not making any assumption about the world. You are like generic world, no assumption you can build an airplane in the kitchen setting that has no idea about anything or any abstraction or knowledge of the environment.
49:38Those algorithms that we have developed, they are as good as you might call it dumb algorithm, but it's just very smart. It's just SGD that we have been using forever in the reinforcement learning recommendation system. Like if you look at Amazon recommendation system, is based on SGD. It doesn't use any of these advanced RL algorithms that we are doing. It just does SGD. But these recent works actually show that SGD is actually optimal RL algorithm when you don't make much assumptions. But now that we are having this LLM in the picture, would you say SGD is a smart algorithm? I would say I doubt.
50:15So for when you have LLM, I don't think STD would be the optimal algorithm. We need to have intelligent algorithm, which directly goes and utilizes the presence of the LLM. And one of the directions that some of us are excited about is related to information gathering. Many of the RL algorithms before were greedy about not losing. That's literally the way you can describe them. They were extremely greedy about minimizing the long-term loss. So that was basically the core of all these algorithms. But if you have LLM, you don't need to be that greedy. You can look at the future and decide, hey, should I ask some certain questions from the LLM for a given task to gather information and understanding about the world before becoming greedy to minimize the future loss?
51:07Or I should go directly for the future loss. That's a point that I think it would enable us to develop a new foundation for reinforcement learning, having access to this knowledge abstraction models. So kind of put another way, classically, one of the big challenges or variables, depending on how you put it in, RL has been balancing explore and exploit. It sounds like you're saying that these classical approaches have tended to be more exploitative, maybe. And now we've got this new tool to help us balance, explore, exploit and explore more intelligently. You know, the knob that we're using to balance these two is not like one number, like a percent of time.
51:56It's situation dependent, state dependent, that kind of thing. It's much more nuanced than was previously possible. That's exactly the setting. We have this abstraction. It can be used to direct our exploration to be extremely informative. One of the things that you said a moment ago triggered a thought for me. You talked about how we have all of these RL algorithms that don't have world knowledge. And it made me think about generalizability as kind of a challenge that RL has faced. Meaning one of the reasons why we have all these algorithms is, you know, one does really good on Atari games. Another is really good on Minecraft, Soda or whatever, or Dota rather, not Soda.
52:41But it's been a challenge to have like a general purpose algorithm. Is that still the case without LLMs? And to what degree does LLMs, do LLMs change that? That's a great question. For each domain that you refer to, we have been developing a specific algorithm for. Okay. And back in 2014, 15 and 16, one of our main goal was to solve games. and if we keep that goal of getting better at games yes LLMs can actually come to the picture and basically help the models to RL agents to do a better job in learning when the agent is dealing with Dota the agent doesn't need to start from scratch now we have millions of videos out there we can describe exactly by using LLM each agent is doing and come with a strategy and describe it and this is going to help us to speed up the rl agents when you're dealing with different games however in last year two or even further back the whole focus has shifted a tiny away from game solving if you remember 2016 17 18 was just every everyone was like in reinforcement learning many people in reinforcement learning were very involved in solving dota solving go chess these things.
54:06But now the focus is a bit shifted to more real world applications that we can improve human life directly after building all this foundational knowledge by tackling game of go, game of Dota. So we are actually going more towards like being more domain specific for which LLMs would help. When we do domain specific things, we probably would not have one model or we actually we don't aim to have it's not it doesn't make much sense to have one model which is able to do stock exchange optimal policy development at the same time is able to fold clothes and clean dishes like i'm like maybe not the best idea so we are getting more dominant specific in each sector of academia and industry is focusing on one of these directions however one thing very important about the time that we have been doing a lot of games is the the expertise and knowledge we have developed.
55:04That thing we should not ever forget. The reason we have many of these LLMs is because we have done a lot of work in Atari and those are like precursors of the RLHF is one of the algorithms that people developed. The algorithm used there is one of the algorithms that has been used in one of the works that tackled robotic tasks back in i don't know 17 or the foundation of alpha go taught us a great lesson both alpha go and tdgammon algorithm both of them also dq and all of these taught us a great lesson that when you develop a reasoning algorithm and you care about reasoning a lot you will have a hard time putting all of it in one model.
55:52So when you play the game of Go, we tried a lot for years to train one model which takes input as a state of the game and outputs the next move. We tried a lot to do that, but this is a reasoning task. And we learned that if you want to do reasoning and if you want to train a model which, given the input state, tells you exactly what is the next move, Training such a model is going to be extremely expensive. But AlphaGo taught us we can, and also some other algorithms based on UCT, which is upper confidence tree search, taught us in order to get optimal reasoning, you can balance compute and model size and data size.
56:34Instead of having one model, which is able to tell you the next move, you can actually run a tree search computation and see if I make this move, what happens? If I make this move, what happens? If I make this move and then this move and then this move, what happens? So you just build a tree locally, do a little computation. But at the branch of a tree, you can ask what is the average good move after that. So you can actually use these types of state-based reasoning to gain a lot of intelligence. And this lesson that we learned, I think, would be at some point useful for LLMs when we want to make LLMs to be more intelligent in the sense of reasoning.
57:17So at the moment, the LLMs, you give the input sequence, they output the next word. But if you use LLMs to build that tree for themselves, making many trees and see which one actually is more reasonable one, you can gain your intelligence exponential, maybe the exponential improvement by just doing that instead of making your LLM model bigger and bigger and bigger. So that's a great lesson that we learned from AlphaGo, I would see that that would be one of the steps in the future of LLMs, that we, instead of having them as a reflexive model, which takes the input directly out with the solution, make them a combination of state-based model and reflexive model.
57:56So you can have LLM to predict many, many possible outcomes and evaluate each and then come with a solution, which is the most reasonable one. So this way we can gain a lot, like good amount of boost in intelligence behind these models, which is a lesson we learned from AlphaGo and a few other great works in the past. So historically, RL was seen as this really promising element of the journey towards AGI in the sense that of all the ways that we do machine learning, it seemed to be the one that was most similar to the way babies learn. It's kind of the classic analogy. And in the game setting, having an intelligence that can excel at lots of different games supported that idea.
58:47It's like, okay, we're starting with trying to create this intelligence that learns like we learn, that's good at one thing. Now let's make it good at other things or a broad set of things. Now you're suggesting that we've reached a level where we're now trying to put these into practice and practical use more. And that kind of, by definition, de-emphasizes broad cross-task generality. Do you think that that slows progress with regard to foundational RL capability? You know, not necessarily that AGI needs to be your goal, but trying to get to general intelligence has propelled a lot of innovation.
59:31Like, Do you think that by focusing on kind of these narrow commercial use cases, that that slows innovation overall for RL? I would say yes and no. Yes, in the sense that we will be working many of the bright minds. The bright minds in this area, they will go and work on different subjects. And each subject is going to, as you said, that requires to generate revenue and money and stuff. And they will make progress. but eventually after we make great amount of progress in each of these directions we start merging things that has been always the case in machine learning you push in one direction some other friends or colleagues they push another direction some point we meet we merge and we keep doing it and then the merge happens bigger scale and bigger scale and bigger so you can see in like in nlp natural language processes we have been developing as many ideas and many architectures in computer vision we were doing the same thing at some point we merged them okay the merge at sometimes the merge happens now we're using transformers for both domains similar ideas similar architectures so from that perspective i think is best thing to do at the moment is let the people naturally go in directions that excites them and develop algorithms for each of these directions we can also take the other direction of all of us keep working on this foundation things but i think compute wise we're not there yet so if you want to go in direction of having an AGI, a reinforcement learning-based algorithm that actually is able to solve many of these multi-bondial problems in many directions and many sectors, and sciences, and also engineering.
1:01:09If you want to train that thing right now, we don't have compute to do that. Going back to our first conversations, you said then RL is all about compute efficiency and trying to see how much juice we can squeeze out of the lemon, which is our compute. But you're kind of also saying we've kind of squeezed all the juice out of the lemon for where we are for our broad goals. So now let's go in these more narrow directions where there's still, you know, there's value in applying this technology and these ideas. There's plenty of value there. Let's go in those directions. And a little bit of it sounds like, hey, take a breather and let's let compute catch up for us or catch up to us.
1:01:52Yeah, we should wait for compute to catch up for us. And at the same time, for each of these subdomains, we need to go and develop domain-specific algorithms. And these domain-specific algorithms that we develop in science, usually when we go in different directions. This is actually, I think, Richard Hamming was saying in one of his talks. You go in one direction and you develop so many things. You can develop so many things because you see so many things in that specific domain that inspire. or basically the answer is in the problem setup. And I work in another domain that is answer is not that clear.
1:02:31But when we talk to each other, you tell me how you solve that problem. I'm like, hey, the same solution actually works for me here for this other problem setup. And then I take your ideas and apply to this new domain and I advance my domain. And then the new things science shows me and the problem setup shows me or I come up with new foundation and I see new observations or new keys to solve more advanced problem. And I solve it and then I tell you, you're like, hey, you can actually do that. So I gave you two people problem, but this happens in broader sense across like millions of researchers.
1:03:06And that's one of the ways that research has been evolving. I come up with an idea in geometry, in mathematics, prove so many theorems. You do so much in number theory. And then we meet and we're like, hey, if we just change our terminology, I can use all these geometric tools and theorems that I have proven. I can use this adaptation of terminology and bring to number theory and solve many, many open problems in number theory. And this had happened many times in history. Most of the advances in mathematics happened this way, same as engineering and same as other branches of science. Here for reinforcement learning is we go and solve these domain-specific problems.
1:03:46And along the way, we develop new foundations that actually allow us to make things to work. and over time we communicate with each other and the solution i come up with for one problem i can tell you and you can say or maybe it works for your case too and we just keep doing it and come up with the generic ideas but at the same time since the community is talking to each other a lot the unification happens very quickly it's not like 1940s unification that used to take like many written letters to happen now we just take a phone call and the communication happens and And many, many researchers are using transformers in their architectures because we just communicate with each other.
1:04:24Or many researchers are using LLMs in their settings. So we just keep doing it and keep getting better and communicate the things we discover with each other. And hopefully we will have a really good foundation soon. On this idea of compute being the limiting factor, what's required? Is it incremental innovation, meaning going from 10 nanometer processes to three nanometer processes, that kind of thing? Or is it like, we're not going anywhere till we get quantum computing? I would say to be optimistic because quantum computers are not there yet. If I say the latter, it would be like, okay, that would be pretty bleak.
1:05:06Yeah, it would be like, we need to wait for very longer. I hope it's the former one. Of course, if we have more compute, we get more intelligence. That's like a no-brainer. That's been known for centuries. We have also seen it here many times. We know that in decision-making, if you build a really, really long tree or deep tree, you can solve it for many of the decision-making problems. The problem is we can't build that tree of knowledge. For the current need of the society, how much of intelligence we need, I would say at the moment if we get better GPUs and faster GPUs. There are many people.
1:05:47Actually, also NVIDIA, there are many, many people actually, they are working to get better, improving GPUs. So GPUs are improving. They are hopefully getting more accessible and many people can have their hands on them. And this way we can develop algorithms that can run faster on these GPUs. And then if we can do very good reasoning-based algorithms and build these reasoning-based algorithms on using LLMs and generative AIs, AI models, we should be able to get sufficiently good and reasonable AGI, which is able to do the task of reasoning. We should be on the right track. So the improvements from algorithmic side are going to be mainly incremental.
1:06:30The compute side also going to be incremental. What I hope is not getting billion times faster or better in compute. If we get like any factor of like 10 better, we should be able to tackle really, really challenging problems that are exciting. But at this point, I would say, I think we talked about this one at one of our earlier conversation. We need more talented people to work in AI and reinforcement learning. We have amazing people working in this area, but we kind of need more. And there's also, you know, there are many great people living academia. So we need to also find a way to teach the next generation, which has been a problem.
1:07:10Yet I'm seeing right now that since we have, academia lost many people who were working reinforcement learning to industry. And it means that we don't get that many people being trained in these areas, which is kind of problematic, but community and society would find a way out of it. But if you can help those things, that would be also help. It would have been great to have many, many reinforcement learning students graduating and joining either academia or industry to advance these directions. I'm not going to be pessimistic, but I'm saying it would have been better. We don't have it, but it's all right.
1:07:46Yeah, I mean, I think we could go into different conversation, maybe beers or whatever, but there's a whole academia push versus industry pull. And probably a lot of that pull right now is happening towards LLMs as opposed to RL, which is not making things easier, I think, for RL as a community to advance. Although, I guess the converse is that, or the flip side is that LLM is this amazing new tool that we can use to make RL better. Yeah, 100%. Also, one thing the machine learning committee has shown is we can switch direction of study in one day. In 2016 to 2017, I think we had quadrupled the number of papers submitted to NeurIPS in related reinforcement learning.
1:08:33so the people in the machine learning are extremely amazing and very agile similar to many other fields and we have smart people working on LLMs we will get it to a great point we will start more working on other subjects until we pick a direction that is very exciting for anyone who is excited about anything should be able to work on it and that has been the case in machine learning which is great Awesome Well Kambiur, once again amazing conversation I learned so much from your notes and then digging into it with you. And I'm sure our audience has as well. Thanks so much for joining us once again to kind of talk through the progress we made and talk a little bit about the future.
1:09:13Yeah, thank you, Sam, for having me. And it was fun chatting with you once again and looking forward to future talks coming out of your podcast and also hearing from you and your team in the future. Awesome. Thanks so much. Thanks. All right, everyone, that's our show for today. To learn more about today's guest or the topics mentioned in this interview, visit twimla.ai.com. Of course, if you like what you hear on the podcast, please subscribe, rate, and review the show on your favorite podcatcher. Thanks so much for listening and catch you next time.
From the publisher
Today we’re joined by Kamyar Azizzadenesheli, a staff researcher at Nvidia, to continue our AI Trends 2024 series. In our conversation, Kamyar updates us on the latest developments in reinforcement learning (RL), and how the RL community is taking advantage of the abstract reasoning abilities of large language models (LLMs). Kamyar shares his insights on how LLMs are pushing RL performance forward in a variety of applications, such as ALOHA, a robot that can learn to fold clothes, and Voyager, an RL agent that uses GPT-4 to outperform prior systems at playing Minecraft. We also explore the progress being made in assessing and addressing the risks of RL-based decision-making in domains such as finance, healthcare, and agriculture. Finally, we discuss the future of deep reinforcement learning, Kamyar’s top predictions for the field, and how greater compute capabilities will be critical in achieving general intelligence.
The complete show notes for this episode can be found at twimlai.com/go/670.
