Inverse Reinforcement Learning Without RL with Gokul Swamy - #643

0:07All right, everyone, welcome to another episode of the TwiML AI podcast. I'm your host, Sam Charrington. And today I'm joined by Gokul Swamy. Gokul is a PhD student at the Robotics Institute at Carnegie Mellon University. Wherever you're listening to today's show, make sure you're subscribed and be sure to like, rate, and review the show. Gokul, welcome to the podcast. Yeah, thank you so much for having me. Hey, I'm looking forward to digging into our conversation. We'll be talking about your research into the fields of efficient interactive learning and making good decisions without observable boundaries.

0:39with a particular emphasis on a few of the papers that you are presenting at this year's ICML conference. Before we dig into that, though, I'd love to have you share a little bit about your background and how you came to the field. Yeah, good question. So I just wrapped up my third year at CMU. I spent a bunch of time there working on different things that are about sort of algorithms for this field called imitation learning, which is broadly speaking about how you can try and learn to make good decisions from data. Before that, I spent a few years at Berkeley where I was a master's and undergrad student.

1:14And there I was working on methods for human-robot interaction. Even before that, I grew up in San Diego and I tried to just maximize the amount of time I spent on the beach. Sounds worthwhile. Tell us a little bit about your research interests and the focus of your work. Yeah. So I think over the last few years, what I've been really trying to think about is how we can try and learn to make sequence of decisions well from observing data, some sort of expert demonstrator doing the same thing. So perhaps the most kind of intuitive example of this is something like self-driving cars. So, you know, we put some person in a car, we strap the car up with sensors, we get them to drive all around Pittsburgh, and then we want to learn a program, lets the car do the same thing.

1:58And I spend a lot of time thinking about what are the right sorts of algorithms for that problem. And I think there's sort of kind of two parts of that that I'm most interested in. The first is how we do this sort of efficiently. And I mean efficiently in various senses, both in terms of the amount of data you need to use, the amount of compute you need to use, stuff like that. The other thing I'm really interested in is if we're trying to get a car to do the same thing as a person, they don't have the same sort of observation space in some sense, right? They don't see exactly the same way. They don't have the exact same sensors.

2:26And at that point, there's all these sorts of things that show up that affect a relationship you care about that you don't really observe. So let's say you're trying to predict from some variable X to some variable Y, and there's some unobserved variable U that affects both. This is usually called an unobserved confounder. And here I'm trying to predict from some sort of observations, some sort of how much I want to hit the steering wheel, how much I want to press the gas pedal, stuff like that. And for example, your self-driving car might not be able to pick up on the fact that a hand gesture from somebody in the car across from them actually means something.

2:59And when you kind of have this sort of partial observability, I think it's a really interesting question of how do you still learn well? So I've been thinking about both in the sort of idealized setting, how do you do it efficiently? And in the sort of more real world setting where you don't perhaps get the access to the same pieces of information, how do you make decisions well there? Awesome. And how does that tie into the research that you're presenting at the conference this year? Yeah. So I think we have a few papers at the conference, which I'm very thankful for. And they sort of touch on different parts of those topics.

3:30So we have one paper at the main conference, which is really focused on the question of how do we do imitation learning efficiently. So that paper, I think, is really cool because it's a paper where the theory is very elegant. It's not a very sort of complicated idea mathematically, but it really, really works in practice. So we're quite excited about that. Is this the inverse RL paper? Yeah, yeah. So I can maybe talk a little bit about that if that would be good. Yeah, let's dig into it. Sure. So just before you do the title of the paper is inverse reinforcement learning without reinforcement learning.

4:02Yeah, which is a intriguing title. Thank you. Yeah. So broadly speaking, right? In sciences, we have sort of like forward problems and inverse problems, right? You know, the forward problem is usually, okay, I have some kind of objective function and I'm going to optimize it to get something. And the inverse problem is, okay, given behavior of the optimal thing, what was the thing I was trying to optimize in the first place? And in the sort of inverse reinforcement learning setting, right? The forward problem is reinforcement learning. So I give you a word function. I want you to find the optimal policy under this word function.

4:37The inverse problem is, okay, I gave you data from the optimal policy and I want you to figure out what was the word function that was being optimized here. So if you think about the driving example, right, I might want to try to extract a function that tells you, okay, how much does this person care about, you know, staying away from the cars that are in front of them? How much do they care about, you know, adhering to the speed limits, things like that. And I think it's a very natural question to ask, like, well, can't I just write down a function that does that? And it's actually really quite hard when you try to do it, right?

5:07Because I can tell you that, okay, I definitely care about observing the speed limit. I care about staying away from people that are in front of me. But exactly how much I care about those two things relative to each other is very hard to write down. So I think it makes a lot of sense to try and learn from data. Yeah. Yeah. It sounds difficult enough to do manually when you're only thinking about one of those relationships. But when you're talking about highly dimensional set of features, it sounds very, very difficult. For sure. Yeah. Yeah. And especially if you really want to have good human like behavior in a variety of settings, right?

5:40Most of people are just paying attention to one thing when they're driving, right? All these things are balancing in their head. So you really do need to try and actually learn from data how to do this. And so talk about the motivation behind trying to apply inverse RL without RL. Yeah. And maybe was that a motivation or was that a result? So I think the motivation here is really like how we can make it inverse reinforcement learning more computationally efficient. So there's sort of like pros and cons, I think, to the sort of inverse reinforcement learning approach to things. I think the pro is basically that you don't suffer from something called compounding errors.

6:17So the simplest approach you could think of trying to do for imitation learning, right, is basically purely offline supervised learning approach. So what I do is I get a set of expert states, I get a set of expert actions, I just regress between them, and I just roll out my car. The issue is that, of course, at some point, you're going to make a mistake, and you're going to end up in a situation in which you didn't have any data before. And then you're not going to know what to do, and you're just going to keep making mistakes over and over again. So if you kind of look at the early self-driving work in the late 80s, this is sort of what they were trying to do.

6:47And, you know, the cars drove a little bit, but as soon as you try to do something hard with them, they didn't work. The benefit of the inverse reinforcement learning approaches is because you're actually, you know, rolling out the learner's policy, seeing how things go. You're able to see, you know, OK, when I turn left, this is actually where I ended up. You're not just looking at states from the expert state distribution. So if you kind of think of it at this sort of set of sort of, maybe in the kind of distribution shift setting, basically what interaction gives us is it lets us kind of get samples on the test distribution because we can actually drive and see where we end up.

7:18So this is really nice because it lets you, you know, kind of see where you're going to end up. So you're not going to make mistakes you don't expect. But the challenge, of course, then is that you have to repeatedly interact with the simulator over and over again. And you have to solve these hard reinforcement learning problems. So conceptually, the way inverse reinforcement learning works is basically it's almost like a GAN, but in the space of two directories. So your generator here is like a policy that's kind of coupled with world models or dynamics kind of give you two directories. And your discriminator here basically is saying, okay, I want to look at the difference between expert and learner trajectories.

7:53And then you do a policy update that is, okay, let me take this learned discriminator and use it as a reward function and do reinforcement learning with that. So the issue then is that you need to repeatedly solve a hard reinforcement learning problem at every single step of this procedure. So that means you're going to spend a huge number of samples, right? We already know that reinforcement learning is really, really hard to actually do in the real world on problems, right? And if I'm asking to do it over and over again, well, it's kind of difficult. So our question in this work was, okay, there's all these benefits, the inverse reinforcement learning approach, but there are these severe computational issues.

8:28So how can we try and actually speed this up quite a bit. And you arrived at an approach that does not use reinforcement learning, in fact? Yeah. So it's effectively an approach that allows us to, like learning to make a sequence of decisions, but without the sort of part that makes reinforcement learning hard, which is exploration, right? So maybe sort of a concrete example I like here is something like, imagine what we want to do is sort of act optimally in some sort of problem that looks basically like going down the paths of a binary tree. And basically, I tell you that, okay, the reward function this person cared about is zero everywhere except for one of the leaf nodes in this tree.

9:16And my discriminator says, okay, I'm going to pick this one leaf node to be one, everywhere else to be zero. Then my learner needs to explore the entire tree to figure out where the one node is. that was non-zero. And that's a huge waste of time and compute and everything, right? The sort of insight we had in this work was that, well, if I know the states the person actually was in, right, I knew the person always went to the left in this tree, went to the leftmost node, I don't actually need to look at the rest of the tree, right? Because they never went there. That's probably the wrong thing to do.

9:50So I should be focusing my optimization just on the leftmost path. And if you kind of think about it that way, I think it's pretty reasonable to say that, okay, if I really just focus on optimizing on states that were, you know, related to what I saw the expert do, I should be able to cut out a lot of this sort of unnecessary exploration. And so did you cut out all exploration or did you cut out unnecessary exploration, as you said, by constraining the search space of the states that you're looking at? Yeah, that's a really good question. So if I cut out all exploration, that's very close to doing something that's like fully offline, right?

10:30And at that point, we wouldn't really have any robustness to compounding errors, right? Because we could just sort of end up in a place we didn't expect. But if what I do is I have enough exploration that I learn to recover my own mistakes, but not so much that I, you know, explore the entire space of the world to figure out to do this one thing, I can kind of balance these two things. So I can be both computationally efficient and not sort of end up in situations I don't expect and don't know how to recover from. Sure. You kind of constrain the search base and then you use some alternate method that's not reinforcement learning to kind of navigate through it.

11:04Yeah. So what we did was actually, I think, a very simple idea, which was that we basically did something very close to reinforcement learning, but we just changed the start state distribution to be that of the policy you want to imitate. And because of that, right, I'm still doing sequential decision making. I'm still learning to learn learning to recover from all these stuff and stakes. But I'm saying, hey, you don't need to start at the top of this tree and figure out how to get to the bottom of a tree every single time. I'm telling you, okay, you're only going to be on the left part of the tree.

11:35Just make sure you're good there, and then you're fine. So this is kind of how you can sort of constrain the search space, but still actually doing search or reinforcement learning. So I think it's really like still, it's sort of, I think, the best of both worlds in the sense that you are getting rid of the part of reinforcement learning that's challenging, which is exploration, while keeping the part of it that is helpful, which is actually learning to recover from your own mistakes. And so talk a little bit about how you kind of assessed and evaluated your results for the paper. Yeah. So we thus far only got to try out things in simulation.

12:10I'm really excited about trying this out on more real world problems soon. But what we did was we basically picked some of these sort of Mojoco sort of open AI gym environments. And what we tried to do is basically see, you know, how well can we sort of imitate some expert policy. So we, you know, we trained some policy reinforcement learning. And we said, okay, we want to learn another policy that does the same thing. How can we do this quickly? And what we did was we picked problems that are very challenging exploration problems. So imagine something like you're controlling a four-legged creature that walks through a large maze.

12:42You don't have to go through every single path in this maze if you're doing it with traditional reinforcement learning. But if we basically tell you, hey, these are the set of waypoints through the maze that you only need to care about, it's a much easier optimization problem. And when we tried it out on this sort of problem, we saw rather remarkable improvements in terms of the amount of interaction with the environment needed. And there's the kind of computational benefit of it. But I also would say that I think there's a bit of a safety benefit in the sense that if your environment is actually something that isn't the real world, you don't want your learner going around and doing crazy things all the time.

13:15You want to sort of keep them in a reasonable place. And in this example, were you worried about kind of extracting way waypoints from expert data, or did you assume that that was a downstream or upstream task, depending on perspective? That's a good question. I guess I would say it's an upstream task, but it's one that wasn't too bad. Basically, what we could do is basically say that, okay, every time step or every few time steps, where was the expert at that time? And then we could just grab that point and say, okay, your waypoint, we can start the learner from this waypoint and see where they go.

13:51And so in terms of did you have existing data sets that you were able to apply to this and benchmark results? Or how did you benchmark your performance? Yeah, so what we did is we picked one of these sort of like standard environments that sort of, you know, a bunch of different methods have been tried on. These environments also come with a set of data. So we actually took some of the data sets from the sort of offline RL literature, and we used that set of data. And then we sort of implemented each of the methods ourselves. And do you see this? You've used the autonomous vehicle analogy several times in describing this work.

14:30Do you see this method scaling up to that application that's a far way from the four-legged open AI gym type of environment? I mean, if you have four wheels, but I think it's actually a very reasonable application. Actually, it's one I'm pretty excited about. Another one I'm super excited about is I recently learned that a lot of the routes in Google Maps are now calculated using inverse reinforcement learning. I think it's actually one of the world's most widely deployed machine learning systems now. And I think that these sort of techniques could be really, really useful there for just drastically reducing the amount of compute need to do to compute routes and stuff like that.

15:08And I think also for self-driving, I think it's the sort of thing where, you know, even to this day, I think a good chunk of the self-driving industry, really the sort of bread and butter of the way their autonomous systems work is using different sorts of reinforcement learning or using inverse reinforcement learning techniques. So I could see this being applicable to a really wide set of real-world systems. And it's also an application that I think is very reasonable in the sense that one of the tricky parts about our method, and one thing we're trying to address in future work, is that I need to have a simulator environment where I can just put the learner in some state to start them off.

15:49Perhaps for real-world robots, this is really challenging. If I'm trying to get the robot to do a backflip, I don't know how I reset the robot to be flipped halfway up in the air and then started. Right. But a lot of kind of stuff in the self-driving space, training is done in simulation. So this is actually a very reasonable thing. Like it's, it shouldn't be hard at all to just basically, you know, just move the car to a different position in the simulator. So it's the sort of thing where I think actually self-driving is a, is a sort of a perfect application of this sort of algorithm. I'm also very excited about the MacBing application because I think both of them are settings where this sort of thing should be able to help a lot.

16:25A sort of third application I'm really excited about is kind of in the space of large language models. So if you think about the sort of training of these models, right, there's often a sort of fine-tuning step at the end called RLHF, or reinforcement learning from human feedback. And there, once again, we're using a very expensive reinforcement learning procedure. But we also have data of what we actually wanted, right? We know this was a data that was generated that people preferred. So it feels like we should be able to do something very similar there to basically speed up that search a lot.

16:56And there, given how computer-tensive is to train these models, I could expect this to provide really strong computational benefits. So I guess what I'm trying to say here is that I think that there is a wide set of problems that satisfy the assumptions required for this method to work out well. And I'm really excited about those applications. But I also think it's a very interesting question to think about, for the problems where we can't do this really complicated agile robotics? What are sort of alternative approaches that could be useful there? Awesome. You've also got a couple of workshop papers at the conference.

17:30One is complementing a policy with a different observation space. Can you tell us a little bit about that one and what you're trying to do there? Yeah, of course. So I think this sort of touches on what I was talking a little bit about earlier, which is trying to make decisions without access to all the observed features. So the setup for this paper, I think, was pretty interesting. So let's say we have data of some doctor or set of doctors interacting with some patients. So we get data of what treatment they gave and what notes they perhaps took down. And we also see whether the patient got better or worse, things like that.

18:07And then what we want to do is we want to learn some sort of decision support system, you know, some sort of computerized agent that's able to help them. But, you know, because this agent is a computer rather than a person, they're not going to be able to see the same set of things. They're not going to get this kind of same set of observations. And the question is, okay, how do I actually figure out how to help here? And let me try to give you a concrete example of why this is actually not an easy thing to do. So let's say I have a doctor who is prescribing chemotherapy to patients. And let's say the sort of feature they observe is some test that tells you with a high probability whether this person has cancer or not.

18:47And let's say this doctor is a really good doctor. So they basically only do chemotherapy if you actually need it, otherwise they don't. And let's say, just for the sake of argument, our computerized agent here doesn't observe the result of this test. Well, what it's going to see is that every single time someone got chemotherapy, they got better. So it's then going to say, okay, the optimal thing for me to do is prescribe everyone chemotherapy because it only makes people get better. And then if you try to say, okay, and you try to trust the system a bit more, unless the vast majority of people had an underlying cancer condition, people are going to get worse as a result.

19:20So the question in this paper was, okay, how do we still learn how to do this well? We spent a bunch of time using some techniques from the sort of causal inference literature to try and basically kind of correctly estimate the sort of effect of an intervention, like giving someone a drug and try to use that to really learn these sort of decisions of four systems that are able to actually help. And so is this an application of causal modeling approaches? Yeah, yeah, fully, yeah. Okay. And so talk a little bit about the specific approach you took. Yeah, yeah. So we sort of considered a couple different settings in the paper and each required a sort of, I guess, kind of different approach or a different set of assumptions.

19:59So I think maybe the easiest setting was basically, okay, at train time, but not at test time, I get to see both sets of observations. So for whatever reason, I also get to see what the doctor saw. But you know, at test time, I'm not going to get to see that. The system doesn't get to get that. You can basically use this technique called deep actor adjustment that was pioneered by Gita Pearl back in the day. And that sort of lets you effectively use an important sampling correction to fix incorrect estimates. So basically, you'd be able to sort of figure out that, hey, you know, it's this feature I didn't observe that actually was causing this positive effect.

20:35So I shouldn't like kind of over misattribute it to something else. I shouldn't like overestimate the value of the chemotherapy. Of course, this is a somewhat unrealistic setting. So then we spent some time thinking about harder settings. So one setting is, okay, you don't get to see what the doctor was looking at, but I do tell you, you know, what was their probability of taking this action? You know, so how likely would they actually give this person chemotherapy? And then what you can do is you can, again, do a sort of important sampling technique to kind of fix things. The sort of hardness setting is when I don't give you either of those.

21:06I just give you the actions and I give you a different set of observations. And if you think about this, this is a really hard problem, right? There's no reason to believe without assumptions you could actually solve this problem, right? Like it's like accessing your self-driving car to be able to stop even though it doesn't see the stop sign. It's a really hard problem. So in this case, you're giving the model a set of actions that doctors took to, you know, treatment actions without showing the outcomes. So we're giving them the actions and the outcomes, but not what the doctor saw when they were choosing to give them that treatment.

21:40Ah, not the observations. Yeah, yeah, exactly, exactly. Yeah, yeah. And if we don't give them the outcomes, that becomes a very, yeah, really hard, yeah. So, you know, it's like, the way I like to think about this, it's almost like, imagine if you and I were playing a game and it's to predict the outcome of a coin flip and I get to see the outcome of the coin flip. I'm definitely going to win this game, right? So you can't do this without any assumptions. So in particular, there's this technique from the sort of econometrics literature called a proxy correction, which simply put, basically says that even if there's something I don't observe that influences a relationship I care about, so long as I actually have proxies for this that kind of vary in interesting ways, I'm able to use these to basically kind of filter out the effect of this thing I don't observe.

22:27So what we actually use was sort of a more modern version of those techniques to basically kind of correctly estimate the effect of actually, you know, giving a person a drug. I mean, it sounds like a lot of what you're trying to do here is to correct for kind of sampling imbalance in your data set. Is that a fair assessment of challenge? challenge? That's an interesting question. I guess it's a sort of correcting for sampling balance. And it's a very interesting sampling balance, where it's one where the sampling was done based on something you don't observe. And because of that, if you make conclusions based on this data without accounting for that fact, you're going to draw incorrect conclusions.

23:10It's definitely that sort of thing, but it's a very precise sampling balance. Right. So the generator function of what's in your observation set and what's not is kind of out of scope until you've got this whole other set of observations that you just don't have access to. And you're trying to, as efficiently as possible, kind of identify ways to modify the actions you would take on your observations based on some knowledge of what the rest of the world is like. Yeah, that's a great way of putting it. Yeah, exactly. And then the other workshop paper is one called Learning Shared Safety Constraints from Multitask Demonstrations.