Building Maps and Spatial Awareness in Blind AI Agents with Dhruv Batra - #629

0:07All right, everyone, welcome to another episode of the TwiML AI podcast. I am, of course, your host, Sam Charrington. And today I'm joined by Dhruv Batra. Dhruv is an associate professor at Georgia Tech and research director of FAIR, the fundamental AI research team at Meta. Before we get into today's conversation, be sure to head over to Spotify or your listening platform of choice. And if you enjoy the show, please leave us a five-star rating and review. Dhruv, welcome to the podcast. Thank you for having me, Sam. I'm looking forward to digging into our conversation. We'll be talking about one of your recent research projects, the emergence of maps and the memories of blind navigation agents is the name of the paper.

0:47It was an outstanding paper recipient at the recent ICLR conference. And looking forward to digging into that. Before we do, I'd love to have you share a little bit about your background and how you came to work in the field. Sure. Happy to describe that. And yeah, it's been a wonderful journey, a bumpy one. And I can get into that with respect to this paper. This paper has been rejected from a few places. It was wonderful to see it, to see the recognition at iClear. But to answer your question, I've been in the AI research field 17 or so years at this point. I started my PhD in 2005 at Carnegie Mellon.

1:24And I think people entering this field now are far better prepared and informed than I was coming in. When I started grad school, I showed up not knowing what computer vision is, what machine learning is. I came from a signal processing and electrical engineering background. So I kind of knew what pattern recognition is and what image processing is. But that in those days meant 2D filtering. Sure. I came from the same background. So lots of common filters and wavelets and all that kind of stuff. Exactly. I, you know, took computer vision. And there was a course being offered in my first semester in grad school on intro to computer vision, intro to machine learning.

2:07I took those classes and just got hooked. My research trajectory since then has gone through a few different phases. My PhD research was on probabilistic graphical models, which was a set of techniques and area of research that was fairly dominant in machine learning before the modern sort of deep learning era, if you will. and it was applying it to image segmentation problems. Since then, my research has gone through a few different phases. I've done a lot of work in discrete optimization and probabilistic graphical models. I spent a few years working on vision and language problems. So building agents that can describe images, that can answer questions about images, that can hold a dialogue.

2:52So chatbots that are visually grounded, if you will. And for the last five years or so, I've been working in embodied AI and slowly doing a lot more sim tutorial and robotics work. So building agents that can navigate 3D spaces initially in simulation and then in the real world that can pick up objects, that can answer questions after navigation, that can understand 3D spaces. And it's been an interesting journey entering entirely new fields. I'm not a roboticist. I've never taken, for example, a motion planning class in my grad school. And today I supervise robotics PhD students. I support robotics research scientists and engineers at FAIR.

3:46And it's a fascinating learning experience. It's an entirely different world, but I enjoy making connections across areas. So it's a fun journey. That's awesome. Do you primarily work in the simulation domain or do you get your hands dirty with real robots that move around and break things? I am no longer running the experiments myself, but my collaborators and I do actually work in sim to real. So we make a lot of advances in simulation and then we deploy them on real robots in the real world. The paper that we're going to talk about today is entirely in simulation, but those same ideas, we have trained policies and simulations and deployed them on real robots, like the Stretch robot from Hello Robotics, which is a startup by my colleague at Georgia Tech, Charlie Kemp, and the Boston Dynamics Spot robot, which we have used in a number of recent projects.

4:41Okay, very cool. And before we jump into the paper, you're also a podcast host, or were at some time, I'm not actually sure this, the current status, but you worked on the Humans of AI project with Devi Parikh, who's also a former Twomal guest, esteemed guest of the podcast. How was that experience for you? It was a fun experience. So Devi started that podcast, Humans of AI, in 2020. I think this was right after the pandemic. And it was really her way of just battling the isolation that came from the lockdowns. So both Tevye and I enjoy structured conversations with our friends. We subject our friends and colleagues to that kind of conversation a lot where there's a question.

5:28Meaning the same 20 questions? You know, it doesn't always end up being 20 questions, But sometimes having a deep, insightful question and just going around the table and hearing people process their thoughts on that question can be the right spark. we've all had those evenings where it's like 1 a.m. The evening has wound down, but there's a group of people still sitting around the table hashing out an idea that they're working through, maybe discussing that book that they read on the topic or they're still struggling to figure out how this recent idea relates to their worldview and so on. And I find those conversations far more rewarding and meaningful than small talk.

6:17What's your favorite conversation starter question or thought-provoking question? Favorites, hard. Favorites really hard, but I can give you one that I like, which is I enjoy asking people what's a recent idea they came across that was either new or challenging to them. It simultaneously tells you where their head is currently at and it tells you how it fits into their worldview. If you ask people about ideas that resonate with them, That gives you one sense of insight into what they're thinking. But if you ask people what they're struggling with, what idea they're struggling to incorporate into their worldview that they find odd or surprising or bizarre, that also tells you a little bit about their worldview.

7:02And so what's that idea for you? Well, I'll tell you that answer in the context of research, because I think that's a safer territory.

7:12Apparently, it's not late enough for the real question. Yeah. The non-research answer requires either a coffee shop setting with highly caffeinated people or a late night setting with, you know, after a few bottles of wine. But the research idea that I think I'm struggling to incorporate into my worldview is, and this is related to the research that we'll talk about today, is there's this idea of, it's called the embodiment hypothesis. There's only one form of generally intelligent system that has existed in nature, and that is embodied intelligence. We went through millions of years of evolution from the simple cell organisms to complex cell organisms to plants and animals and ultimately mobile animals and predator and prey systems.

8:01And when you look at evolution, it becomes clear that evolution spent most of its time evolving us to be very good at sensory motor control. There are tasks that we take for granted. The fact that you and I can pick up an object and twirl it in our fingers, this seems trivial to us. Whereas if you asked people, is playing chess or go easy? Most people would say no. Playing it at an expert level is hard. But that's not what we evolved to do. It seems hard to us, but it's not clear whether that's computationally a hard problem. And so this is also known as the Moravec's paradox, the idea that the hard problems in AI are in sensory motor control.

8:45And what seems easy to us is actually hard, which is controlling our bodies. And what seems hard to us is actually easy. And that's the hypothesis that I've been pursuing for the last few years, which is if we want to develop generally intelligent systems. We must give them a body. We must put them in 3D environments. We must learn from large amounts of data because we're essentially trying to replicate evolution. And yet at the same time, when I look at the progress in language models or disembodied language vision models, so chatbots or chatbots that can absorb images, it is clear to me that there's a lot of progress being made.

9:24And where is the ceiling of those kinds of intelligence systems? Is there a ceiling? How far could we get with language models alone? I'm struggling with that question. I'm struggling if the landscape of intelligence is multimodal, meaning that there are multiple hills in the landscape of intelligence, and maybe some pathways up those hills do not involve embodiment. This is not something that one can rationalize or reason through from first principles. I think it's a hypothesis. We need to conduct experiments and find out what the right answer is, but it is something that I don't have a good handle.

10:02Along the same lines, also not specific to the paper that we're going to eventually get to, how are you processing everything that's happening with LLMs in terms of kind of thoughts on and future implications of it? Do you have easy answers for that or is that, Do you find that's kind of equally challenging to fit into a coherent view? I think there aren't easy answers. There is clearly a lot of progress being made in language models. I tend to view them as tools that might augment human creativity and workload. The history of AI systems has always been automation and freeing up humans from doing certain tasks that we can do more efficiently with machines.

10:51At the same time, there are, of course, things that we have to be careful about, about training on the right data sets, making sure that they're used responsibly and things like that. And we have to engage. This is not just a question that simply AI researchers will figure out on their own. I think this requires a concerted effort across a broad range of stakeholders. So you talked a little bit about embodiment and kind of queued up a very broad context in which the paper that we were discussing came about. Let's maybe go a little bit deeper into the motivation for this specific project. You mentioned that it, I forgot exactly how you put it, it was not without its challenges.

11:37The research was fascinating. The pathway of getting the research published was bumpy. I can tell you how this project started. In 2019, we built a 3D simulator called Habitat. It simulates 3D experiences or an agent navigating in 3D virtual worlds, where we load up scans of indoor real world environments. So houses, buildings, office spaces, we can load those spaces up in a 3D simulator. We can have an agent navigating in those spaces, experience rendered RGB images and depth images in those spaces and train that agent to navigate. The question that this ICLR paper is asking, the question is, do navigation agents build maps, internal spatial maps of their environment as a consequence of learning to navigate?

12:37Why is that the question that we're asking? The historical context here is that animals, we know from decades of literature on animal navigation that animals build what are known as cognitive maps or mental maps. And there's a fascinating history here. In 1948, a researcher at Berkeley, Edward Tolman, actually conducted these experiments with rats in mazes. And he wrote a paper describing that experiment, which is that you can train rats to find food in maze-like environments, which was in itself not a novel finding. We knew that rats can, with their sense of smell, find out their pathways in mazes.

13:19And essentially, if you keep the food at the same place, they know how to get to that food location in a maze. What was fascinating for Tolman is the fact that when he made a change to the maze, that he blocked certain pathways and actually changed the layout of the maze where at the start of the location, you actually have multiple new pathways that you can follow. the rats seemed to take a new path, which was a direct shortcut to where they had last found the food. So their old path was blocked. They see a new pathway, and they take one that is directly taking them to where the food was found.

13:57And that in Tolman's mind was evidence that rats are not memorizing paths. They're not memorizing that I have to go straight, turn left at this corner, turned right at this corner. They are building what he called cognitive maps or mental maps of the environment so that when a pathway is blocked, you can take a different pathway towards that goal. In 1971, John O 'Keefe actually established a mechanism. The hippocampus in the human brain has what are known as place cells. There are certain neurons that fire when organisms visit certain locations. In 2005, some collaborators of John O 'Keefe, the Mosers, they discovered what are known as grid cells.

14:41So essentially, your brain has a GPS-like tracking system where as you're moving around, you can use that to perform path integration. And this line of work culminated in a Nobel Prize in 2014, the fact that biological organisms build mental maps of their environment as they are navigating. Now, flip that to the AI or the robotics world, we have known for decades that the way we built self-driving cars, autonomous vehicles, Roomba in your house, the way we build those engineering solutions is that we break down a problem into sensing, mapping, planning, actuation. You take input data from your cameras, you build a map, you plan a pathway on it, and then you navigate.

15:24The question that we were asking in this paper is, must we design mapping or is mapping an emergent phenomena? Could agents learn to build internal mental maps without any previous predisposition to mapping? Why would we want to answer that question? Because if we find out that if you train an agent just to do navigation without giving it a mapping module, without giving it mapping supervision, without putting your thumb on the scale, if you will, towards mapping, and mapping still emerges, that suggests that mapping is playing a fundamental role in navigation, in the problem of navigation, because now you're seeing it emerge both in biological systems and in artificial systems.

16:09And there's this phenomenon on biological evolution called convergent evolution, the idea that different organisms evolve similar structures to solve similar problems. I suspect many people know this already, but maybe not as clearly, that the ability to fly evolved completely independently in bats, birds, and in flying dinosaurs named pterosaurs. Their common ancestor did not fly. All three of these organisms, they fly. And the invention or the evolution of wings happened completely independently in three separate line of organisms. And what that tells you is you're getting a glimpse into the underlying pressures that these organisms faced from their ecological niche.

16:56And that's why wings are the natural response to flight. There's a fascinating process there. You know, the eyes evolved multiple times. So cephalopods like squids and octopus, they have a better design of their eye than mammals do. In mammals, we have the optic nerve passes in front of the retina, which is why you get a blind spot. Each of your eyes has a spot where it does not register information. You don't realize it because your brain basically fills in that information. But octopuses or octopi don't have blind spot because they have a design of their eye where their optic nerve passes behind their retina.

17:36That is telling us that eyes evolved multiple times in two different lines of organisms. So if it turns out to be the case that mapping emerges on its own in artificial agents without any previous predisposition for mapping, then that is telling you something fundamental about the nature of mapping in navigation. So that was why we conducted this research. What we did is that we took an agent that is essentially blind that can only tell ego motion. So you have an agent that can tell how far it has moved. This is similar to the sensory mechanism of blind mole rats that can use Earth's magnetic field to perform path integration.

18:18So imagine a virtual robot in 3D simulation or a virtual agent in 3D simulation. When it moves forward a little bit, it can know how far it has moved. And when it turns in its place, it can know how much it has turned. That's the only sensory mechanism it has. It has no sight, no sense of smell, no other sensory mechanism. and we're going to train it to go from point A to point B, where A to B is specified in relative coordinates. So go forward point two meters or go forward 20 meters, go left 30 meters and so on. You are in a sophisticated real world environment like a house. And so there are multiple rooms, there are walls, there are tables, there are chairs, there are obstacles in your way.

18:59We are not going to tell you anything about the obstacles and we're not going to equip you with any mapping module It's just going to be Egomotion fed into a recurrent neural network. So Egomotion goes into an LSTM or a GRU recurrent neural network. The agent is going to output some actions like move forward, turn left, turn right. We're going to give it positive or negative rewards based on how far it is making progress towards its goal. That's it. No expert demonstrations, no 3D module, no mapping supervision, nothing else. And we find essentially a number of surprising phenomena. First, we find that blind agents can navigate, which wasn't clear going into the project.

19:48Blind agents are actually highly effective navigators. They're just not very efficient navigators. Blind agents will make their way from point A to point B essentially 95 % of the time in the scenarios that we tested them in. But the way they do it is very interesting. Most of these blind agents and the videos are available on our website will effectively go slam their face into a wall and follow a wall along. And that is an emergent phenomena. There is nothing in the architecture or the reward function that is encouraging agents to follow walls that emerges. And interestingly, there is a algorithm known as the bug algorithm where people were trying to code up essentially these solutions.

20:30and we find the emergence of that solution automatically, we find the emergence of collision detection neurons. So we can take the LSTM representations or the RNN representations that the blind agent is building and we can try to predict collision or not. Does it know in the last step whether it collided or not? And that can be done with essentially 98 % accuracy. And so blind agents know when they have collided in their representations. and more importantly, we find that these representations contain maps or spatial information about their environment. So you can actually take one blind agent, put it in a new environment, ask it to navigate from point A to point B, then take its LSTM memory, transplant it into a new agent, into a new LSTM, and it will take shortcuts.

21:22It will ask it to traverse point to point B again, and it will not do the exploration that the first agent did. That is evidence that what is happening is that the internal representations are building spatial representations that enable shortcuts in these new spaces. And we've extended that to also extract full top-down map decoding of the environments as well. All right. In a lot of ways, it sounds like some of the work that was done around CNNs to kind of look deep at the kind of initial layers of the CNN and you see kind of these, the emergence of layer specializing in straight patterns, angles, different textures, that kind of thing.

22:09Like you are introspecting, you're trying to introspect these sequential models to determine what kind of fundamental properties they're picking up on. Is that a reasonable analogy? Yes, that's exactly the right analogy here. In visual backbones like CNNs or visual transformers, you find the emergence of, at least in CNNs, it was very clear that you found the emergence of part detectors, like line segment detectors, parts of object detectors, ultimately leading up to full object detectors, and then scene classification in scene classification networks. And for instance, one of the fascinating papers on this was by Antonio Talba's group.

22:50This is analogous to that in the navigation space with sequential models. At the same time, if I can jump in, it's very counterintuitive to try to apply that analogy to a sequential model that doesn't have the same notion of layers or kind of a module or architecture. Yeah, exactly. Where is all this information stored? Is it just in kind of memory coefficients somewhere? Yeah, you're absolutely right that you can at least try to make an argument that there's something about CNNs that is a particularly good inductive bias for 2D data like images. Sequential models like LSTMs, it is not clear whether they are a particularly good inductive bias for GPS coordinates over time.

23:36I mean, it is a sequential model and you are receiving sequential data, but maps inherently are not sequential in that sense. And so the fact that maps emerge in sequential models is an interesting phenomenon. Your latter question of where is that information stored? I think that is a particularly tricky question, one that we have not yet answered. We can point to the existence of these representations. So it is essentially a memory that you are building over time. We can tell you that memory is extremely important. So if you put a bottleneck on the memory, if you chop off from the trajectory that the agent is seeing, you know, some k time steps in the past.

24:14If k is too short, so if it is essentially a reactive agent that just remembers it's past one step or two step or five steps, the performance of navigation will tank. We can confirm that, that these agents are remembering something like the past 1000 steps in their navigation trajectory. We can confirm that we can decode collision detection neurons or representations. We can decode explicit maps. But in the context of the hippocampus and the human brain, the discovery of place cells and grid cells are a mechanistic explanation. How does the human brain build maps? Well, there are certain neurons that keep track of places, certain neurons that keep track of path integration.

24:56That we have not yet provided with neural networks. We don't have that answer. So we're in the AI world. We are essentially at the 1948 discovery, not at the 1972 or 2005 discovery. But to be fair, the 1972 and 2005 discoveries led to Nobel Prizes. I'm not sure we're making that bolder claim yet. And you touched on this, but does the results that you've seen kind of give you confidence that the LSTM is the right, the right even saying it sounds very strong, but you know, is the the right architecture for navigation types of problems? Does this give you some comfort that there's some fundamental sequentiality to them as opposed to trying to superimpose a 2D representation or something like that?

25:45No, actually, I wouldn't make that claim. One of the reasons why we went with LSTMs is because they are general purpose sequence models and we were dealing with a trajectory is essentially a sequence of observations along that trajectory. And we did not want to put our thumb on the scale. We didn't want to go in and create a map-like architecture, a spatial architecture. If you do that, then you cannot answer the question. Oh, do maps emerge? Yes, but let's start with a map. Exactly. Right. At various conferences where I've presented our works along the way on building mapless navigation systems.

26:26I often get asked this question, why don't you put in a map? We understand mapping. We have decades of work on mapping. I have the same reaction where if I'm building an engineering solution, of course I will use every tool available in the toolkit. But you do not answer, is X necessary by putting in X from the start? You answer questions of, is X necessary by trying to see if a system without X can solve problem Y? And if so, X is not necessary. or by trying to solve problem Y directly and seeing if X emerges, then yes, it seems like there is something about this problem that X seems to be necessary.

27:04So that's why we started with LSTMs. I think the fact that we get this glimpse that mapping itself seems to emerge. One of the problems with maps and mapping is that the word map is a suitcase word. Different people use it very differently. Cognitive scientists mean a very specific thing. They mean a cognitive map. They mean how the brain implements mapping. Roboticists often - Can you elaborate a little bit on what that specifically means to the cognitive scientists? To cognitive scientists, a cognitive map is a mechanistic claim. So it's a claim about a capability and a mechanism. The capability is that an organism should be able to navigate against a map, meaning that if you have experienced a space and build a cognitive map, then I should be able to spawn new or you should be able to go from point A to point B where A and B are now new locations and the environment has changed and you can adapt to it.

27:59A mechanistic claim is about how is that map implemented in your brain and that's what grid cells and place cells answer. That term cognitive map means to cognitive scientists. To a roboticist the word map can mean something like a point cloud of a 3D environment. So you put a LIDAR on an autonomous car and it's giving you X, Y, Z locations of certain obstacles, of certain points where you're getting those rays back to the LIDAR. For example, it can be a purely geometric structure and a sparse geometric structure that may not tell you anything about semantics. All of the semantics about what are objects, what am I looking at, what can objects do, what are parts of objects, where can I pick up a mug, Those are all higher level things that, according to some definitions of a map, are not part of the map.

28:50Maps can also mean what are known as semantic maps, so that you can construct a full 3D graph of where are the parts of objects, what are they grouping towards into. Maps in modern sense can also mean implicit neural maps, so that there isn't an explicit geometric or semantic structure. there are weights of a neural network that I feed in XYZ coordinates to, and it tells me something about what is present in that XYZ coordinate. And this is building... So starting to get to a mechanistic view. Yeah, in a way, but also building on top of the recent developments of NERFs, so neural radiance fields.