What’s Next in LLM Reasoning? with Roland Memisevic - #646

0:09All right, everyone, welcome to another episode of the TwiML AI podcast. I am your host, Sam Charrington, and today I'm joined by Roland Memesevich. Roland is a Senior Director at Qualcomm AI Research. Before we get into today's conversation, be sure to take a moment to head over to Apple Podcasts, Spotify, or your listening platform of choice. And if you enjoy the show, please subscribe, rate, and review. Roland, welcome back to the podcast. This is three for you, right? Yeah, thanks. Great to be here. And I think it's the third time. Yeah, it's great to be here again. I'm really looking forward to digging into our conversation.

0:45We're going to be talking about LLMs and reasoning and how your work over the years, since that initial conversation has evolved to really take on these topics. Since it's been a while since we've spoken and the first time since you've joined Qualcomm AI Research, I'd love to have you kind of talk through your background and how you got from 20 billion neurons to where you are today. Cool. Yeah. So last time we spoke, I think it was around two years, two or three years ago, something like that. And the first time about five, I think. Yeah, something like that. And we talked about the company that I co-founded and that was 20 billion neurons.

1:26The things we've been doing there, which essentially has been the stuff we've been doing before and then during the company and are still doing now at Qualcomm. We all joined Qualcomm as part of an acquisition around two years ago. So not too long after the last time we spoke. And we are now with the team over here and pursuing essentially the same research direction around combining perception and language so that you can build agents that can drive an end-to-end vision for AI further than we believe other approaches to AI have been able to do that. And so that has been a very long, long story over the years.

2:03I was actually already interested in this particular kind of thing, like building a situated agent rather than modules that solve certain subproblems. Previously, when I was on faculty at the University of Montreal, and then I realized it would be best to pursue this at a company, compute resources, data, all of these things. And now we moved all over to Qualcomm and continuing along the same lines. Awesome. Yeah, I remember what led to our first interview was I think I ran into you and a couple of your colleagues. I think it was like a rework conference in Montreal. Yes, that's right. And you were demoing this data set that you'd created that was trying to capture actions.

2:44Like, I think the way you described it at the time, which I thought was really interesting, was, you know, all of this work we've been doing in computer vision was noun-centric and you were trying to do verb-centric data sets. So how did that work lead you to what came later with the Fitness Ally project and your interest in reasoning? So you are talking about nouns and verbs, which already kind of alludes to the whole reason for all of this, which is essentially language. There has been a really long standing tradition, I guess, over the last 50 years, probably 50 plus years. A tradition or let's say assumption that language is a key ingredient to any kind of human like intelligence.

3:25And there have been many, many efforts like Eliza and all of this stuff over the decades. And I'm definitely a deep believer in language as being a must-have ingredient to build intelligence systems and to make them human-like in some way. So that it's the ingredient to human-like AI. And interestingly, the community at large, I would say, discovered around 2012 after the ImageNet incident happened and people were training networks on ImageNet, which A, worked, okay, which is cool and it kind of revolutionized vision and vision in many ways and so on. But more importantly, that the features, penultimate layer features or whatever, that the model learns in response to being trained on those thousand classes.

4:08Highly universal visual features that can solve many other vision problems besides categorizing cats and dogs and houses and cars and such things. So that was from a language, blunt language perspective. It's true. It was thought of about nouns, not about verbs. but I think it missed a more fundamental point that is behind all of this which is the moment you think about labels not in terms of one of thousand like you say label 37 but you actually give them names like fish but then later you do it for verbs as well and maybe for adjectives and adverbs and etc etc so once you have a rich representation of the kind of concepts you want the model to acquire then you have a much much more fertile ground for the model to develop features that are then useful for downstream tasks, right?

4:55And so the name of the game has been also at the company, how can we drive this end-to-end vision where you train a network supervised on one task, a bunch of tasks, however you want to call that, on something, you train it supervised, not to solve the task, but in order to instill features in the network, representations that are going to make it in some way universally intelligent. And so once you want to do that, want to instill capabilities in the model, you wind up using language. There's no other way than using language to represent the concepts that you want to have as a supervision signal to the network, right?

5:31So in that data set that we discussed some five years ago or something in one of the earlier discussions that we had, it was called Something Something, and it was about adding verbs, but also adding adjectives, adverbs, et cetera, et cetera, et cetera. In other words, adding captions to videos and training a network to make predictions, so predict those captions, detailed predictions like there's an object that fell down over here and it hit the other object whatever crazy weird prediction stuff again not so that the network is able to do this kind of weird stuff but so that the network learns about things like occlusion temporal assistance of objects learns about objects in the first place right when an object moves it doesn't usually teleport itself and just the idea of object comes about partly because there is something persistent about it across time, even though you have motion and these kinds of things.

6:20You can learn about material structure, know something about rigidity and so on and so forth. And so it's a way to instill cognitive skills, base skills, perceptual skills in that case, in the model, hoping that it would make the model smarter. And that whole vision then got pushed further and further to cases where the system was supposed to be chatting with the user in real time, sharing a joint space. So you're like in front of a table with some objects on it or the user is doing things or something like that. And so you have a face-to-face conversation just like we have right now, and you can show things and discuss things and basically immerse the model in a real-time, real-world environment so that it learns common sense capabilities.

7:04So it's common sense that language can easily be used. Maybe some people would say misused, but maybe that's one of the main reasons why language is so successful in evolution, can be used to instill in models, right? So having a shared name for things, we call things fish, and the one individual doesn't call it label 37, and the other one calls it label 501 or something like that, but there is a joint notion, and we both call it fish, and so that way we can teach each other about what the objects are, and we have learning signal in human societies and whatnot. And the same thing carries over, I deeply believe, to AI systems, where once we use language, we can align the representations with the ones that we have fairly well, and that way get some kind of common sense and human-like AI going.

7:50And how did this lead you to Fitness Ally, which was ultimately the product that you were building at 20 Billion Neurons and continues today as a kind of a platform for research? it's a bit broader than that. It was one product. It was one instantiation of getting value out of this weird system that we were basically up to, right? So the idea being, okay, we want to build end-to-end systems. We want to drive this end-to-end learning to the end. The system should be able to have auditory input, visual input. We didn't go as far as sensory input and robotics because we don't want to leave us mechanics and whatnot and didn't feel robotics is quite up for it.

8:28But it's all kinds of sensory and typical human-like sensory input, and specifically vision and audio. And it has to have a body, so it has to be able to convey things by moving around and so on. And it has to give you auditory output, speaking to you, and also visual output in the sense that there's a camera looking at that creature that we built so that it can basically have a sense of a conversation with you. So there's this end-to-end learning pushed all the way to the end, save for robotic, like true embodiment. Sometimes we call this virtual embodiment. So we wanted to build a virtually embodied AI system, a virtual robot, and exposing it in real-world conversations with real people in real time so that it learns something about the world that it would just never learn otherwise if we train it end-to -end, supervised on image-style labels and action labels for videos, etc.

9:16etc. So that was a vision and we were convinced that this is the only way to build human-like AI systems, train them end-to-end on the right setup of tasks. And those tasks have to be of this nature, sensory input, language, back and forth, and learning that way. And so we tried a whole bunch of stuff, turned this into a vision that is also commercially viable. And so we built things like retail assistants that would teach you things about objects at an end-of-aisele display. And And then they tell you, do you want to try this on? Maybe sunglasses. There was one demo that we showcased at NERVES at some point.

9:50And then we had various other use cases where this avatar would just provide value. And then fitness was one of those explorations that we did that stuck very well. And this is probably not surprising. Fitness is a case where you can have real value if you have a companion of sorts that knows you through the days, weeks, months, years, knows how you've been performing your exercises and so on. then guides you by giving you feedback and so on. There's an obvious superficial utility in that system can tell you what to do to avoid injury maybe or to be more efficient. There's a much more subtle and much more powerful one, which we were also after, which is companionship.

10:30So that system, by knowing you and knowing how you've performed your pushups in the past and how many you will be able to do, it's much better at holding you accountable than some bean counting app that basically just has a dial that goes up, a number that increases every time you do one push-up or something, but you feel a little bit more, you know, you don't want to disappoint that trainer in front of you that remembers yesterday you managed to do this many, and today, come on, you're going to do maybe one more, and so on and so forth. So some degree of companionship. And so that got stuck and worked very well, and users liked it, and so we pushed really hard to push that further.

11:07It took a while to land on this particular type of use case and vision as a use case that straddles between these two, which is crazy, right? If you think about it from a business point of view, maybe from a research point of view, it's a system that is really useful and provides real value. And at the same time, it's intimately linked to this end-to-end vision of having an AI system that has to have a real-time language-driven conversation with you about stuff that's happening in front of the camera, which I still believe is the only way to build systems that are human-like, sufficiently human-like.

11:40And so now FitNagala is a system that we had in the app stores, various app stores, and people were subscribing and using. And now after the acquisition, we're still pushing this further as a demo. And for example, combining the language capabilities of typical LLMs, we train with the visual capabilities, also trained in part through language of that perception stream that comes in. It's still one end-to-end system that has to learn and be fine-tuned end-to-end in order to do what it's doing. I was going to mention that a lot of this research predated LLMs. How have LLMs kind of impacted Fitness Ally, but really more broadly, your research interests and research agenda?

12:27Yeah, it's interesting. So we did give us predecessors of LLMs, which used to be LSTM-based RNNs, right? So RNNs had to be a part of this whole stack because you want to have language in order to instill capabilities and so on. So you need to have an agent also that uses language in order to do stuff. We had early experiments of various nature, maybe, for example, showed that if you train a model that has an RNN-based language decoder, autoregressive language decoder linked to the visual input, and you train it across captions, so like, for example, in the something-something scenario of increasing complexity, so instead of doing classification, you do various levels of increasingly sophisticated, detailed description of what's going on in the scene, that then you learn better features for downstream tasks that are better generalizing and that are better infused with a degree of common sense understanding of what's going on in the world.

13:21So we had these kind of studies. They used to be RNN-based. The big difference, the big transition, which I think is going to be a transitory transition, by the way, historically in AI, but the current transition to attention and the lack of true recurrence has some benefits in training. And so they are much better at considering language capabilities that are so deprived of On the other hand, deprived of visual input, perceptual input, but they are better at just handling language and so on. So that's much better right now. And so it's easy, though, since those models are also autoregressive.

13:55So they are recurrent in quotes. They generate one token at a time and they can still be agentic because of that. So they can still have a back and forth conversations and leverage all of those aspects of language. But the good thing about that way of getting at language is the current way of getting at language is that you can train them on parallel hardware, which you cannot as easily do with RNNs. When you have an RNN that generates one token at a time, you generate a linear sequence of events that you're going to have to backpropagate through. So you have a factor T, where T is the length of the sentence or whatever output that you generate as a multiplier in the computational time complexity of training.

14:35There's just no way around it. You're going to have to generate one token at a time for a certain amount of time until you have your attempt to, say, replicate a training case or something. When you look at a transformer, that disappears because now you can just have an input. Every column that processes the inputs can just try to predict the next token. You get losses across and you can just do one. In one step, so to speak, you can process this whole sequence. And so this factor T magically disappears as a constant that influences the training complexity, time complexity. And so you can train them much faster.

15:10So I think a reasonable assumption is that if you would take an LSTM-based old school LLM, whatever, small SLLM, small large language model, it has to be large. So and you would train it for like 100 years, then we would probably get something as good as or probably even better than a top shelf GPT pre-trained language model. It's just that you would have to wait 100 years to train it. And so that's why it's currently the field is dominant. Any language stuff is dominant. large is dominated by parallel stuff. But it's a coincidental technical. I think the point, just to kind of linger on that last point you made, I think you're saying that LSTMs, RNNs are not fundamentally inferior in their ability to capture language, but rather from a practical perspective of training them, they are not as efficient by a significant degree than transformers.

16:10Exactly. In spite of like the difference in, you know, transformers have this ability to capture longer context. Are you saying that if you can, you know, if you can make T arbitrarily long and you had sufficient compute to, to train with an arbitrarily large T, is that part of what you're saying? It's part of it. So you couldn't, you can add other things to that, right? I think the complexity is a key factor. There are benefits, obviously, and the verdict is out. Nobody knows, right? Like what if you had recurrence, yet you have the ability for some long-range stuff. Essentially, what humans are doing is some kind of reverberating shorter memory, right?

16:51So things that you just said, there's an auditory loop. You remember that for a short amount of time, but it's quite accessible to you at any moment. So the last few seconds, you still have them somewhere reverberating around. And so the ability to just grab that and take it and use it and stuff is probably another aspect in which, say, GPT, whatnot, is superior over some LSDM. But there wasn't LSDM or RNN or something like that. Or even back then, people were exploring many, many different incarnations of RNNs with many different clockwork type models where you look back a certain amount of time.

17:26And then RNNs were the initial type of architecture where even attention in an encoder, first in an encoder, decoder kind of scenario got explored. And so it's very natural to have this, to deviate from this linear state-by-state transitions or something like that, even in RNN. By the way, I think it's probably a bit of a trend also right now with papers that are pointing this out. But there's an interesting fallacy for taking that shortcut, pretending to be recurrent, but actually being fully parallel and momentary like GPT style model is. And that is the fact that they can be pseudo recurrent in the sense that they can generate sequences and seem to rely on past actions to do current actions and this kind of stuff.

18:12Where in reality, they can also fall prey to being tempted by pattern matching, essentially, right? So they can basically find similar patterns or they can be doing certain actions just because the context up to now just has a certain signature that makes it look similar to certain training cases, after which that action is also the right action to take or the model learned to take. And so some of the capabilities that seem very striking can maybe be explained away by a certain degree of pattern matching and then something much simpler. And one problem in which this is actually turning into a real problem is length generalization.

18:49So there are various studies lately popping up that study what happens when you take an LNM, train it on a certain length sequence of a trivial task, like multiplying things and parity and whatever, like certain symbolic tasks. And then you try to have it generalized across beyond that window that it was trained on. And so I'm not talking about context window, right? The context window might be much larger than any of those either training or test situations, but it is just that they were trained to solve instances of a certain tiny length and then try to generalize into a slightly larger length and it completely falls apart because it doesn't understand the gist of the problem.

19:27And where does that come from? Well, it comes from the fact that the model isn't actually recurrent. So it doesn't have the same notion that you have about in order to solve this multiplicative what not problem. I'm going to have to take this token here. Then I have to combine it with that one here. And then I have to do this. So this whole sequential nature of arriving at the answer is not there. The model learns this on the training data because it, again, it gets tempted to just do subgraph matching on instances that look similar to training data. And then it just comes up with the answer. And so it seems to being serial in sequential processing those multiplicative problems but in reality it's not and that is a big problem that is actually probably going to require recurrence to come back but now people can approach it from a different angle you can start with some model like that and then sprinkle in recurrent connections and fine-tune them and see what yeah but i think recurrence will be back right it's not i was going to i was going to ask when you say that you think llms are a bit of a transient state it sounds like the next state or n plus n you know n plus one state is some degree of recurrence coming back into the model is that kind of what you're thinking yeah exactly that that's at least one one way in which those models are going to have to evolve and will evolve and it's it's okay right so they provided a lot of value a lot of insight now i want to shift gears a little bit and jump into this idea of reasoning particularly in the context of language.

21:05It's been a topic that I've been super interested in as have many of us in this field. And, you know, one way I've kind of come to, you know, snip or soundbite that I've come to is that LLMs do not reason or cannot reason, but they can produce results that seem like reasoning. And the idea is to reinforce that, you know, they're next token generators. They're not following a process that we might think of as reasoning, but somehow the result of running this next token generation, you know, some number of times looks like or produces the results of reasoning. And I just wanted to throw that out there for you to pick apart.

21:46I don't agree with that. You don't agree with that. Yeah, let's get into it. Tell me why. Tell me a human being that is not producing one micro action at a time, one after the other, right? It's not possible. So at the end of the day, atomically, whatever you want to call this, the robot that we are, the robots that we are, whatever, or when we talk, right, the language that we output, it's always a string of, let's call it tokens, you might call it whatever you want. You can also say phonemes, whatever. You could have done it on the audio level, even if we had more compute and stuff, whatever.

22:21Let's call everything tokens that are model outputs, that are agent outputs. You never do anything but generating one token at a time. And that's just a fact, right? There's space and time, right? The constraints of our physical world and time just advances and we can just do one thing at a time. And that's just what it is. It's just a fact of nature. And so if we build an AI system... I don't know that I necessarily agree with that. Like what we know is that, you know, because we've got, you know, one vocal apparati, the output is one, the output is serial from a word perspective. you know of course but from a biochemical perspective there's lots of things happening at the same time they're just serialized by this output device and i guess that's absolutely fair i agree with that and i agree with partly with the idea that some of the the chains of thought that you see are pretend reasoning and the recurrent the lack of recurrence is just one example of that that's for sure on the other hand want to build an intelligent system i think we We have no other, that is human-like.

23:26So we really want to advance this agenda, like building more human-like intelligence. So we have no other choice than building an agent that has a perceptual input and has one output at a time, an actuator, and it goes through, it is an autoregressive model. It has to generate one thing at a time, influenced by whatever it sees right now and the history of whatever it saw before and whatever it has generated before. So that's it, right? That's a universal definition of an agent, of whatever reinforcement learning agent. And there's a lot of benefits to formulating a system like that versus going for feed-forward networks and solving vision and then solving this and then solving this.

24:03The benefit is that the driving force in neural nets in general, starting with image-net, creature-networks, speech and whatnot, it's end-to-end learning. So you can do supervised learning and you get domain transfer. So that benefit is real and that's what I believe is the 99 % of what drives AI forward. That benefit can carry over to agentic situations where you want to build an agent just as well if you take as the artifact that you're building an agent that has to do one thing. And this is a real agent. It's embedded in a real world that has time marching on and it has to do one thing at a time.

24:39And the environment is unforgiving. And so sometimes it has to do things fast in order to catch up and whatnot. But it is an agent at the end of the day. When we train agents like that, so when we define the AI model not as a function, but as a procedure, right, as an agent that does things over time, then transfer learning, I believe, can play out in the realm of action sequences and decisions and all of this kind of stuff. And that, I think, was a big problem with a lot of reinforcement learning in the past. So in lots of value-based approaches and the whole RL history of many, many years, decades, actually, the engineer was in the driver's seat.

25:16And you have a value function and you design the system based on a time slice that captures the history and the assumptions about the future and so on. And then you use that in order to define an agent yourself. What this is transitioning to now is a situation where the system is a neural net and you leave that damn neural net alone and just define it as an agent and then expose it to various tasks that are agentic in nature. So that are sequentially, you have to do this and this and this and so on and so forth. So that the cognitive skills in referring back to things you have done before and understanding words like first, second, third and so on and so forth.

25:53understanding that there are actions you can choose from and so on, that this emerges in response to the tasks you train it on. And so now transfer learning can play out. And that's this domain and the domain of making a better agent. So from that point of view, I think there's just nothing other than an autoregressive model that will lead us to better AI because you're just missing out on the agent aspect. Humans are agents and our language is deeply informed by humans being agentic. We know about past and future and first, second, third, and this and then that, and so on and so forth. So I think you could claim it's hard to find a significant number of words in the English or in other languages that are not in some way infused with semantics that has to do with gentleness of that human that uses those words.

26:43None of those words are represented in the same way in a GPT-free term language model yet fully because they are not trained in agentic scenarios fully yet, etc., etc. But since it's an autoregressive model, it has the bucket, the potential to absorb this meaning that goes beyond what it sucks up from Wikipedia and web language and so on. And so that's what's exciting about this. So when you think about kind of crafting a research program to explore these kinds of issues, to understand what reasoning means, because that's one of your main focuses that Qualcomm on the research team there, like, you know, talk about how that plays out some of the individual works that you're exploring to kind of get at these questions around what is reasoning and how do you create reasoning capable agents?

27:38systems, agentic systems with, you know, these building blocks that we're left with, LLM for currents, whatever, these autoregressive models? So first of all, I think it's reasonable to proclaim that reasoning exists because of language. That's one distinguishing factor that distinguishes human intelligence from other animals for large part. Not everybody would agree with that, but I think language has to play a key role in that. So when we talk about reasoning, we foremost talk about language. And that incidentally is also the way in which reasoning has made a huge advance in the last few months, actually.

28:15It's because language has entered the realm, right? So it's like, arguably, it's been language had something to do with it. There is the longstanding theory in psychology and cognitive science of dual process scenario or architecture, whatever, that the human brain adopts, which is, there's a system one and a system two. Dan Kahneman, Nobel laureate, has been a very, very outspoken proponent of that view, has written a very famous book, Thinking Fast and Slow, about this and so on. And so that viewpoint says that there is a fairly clearly, a somewhat clearly distinct set of cognitive capabilities, which can be called system one and system two, where system two is the stuff that is serial and slow and deliberate, very human-like, actually, because of that.