Video as a Universal Interface for AI Reasoning with Sherry Yang - #676

0:07Hey, what's up, everyone? Thanks for joining us for today's episode of the Twinwall AI Podcast. You know me. I am, of course, your host, Sam Charrington. And today I'm joined by Sherry Yang. Sherry is a senior research scientist at Google DeepMind, as well as a PhD candidate at UC Berkeley, where she is advised by longtime friend of the show, Peter Reveal, who I interviewed back in 2021 and 2017 before that. If you're watching us on YouTube, be sure to take a moment to hit that subscribe button. And if you're listening in on Apple Podcasts or Spotify, It is the follow button that you're looking for to make sure that you don't miss any of the great episodes that we've got in store for you.

0:44Sherry, thanks so much for joining me and welcome to the podcast. Thanks, Sam. Very happy to be here. I'm looking forward to digging into our conversation. We'll be talking about your recent paper titled Video as the New Language for Real-World Decision Making. Sometimes I ask guests for their controversial opinions. And I think in this case, it's pretty clear that yours would be that video is underappreciated and that its chat GPT moments should arrive very quickly. Is that fair? Yes, that's correct. I think gradually more and more so people started to realize maybe after Sora was released, but this realization could have really came much earlier.

1:24How did you get started in video? So I guess a little bit of background. When I started in research, I was actually doing a lot of reinforcement learning. So this is about, you know, the technology that empowered AlphaGo, that actually this is what enabled AlphaGo to search beyond human capabilities and eventually outperform humans. But then for AlphaGo to work, there is actually something very special about it, which is the simulator of the game that AlphaGo uses to imagine into the future is the ground truth chess or goal simulator. Back then in 2016, people were like, well, AGI moment has arrived, but nothing happened afterwards, right?

1:59Because oftentimes in real world, there is no simulators. And the simulators that people use for robotics or dynamical simulation is actually quite different from how real robots look like in practice. So as a result, you know, any algorithms that people develop in simulation wouldn't work in the real world. So what's so special about video generation models and large language models is that now with these internet scale data sets, we see that these large foundation models, video models and language models being turned into agents and environments and world models and simulators so that we can actually run these algorithms for searching for better decisions in simulation.

2:37But then because how simulators look so realistic to the real world, then any of the decisions we can find can also apply or transfer to the real world. And we can achieve superhuman performance potentially in a lot of real world scenarios. This is why I'm very excited about this direction. And maybe the question is why specifically for videos? So I guess that goes into really the motivation of this paper, which is kind of the recognition that video is, in fact, a unified data format for a lot of different information. So, you know, for language models, one of the reasons that they work so well is because text is a unified format, right?

3:14Because Wikipedia articles are in text, code is in text, math formulas are in text. So when people have a unified representation of information, they can train a single model using a unified objective, which is just language generation. Translation tasks are language generation. Summarization tasks are language generation. And solving math problems are also language generation. So with a unified representation of information and a unified task interface, this is why people can use language models to solve any tasks using kind of Internet scale knowledge. So I think this realization wasn't so clear for people working on video generation.

3:50Most of the time, you know, there was Pika Labs, there was stable diffusion, there was runway. A lot of people were looking into videos, but it was mostly for entertainment purposes to generate video according to user instruction, but usually some cute panda running on the street or something like that. So back in 2022, in September, a few of us from DeepMind and also from Berkeley started to look into, can we really use video generation models to solve real world tasks? Because video is also a unified representation of information, right? So I don't know how long I should go down along this path before you ask another question.

4:25But if we think about it, because I'm just like so excited about all of this. No, it's clear. I think what I heard there and got from the paper was really your excitement about video and almost like a sense of awe. Like everybody, you're all getting so excited about what we got out of text. Like we have these quote unquote emerging behaviors like reasoning and all these other things. Well, video is so much more rich than text. Imagine if, you know, what we can get if we can fully take advantage of video. I think the big question is then how do we do that and how do we overcome some of the challenges associated with taking advantage of video?

5:10And maybe we can start with some of those challenges and why it's been so difficult. For example, with text, I think a big part of the breakthrough is figuring out that we can come up with self-supervised training approaches. And some of the same things are available in video, but also it's different and harder in a lot of ways. Can you talk a little bit about some of the challenges with the way we currently treat video in terms of achieving some of the goals that you're looking to achieve? Yeah, there are many challenges. I guess one of the challenges is data coverage. So any of the things that we're saying now is unlikely to be completely out of the distribution of internet text.

5:51It's a little sad to say, but it's probably true. You know, a lot of the words I'm saying now are probably somewhere hidden in the corner of internet text. But for videos, I'm pretty sure like the bedroom I'm in right now is not on the internet. So in some sense, like the image of my bedroom is indeed out of distribution. So this is kind of like the data coverage issue. If we want to generate videos starting from arbitrary image frames, that is actually much harder than trying to generate text, which is just to complete a sentence. And data coverage is one such issue, but we also run into this kind of lack of labels.

6:25So for language, there is something actually quite nice about it, which is the words that come later in a sentence is somewhat of a label or supervision for words that are earlier in a sentence. If we just cast all of the tasks into a language generation task. But for videos, oftentimes they are really only useful when we do conditional generation, meaning that condition on the image frame would give it some input, like move forward, and then we generate this motion, which means that we will need to collect labels for specifically controlling the type of generation that we want. We probably have a lot of internet scale videos from YouTube, for example, but oftentimes there aren't really good labels for these videos.

7:04Maybe there are captions, like some transcriptions of what we're saying right now, but it's not really characterizing what is happening in the scene, right? It's just a transcription. Why doesn't something like infilling or frame interpolation as a task or objective solve the problem of self-supervision for video? Yeah, so it's not very clear. Just predicting the next frame without any external input is actually doing anything useful. Whereas for predicting next words, because a lot of different tasks can be casted into predicting next word. So predicting next word already solves a lot of the tasks.

7:42But for videos, it probably learns good representations, but how do we use those representations for downstream tasks is not very clear. I guess there's another challenge, which is for language models, people were working on different architectures, LSTMs and transformers, but pretty quickly people have converged on autoregressive transformers being the architecture and people start to work together, iterate on the same architecture. But as of now, video generation models are quite heterogeneous. There are diffusion models, there are mass models, and there are autoregressive models. And it's not super clear whether any one of these models will win or a combination of these models would be the best.

8:21So each person or each organization is working on their own kind of best model architecture. So we haven't really come together collectively as a whole to discover the best architecture, Even though perhaps more recently, there might be a little bit more convergence happening where these higher level semantics are sort of tokenized and people use autoregressive models to generate these high level tokens and use diffusion models to really turn these tokens into high resolution pixels. It just took us a while to come here. When you look across the space and the research, how close are we getting to achieving some of what we're seeing on the tech side with video?

9:00Yeah, so if we were to ask what videos can empower us to do that text cannot, and this oftentimes is the type of information in videos is oftentimes much richer. It contains very detailed physical laws of the world, like friction and torque, implicitly all captured in the pixel space. Language is okay when we use it to describe a car driving down the street, but it's fairly high level. It's not really telling any details of how we can do it. And oftentimes people might prefer watching a YouTube video to learn how to peel a pineapple or how to replace the tire on their car as opposed to reading like a step by step manual.

9:38It's just that because the video has a lot of implicit information that helps people understand. So I would say, you know, one such idea would be can we have chat GPT, but generate videos to answer people's questions in specific scenarios. Right. I don't care about a panda running on the street. I want to know, you know, I'm halfway through cooking this meal, but I'm kind of stuck. So what do I need to do next? Or I'm halfway through fixing my laptop, but I'm not sure what to do next. So now I want to customize answers in the form of videos that I can watch and understand how to perform these tasks.

10:10So that is something that text could only help us to a certain degree, but watching a customized generated video as answers is much more helpful. And there are also these visual reasoning tasks. For example, solving these puzzles, you know, you have a circle, circle, square, and what comes next? Visual reasoning is not very easy with language models as well. My background is in reinforcement learning decision making. So there's been a lot of excitement in robotics about leveraging these large models. The issue is that these language outputs from these models are generally pretty high level. If the goal is to put the fruits in the drawer, then language model will output something like open the drawer.

10:49This is still quite some distance away from the low-level controls the robot needs to know exactly how to open the drawer. If we can generate a video of a robot hand opening a drawer, that actually already has all the information pretty much we need for the robot to know this is how much I need to move in the pixel space to reach the handle of the drawer, for example. So video bridges this gap between these very high-level information from language and these very low-level controls and these continuous values that people have to derive in order to control these systems. Continuing your argument from earlier, one might say that if we do a good job of turning the picture into the thousand words and we can do a good job creating pictures or videos from thousands of words, then why do we need to work natively in video at all?

11:38Are there very concrete, succinctly stated examples of what you could never achieve if you have to use text as an intermediary? I mean, it's actually really hard to say because it's hard to define what is a text because, you know, byte streams are text, but people are not doing language models using byte streams. Text has some structure that language model leverage to solve tasks as opposed to just using raw data in its own format. And the same with videos, there is prior knowledge about videos and also shape and rotation invariances in images that this prior knowledge kind of defines this set of data being a special format that is potentially useful for tasks.

12:21That's why, you know, as humans, we have visual system and also language systems to solve a lot of different tasks. Yeah. I think what I hear you saying is that in the end, it's going from some pixels to some representation of those pixels that's expressed as vectors. And whether those vectors are numbers or letters, you still need to get back to some pixels or actions and so you know whether it's text per se or not it's all about the transformations that are getting you from the pixels to the to the actions or the outputs that you're looking for yeah exactly it's really kind of how we represent these data formats right that comes back to the first part of the point that i was making is well as you said if we just convert every pixel into some discrete token describing that pixel, then there is no difference between videos and language.

13:14But it is indeed kind of like the level of abstraction in the sense that language is not raw bytes. It is some structured information. And same with videos. It is some structured information. Yeah. And so the paper, is it a survey of the challenges and kind of a projection of the road ahead? Or are you proposing specific methods that you think will help get us to richer representations for video and being able to the specific task performance that we're targeting here? I would say it's both. It's really to scoping out this problem space to first really recognize what is the thing that made language model work so well.

14:00And the point that the paper makes is this notion of unified representation of information using text and a unified task interface using text generation. And then it's really like the recognition that we can do the same with videos because as we said, videos has all of these information, detailed spatial and visual information about colors and shapes of objects, distances and orientations, for example, they're all captured in video and also dynamics, interactions, behaviors, actions from humans, from robots. And all of this information is in the video as a unified data format. I wanted to drill in on just that point.

14:38When you say unified task interface, what exactly does that mean in the context of text? And then extend that to video. Yeah, I'm glad you asked. So for language, if we look at how natural language processing community has evolved over the years, it's very interesting. So initially, people had, you know, people focusing on specific topics within natural language processing, there are people working on semantic parsing, there are people working on translation, there are people working on summarization, sentiment analysis. And oftentimes, you know, the focus groups probably don't even talk to each other all that much.

15:09And it turns out that all of these different tasks at the end of the day is a language generation task. Because we have a single model and a single objective, then that's why we can just pour internet data into the single model, single objective, and as a result, get a very general model that can perform different kinds of tasks. So that, in my opinion, is really kind of like the takeaway from the success of language models. And now coming back to videos, why is this true for videos? Well, let's think about what kind of things are in videos, right? Maybe there is how to cook or how to fix something.

15:43Those kind of tasks could be characterized as a video generation task. And there are also tasks like, how can I teach my robot how to open the drawer? And that is a video generation task. And also simulation, physical simulation, dynamic simulation, fluid dynamics, and cloud movement. All of those tasks, knowing what happens next if I give the model some control input, these are all video generation tasks. So again, we see the common theme. Once we have a unified task interface, we can pour all of the videos into this video generation task. Yeah. And do you see the interesting problems in this space as inherently multimodal, meaning you've got some video input and then some conditioning that's outlined or provided as text and then a video generation?

16:36Or do you think a single-moded video in, video out has equally useful use cases? I see it as a multimodal problem because, as I said, there are information at different abstraction levels. There are these pixel information that are very detailed and there are these high level abstractions using text. And oftentimes, if we were to look at it from a decision making perspective, which is what a lot of my research focuses on, we have an agent or actor in this setup that's deciding on actions with a policy, which is kind of like a strategy for choosing actions. And we have some environment, which is taking these actions and tells the agent what happens next if I were to take this action and whether this action was good or not.

17:25So clearly we see kind of two different components in the system. One is an agent deciding what to do. Another is an environment telling the agent, what happens next. So people have gradually realized, oh, we can actually make language models into agents. So there's been a lot of work on using language models as agents to interact with external tools like websites or with humans, for example. So I think there is not too much of a recognition that, well, these large models are actually also good world models in a sense that they can actually provide information on what happens next if I were to take an action and whether this action is good or not.

18:04So there has been some work perhaps of using language model to provide feedback for itself so that it can self-improve from this feedback. But that really boils down to this basic setting of decision making, which is when we have some agents interacting with environments and learn on its own, which is how we get superhuman performance. Because if we just do supervised learning or unsupervised learning using the data set, maybe the best we can get is the behavior in the data set. But once we're in the setting of having an agent, interacting with the environment, and learning to self-improve in the same time, this is how we can get superhuman performance.

18:38And this is really kind of becoming more and more possible because we have language model as agents and perhaps video as world models. And this is really kind of like enabling new applications that in the past wouldn't have been possible. Yeah. Do you see specifically an analogy in the video domain to tools within an LLM? Yeah, there's definitely the analogy in the sense that, for example, video models also need to know how to retrieve information. Like, you know, if I'm generating a video, I put a fruit in a drawer and I closed the drawer like five hours ago. Then when we generate the video of opening the drawer, the food should be in the drawer.

19:24So does this mean I have to store five hours worth of video in the context of the video generation model? Or do I need an external database and have a video model to retrieve information from the database? If that's the case, then it's quite similar to this kind of retrieval augmented language models. Reg style. Exactly. Yeah. So there are more challenges associated with it because if we look at the interface is probably some kind of programming language. And as a result, language models are probably more well-equipped for directly outputting those programming language code to kind of use external tools.

20:02But for videos, there is additional information and there has been work on converting images or videos into code. It's a different form of input with different form of information. And we will still have to require a language model to generate some calls to the external tools, but at the end of the day, we still have more information in the video generation model. One example that just came to mind for me was, say you show a video that is part of tossing a ball and you stop after some period of time, the ball is still going up. The generator might want to call out to some tool to run the physics model on what that parabola looks like can provide some parameters and use that to condition the completed generation.

20:49Is that something that you think is, is that the direction that we're headed? Or do you think that that's a viable idea? Yeah, totally. So I think video generation actually has a lot of implications for domains, like you said, science and engineering, right? Because a lot of the scientists I've talked to were like, well, what do we need to do now? Do I need to convert everything I have into text and try to use a language model, right? It's really difficult for them to believe that. But then oftentimes, you know, a lot of the tools that people have built for simulating fluid dynamics or taking, you know, electromicroscopy scans of samples from biology and chemistry and materials, a lot of these data actually come in the form of images and videos.

21:32And a lot of the end result is being able to simulate the dynamics accurately and as a result, being able to optimize some sort of control input to the dynamics well. So in these domains, perhaps video is a very much applicable form of modality than text, because oftentimes a lot of things are just not in text, right? The first observation is in the form of images and videos. So it's a lot easier in these environments to learn a visual dynamics. So condition on some control input, some turbulence that is asserted into the system. And then to try to simulate the visual effects of these external inputs.

22:11So this is really kind of coming back to the question of like, why isn't text alone enough? It's because, well, a lot of these scientific domains are just interested in the visual information. And I think you had a really good point about these physics-based simulators. So as a person who worked in some robotics questions, I realized the sim-to-real gap is big because there's a certain extent to which people can simulate physics using human knowledge. So then the good question is, well, instead of the graphics people and the machine learning people working in separate areas, while using data-driven learning approach, while using physics prior knowledge to build simulators, maybe it makes more sense to kind of bring together data set that has different information or different pieces of the puzzles to solve the problem of simulating the world.

23:00Because for the graphics engine people, maybe they have very precise control of what's being simulated, right? Because they build things from bottom up so they know exactly this is a table and this is a chair. Whereas for more end-to-end learning people, we have a lot of data from the internet. A lot of them are not labeled. A lot of them don't have precise motions or controllable motions in the video. So can we kind of, you know, somehow combine knowledge from both research communities and think about jointly together, how could the graphics community and the video generation community kind of come together to simulate these highly realistic interactions with the real world, but are also very controllable.

23:44So right now, like people use text to control simulation, right? But if I were to specify, okay, make this person run, you know, 1.36 faster, like it's not clear if we can achieve that level of specificity in controlling the video generation. I think that calls to mind Sora and what we have all seen and got excited about with regards to that model. Talk about, in light of this paper and research direction, how did you parse Sora and the degree to which it is a competing idea, a fulfilling idea, a proof point? Talk a little bit about Sora in this context. Yeah, so it's quite interesting because this paper was written before Sora became available.

24:33There were some delays in terms of putting it on archive. So that's why you actually cannot find any information about Sora from the paper. I think the research community, which is what this paper represents, and Sora, which is probably what the industry community represents, are actually heading towards the same direction, but from different angles. From the research perspective is how can we learn good simulators for training agents for improving algorithms? And from OpenAI perhaps is how can we have hyper-realistic generative models for videos? But then I think this realization that generative models can indeed serve as a real world simulator has kind of been unanimously agreed on across the industry and academia.

25:15Because once it can simulate the world very faithfully, there are just very diverse set of applications, implications that has been opened up, like I said in science and engineering, robotics, filmmaking that has just not been possible before. So I guess this is also somewhat why Sora's framing is a little different from previous work in video generation, which is more targeted towards entertainment media, whereas Sora is more targeting towards treating video generation as a real world simulator so that we can do various things with it. so I had another paper called learning interactive real world simulators it's called Unisim it is really kind of conveying the same idea as Sora last October on how video generation is the real world simulator and the paper went on to like using this simulator to train robot policies and of course maybe that's something that Sora hasn't been doing or maybe Sora will be doing as OpenAI is investing in maybe robotics companies.

26:16But I would say it is kind of using similar lines of ideas to kind of tackle the same set of problems, which is really kind of using video generation as a simulator. I wanted to go back to one of the other parallels you mentioned in the paper between language models and video models, and that is the idea of reasoning and chain of thought in particular. Can you talk about how you see that playing out from a video perspective? Yeah, so definitely. So I give some examples of like maybe solving visual puzzles using video generation models. For example, geometry problems, right? Oftentimes, drawing the right auxiliary line is pretty much equivalent to solving the geometric problem.

27:03So in that case, we could ask a language model to describe, you know, these are A, B, C, D, E points, between which two points should I draw the auxiliary line. But oftentimes, you know, when people solve these geometric problems is to reason, well, once I have a line here, I have a 90 degree angle here and so on and so forth. And then being able to kind of generate that as an answer to this geometric problem is one way of video models can reason and solve these geometric problems. There are also kind of other approaches, which I have done some of the work. So some of the algorithms, when we actually execute them, for example, breadth-first search, right?