Unifying Vision and Language Models with Mohit Bansal - #636

0:07All right, everyone, welcome to another episode of the TwiML AI podcast. I am your host, Sam Charrington, and today I'm joined by Mohit Bansal. Mohit is Parker Professor and Director of the Merge Lab at the University of North Carolina, Chapel Hill. Before we get going, be sure to take a moment to hit that follow button on Apple Podcasts, Spotify, or wherever you are listening to today's show. Mohit, welcome to the podcast. Thanks for having me. Really excited. Same here. So we're going to be diving into your work on multimodal, large-scale language models, and in particular, the themes of unification, efficiency, and evaluation, among others.

0:48And I'm very excited about exploring that topic with you. But before we get to that, I'd love to have you share a little bit about your background, how you came to work in the field, et cetera. Great. Yeah, definitely. So way back, I was, I guess, 15 years ago now, I was in India. So I did my undergrad there. I don't know if you wanted that much background, but yeah. So I started some AI and machine learning. IIT Kanpur, multimodal was not really a thing back then. And then I did my grad school at Berkeley, where we were always sort of joking between the NLP and vision groups, like, hey, like, when should we start multimodal?

1:22And then we would say, like, it's not really ready yet. The vision and NLP components need to be sort of smarter to talk to each other. So I think my first multimodal project was around 10 years ago when I was in Chicago. And then, yeah. And then since then, it's been a big part of my group and my lab. We basically work on several things. We have large scale language models for multimodality. We do a lot of video based data sets and tasks and models for video summarization, video future prediction, video question answering. We've also done a lot of work recently on trying to evaluate these very hot text-to-image and text-to-video models, which you see a lot of samples probably online.

2:02But how do you actually evaluate right there and compare these models? So, yeah, a whole spectrum of multimodal aspects. But like you said, I'll focus a bit more later today on themes of unification, efficiency, and evaluation because I think all these are very important in today's context. Were you surprised when the news came out that GPT-4 was multimodal? No, I feel like there was a lot of people already somehow predicting that. I mean, not somehow, they had probably some knowledge about it. Yeah, so we were expecting more multimodal elements to be getting integrated in all these models. I think it does have some pretty interesting elements in terms of zero-shot image understanding capabilities, being able to also do some step-by-step explanation of how it did that reasoning.

2:45But I mean, it's still a long way to go. I mean, there's a lot of capabilities like step-by-step generation and also things about explainability and efficiency. I would say it's a balanced excitement. It's an exciting time and it's giving a lot of people the sort of excitement to focus on this area while still tons of work left to be done. Yeah. And when you characterize the work left to be done, I imagine these themes that we've mentioned a couple of times now are a big part of the missing pieces. Right. Definitely. So it would be things like when I say unification, the idea is instead of having thousands of specialized models for every kind of vision and language tasks that have popped up in the last, I think we did LexMIRD, which was one of the first multimodal BERTs, right?

3:31You know BERT, obviously. So the first sort of BERT to combine vision encoders and language encoders and then have a cross-modal encoder for multimodal interactions has been almost four and a half to five years now. So since then, there's thousands of specialized models for very specific kinds of tasks, like visual question answering versus referring expression or visual pointing, generating a bounding box or segmentation, or even image generation and so on. So recently, the focus has gone much more on can we unify these models? So first of all, that's more simple, that's more efficient, because it can share parameters, but it can also learn knowledge from each other's parameters and modules.

4:10So there's a lot of advantages to unification, right? The shared knowledge, like I was saying, but also generalizability. Maybe because you've learned all these tasks together, you might be able to also compose a new task, right? An unseen task when you come across that. So that's obviously one angle. And then efficiency, I don't have to say much, I'm sure, right? Because as you know, all these models are going crazy in terms of number of parameters, like 100 million used to sound big, then a billion used to sound big in terms of parameters, then 10 billion, right? So, I mean, the definition keeps changing, right, of like what we are surprised by.

4:41The sad part is that these are still extremely out of sort of reach in terms of the resources available to most people, right? I mean, obviously, most of academia, but also just in general, right, any underrepresented community or anything that's not a big organization or like the four or five such options. So, that's the other theme that I can go deeper into later where how do we make these models efficient in all aspects, right? So number of parameters that need to be updated, the memory, the sort of speed, even, right? And there's many other angles, right? Like the environmental aspects, the carbon footprint, and so on.

5:11And then the third thing you mentioned, evaluation. Yeah. So evaluation is completely like an open challenge right now, right? I mean, I would say the first two, at least we've been, right, the community and our lab and all of, like a lot of folks have been at least focusing on the last two years, much more. But evaluation, especially for things like generative AI, right? So what we call generative AI, I don't really call it that in my life, but a lot of us have already been doing it for so long. But if you want to call it generative AI, just the term itself, generative AI, at least the good thing about that term is that it should clarify how hard evaluation is.

5:44Because even in language, there's a 30, 40-year-old field of just focusing on how to evaluate generated text, like automatic summarization and so on. Because there's thousands of ways to say the right thing. right so you cannot just match it to the ground truth which the humans wrote so now imagine doing the same thing for generated images and videos right which will have even more infinite sort of variations right that can still mean the same thing and they don't really matter right they're not really important changes but the model might get fooled by it right so yeah so evaluation metrics is one aspect of it right so we have one of the first works called dally eval that tries to take tally and other diffusion and image generation models and actually quantitatively try to evaluate their fine-grained reasoning skills like can they get the count right can they get the spatial positions right like left of something versus right of something right generate a horse left of the astronaut versus right of the astronaut got it and also social kind of issues like biases right gender bias race bias and so on color skin tone bias but there's so many other aspects to evaluation right are these models even if they're getting the answer right is it by spurious correlations or is it actually causally correct right is it right for the right reasons or right for the wrong reasons then there's this whole angle of also explainability right can the model explain the path to its answer right so we have some work and there's also some recent work from the community on basically being able to show the program right that sort of generated the actual answer could almost be like a graph or a tree of different modules that you're using so that opens a lot of very exciting doors actually as future work right because you've probably seen some of these works where, in fact, you can generate something as a program of different APIs or different tools, right?

7:25And in some sense, you're basically using this API or even apps, right? Different kinds of tools that are just getting combined and you execute that program, right? And the answer comes out. So there's a lot of recent work from different groups on that. Awesome. You know, when thinking about unification, one of the drivers for a long time for multi-modality, at least from the perspective of some NLP community, was this idea that in order to really kind of master language, we need grounding. And one of the main sources of grounding is like visual information. And so a driver for kind of multi-modality was to achieve grounding.

8:05In a sense, you know, the success of large language models says maybe we don't need other stuff in order to create these models that are really good at language. So I'm just curious your perspective to what degree does grounding factor in when you think about multimodality? And what are the primary drivers for your research in the area? Yeah, so maybe I'll come at it from two different angles. So one is the perfect question you asked, which is about what is grounding and is it needed today, right? In fact, the first multimodal project that I worked on 10 years ago was grounding where the topic was you have some 3d images and you have this sort of maybe caption right describing the image and this was in collaboration with other researchers at TTI Chicago and basically the idea was that can you actually learn the perfect definition of coreference so coreference means like figuring out which two words in a sentence right refer to the same entity so usually it's just been done as text alone right because you just try to figure out okay does this pronoun refer to like this person or the other person right but if you look at the actual definition of coreference it's all about grounding right do they actually ground two words ground to the same entity right in the actual world so that's what we did we said okay like can we improve coreference of different things in the long caption it was a paragraph level multi-sentence caption can we improve the coreference by actually trying to ground different words to bounding boxes in the 3d image like cuboids not boxes so that was the basic idea and in fact you could do the reverse you could also improve the box prediction and the scene prediction using information from the text so it was sort of bi-directional mutual improvements and then six years later we did this paper called vocalization which was basically the first attempt at taking something like BERT like a pure language model with masked language modeling objectives right which I usually just call fill in the blanks which means that you just remove some words right and then in context the model has to predict the the missing words.

9:59Now, what we said is that doesn't sound enough for Bert to just be basically be a world model by only filling blanks in Wikipedia. So what if we also ask it to do an additional pre-training task objective of for every word, you also need to predict a visual mapping, which is exactly what you said of like, can each word in context also try to sort of come up with some sort of image, right? That represents that word in context, right? Because the word on its own can have multiple meanings or can be ambiguous. So that's what we did. We first trained a Vokenizer. This was... But not from a generative perspective, from a retrieval perspective?

10:34From retrieval, because this was 2018 when we started this project, 2019. So now, yeah, you could do the exact same thing with generations. But back then, it was basically first training something like... This was like a small version of Clip before the Clip era. And what you would do is you could train a Vokenizer on MS-Coco kind of multimodal data sets, which have images and captions. And then you would basically take that visual mapper that's trained on MS Coco and get all kinds of visual predicted visual mappings for all of Wikipedia because that's what you trained Bert on right back then so basically you would want longer sentences from Wikipedia and much more text because Bert cannot just be trained on small short caption style sentences so you would transfer this you would train the vocalizer on MS Coco but you would get all the visual mappings from Wikipedia based on that tool and then you would basically retrain pre-train Bert using two objectives now the mask language modeling as well as the visual like retrieval prediction and then we did a more advanced version of this at NeurIPS 21 called WIDL and KD where we were not just using images but we said let's distill knowledge from videos right because videos have much more temporal and physical knowledge so then what we did is we trained a teacher model which was trained on video and language discriminative objectives took the just the language part of that teacher model the language encoder.

11:51And we basically distilled its knowledge to a student language model. And then we only use that student language model to perform language only tasks. And we showed that basically you can So you're trying to show that you can ground in training visually and then take just the language part and separate and it retains some of the benefit of grounding. Yeah, exactly. So we actually showed that it can retrieve relevant videos just as a language model, even though it has never seen videos before. But more importantly, we took NLP only tasks, right traditionally, which are not multimodal tasks, just text only tasks.

12:24And we showed that you can get better results, because this language model has distilled knowledge from videos. And in fact, we showed it on very specific data sets that intuitively should need temporal or physical knowledge. So one example is physical common sense task called PICA from UW, where basically it needs kind of knowledge of whether like how to screw a bulb clockwise versus anti-clockwise. And then also something different would be like temporal knowledge, where if you mention a very complex ordering of events, like can you actually figure out which event happened before which other event?

12:56So these are the kinds of things where you would imagine, right, this kind of knowledge being useful. And that's where we showed the bigger gains. So that sort of comes back to your like last point, right? Is it relevant as the language models keep getting bigger and bigger? So that's sort of an open question. I have two answers to that. One is, I feel like there'll always be certain kinds of physical common sense and temporal common sense, right? And dynamic common sense kind of knowledge that's just not written in text, right? I mean, this is a very old concept, right? That some things are more visual and some things are more verbal.

13:29So unless you actually find artificial data that has described every possible thing in the world, like verbally, if you look at natural data, right, then there are a lot of things that are inherently visual and also temporal and physical. so you don't expect them to be having very concrete definitions in text very verbal so my short answer would be that right but the other more interesting answer would be also data efficiency right that connects to the like maybe in a nice way the second point of efficiency that even if you did find a way to get 100 billion tokens of text to be able to capture all this knowledge like why bother right i mean if you can basically use much smaller amount and much more natural and much easily collectible similar information from another modality right like that's in some sense how we learn right as humans because we use all our senses as opposed to if i forced you right to like no you're not allowed to just read ground anything yeah you just have to learn your whole you'll be 40 years old and you have to just learn everything by getting lucky by finding text right so so that's the that's the trade-off i guess it's also about data efficiency It's a bit counterintuitive to think of video as being efficient relative to text because the files are so cumbersome, the volumes are so large.

14:41I completely agree. But I think there's a difference in terms of senses, the sense being used and the modality being used, it's more efficient. But in terms of engineering, it's less efficient, right? So I think you can pack the same amount of information in a video if you are learning as a human by watching a video, right? So in that sense, much more information can be packed. But in terms of engineering, yeah, just the way we represent videos right now is just more cumbersome. So that's been a big focus of our work, actually, right? How to make video and language models much, much more efficient.

15:09For example, by using very specific keyframes and not all the frames. And we have another paper where we showed that you don't need to use the video all the time. Sometimes you can decide which parts are important to see the video. And some parts you can just listen to the audio, right? Imagine watching a movie, right? Sometimes you're just working, right? when watching a movie and only a few times you glance at the TV, right, for the important scenes, but the rest of the time you get most of the material from the audio. There's a lot of, right, other ways to get efficiency in video and language.

15:35So one of the things that is enabling kind of this next level of unification for multi-modality is kind of this emergence of transformers as a standard substrate for machine learning applications. Is that necessary for unification as you've defined it? Or were you doing experiments around unification pre-transformer? Is there an idea of pre-transformer unification? You know, in some ways, maybe there's like architectural unification, but there's also like task unification or something like that. Like what are the right words to talk about this kind of stuff? So there's different levels of unification is what you're trying to ask.

16:14So the unification thing that we did in 2021 ICML, we started it early 2020 was called VLT5. So that was definitely based on transformers, right? And the T5 model. And the idea there is, can you unify based on treating everything like language generation? Since it's a multimodal interview, a lot of it is visuals. I wish I could show you slides, but I'll try to describe it visually. This is a perfect example, right? Of the efficiency I was talking about. I have to describe everything verbally, but I can just show you a figure and we would be done in five seconds. So basically the idea is that we said, why can't every task be treated as language generation?

16:50So this was kind of like the language version of generative AI, where we said, okay, imagine you're doing visual question answering. Then it's obvious, right? You take the image and the question and the answer can be generated as text. But suppose you're doing image captioning, then also it's obvious, right? You take the image and the caption gets generated as text. But you can also do something like image text matching. So that's a task, right? where you have to say false or true, whether a pair of image and text match each other or not. So that can also be generated as true or false. Instead of classification, right?

17:17You can do a generative version of that where it generates same as VQA. VQA traditionally is a thousand way classification. You already know all the thousand possible answers and your model has to do a sigmoid right over all kinds of answers. Whereas what you could do is you could try to generate all these answers, right? And this way it also allows you to handle rare answers or unseen answers. So that was sort of the idea. But what you could do is you could also handle very unexpected tasks. So what you could do is you could also say that I want to basically do the task of visual grounding, which is also called referring expression comprehension.

17:49And the idea is I give you a complex image and I say the third horse from the left, right? And it might have many horses. So basically you have to draw a bounding box or segmentation around the third horse from the left. So traditionally, this is basically the task has to generate a, like give up, like it has a lot of boxes and it has to predict which box am I talking about? So in this case, let's say box number 31. right so what we said is now you can also handle this task as language generation right where you could basically just ask your language decoder to actually just generate the id of the box as a word so that's what the idea of vlt5 is right that you could handle all kinds of seven eight different very different multimodal tasks like bqa captioning image text matching also visual grounding but also things like multimodal machine translation where you're given english plus image and you have to generate a second language and we showed all of these can basically be treated as image and text inputs go in you do a bi-directional transformer and then the output is generated by language decoder it's basically just like conditional language modeling all right you just have a conditional language decoder and yeah so that way we showed that you can actually have seven times less parameters by basically sharing parameters across all seven tasks and in fact get pretty similar performance to all the specialized individual models for all these seven tasks And then there were also other side effects, right?

19:05Like I said, you can also handle rare categories or unseen categories in VQA because you're generating things, right, word by word, or even character by character, as opposed to classifying from a vocabulary. So this was 2021. To be clear in that particular research, the VLT5, was that end-to-end trained or did you have a pre-trained kind of standard LLM and input stage that kind of fed into it? Yeah, so we started with T5 parameters. Yeah, T5 parameters could be used to initialize the decoder. And then you would have the visual and textual input. So then you would think of it almost like visually prompting a T5, right by giving it an image and a question, these kinds of things.

19:46We showed both ways like with and without using the T5 parameters. And then what happened is this was 2021. And then a lot of good follow up work from Google and AI to so you probably heard of models like flamingo and also unified io gpb right so a lot of these models followed up on vlt5 and they took it several steps forward where instead of only generating everything as text you can also handle tasks like text to image where you could also be right like the decoder is also multimodal so based on the task you can decide whether to only generate language or generate an image or both right so this is sort of now it's many to many unification Many things can come as input and many things can come out as outputs.

20:26But now coming back to your question of different levels of unification. So VLT5, there's three levels of unification, right? You could unify the parameters that you can share the parameters, but the pre-training object might still be different. And then architecture might also be different for some parts of the model. So another more interesting thing could be where you also unify the pre-training objectives. So one example of what we did there was called TVLT, which is called textless vision language transformer. And the idea there is that you basically want, so this goes back to your question about video and language efficiency.

20:59What we figured out is we said that if you're using video and language, and there is a lot of issues in the text. So what happens is you usually use the automatic subtitles, right, that are obtained by getting like automatic speech recognition, right, spits out the subtitles. So there's a pretty well known problem here that this is all very noisy and misaligned with the videos and so on. So what we did was we basically said the intuition is that you can actually skip everything on the text side. You don't want any ASR modules. You don't want any tokenizers. These may not even exist for a lot of domains and languages.

21:30And what you can do is you can treat the audio spectrogram images almost the same as video frames. So basically, again, visually, it would be easier to explain, but you probably have seen audio spectrograms, right? Like these sort of images, right, that have different levels of the audio features. so what we did is we basically treated the video frame images and the audio spectrogram images exactly as the same image encoder with the same shared parameters but also with the same shared pre-training objective now and what that means is that you can actually just do the masked auto so are you familiar with the image version of masked language modeling so in in bird what you would do is you would remove words right mask out words and fill in the blank in context in images you do the same thing where you mask out patches of images, masking squares and stuff.

22:17Yeah, right. And then you have to reconstruct them. So basically, we did the exact same thing, both on video frames, as well as audio frames. And this way, we could share both the pre training objective and the parameters and the architecture. And what we showed is using this, you avoid all text modules and all the noisy situations, you get 30 times speed up in inference, and one third of the parameters while still maintaining the same results as text-based models. But you also get improvements when there are tasks that actually need audio features, like multimodal sentiment analysis, where you need some sort of emotion and pitch and tone, right?

22:51These kinds of elements. So that's sort of one example of going one step further, right? In terms of like the kinds of things you can unify. So on this topic of the image masking, just as an aside, I guess, Is that the same as kind of infill as a task? Yeah, it's similar to infilling or inpainting, these kinds of things. So that's also being used for text to image generation now. So that actually also connects well to the third example I was going to give for unification, which is this new work we had in collaboration with Microsoft called UDOP, Universal Document Processing. And the idea there was that now you have even more complex modalities.

23:26You don't have just text and vision, but you actually have them spread out in a very complex layout, right? So it could be very... So think about like a newspaper page or an annual report or something like that, where you've got text, you've got images, you've got tables, et cetera. Exactly. So it could be even things like random websites, finance reports, academic papers. So very, very diverse kinds. There's a whole actually benchmark, which has all these kinds of documents, finance reports, academic papers, websites. And the best part is there's a whole benchmark on this, where you have a set of questions that you need to answer about these documents.

24:01So it's called document question answering. And there's several other related tasks called document understanding tasks. So it's called the DUE benchmark, document understanding evaluation. So basically, this UW model that I'm talking about got the new state of the art sort of frank one on this benchmark. And the way it does that is that it unifies text, image, vision, and layout, all three of these together now, right? So it's not just about text and images, but it's about text, images, and layout. So the way it basically does this is it just has these joint pre-training objectives where the text layout and vision goes in and then it has to reconstruct the text and layout together so instead of just reconstructing the patches or the text independently now it treats the text and the layout kind of boundaries and locations indices right like the the coordinates you can call it it starts merging the text and coordinates together as just text and it tries to reconstruct the text and the coordinates of that text all together and then it also does the similar thing for the vision side.

24:57It's very cool. I wish I could show you images, but like what it can do is it can. Well, if you send me links to images or we'll of course include the paper in our show notes, but if there are particular images or videos or anything like that, we can include those in the show notes page as well. Sounds like a little bit of, you're doing two things here. One is you're making the layout information explicit to the model. Right. Right. And the second is multitask kind of thing. You're having the model attend to thinking about layout as it's making predictions. Exactly. So it can sort of learn that like what kind of text appears on which part of the document, right?

25:36It's sort of learning this together. That suppose I see something like a signature, maybe already know that it's usually in the bottom left or the bottom right, not on the top left or top right. So that's exactly what you mean by making it think jointly. It's not just generating the text. And by here text, by the way I mean also OCR like it can be handwritten not just like type okay so that's why I wanted to show this to you because we have these very cool examples where this is one of the first models that can actually edit documents so I can tell it exactly like which part of the image sort of to change right and like replace it with something else with the same exact format or handwriting and so on and so the technically this is important obviously ethically there's a whole separate discussion on this right that's why we didn't release this image encoder we only decoder We only release the importer.

26:16But technically, it's extremely important because this way, the model is actually much better at understanding by being able to generate and edit the input, right? When you're actually able to edit and generate things, you understand much better, right, than just staring at things or just looking at them, right? So that's sort of why this model is really state of the art, because it's understanding by generating and editing documents. I want to draw a parallel. Well, and you tell me if you think that this is applicable or whether I'm kind of overloading this idea. So we've kind of long explored in the podcast, this kind of swinging pendulum between pure statistical systems and trying to incorporate kind of physical knowledge into statistical systems.

27:00For example, in the case of robotics, like do we incorporate the physics information or do we just do kind of an end-to-end statistical treatment of the problem? And in a lot of ways, incorporating in the layout is kind of analogous to incorporating in the physics of the document as opposed to an end-to-end, hey, everything is pixels and we don't care about anything. We don't care about this additional kind of featurization of the layout. The structure. The structure. Is that a fair analogy? Yeah, it's a fair analogy. And I'll take it one step further, which is I think already what you said, is that I could do it in two ways.

27:36I could actually pre-define the structure, right? And like sort of hard-coded or give it as priors, right? Whereas the other way to do it is to sort of learn that, right? Through generation or editing, right? So that's the sort of difference I was trying to create that also because this model is not pre-assuming anything during understanding, but it's actually learning by making edits and changing and seeing whether the answer goes wrong, right? That's why it can learn much more in terms of almost like exploration, right? As opposed to given some priors, right? Okay, this is how the world works.

28:04That doesn't work for very, very diverse document layouts, right? Like different finance companies versus websites versus academic papers, like different style files, right? Things are so diverse that the generation element is very important here. This UDOT paper is a CVPR 2023 paper, and we'll also include in the show notes, Cha Zhang from Microsoft is a co-author on this paper. And he was previously on the podcast, at least a year ago, I want to say, talking about kind of some of the ways that they're approaching advanced OCR, which I'm imagining some of that is like a precursor to this work.