Why Vision Language Models Ignore What They See with Munawar Hayat - #758

0:00Thanks so much to our friends at Qualcomm for their continued support and sponsorship of today's episode. Qualcomm AI Research is dedicated to advancing AI to make its core capabilities, perception, reasoning, and action ubiquitous across devices. Their work makes it possible for billions of users around the world to have AI-enhanced experiences on devices powered by Qualcomm technologies. To learn more about what Qualcomm is up to on the research front, visit twimlai.com slash Qualcomm. If you take these foundation models, the proprietary models, and you should give it very simple images like an image of two boxes, two cardboard boxes, very plain.

0:40One is bigger, one is smaller, and you ask it to generate an image where we unstack these boxes. So once it unstacks, physical properties of the boxes change. They are not exactly the same boxes. Their shapes might be deformed. Their sizes might be different. And this is problematic.

1:11All right, everyone. Welcome to another episode of the TwiML AI podcast. I am your host, Sam Charrington. Today, I'm joined by Munawar Hayat. Munawar is a researcher at Qualcomm AI Research. Before we get going, be sure to take a moment to hit that subscribe button wherever you're listening to today's show. Munawar, welcome to the podcast. Hey, Sam. Thanks a lot. Thanks for having me. I'm really looking forward to digging into our conversation. We're going to be talking about some of Qualcomm's papers on multimodal AI and visual AI from the recent NeurIPS conference. To get us started, I'd love to have you share a little bit about your background.

1:51How'd you get into the field? So I did my PhD in 2015 from Australia. I got interested in computer vision and I took some courses in my undergrad on image processing. That's what fascinated me. And during my PhD, I worked on visual analysis of facial data. And then afterwards, I have been contributing to mostly computer, different subfields of computer vision. And yeah, and then after my PhD, I stayed in academia. I had a faculty position, and then I moved to Qualcomm in 2023. What's your research focus at Qualcomm? At Qualcomm, I have been mostly working on multimodal generative AI, so on vision language models for understanding generation and retrieval.

2:52And those are the three areas where I have my papers at NeurIPS that I believe you're going to discuss deeply. So when I joined Qualcomm, the first project I worked on was making diffusion models run efficiently on Qualcomm hardware, that's a mobile phone. And we're generating images in under half a second. And then we also worked on visual understanding, that's visual question answering, given an image and a question, a text question. We want to respond to the question. And for visual question answering, we also had a model running on Qualcomm hardware. So basically here in Qualcomm, I have been working primarily on multimodal generative AI, on generating visual content, understanding visual content, and also retrieving cross-modal information.

3:53When you think about how the field has evolved over the past couple of years since you started there, what's most exciting for you about this moment right now? I guess we have come a long way. Like for decades, we were focused on solving niche, very specific problems. Think of we spend so much time in just solving, classifying different objects from visual images. and then we moved on quite a bit. But there is a lot to do, I guess, especially in visual understanding and also in visual generation. In visual generation, physics-based generation, I guess we largely lack on that. And also in visual understanding, once we marry a vision model with a language model, there are lots of works that show that vision largely gets ignored and the language model takes over and we are responding basically from the parametric memory of the language model and not by looking at the actual visual content.

5:06When you talk about physics-based generation what do you mean by that? So physics-aware generation meaning we had a submission recently in which we observed that if you take these foundation models, proprietary models, and if you give it very simple images, like an image of two boxes, two cardboard boxes, very plain, one is bigger, one is smaller, and you ask it to generate an image where we unstag these boxes. So while these models can generate very fancy images with intricate visual details, but a simple task like this where we have very plain objects and we want them to unstack them. So once it unstacks, the physical properties of the boxes change.

5:56They are not exactly the same boxes. Their shapes might be deformed. Their sizes might be different. And this is problematic like in future if you want rewards to work within human environments and they want to perform normal tasks, then they have to have understanding of the physical world. So simple tasks like how would an image look like if I open a drawer? So current models fail. They would hallucinate. They would maybe generate new objects in the air. Or once they open the drawer, its physical properties are going to be different. And humans are very much kind of accustomed to that kind of thing.

6:44We just get by those things very easily. We can create a mental simulation once we're opening a drawer, for example, that the drawer is going to open this way. So I don't have to stand in its way. I have to position myself at a certain position. So when the drawer opens, it doesn't hit me and so on and so forth. and how much it's going to open and we can predict affordances like where do we grab to open the drawer and so on. So those kinds of things, while we take it for granted, but for the current models, it's a major limitation. They can't tackle it reliably and robustly. Now, some of those things, those challenges, when I hear them, strike me as being fairly solvable with more training data, like, you know, affordances and like where a box is, where a drawer is movable from, like what are the handles, that kind of thing.

7:46Others strike me as more, you know, found, you know, where we're lacking foundation, like memory, you know, for example, as a more robust structure in VLMs. Yeah. How do you think about what, you know, maybe what's lacking or where kind of opportunity lies in terms of overcoming the challenges that you're outlining? Exactly. I guess training data is a big, has a big role to play. So for this task, it's not as simple as crawling the web and collecting large scale training data. We can collect image text pairs. We can even generate description of images. But even those descriptions, they don't carry the physics information, the physical properties of the object.

8:39So the models can kind of implicitly learn those. So that remains a challenge. So one thing we observed was during training of these models, which is generally done with paired image text data, like image and its description. So if we expand the description of the images, we do prompt expansion. And during that expansion, if we try to describe the physics of the object in the scene, it really helps. So for example, if your task is to unstag boxes and you describe it, unstag the boxes, keep their structure intact, keep the lids closed if they are closed, and make sure the physical sizes of them, they stay the same.

9:31It seems like here we're taking advantage of the fact that the L is a lot stronger in VLMs than the V. That's exactly true. So that in itself, I guess it's a big challenge. So I guess from Travel Daryl's group, there was a recent very interesting study where they show the vision models the vision foundation models like Dino or Clip or Sam. So they, by themselves, have very strong vision capabilities. Simple tasks like spatial correspondence. So if we have two images, one is, and both of them are taken from slightly different viewpoint of the same scene. And if you ask which point in this image corresponds to which point in that image.

10:18so the standard vision models can solve this task fairly reliably but then they show that once you combine the vision model with the language model the performance of vision language models falls below chance so which is very interesting like they kind of fit so this is what we observed as well and this is one of our papers in NeurIPS that tries to tackle that to some extent. I don't claim it's a solved problem. We have a long way to go. And which of the papers is this? So in this paper, we are basically, we provide a systematic analysis of why do vision language models hallucinate. And what we observe is we take intermediate representations of the vision tokens recontextualized while they go through the language model.

11:23And then we have pair data of the same image text pairs and we have data of different image text pairs. So if you get the distribution of the similarity of image embeddings and the text embeddings in the embedding space of the language model, then for the distribution of the same versus not same we would expect those to be disentangled but instead what we saw was that they were highly tangled like there was a huge overlap and this kind of intrigued us and we did some further analysis and what we observed was so remember in vision language models we have vision tokens and text tokens we concatenate them and then vision and text tokens, they go through the language model.

12:13So while predicting the next token, we would expect that the language model would attend to the vision token. For example, if I ask what's the color of this box, so we would expect the box to be attended to, the visual tokens corresponding to the box to be highly attentive to. Or in other words, if we visualize the attention scores that we can easily do in the language model, we will expect that the attention scores corresponding to the visual tokens of the object of interest are higher. But what we observed instead was that was not the case. And this kind of intrigued us and we wanted to dig deep and try to tackle this problem.

13:00And that's basically what our work in Europe does. And this specific paper is attention-guided alignment in efficient vision language models. That's exactly right. So the thing is, how do we solve that, right? So we made some right observations that vision is not being attended to, vision is not being paid attention to. So just a question that just jumps out at me is, you know, when we think about how language models are trained, they're trained on, you know, lots and lots of textual data. Is, you know, part of the challenge that the, you know, these vision tokens somehow don't exhibit the same, you know, patterns and distribution as text tokens and the language model doesn't know what to do with them as easily as it does with text?

13:58That's right. So if you see, we have a pre-trained language model, as you said. It's trained on large-scale text-only data. And that scale of the training data is much bigger. Think of trillions of tokens, 15 or 20 trillion tokens, even for the smaller models. And then we have a pre-trained vision encoder, which is pre-trained on, think of, a billion image text pairs. and the embedding space of the vision to encoder and the language are different because they were pre-trained entirely differently. We want to combine them together and we do some alignment of the vision and the text. And this alignment is done on a very small scale data, generally.

14:41So that's kind of, I think that's also, that has a role to play. So you want to combine these two foundation models, pre-trained entirely differently, and then you're combining them, and the scale of the data that you're combining, and the quality of the data, and what kind of descriptions are in there, that has a huge role to play. And that is generally much less.

15:10So we observe that in some of the even benchmarks that exist for vision language model analysis, is if we ignore the vision and only ask the question to the language model and the responses we get just by using the language model and on purpose ignoring the vision part on these benchmarks, it's comparable to if you will feed the vision language model with both image and text, image data as well. So there is a problem with the benchmarks that we have as a community. And then I guess there are recent vision-centric benchmarks coming in. For example, if you ask, what's the color of an elephant? For that, the language model probably knows what the color of an elephant is.

15:56It doesn't really need to look at. Training data and the way they're trained, that has a huge role to play. To tackle it, what we observed was that we want to inject the visual information at different hierarchical levels of the language model, meaning language model has different layers or different blocks. It's a transformer with different blocks. So we interleave cross-attention modules after every fourth block, for example. And this way, right from the beginning till end, every fourth block, we have a cross-attention module through which we interject or we inject the visual information, visual tokens.

16:48So we make sure that visual tokens are there. And then the loss formulation we also change. So we make sure that the seminal region or the salient region within the vision is being attended to through an additional auxiliary loss. apart from the standard next token prediction loss that's used to train these models. And what is, like, how do you articulate the intuition behind this loss? So think of, if my question is, what is the color of the shirt of person sitting next to maybe a dog, right? So now we can get a segmentation mask of the person. That's easy to get with the off-the-shelf segmentation models.

17:46And the intuition is attention scores for the visual tokens must be highest for the visual tokens corresponding to the person because that's what the question is, right? And so if we ask a human to answer to that question, they would focus on that particular region. So we get these segmentation masks and we can come up with a loss formulation. We know where visual tokens are and we know where tokens corresponding to the question are, the visual tokens. And then we know what was the attention score, what were the attention scores while responding to this question. So we can maximize the attention scores corresponding to the relevant visual tokens.

18:33So, and we basically do it via segmentation masks through a SAM-like model. Is that implemented as an auxiliary model or is segmentation formulated as a loss component itself? It's formulated as a loss component. The segmentation masks are computed offline because this is just part of the training. So at inference, you don't need that. you can live without that. But during training, we have these segmentation masks computed offline, and then we can use these segmentation masks to compute the auxiliary loss. How does that impact your computational performance at training time? You said at inference time, it doesn't impact it?

19:27That's a great question. Just because we are injecting the visual tokens through cross-attention modules and we're not concatenating them at the beginning. So if you think of like, if M is your number of vision tokens and N is number of text tokens, through standard design, your compute complexity through the self-attention modules would be m plus n whole squared. But now, because text tokens are being self-attended to and vision tokens are being cross-attended to, our compute becomes much less. Text, if it's n, n squared plus mn. So it's much less than m plus n whole squared. So we observed that compared with a standard approach, the training is actually faster this way.

20:19Yes, actually, there is like an additional cost to compute these masks offline. But that's just done once. That's just inference. And that N-squared relationship in terms of complexity is one of the challenges in achieving large context for language models, VLMs. Does reducing that complexity allow you to do things in terms of longer context as well? Is that something that you've explored? It does. We have explored. It's not part of the paper, but it's a continuing work. So this allows us to reason from longer videos, especially for retrieval purposes. Yeah, and there are other techniques as well, but standard way of concatenating video tokens with text tokens for longer videos.

21:23We do have challenge in terms of compute and square complexity, as you mentioned. And via cross-attention, we find this is a good solution. You mentioned that the benchmarks are lacking in their vision centricity. Can you talk a little bit about the benchmarks, the key benchmarks for this problem that you use? And did you explore any of the newer vision centric benchmarks? So benchmarks like ScienceQA, which has an image, high school science related questions, AI2D, they generally, for most of the questions, even if you don't look at the images, you can respond to them. So more lately, people have come up with new benchmarks like CVBench, which makes sure that the question cannot be answered without looking at the image.

22:29For example, which object is closer, person or the table, right? So for that kind of information, just relying on the language model's parametric memory, you can't respond to that. You really have to look at the image and then respond to the question. So these kinds of questions where you don't need to articulate the answer in terms of language, but you have to, your answer might be short and crisp, but you have to look at the visual data.

23:04Those are the benchmarks that really, or spatial correspondences, Like we are given paired images and we say on one image, we have four points and on another image, ABCD, and on another image, we have one point and we say which one of the four points this point in image B corresponds to. So for that, you have to really look at the visual data. So these are the benchmarks, basically, that we need to pay attention to to make sure that vision is not being ignored. We are really solving the vision-centric problems. earlier in introducing this challenge you talked about object hallucination in vlms do do you think of object hallucination as like a general failure mode that contributes to like all of the problems that vlms have or is it you know a particular failure mode and And there are particular benchmarks that you looked at to look at hallucination performance.

24:19So, of course, we looked at a particular hallucination bench, which is a benchmark. But I guess it's a problem in the sense that if you ask a question, for example, how many cats are there in the image? If the cat is sitting in front of a mirror, right? so it would hallucinate and respond to. Or if there is a scenario which is very common and you ask a counterintuitive question, so we know elephants are of a certain color, but if you provide a counterfactual example like an elephant painted pink and you ask what's the color of the elephant, so it would hallucinate and say... So those kinds of scenarios where I guess we can see and study if the model is hallucinating or is it really paying attention to the vision information.

25:23yeah okay yeah that the way you answered that kind of gets at i think a thought that was underlying my question and that is like when you know there's a cat looking in a mirror and the model says two um is that you know is that hallucination or is that some other kind of error, like, or when the elephant, you know, when it says the elephant is gray, but the image says, you know, it's a pink elephant in the image, you know, I guess it's, you know, maybe I'm probing at like the definition of hallucination in a VLM context, but, and you kind of suggested that, you know, maybe one way to define hallucination is if the, if it's not attending to the visual tokens and it's attending to the textual tokens only, but it suggests to me that there's maybe a more granular way to talk about error modes in vlms where you know hallucination is not necessarily the same as relying on you know textual uh you know what it what it thinks is knows from text parameters and maybe there are other ways to like kind of chop up these errors that, you know, might be insightful.

26:50That's right. So think of scenarios like which object is closer to the camera, who is on the left of a person, or from third person's perspective, like if there is an image, there is a car, there is a motorcycle, and then there's a person. And we ask the question, if you are a motorcycle and you're looking to your left, what do you see? So those are, I guess, there are different granular levels to which we can break down to the problem. And maybe hallucination is not the right word to kind of club them all together. We are borrowing that term from language community. But we are kind of identifying challenges related to them and then seeing if current VLMs can tackle that.

27:45Counting is another one. So counting is an integrative reasoning problem. And we see that current foundation models struggle at that. And also iterative reasoning or multi-step reasoning. So, for example, if you took an image of the price board of a gas station and it says Supreme is$3.40 per gallon and you say, I want to fill how much of fuel I'm going to get for$50. So that's more of an iterative multi-step reasoning problem. Like it first needs to OCR, read at the visual data, then extract that information, go back, reflect on your question, and then do this compute and come up with a response.

28:37So currently, we just do inference in a single step in an end-to-end manner. We are given an image and a question, and we just start spitting out tokens one at a time. So you're right, like granular definition of these different problems, they might not all club under hallucination as such to say, but yeah. Yeah. And as you were, as you're responding, I'm also kind of thinking about it from the text perspective and like, it's probably true that like there, like there's a whole set of like reasoning failures and other kinds of things that we kind of lump under hallucination in the sense that, you know, somewhere under, you know, somewhere down there, the model is, you know, making up something that isn't grounded in the reality of it.

29:32the prompt or the input. And that's like the foundational cause of the problem. And so that's, you know, from that perspective, the term makes sense here as well. Yeah, that's true. That's right. So basically, we try to identify all those problems. And then a common approach has been people identify a limitation, go back, collect data around that, include that data as part of visual chain of thought reasoning. and that helps us tackle that problem. But yeah, you're not basically grounded in the visual input or it's just outside the reality. Awesome, awesome. Let's switch gears to another paper that you have at NeurIPS on generalized contrastive learning, better search across text and images.

30:26Talk a little bit about the setting and the motivation for this paper? Yeah, sure. So as you know, Qualcomm has a strong interest in mobiles. And one of the key use cases in there is we have a gallery of images and we want to retrieve. So generally, we do retrieval based upon text query. But there are scenarios where a query can be composed or it can be fused. From that, I mean it can be composed of both text and visual information. So show me images, all the images of a person looking like this, for example. So our query is both image and text. So what we see is for scenarios where either query or key is composed, it's not single, either text or image, we see that the current vision language contrastively pre-trained model, like clip-like models, they struggle.

31:34And there's a clear reason for that. Why do they struggle? So this paper basically tries to tackle that problem. So it tries to tackle composed multimodal retrieval. And you said there's a clear reason why they struggle. Why do they struggle? If you see the way they are trained, So we have a bunch of images within a batch, and then we have corresponding text descriptions. So images go through the image encoder, text descriptions are encoded via the text encoder, and then we have a shared embedding space where both image and text speak to each other. And in this space, we compute similarity of image and text.

32:16And within a batch, then we make sure that image-text pairs corresponding to the same should be closer together. And if they're different, they should be separate apart. That's how we train these models. So there's no component of training where we're corresponding text and images in the same space. Exactly. So the general way people try to tackle this is they collect training data of triplets where if you think of QD and QD, QD can be image text and QD can be image or it can be image and text plus image. QD can be image and QD can be text plus image. So we can think of different possible combinations of that.

33:00So a general way people have been tackling this problem is collecting this triplet data of different possible combinations and then training the model with that newly collected data. but this has its limitations like it's laborious it takes time and it takes lots of effort manual effort so what we thought what we came up with this in this paper is let's not collect any data let's see if we can do something in the loss formulation so the loss is basically applied so on one axis we have all the images in a batch let's say n images and on the other axis we have n-text descriptions. Then we compute similarity between these n-text and n-images, and we apply the laws on this n-by-n matrix.

33:49So now instead of this n-by-n matrix, we try to change the possible combinations. So we say we have image embedding, we have text embedding, and we can get a fused image text embedding by just doing simple fusion. so now we have three embeddings image text and image plus text and we can come up with multiple combinations like we have three and we want to select two so we can come up with multiple combinations of those and then we can compute similarity between those multiple combinations and reformulate our laws based upon those similarities that come from combinations of two from three different options, image, text, image plus text.

34:33And this helps us generalize to scenarios where our query and key can be any modality, any single or fused. And so in the same system, you mentioned this triplet and you've got different permutations of this triplet. Are you training only on a particular permutation or are you training on all of the permutations of this triplet? We are training on all possible combinations of the triplets because for us, it's very easy to get those. Like, we don't have to collect any data, right? And we gain in terms of generalization to unseen scenarios. And one of the things we observed as kind of an evolving characteristics of this loss formulation was that it even evolved and generalized to videos.