AI Trends 2024: Computer Vision with Naila Murray - #665

23:50Of course, you can think at the boundaries, things might come in and out. If you have dynamic elements, they might come in and out. But generally, if you have a scene, you're going to have a lot of the same elements. And then the process of creating a video out of it is a process of sort of tracking the individual elements. And so that gives you like a very interesting way to go from a static scene to like a dynamic scene or to go from one frame to a sequence of frames as well. Is the approach used primarily for generation or is it primarily for analysis? When I think about the description of kind of superimposing Gaussians in a space, the most obvious thing for me is like representing the structure of the space as opposed to kind of a rich visual generating a new image or a video or something like that.

24:33But it sounds like you're saying that it can be used for the latter. It can be used for that. And it's astonishing. So the interesting thing is that in the original paper, I believe they used in general anywhere from 1 million Gaussians to 5 million Gaussians. So the combination of the Gaussians with the kind of alpha matting on top of the Gaussians allows you to have a really exquisitely detailed image. You know, you generate these reconstructions that if you think of what image reconstruction looked like 10 years ago versus this, it's indistinguishable for me anyway, from someone having taken a picture of it with an ice camera.

25:07Is the effect ultimately, it's kind of a lower dimensionality representation of this very intricate thing. And so it allows us to, it's kind of an efficiency play in a sense. I would say, I think that's one of the main advantages of it, because you can kind of think of it as saying like you're representing a 3D scene with indeed like one to five million Gaussians with some kind of weighting and a bit of information, metadata associated with each Gaussians. You can see I have these millions of Gaussians, each of them I have associated like an opacity, an extent, a location, that sort of thing. If you compare that with a typical Nrf, which is like a neural network, right?

25:42So it has many millions of parameters, it's generally more efficient. And so that's like what I find most exciting, although I'm sure there are other advantages as well. Awesome. I'll be on the lookout for that. Maybe that's why I saw NERFs everywhere, but I think not. I think it was totally organic. Yeah. Search for Splat Tank, your 3D Gaussian Splat Tank. You'll find a, I think it'll be unavoidable anyway next year. You won't have to look for it. Awesome. So the next category you identified is called Vision plus LLMs, but we've already been talking quite a bit about Vision plus LLMs. What exactly are you trying to get out here?

26:16Yeah, so it's true. I was actually thinking about this and I was sort of like, well, that's kind of the whole game right now. So what does it mean to separate it? One thing I thought was interesting is, I think I was thinking of this in terms of maybe going towards multimodal. So kind of going a bit beyond what we call language, large language models and saying there's sort of just large multimodal models where you're thinking of vision and language almost as first citizens to a large extent. So there are some works that are very interested in, as opposed to kind of like tacking a visual encoder onto a language model, mixing tokens from the get go.

26:48So you're sort of like interspersing language tokens with visual tokens, for example. So I think that's something that I expect to see some more of, at least as exploration, because I think there's this fundamental question of what's the way to go for multimodality? Should we think of multimodality as a combination of separate submodules that are trained relatively separately? And then you sort of like combine them kind of in what you might call late fusion or postdoc versus what if we try to make everything into some, like tokenize everything basically. So you take audio, you take video, you take text and you create tokens out of everything.

27:20And then you sort of input them into a model in a kind of agnostic way. So I think that's going to be continuing to be interesting going forward. I also thought there was this interesting work this year. Do you have any bets placed on one or the other of those approaches? I think there's two ways it can go. And I don't have a good intuition on which one. So I think that there's a hypothesis that you sort of just need more tokens. And if the tokens are kind of multimodal tokens that's to some extent irrelevant. You just sort of need more data to be able to scale up more powerful models. And so let's say, for example, that we want to get a lot more tokens to train models and to scale them beyond to larger and larger sizes.

27:57Maybe we should be going multimodal to get that extra information. So that's one approach. I think another approach is that these modularity sort of allows you to be more fine-tuned and more fine-grained about the kind of capabilities you want to embed as well. And I think that's also very compelling. So I think it's a bit early to bet. I'm not going to bet at each one right now. Let's let a million flowers bloom and we'll see where they go. So the specific papers that you identified here, visual instruction tuning and visual chat GPT, talk about those. Sure. Yeah. So visual instruction tuning, what I thought was interesting about this is that it really takes this analogy of training language models and it takes it to kind of like its full extent.

28:37And by that, I mean that a key part of getting language models, really unlocking the potential of language models is being able to fine tune them to follow instructions. Right. So to take them beyond sort of like simple auto completion to having them respond in ways that we find we as humans find useful using instruction fine tuning. So either supervised fine tuning or reinforcement learning from human feedback. And so this work took the similar approach, but for visual instructions, right? So it took this approach of collecting the type of instruction tuning data that you would typically do for language, but doing it for vision instead.

29:14So having these very sort of complex prompts, for example, and then actually not only doing these input prompts, but also having conversational data. So this work took the approach of collecting this type of data and then fine tuning a visual language model on top of that and showing that after that process, it gave, you know, like dramatically better performance on these tasks. Interesting. Some of the tasks on the vision side are kind of inherently more easily evaluated on the language side. There's like, it's right or it's wrong. It used to be. It used to be. But now when we're talking about generation, it becomes very subjective and other tasks.

29:52We're in the same mode now as language. I used to feel like kind of sorry for folks working in language because I thought it's so hard to evaluate. It's so hard to evaluate. you don't have any reasonable ground truth. You have to have these strange blur scores. But now we're exactly in that world. Because I think now that the vision community is trying to tackle those really tough problems, we're also the same space now where it's very hard to evaluate things. And you need to resort, like resort, I should say, but you need to really have human evaluations very often to get any sense of if you're making progress.

30:20And so I expect actually that in the same way that for language community, folks are focusing a lot more on automatic evaluation. So using very powerful language models we evaluate correctness as opposed to having humans in the loop because it's very expensive. I suspect we're going to get there as well for language, for vision, I should say. Interesting. So visual chat GPT, is that what we're used to, just putting pictures, images in chat GPT and asking it to do things with those images? Or are you referring to something else here? To be honest, this is actually very similar to the visual programming works that we discussed earlier.

30:53Okay. In the sense that it once again takes a set of models that are tailor-made for specific visual tasks, right? So things like segmentation or object detection or image retrieval or things like that. It once again, uses sort of like in-context learning to be able to solve specific prompts or specific problems that you're being asked. So, you know, there's this interplay between visual programming and sort of like vision and language models, because very often you're sort of using these language models for the programming itself. So I think you can kind of put them in both in sort of the same bucket.

31:23It's interesting because this one, it also reminds me of some interesting work in the purely LLM space, in the purely NLP space around tool usage. So for example, there was this work called Toolformer earlier this year that made a big splash in terms of using different varieties of tools to solve tasks. This is, and I think to some extent, the Vizproc work, the Vipo GPT work are all in this vein of saying, how can we have these elements and then combine them in ways to solve like unique and like zero shot tasks. Awesome. Very cool stuff. So we covered kind of our top four categories that you found most interesting from a research perspective.

32:00I want to switch gears here to tools or open source projects that you came across that made an impression with you. The first one you have is segment anything. What's that one? Right. So full disclosure, this is also from Meta, but it really is like seminal. It's made like a big splash in the community. So this is a work that went full on in providing a tool and building a tool that allows us to do, as the name implies, to segment anything. It takes a page to some extent out of what you might consider the LLM space in the sense that it's focused on using different types of prompts. But these prompts are not necessarily text.

32:35They can be text, but they could be things like text. They can be things like clicks and they can be bounding boxes. And so it's a tool and it's a research work at the same time, but it's also like it's been open source commercially. And so it's been crazy amount of adoption it's had, right? Because it worked really, really well. But basically, it's a tool at the end that allows you to have this interface where you put in, for example, an image or a video, you can click on it and immediately segment it. And for example, this is being used in a lot of different applications, basically, to do things like removing backgrounds from images very quickly, and not just very quickly, but very precisely as well.

33:10So I think that's very cool. And I think it's going to continue to see a lot of adoption. And it's really changed the way that people think about doing segmentation research, because it's sort of, you can think of it as almost like a foundation model for segmentation. And so you can do a lot of like more complex segmentation problems on top of that. And so segmentation is a very fundamental problem for vision. And so that is going to, you know, that's going to continue to make waves, I think. And was the tool a implementation of new research or meaning is there some kind of fundamental new research and new models that drove the tool or is it just the implementation and the experience that is what's made the splash?

33:48Yeah, very much the former. So there was a research paper that went out of it. I think it won Best Honorable Mention, I believe. I think it was CVPR, it might have been ICCV. So yeah, so there's very cool research around there as well, is how do you actually get the prompts to work correctly? And then how do you actually use these different prompts, basically, in this model? And when you mention prompts, is it a vision language model that's made it work? I imagine there is a vision language model in there. So I know that textual prompts are indeed possible. I think the types of prompts that are used are relatively simple.

34:23So it's not going to be like a very long paragraph or something like that. But you can use text to sort of like pinpoint what you'd like to get segmented as. You can think of it as an open vocabulary segmenter in the sense that you can use specific language that maybe might not have seen before and then be able to kind of segment that object as well. And so was the language element of that the kind of the core innovation from a research perspective? or is there a more fundamental segmentation innovation that's enabled segment to anything? So I believe it's more the interplay between how do you prompt this effectively?

35:00And then also how do you create this data flywheel of continuous improvement in terms of the model? Because what this work did was they took a sort of semi-automatic approach. So basically they had this flywheel of how can we have good enough initial segmentations and use that and then correct that with time. And so, for example, you would start off with a model, a segmentation model that gave you some sort of initial segmentation. You could correct that and then use that correction to improve the model. And then you kind of go iteratively. So there was a lot of work to do in order to use that approach to scale up to a really large, I think it was in order of billions of annotations in a very efficient way.

35:37The next project that you had noted here is ControlNet, which we spent some time talking about. This is another example. Some of these can go in different places, right? So this was fundamental research, but also when you put the two together, so when you have a tool that's well done, you have code that's well executed and released, then what starts off as a research paper can very much become a fundamental tool that people take and run with it. And so I've seen that. You've seen this used for lots of enthusiasts, developers, et cetera. And so I think it's been a fun tool to see people play around with this year.

36:10What's Dinov2? So Dinov2, also from Meta. also fundamental research as well, but also being released with a commercial license. And this one is a visual encoding. So this is a work where the authors have developed and trained, you can think of it as a visual backbone that's been trained with self-supervised learning with the goal of being sort of universally applicable. And by that, I mean, you could take this model and then you can sort of use it as is off the shelf and add on maybe like a module to it that can be applied for or your specific end task. And the goal is to have it be usable for all sorts of sort of like exotic applications where you don't have access to a lot of data and you can't just train, you know, take millions of examples and train a very good model.

36:52What are some examples of the kinds of tasks that it's trying to enable? So it can enable your standard things like object recognition, but also segmentation, object detection, depth estimation. And very interestingly, these are all problems in and of themselves, but they have sub problems, right? Because it's one thing to say, I want to do object recognition on a natural scene where you have cars and you have houses and it's sort of like, these are very common things to see in the world. But let's imagine you have an application that's less common. Let's imagine you want to segment, for example, an ultrasound image where there's just not a lot of public data, for example, or it's hard to get that type of date.

37:29So there's this interesting example from the paper where the authors apply a depth estimation model on a painting, right? And so you don't have ground truth depth for paintings because paintings aren't the real world, right? But you can actually put in a painting of like, for example, an indoor scene and get a depth estimate that seems realistic, right? If that had been a real theme. So this kind of like, let's say adaptation to different subdomains within a specific problem, I thought was very interesting as well. And so this has also been released and it's also seen a lot of adoption as well across the community.

38:01That sounds huge. Would you say that one of the kind of top line things that's happening in Computer vision this year is kind of getting closer to a universal model. Are you overselling that or am I overbuying that? When I think about computer vision historically and compare it to LLMs where we've got this one kind of Swiss Army knife model now, computer vision historically has seemed to be more individual task-specific models in terms of state -of-the-art. But now you're talking about a model that sounds like it has very broad applicability across tasks. Am I mischaracterizing the state of computer vision?

38:41So I would say that we actually have a pretty long history. Like if you look for universal model for vision, you will see examples from like probably 15 years ago, at least, because it's also very interesting. So I would describe many backbones you've heard about, like ResNet, for example, as a foundation model in the sense that people have just built so much on top of it. I see this as a kind of new paradigm that has been ongoing for at least two years, right? where we're taking a self-supervised learning approach, which I think is newer, and then using those as foundation models. Because there was this question for a long time of our supervised models, do we need supervision to be able to train these foundation models?

39:17And I think we can kind of convincingly say that no, we don't, right? Supervised, self-supervised models are state of the arts. And so I think there is interesting work to do to kind of continue to see how much you can scale with that. So can you go to even bigger universal visual embedding models as well? Awesome. Let's maybe switch gears and talk a little bit about kind of what you see coming up for 2024 and beyond. What do you think are the most exciting opportunities in computer vision, you know, in the upcoming years? Maybe in the interest of time, I'll stick to like two. So one thing I think is very interesting is, and I think this is also something I always have in my head when I'm thinking of language modeling, is the balance between memorization and creativity or sort of like knowledge ingestion and knowledge retrieval, I guess you could say, versus creativity.

40:06Because I think there's a fundamental tension between at the same time expecting language models to implicitly encode like a lot of world knowledge, like even like just straight up facts, right? Like who's the president of the United States right now or something like that. It's hard to at the same time expect that kind of factual accuracy and at the same time expect creativity and hallucination. And I think there's an interesting problem there. So I think both for language, but then also for vision, I find it a very interesting research problem to think about how can we control those two things and what is that going to take?

40:37And not only that, but we have these facts that are embedded. How can we adapt that? How can we remove memories? How can we insert them? What are techniques to do that? Should we be focusing only on external memories? So things like vector databases, do we have ways to kind of control that more implicitly? I think that's like an interesting area for research. And it's going to be important, I would say, for all foundation models, be they visual models or language models or multimodal models. Give us an example of how some research that might explore that domain. So there's maybe two avenues. There's a lot of work around, like, for example, retrieval augmented generation.

41:15So the way this looks like for vision is something like, generate for me a picture of the cast of Seinfeld on the moon or something like that. And so you need to retrieve some knowledge, right? So in some sense, you need to have a memory that you're being able to access. So the question is, what sort of memory should this be? How should this be encoded? You can think of it being encoded or like modeled externally as basically like a database of pictures of cast members of movies, for example, or TV shows. And then you can sort of access that and say, I want to retrieve this. And then you retrieve the visual content you need.

41:48And then you proceed to kind of like generate using that knowledge. Or you can expect that it's somewhere in your visual model. It's just somewhere in the weights, right, that it's being encoded. For example, you would have to hope that in all of the images that the model has seen previously, there are enough images of, you know, casts of Seinfeld that it can sort of retrieve that, assuming it has a good alignment between vision and language, right? So that's one way to do it. You have the external one, you have the implicit one. And let's imagine you have the implicit one. I think there's a key question around how can you modify this memory?

42:19And there's been a lot of interesting work. I'm not sure in vision, to be honest, but there's been a lot of interesting work in the NLP domain into really looking into mechanistic interpretations and mechanistic analyses of, for example, memory modules. That involves things like actually doing analysis on the specific activations of the weights in a language model to be able to figure out when the model, for example, is generating the specific token that corresponds to an entity, what was the activation pattern, like which neurons were involved in sort of like putting high probability on this specific token.

42:52It's a kind of emerging field, but I think it's a very interesting field to start to have, really think of these models as glass boxes that you can really look at them and analyze them and have this sense of, you know, what are specific weights doing and specific activations and connections doing. Awesome. What was the next opportunity? So yeah, another one I was thinking of is using simulated data as opposed to relying on, I guess you could say sort of real data. In the case of vision, I guess that would be images, videos, sticking with cameras. How can we go towards actually simulation in some sense?

43:27So there's been a lot of interest ever since we've had GAN starting to do reasonable things. We've had interest in saying, oh, can we use these simulated images to actually augment our data, for example. And there's been a lot of questions about to what extent is that really expanding your data, right? Because you can tell yourself if you've learned a distribution over data, if you generate new samples from that distribution, are you really generating anything new, right? Because you already have the information kind of implicit in the model. But I still think there's interesting work to do as we start to have different approaches to generating content.

44:00So for example, things like Gaussian splatting to represent scenes, right? Because, you know, Gaussian splatting or neural radiance fields, they take like a sparse set of images and then they use it to generate this scene. And then from this new scene, you can sample new views, right? So is there a way that we can use these new views to generate new data that we can then use, if they're diverse enough, that we can then use to augment the amount of data we have kind of artificially? So I think there's a lot of interesting questions there to do about how can we take the different approaches to dramatically expand the amount of data we have for training?

44:30This in recent years, this has become really popular in domains like automated driving to augment the visual data that autonomous vehicles are trained on. It sounds like you're suggesting a broadening of this kind of approach and applying it to lots of different tasks. I do. Yeah, especially because I think the quality is increasing. So there's always been this concern with simulation data that it's not realistic enough. Like it's not diverse. It's not bringing in a lot of diversity. And it's also just the kind of stem to real transfer problem is too big a gap to cross. But I think we're getting to the point where we're starting to have like fewer issues there.

45:08And in terms of autonomous driving, like there's a lot of interesting work recently that's come out also and sort of using indeed, like using different types of generative models to start to come to the point where you can simulate actually using generative models as well for autonomous driving. So there's this startup based in London called Wave that released some notes about their attempts to use generative models to do sort of course and analyses or what-if scenarios, which is very, very, it's really the holy grail for autonomous driving, right? You want to be able to kind of understand how is my car going to perform in this scenario, I absolutely do not want to happen in the real world.

45:43But I need to know how it performs in that scenario too. So now let's kind of shine up the crystal ball and peer in, in terms of your predictions for the field for 2024, you know, what do you see happening? I think we touched on it a little bit. It's kind of hard to talk about 2023 without talking about 2024. So we've focused a lot uncontrollable images mostly, right, for 2023. I think we're definitely going to continue to go towards video. So streams of images and really tackle the problem of modeling motion, right? Because one of the key things of going from images to video is how do you actually have, especially when you have dynamic elements, like people moving or things like that, how do you realistically generate streams of images?

46:28I think also I expect us to include audio in that. So I think it's going to be natural to want to have like a quote unquote video with audio that's synchronized and realistic at the same time. So I would expect to see more work happening there. And we've already seen some really interesting things coming out of places like runway with text to video that have been really interesting. And I'm already hearing people predicting the demise of Hollywood and I wouldn't bet on that folks get excited. I wouldn't quite bet on that, but I would expect to see pretty compelling shorts and pretty compelling approaches.

47:04I think it may not look like a Hollywood video, but it doesn't mean it wouldn't be compelling in its own right, the kind of things that you can come up with here. But I think what's going to be very interesting is really going into that multi-modality. So video plus audio, generating realistic audio, which is in itself an extremely complex problem, not just in terms of speech, but in terms of sound effects, for example, and combining that in a realistic way. So I expect to see all of that come together. I think there's a lot of data here that can be used, maybe not by many people, but if you think about where can folks get millions of videos of realistic audio and video synchronized happening, I definitely think there is space for that.

47:41I expect to see a lot more work here. Another place where I expect to see more work is in embodied models as well. The models that are really taking an egocentric point of view on this. And I think there's just so many places where that's just going to be a very exciting space. So for example, I mean, I work in meta. We are very interested in AR glasses, right? And putting intelligence sort of like close to us so we can interact with. And so I just think there are lots of compelling applications for that and I expect to see that continue. So there's been a lot of recent research in the space of taking robots, giving them a language model and the ability to be prompted with language, right?

48:18Using sort of speech to text models and having them interact with the environment, right? So having them that could be placed on a robot, and then you can interact with the robot by sort of saying, pick me up the can by the table. And then so the model has to be able to understand the text, make sense of the environment, navigate to the object, then grasp it, and then ideally navigate back to you and give it to you, for example. That starts to become much more than vision. That's much more than vision. But vision is a big part of that, right? because we're starting to go into these multimodal models that are taking vision and language, maybe audio eventually directly as a modality in being able to perceive the environment, being able to navigate within the environment, and then of course, act in the environment, which is a bit of a different problem.

49:00But I expect to see a lot more work here as well. And are you predicting more work or are you predicting milestone achievements? You rattled off in a scenario, half a dozen individual tasks that researchers have been working on for decades. right? Do you think it all comes together this year? Yeah, good question. I'm not a betting person, so I'm not sure I would bet money on it. I think there is a lot of scope for that. I think a lot of pieces have come together very quickly, very recently. And so I think it's a real possibility. And then maybe lastly, like the last one I was going to mention is kind of all things reinforcement learning from different kinds of feedback.

49:40I think I've mentioned a few cases of that already. And I expect to see that find more and more applications of folks thinking, how can we thinking of machine learning as in more and more of a kind of alignment problem where you're taking these RLHF approaches or these RLAIF approaches and then applying them to different places. So there's been, you know, a recent work this year that did this for different types of visual tasks. And I expect that. I think that has a lot of promise too. Awesome. Wow. This has been an exciting year for computer vision. It is. What's fun is that so many elements, like the advancements in language, the advancements in robotics, the advancements in different places can kind of come together.

50:20And I think we start to see this flywheel of rapid progress in one area leading to progress in the other. And then, you know, I think I can see this snowballing a little bit. So yeah, I'm excited for 2024. Any parting thoughts as we wrap up? My parting thought is I am very happy that we continue to have a very vibrant research ecosystem. I think I'm looking forward to folks continuing to kind of contribute to open science, because I think that's the way that we all make progress really quickly. Awesome. Well, Naila, thanks so much again for all of the work you put into pulling this together for us and for walking us through your take on 2023 and beyond in computer vision.

50:59It was a pleasure. Thanks so much, Sam. I love the conversation. Thank you. All right, everyone, that's our show for today. To learn more about today's guest or the topics mentioned in this interview, visit twimla.ai.com. Of course, if you like what you hear on the podcast, please subscribe, rate, and review the show on your favorite podcatcher. Thanks so much for listening and catch you next time.

51:30Thank you.