Why Image Generation Needs More Than Bigger Models with Fatih Porikli - #773

0:00Sam Charrington:Thanks so much to our friends at Qualcomm for their continued support and sponsorship of today's episode. Qualcomm AI Research is dedicated to advancing AI to make its core capabilities, perception, reasoning, and action ubiquitous across devices. Their work makes it possible for billions of users around the world to have AI-enhanced experiences on devices powered by Qualcomm technologies. To learn more about what Qualcomm is up to on the research front, visit twimlai.com slash Qualcomm.

0:54Sam Charrington:resolution or local generation, and quality, speed, and memory quickly become constraints. The next frontier in image generation is closing this gap between plausible images and precise results. It's making these systems controllable, efficient, and reliable enough to consistently produce a high-quality rendition of the image you actually ask for. One researcher at the forefront of this work is Fatih Parikhli, Vice President of Technology at Qualcomm, whose team presented more than 20 papers at this year's CVPR, the Computer Vision and Pattern Recognition Conference. Here's Fatih explaining why image generation still has plenty of hard problems to solve.

1:28Maybe we are also asking a single model to solve too many difficult problems at once. Think about what happens when you generate a scene with several people. Now, the model has to understand the prompt, but I'm asking the model, and then decide how many people should appear, determine where they should be placed, like the composition of the scene, reason about their interactions, because if there's a person, if there's another person, most likely there is some connection. Preserve the identity of the person. We can give, okay, this is my daughter, this is my son, and I want them to be in the picture, not like any random person.

2:07And finally render everything together in all a single process.

2:11Sam Charrington:So maybe this is too much. I'm Sam Charrington, and this is the TwiML AI Podcast. For over a decade, I've been exploring the ideas and innovations shaping the future of AI through conversations like this one that help you understand what's real, what's next, and what matters. Let's jump in.

2:37Sam Charrington:Yeah, that's a big difference that I see between, I think, even this conversation and the conversation we had last year, the performance and capability of text-to-image models has improved significantly. And not just performance and capability, but I think accessibility. Like now, ask whatever your favorite LLM is to generate an image, and it will do a really, really good job. And it does kind of beg this question of the computer vision community, like what's left to do? If this problem is solved to this degree, what's left for us to work on? That's a fair question. T2i models, these text-to-image generation models, or image-to-image generation models have become incredibly good at producing very realistic images.

3:30The lighting looks natural right, the details look right, and the overall quality can be amazing. As many things we do, there's initial excitement, There is great work coming up. But still, if you look into that one dive deeper, you realize there are many things to be accomplished. One was, you know, like the controllability. Generating multiple people in the same image, people look almost identical, faces kind of blend together. And also, we showed in the past, we can run such models on users' devices. You do not need to rely on a cloud service provider. I think most of the models still are limited to 1K, 1K resolution, but then you want to go beyond that.

4:17That is a challenge, which has not been actually addressed before.

4:21Sam Charrington:Okay, so what I'm hearing in there is that we've made a lot of progress, but there's still work to be done. And when you think about that work, some of the big buckets include controllability, the ability to really get the models to focus on the way you describe the task or focus on the output that you want. And then you mentioned in there quality, so fewer artifacts, sharper images. And then you mentioned efficiency. We need to keep up with our ability to run the latest and greatest models on the device. So those sound like three chunky buckets for researchers to continue to work in. Yeah, that's a very good depiction, Sam.

5:07Thank you. You know, kind of you said it very well. It's not only Qualcomm, but the community also trying to make sure the models, such models, genetic AI models, can also match the guidance or the quality or controllability expectations or efficiency expectation goes off kind of like a real, like a camera. But you are taking pictures using your birds, your prompts. So there are still many things to be done. And that's why, you know, we publish, present many papers addressing such challenges as CVPR this year.

5:49Sam Charrington:The first one we're going to talk to actually talks about this example you gave, the diversity of facial attributes. What's this paper called? The name of the paper is DISCO, Resolving the Identity Crisis in Text-to-Image E2I Generation. I'll mention to folks listening that a couple of things. One, we'll have links to all these papers in the show notes, but also we will, I'll try to get some of the illustrations from the papers into the videos. If you're not watching on YouTube or watching video, look for that because all of these papers have really good illustrative examples that will help with following the conversation.

6:34Sam Charrington:And so DISCO and Artican, when you think about them relative to these buckets that we've talked about, which of these buckets are they really going after? They are about controllability and how to train a T2i model better aligned with the guidance from the user. Let me go a little bit deeper about DISCO, if it's okay, San. When we look at, you know, kind of T2i, models and our own models also. What was interesting that this is not an image quality problem, the problem that I mentioned before. Like you are asking the model to generate faces and certain number of faces and it keeps generating same faces, almost identical faces over and over again.

7:26So quality-wise, image quality-wise, if you look at the pixels and noise and everything, it looks realistic. But the missing piece was that those models, space models, amazing models, had not really learned to create truly distinct identities. Because existing training objectives focus heavily on the realism and matching the user prompt, but they don't explicitly encourage diversity between people. And that is very important. So that observation led us to a simple question. What if the identity or facial appearance or any diversity, you know, kind of itself becomes an optimization objective? So that led into the idea behind Disco.

8:16instead of creating a completely new T2I model. We kept the underlying model and fine-tuned it with reinforcement learning. We designed rewards that encourage several things simultaneously. For instance, different people within an image should have distinct identities. You don't want to kind of duplicate, create duplicate faces. That's something we call in the paper as intra-image diversity. And across different runs of the same model with similar prompts, same prompt, we shouldn't generate, we shouldn't keep producing, generating the same faces. So this is inter-run, inter-image diversity. So these are explicit new objectives when we train, fine-tune the model.

9:06And also, you know, we want model to generate the correct number of people. If I'm asking generate two people, that should be two, not three. VRX also, you know, kind of incorporating that in the objective. And of course, we still have the previous, you know, image quality objective there. So putting all together, to put everything all together, we use reinforcement learning, something called as group relative policy optimization, GRPO.

9:39Sam Charrington:Taking a step back, it strikes me that, you know, yes, this as an objective diversity of faces is an important one if you're generating images with multiple faces. But it strikes me as, you know, one of many, you know, possible ways that you might want to or characteristics of an image that you might want to influence. And it seems like tuning the objective for all of the possible ways that you might want images to kind of generate correctly seems, you know, not just difficult, but like anti-bitter lesson. Like, is it, you know, why not just collect more data with lots of faces and train the model with better data?

10:34You know, that is what we are doing. But we are sure that you do not need a lot of data. And you are right. You know, kind of there might be many attributes, diversity, facial, you know, kind of diversity is one objective. A number of people is another one. But then let's say we want to generate a certain action of a person or pose of the face or location of the person in the image or their body pose and everything. So for those, but we are saying that, well, you can incorporate such objectives in addition to overall, let's say, perception or image quality objective when we are training such models.

11:18But you do not need a lot of data if the novelty of the paper in this work has to incorporate everything into a reinforcement learning framework that would make it possible. And training, fine-tuning such models would be also kind of very affordable. By the way, there is a difference between the quality of the input images going into such models. There is another thing called this curriculum learning, starting from clear scenes and gradually increasing complexity makes learning, this reinforcement learning, much more stable.

12:01Sam Charrington:I remember conversations I've had many years ago about curriculum learning, and it was always a theoretical improvement. It's exciting to see it being incorporated into practical algorithms, training algorithms nowadays. Yes, you are absolutely right. Curriculum now is making impact. not only, by the way, our papers, but there were several other papers I see talking about how wonderfully curriculum learning makes, let's say, multimodal models better. One broader lesson from the DISCO paper is sometimes the model simply needs the right objective in training, like the things that I mentioned. And also, it is not mainly an architecture limitation, but it is how you are providing these objective and training data to the algorithm.

12:59The way I would summarize your answer is that, yes, there are lots of different attributes that you might want to exert some control over.

13:11Sam Charrington:And, you know, there is a kind of a, you know, a mental or human cause to going after each of these attributes. But from a perspective of the training process itself, it's fairly efficient and nothing like when you think of like traditional fine tuning. It's, you know, very data efficient and you can apply techniques like curriculum to make it compute efficient as well. You know, Sam, something I like to also add because it is related to what we are talking about now. So optimization, of course, is right objective. Optimization at the right data is definitely very critical. That is what Disco paper is talking about.

13:56But we may also kind of think that maybe, I mean, these are models, genetic BI models. Maybe we are also asking a single model to solve too many difficult problems at once.

14:10Sam Charrington:That's kind of asking the question, could we possibly optimize for all the attributes that we care about? So in doing so, would we be asking the models to do too many things at once? That's a good point. Maybe we shouldn't. And also, I can give you an example. Think about what happens when you generate a scene with several people, going back to that core example. Now, the model has to understand the prompt, but I'm asking the model, and then decide how many people should appear, determine where they should be placed, like the composition of the scene, reason about their interactions, because if there's a person, if there's another person, most likely there is some connection, preserve.

14:54And we also, I didn't mention about this, the identity of the person. We can give, okay, this is my daughter, this is my son, and I want them to be in the picture, not like any random person. And finally render everything together in all a single process. So maybe this is too much, And that's what we explored in the other paper, R2CAN paper. Instead of asking one model to do everything, what if we separated, for instance, planning from rendering, similar to what human artists might do? And that kind of led us into this paper.

15:33Sam Charrington:Before we dive into Artican, thinking about that comment applied to Disco and again, this idea about different attributes, possibly a thing to keep in mind is how we've matured the way we think about like using model routers. so you know maybe you have you know a suite of models for the different attributes that you care about and then when your prompt comes in you've got some kind of router that says oh this you know the image that's required here you know will probably have a lot of people let me route it to this model that's been tuned for that as opposed to trying to again you know have a single model that's optimized for all the attributes you want.

16:25Sam Charrington:It's a different approach to what you've taken with Arcane, and we'll dig into that. But it is a way that, you know, that Disco could be put into production practically. Okay. You brought up again another, you know, amazing perspective, and we are working on it. This sounds like more like an agentic orchestrated image generation framework, part-time, right? Depending on the input prompt, maybe we want to generate, for instance, a realistic scene or like a cartoony scene or something, maybe there's a text or some human-generated graph or something like that in the image. So for all of it, different attributes like diversity, like facial identity, we may have specialized models, specialized processes within those models.

17:23So how to pull the right one is we are working on such agentic also pipelines. Depending on the input, it goes, it's, or determines the right tool. This is like, if we consider this call, one instance of, let's say, diversity is one instance. The other one is better text generation. So it can go find the right one depending on where we want to apply them and then orchestrate the final generation. I think this is going to be the ultimate solution. One, maybe size doesn't fit everyone. If you really, really want to generate something amazing, top of the line, so we need such specialization. Yeah, that's a very good perspective, Sam.

18:15Sam Charrington:So I will, again, the images with regard to Disco are really interesting. Like I never really thought of, I don't know that maybe I've just never asked for an image with multiple people, but there are some really good images where you have a prompt that, or several prompts where you're asking a variety of different models, GPT, Nano Banana, Flux, Hydream, a bunch of them to generate multiple people. And you're absolutely right. You look at these images and it's like the same person cookie cuttered across the entire group. It's kind of surprising that the models do that. When you talk about or when you think about evals beyond this idea of like running across the different models, did anything in particular jump out at you in terms of, you know, evaluation for this model?

19:12Oh, yeah. For instance, for this group, we look into existing benchmarks and then we pull together a new benchmark to specifically help people facilitate further research here and also, you know, kind of provide some standardization. We call it as diverse humans. It's also kind of available in the paper, the link for that data set, that benchmark. And, you know, we invite everyone to take a look at it. So we look at that, we created this benchmark and we evaluated this code on that benchmark and also on the other benchmarks data sets. We see that when you explicitly impose such objective, diversity objective, the score, for instance, unique face accuracy detection score, significantly improves.

20:10This score is around 98, 99. But, you know, kind of like the models that, these models where we started, that doesn't have such explicit diversity objective, they are very low. There is maybe more than 10, 20 percentage gap. So kind of this solution that we also talk about in the paper sets the new sort of for diversity in text image generation. In addition to like human preference score, which is very important, evaluating the quality of the output is also superior to the kind of like the base model.

20:53Sam Charrington:And it's interesting how you built the reward function. And I'm imagining that you would do a similar thing from an eval perspective. So one of the components of the reward is what you call intra-image diversity. And that is, you know, looking at one image, are the faces the same or are they different? and essentially you take your image, you apply a face detector that kind of identifies where the faces are with bounding boxes and then you kind of extract those faces and then you embed them and then you can just do similarity across pairwise similarity across the faces and kind of get the distance from face two and one face to the next to determine if they're, you know, all the same faces or essentially to get a measure of the diversity of the faces.

21:45Exactly. Yeah, we talk about that in the paper kind of even there is a nice flow diagram. I mean, I invite everyone to take a look at it. As you said, there is a face detector applied and this is off the shop, you know, kind of. And then we take these detected architecture faces, we embed them into some, you know, kind of space, canonical space to be able to compare them. Then we compute such pairwise similarities and then it gives us a score for that image. What would be the diversity score? Of course, that is one score. And we can do the same thing. Let's say I generated one image and I have another image with the same prompt.

22:34I have another image with the same prompt, but the initializations are different seed points. So kind of the prompt is same. We run it and then we end up with a different image.

22:45Sam Charrington:So you were starting to talk about the next paper, Artican, and essentially how this model kind of breaks down the problem of text-to-image generation. Talk us through that. So what motivated about that one, Disco is great. We, as a pivotal example, we, of course, talk about this identity, diversity. but then in the other paper we are making the point that maybe it's too much for a model to try to do everything going back to you know agentic flow again maybe it's easier to approach some of these genetic challenges like a human being we do not like try to solve everything ourselves. Well, not just ourselves, but even in this domain, art, like you, an artist approaching this problem wouldn't necessarily think about it pixel by pixel.

23:50Sam Charrington:They think about what's the subject, what's the background and... There's a planning, right? There's a planning aspect to it, yeah. In this paper, we build on it, build on that idea instead of asking model to do everything we separate planning from rendering similar to how a human artist would work. So we have two components. There are architect, which is, it doesn't generate pixels, but instead it creates the structure or composition for the scene. Then kind of decides where, for instance, if there is a person is going to be in this image where it should appear. And if there are multiple people, how they should be arranged.

24:33You know, it should look natural and, you know, realistic, then artist starts with that composition structure and generates the final photorealistic image while, of course, one objective also preserve identities if you provide identity. So, yeah, this is like planning an architect than an artist type of framework.

24:58Sam Charrington:The underlying technical approach is also using RL just like Disco. It's also a GRPO-based approach. You are right. We also have this reward function leveraging on different objectives that we want to optimize. In this school, we had intra-image and intra-group, like inter-image at human perception score and count accuracy. You see, here in R2CAN, we also have, well, we have the composition. Can we put the right face into right place in the composition type of objective, which is a part of the reward function for GRPO, and also pose of the face? Because now we are composing. I mean, we don't want people to look this way, the other people look the other way.

25:51If, let's say, we are taking a group picture, we could, you know, kind of, we could say that. So kind of those type of things are all combined into the reward function goes into reinforcement learning. And then we kind of like optimize that one. The model now learns how to pull all of it, optimize all of it at the same time.

26:15Sam Charrington:So to make sure I understand this, you have a prompt. the one in the example I'm looking at is three best friends riding unicorns on Mars. You pass that to what you call the architect. The architect says, okay, I want three faces, and they're going to be here, here, and here in this image, and kind of plans out this canvas. And then that essentially becomes a kind of grounding for your reinforcement learning loop. And so the images are all actually generated, but they're kind of optimized to ground to this canvas representation. Is that the right way to think about it? Right. So that model, that architect knows where to position such faces and artists rendering the base model where we started, now fine-tuned to actually follow the guidance based on the locations coming from the centroid of those face areas.

27:25But there are two things to keep in mind. There is a training phase offline. So this model, the architect and the artists are optimized to do better. But in inference time, you know, first we take the prompt and if you want, for instance, a spatial person identity to be in that image composition, we provide those images and architect generates the locations of the faces. And that is, there's no learning there. It just, you know, builds on what it learned before. and then the artist, the fine-tuned model, renders it. So this GRPO is on the fine-tuning offline ferry.