Demystifying AI Image Generation: Exploring Diffusion Models

0:00Today on the podcast, we are talking about two major updates and announcements on Google tools. and both of these are related to photos and images, one with Google Lens and another that works more with their shopping graph data set. So on the podcast today, we're going to break down what those are and why these are impactful for the industry as a whole. So the first one I want to talk about is Google's new, it's kind of like this new AI clothing try-on tool. And essentially what this does is it allows you to take a picture of yourself and a picture of, you know, a shirt or a piece of clothing and then it is able to show you what you would look like while wearing that clothing.

0:39So this entire process is using something called diffusion and I wanted to break down what exactly that is and how it works. I think this is a pretty prominent technology used in these kind of image generators and I think it's pretty important to understand kind of how this works. The first thing I want to bring up is that the challenge they kind of ran into while they were doing this is that, you know, currently the techniques that people have been using to do these, they're called virtual try-on technology. And the current techniques, essentially, they do something called geometric warping, where essentially they can cut and paste and then they kind of deform clothes and the image of the clothes to kind of fit a silhouette or your person.

1:20But the problem is this, is that the final image is never really spot on. The clothes don't actually realistically adapt to your body. They have visual defects like misplaced folds and this overall just makes the clothes look kind of like mishappen and really not very natural. So when Google kind of set out to go and build this new virtual try-on feature and this new technology, they were really focused on making sure that they were generating every pixel of the clothing from scratch so that they're actually producing a really high quality realistic image similar to something you know the same strategy that someone like Midjourney is currently using.

1:55So they found a way to do it with Diffusion and they created a Diffusion-based AI model to do this. But essentially to understand how this works, you first have to kind of understand Diffusion. So Diffusion essentially is a process of slowly adding pixels or what we call noise to an image until they become like unrecognizable. So think of like kind of like a TV screen when it's just white fuzz and it's just, you know, fuzzing or whatever. Imagine that, but with more colorful pixels. And then once you get it to that point, then essentially you're slowly removing the noise or these kind of fuzzy pixels completely until the original image is slowly reconstructed into perfect quality.

2:40It kind of sounds fancy, but essentially just a lot of colorful pixels are on one screen and they're slowly pulling out the pixels that don't match the image that is behind until they've reconstructed the image in perfect quality. So right now, text-to-image models like Imogen use Diffusion and text from a large language model to generate these really realistic images. And they do this exclusively based on kind of text-to-image generators like MidJourney or something like that. So Google actually said that they were inspired by Imogen and they decided to kind of tackle this entire project with his virtual try-on technology using diffusion inspired by Imogen.

3:21So they did this with a twist though. So instead of using text, like a text input, like you would with Midjourney, they decided to use a pair of images. So the input would be the clothing, you know, the t-shirt or whatever. And then the other one would be the person. And each image is sent to its own neural network, which is a UNEP. And they are able to share that information with each other in a process called cross attention. And so essentially, they're generating an output, which is a photorealistic image of you wearing this piece of clothing. And this kind of combination of image based diffusion, and also cross attention is what they've used to make up this new AI model, which is really cool.

4:03So as far as training goes to make this, you know, this new feature possible, and also as realistic as possible. They put their new AI model through a lot of rigorous training. But rather than just training it with something like an LLM, like Imogen does, they actually just tap straight into Google's shopping graph. So that's essentially, it's one of the world's most, like one of the world's largest data sets of products, you know, sellers, brands, inventory, all of this kind of stuff. And so they have this really massive inventory. And this is something been saying for a while, is just the fact that Google really does have so much infrastructural and so much data, so many data advantages in this AI space, because they are able to leverage so many different data sets that, you know, the sellers of these clothings, for example, have given them permission to use their data and their content, you know, if it's being posted on Google Shopping.

4:57So they're able to leverage it to make tools like this that are a lot trickier for other people to do without such a comp, uh, you know, comprehensive data set. So essentially they trained their model using a lot of different pairs of images. Um, and each of those would include a person wearing a shirt. Um, and then in, they would do it in two different poses, right? So they get the model, the model will take a picture with the shirt in two different poses. Um, and then they would get the AI to learn to match the shape of the shirt, um, in the sideways pose with the person in the forward pose and vice versa so they're essentially training it to understand all of the angles and what that shirt looked like from all of the different angles by giving it you know to two images of the same person with the same shirt but from different angles they do that until essentially it could generate really realistic images of that shirt on the person from all angles and then in order to kind of take this to the next level they repeated the process by using millions of random image pairs of different clothing and people and by doing this essentially they've created a model that allows you to see you know what a t-shirt would look like on a you know a number of different you know models so starting today i believe you can go and use this virtual try on tool for women's shirts and you can use a bunch of different brands on google shopping graph so there's things like anthropology loft h &m everlane over time they said that they're going to get even more precise and expand to a lot more many more brands but you can go check this out and if you're interested google recently wrote a blog post explaining all of this with some graphics called how ai makes virtual try-on more realistic so a really incredible use case in ai from google a really powerful you know application where you can actually see a lot of customers getting some serious benefits out of this so i think this is a really cool a really cool move by google in furthering ai the second story I wanted to talk about today is again one that I think is a really big win for Google and for their image generator that I think has the potential to do a lot of good.

7:04Essentially what it is, their new feature is on Google Lens that essentially can now search for skin conditions. So Google expanded the capabilities of Google Lens, its computer vision powered application. And essentially if you don't know what Google Lens is, it's a product that you can take a picture of something and it will tell you what it is. So like if you saw someone, you know, walking around outside and you thought, you know, you liked their sweater, you could take a picture of it and Google lens would be able to pull you up, uh, you know, where you could go and buy that sweater. So obviously this is a good tool for Google because it can help with their Google shopping experience.

7:38You know, it will help them to generate revenue. Um, but it also can do a lot of really interesting different use cases. You can use it, take a picture of a plant. It can tell you what the plant is a lot of different things but now you can actually use it to identify skin conditions and search what they are so you'd be able to take a picture of a rash or a mole or a lot of different skin conditions and I think this is a really cool tool to really help you know people understand if they have some sort of you know cancerous or dangerous skin condition of course this isn't a replacement for a doctor or something like that but I think this is a really good idea.

8:15If you took a picture of something with this, it gave you a warning, this would be a really good sign to go and see a doctor and get that looked at. And I think this is going to be quite a powerful tool. So essentially, users can now include images in their interactions with Bard, and Lens is going to help Bard in understanding what the visual content is. So, you know, for example, if you uploaded an image of your skin to Bard and said, hey, you know, what is this that I have, it would be able to look at that, use Google Lens to identify it, you know, tell you maybe a little bit about it, and then hopefully provide you with an app create feature.

8:51Now, again, this is, I think, a really powerful ability that Google has is the fact that they have this really wide variety of tools and software, and by integrating all of them, they're able to make some really powerful products, right? So Google Lens has been around for quite a while now. They're adding a lot these really impressive AI features, but obviously it's been using AI to do image, you know, identification for quite a while. And now being able to pair something like that with Google BART makes Google BART an incredibly powerful tool. And so it's going to be interesting to see how, you know, companies like OpenAI and ChatGPT try to respond as Google starts inevitably implementing more and more and integrating more and more of their softwares into their core AI tools and products.

9:33And it's going to be interesting to see who kind of the major winners are in these spaces and how this continues to benefit society.