The Future of AI Artistry with Suhail Doshi from Playground AI

0:05Hi, listeners, and welcome to another episode of No Priors. Today, we're talking to Suhail Doshi, the founder of Playground AI, an image generator and editor. They've been open sourcing foundation diffusion models, most recently Playground 2.5. We're so excited to have Suhail on to talk about building this model in conjunction with the Playground community and the future of AI pixel generation. Welcome, Suhail. Thanks for having me. So this is your third company. You started Mixpanel, Mighty. Now you're working on Playground. How did you decide this was the next thing? I think like back in April of 2022, I think that was just a place that time it was like GPT-3, 5 kind of came out and then Dolly 2 came out.

0:47And I was actually working on the second company, Mighty. And at that time, I was trying to figure out how to like do something with AI inside of a browser address bar. But when I saw Dolly 2 came out, it was just this like very big, strange eye-opening moment where I think a lot of people didn't think that we'd be able to do like weird, interesting art things so soon. And so, and I think then soon after that, I think Stable Diffusion came out around June or July of that same year. And I got early access, maybe a couple of weeks access to early to SD14. And I just kind of blew my mind what people could do with that.

1:24And I just thought that it seemed odd that all of this was being done in a Google Colab notebook. Shouldn't there be like a UI that makes it really easy? That sort of thing. From the start, were you just thinking we will open source? We will train our own models from scratch? Did you think about other modalities? Yeah, I mean, there have been a lot of people that thought I should do like something in music. But I just, because music has been like a huge hobby of mine for like six years or so. I like produce music. But I just couldn't wrap my brain around like what useful thing I would end up making for people.

1:59Although now there's like a lot of very interesting, cool, useful things for music. And then it seemed like a lot of people were very focused on language. And I had really enjoyed, I already work with lots of creative tools. Like when I was in high school, I used to make logos or I would make music or whatever. So I was excited that finally I could find something where it was a combination of creativity, tooling. Images have like really amazing, like built in distribution. People want to share those kinds of things. So it ended up just like being this perfect thing that I was excited to work on.

2:31How do you think the like overall landscape for competition is different in language versus images versus music? How did you think about in what ways you guys would want to build advantage and stand out? I think with language, there's like, I don't know. I don't know how many language companies there are. You guys would probably know better than me, but it seems like there's like over 20. And then maybe like five or like eight of them have a billion dollars worth of funding. I also didn't want to work on something if there were already extremely passionate people really working hard at that thing.

3:06People that I like really respected that were working on that thing. And so at the time with images, there was just sort of, I think there was MidJourney, there was OpenAI doing some Dali stuff, and then you saw sort of stable diffusion. But for some of these companies, it didn't seem like there was going to be a longstanding concerted effort to keep making them better. It was sort of unclear who was doing this as a fun demo versus who was doing this as something they would spend and invest tons of their time in. And so once I had kind of figured out to what extent OpenAI was going to invest in it, or to what extent seemed like the folks at Stability AI were like sort of focused on like seven different kinds of things.

3:43And I just thought like, hmm, there's just not enough people that want to do this one thing and do it really, really great. So I think for me, it was just about were there enough capable people that wanted to do this? Can you talk a little bit about the specific direction you decided to take with Playground as well? And I know you thought really deeply about some of the applications or use cases for it. So I was just curious if you could share a bit more about that. Yeah, I think one thing that has sort of been surprising and it hasn't changed too much actually from maybe around June or July of August, 2022, was that like a lot of people think about like text to images, just it's right now it's kind of not, it's not even text to images, like text to art.

4:21Sorry, what's the difference? The difference is that these models have, they haven't quite reached the potential of like what maybe what its utility could be. Right now, for the most part, we take, we formulate a prompt, which is really just a caption of what the image is. And then it diffuses into an image, a set of pixels. But a lot of those pixels are primarily used for art. But what we haven't done is we haven't done anything beyond that. We haven't really done something like editing, for example. Like, why can't we take an image that you already have? And why can't we like sort of insert something into that with like the correct lighting and stuff?

4:57Why can't we like stylize an existing thing? Why is there not a blend of like real and synthetic imagery into a single image that could then be used for a lot more things than just pure art. And so right now it's a lot of just people making art, but not a lot of people. Sometimes that reduces its practicality or its utility. Yeah, that makes sense. I think one of the things that you folks did as well that I thought was really interesting is you built your own models, right? Or you train your own models and a lot of people in this space just take stable diffusion and fine tune it or do other approaches like that.

5:28And you've gotten, you just launched 2.5. The model is performing incredibly well. It creates really beautiful imagery and it's super high quality. And I'm a little bit curious if you could share a bit more about how you went about training your model and hiring a team specifically for that purpose and how you thought about it and approached it. Yeah, it turns out that, you know, I think when, you know, a set of strong engineers, like their first thought is that like you just take a model architecture, you find a lot of data, you get, you fund yourself with enough compute and you just sort of like throw these things into like a mixture of sorts and like out comes out like something like Dolly 2 or Dolly 3.

6:04It turns out that it's just way more complex than that and way more complex than I even imagined. I had a sense that it was more complicated than that, but then it's still further, it's more complicated than even that. So I think there were a couple of things that we did. One of the things that we were really focused on with that model was that we wanted to see how far we could push the architecture of something that already existed. This was mostly like a test. It was a test to see whether how far we could get as a research team before like the next model change. And so we We wanted to take something that we knew was a recipe that worked already, which was StableDiffusionXL's architecture, which is like a UNet, right?

6:41And Clip and the same VAE that Robin Rombach trained, all this stuff. And then we sort of said, okay, what if we try to get something that's just at least better than SDXL, better than the open source model? We weren't really sure by how much. And so our only goal was to just be better and try to deliver on the number one state-of-the-art open source model that we could release. And so we kind of learned two things. One is that when we looked at some of the images from something like SDXL, we noticed that there was sort of this average brightness. It was really confusing. It didn't quite have the right kind of color and contrast.

7:18And in fact, I became so used to this. I became so surprised about the average brightness when comparing it to the images of our model that I thought it was a bug during eval. I literally was looking at the images and I was like, these cannot be the right images. And my team was sort of like, hey, I think you're actually just getting used to the images of the new model. And so we employed this thing called this EDM formulation, which samples the noise slightly differently. And it's a really clever kind of math trick. And there's a paper that you could probably read on it. But it's surprising how this one little very clever trick can produce images that have an incredibly great color and contrast, where the blacks are really vibrant with a bunch of different colors, and this average brightness kind of goes away.

8:00So that's one thing. That's a really interesting example of really optimizing for one aspect of creating aesthetically pleasing imagery. And there's a few other aspects like that. So I'm just curious, how much do you have to hand-tune different parameters versus it's just something that you get as you train a model or post-train a model? Yeah, I mean, there's different dimensions of these models. I mean, one is just understanding of knowledge. But then for aesthetics, it's really tricky. Um, I honestly, I think like the field itself is just so nascent that like every month there's like a new trick.

8:37There's like a new thing that we all sort of develop or find, find out. I think there are, there's an element of some of that being like a lot of different tricks. Like there's like this new trick that hasn't been like, uh, um, well employed or well exploited yet by this guy named Teo, uh, Keras. And he basically does this weird thing called power EMA anyway, like basically helps converge training really fast. And, and so that's like one trick. And then there's EDM trick and there's this thing called offset noise. And so there is a lot of tricks for things like color and contrast. There's even a trick called DPO for like, that I think works in the language model world and also the image world.

9:13So I think there are all these, there are like lots of tricks that sometimes get you like 10, 20, sometimes 2x improvements. But I think the number one trick is like really just like that last phase of, you know, a supervised fine tune where you're finding like really great curated data. And it's hard to say how much of that is a trick because it's actually just a lot of meticulous work. So I think there's a kind of a combination of some of these things being tricks and techniques. And then there's just like this other thing that's just like really hard, meticulous, like there has to be deep care.

9:40And with images, maybe more so than language, there has to be like taste and judgment. Yeah. How do you think about that from the perspective of the e-value dude? Because not that many people have amazing taste aesthetically, right? And so I'm a little bit curious how you end up determining what is good taste, or is it just user feedback, you know, thumbs up, thumbs down? Like, how do you think about that? One thing that I've noticed is that every time we do an eval, we try to make our evals better and we try to make them better than the predecessor eval. And so one thing I always notice, though, with each successive run is that I find out much later after eval that the model has like all these gaps.

10:16So an example of a gap that we recently had was like, we did well in our eval, but one area that I thought we did poorly that I wish we had done better on was photorealism. Sometimes it would make faces look like they hadn't gone to sleep for three days or something. And so I think that most evals in the industry are relatively flawed. Like a lot of them are doing like benchmarks on things that maybe are valuable from the purposes of marketing, but are not necessarily well correlated with what maybe users care about. And so like a simple example would be like with large language models. Like there's probably a good reason.

10:50There's a reason why they're probably good at homework. It's because like a lot of the evals are like related to things that are related to things that could be like homework, like solving like an LSAT or a bio test or math test. And so some of these evals just don't have like the necessary coverage. So I think with things like judgment and taste, my feeling is that overall, the evals need to get like way stronger. And so one thing that we tend to do is we just tend to like really look at like a lot of images across a lot of grids. And we're really like being exacting about like what thing could be off, but you have to look at like thousands of images across that, you know, lots of, lots of different grids across different checkpoints to basically find and pick like sort of release candidates.

11:31But I still think that our own emails are not sufficiently strong enough. And they could be better at like world knowledge, whether that's like its ability to reproduce a celebrity, if that's what you want, or paintings. Sometimes paintings are like difficult or like 3D or like illustrations or logos. Those kinds of things are all like overall, I think coverage is like pretty tricky. So one of the things that you guys do is have like voting schemes or user studies within the product itself. So I don't know if it's grids, but you're asking users to, you know, express preferences more so than I think perhaps other research efforts are.

12:12Can you talk about just like generally your data curation strategy, if there's some sort of overall framework or if community is a big piece of it? But generally, we try to keep something very simple because we know that users are there to make images. They're not there to necessarily help us label images or annotate things or tell us everything about their preferences. And so we kind of have a very sophisticated process of how we sort of curate images and how we're collecting data from these users to help us rank and make sure we're choosing the right sort of things that we want to curate. And so I think these things are like very, they might seem like very simple when you encounter it.

12:53But like beneath that is like something very, very complex. But yeah, it's a little tough to go into it too deeply because, yeah, it does feel like a little bit of a secret sauce, I suppose. Yeah, well, at least I'll feel good that my guess as to what's interesting is right. Can you just characterize like where Playground does really well today? Like where you stand out and what sort of use cases you're focused on like winning? we're maybe like probably like number two i suspect i guess at like text to art at the moment just because we're training these models from scratch we're getting we're closing the gap really rapidly as as rapidly as we can around all the various like kind of use cases but i think that we'll probably diverge from some of the other companies in part because we care i think we start we're going to care a lot more about editing um you know people just have like a lot of images on their phone or they want to take some image that they love whether that's made as art or something that they found and they want to like tweak it a little bit.

13:46It's a little annoying that like you make this image and then you can't really change too much about it. You can't change the likeness of it. Maybe there's like a dog or like your face or like something character consistency issues. It feels a little like a loot box right now. And I think that because it's so much of a loot box, it feels like it's too much effort, I guess, to get something that you really, really want. So I think more where we're navigating is like, how can we help you take an image that you love? Maybe it's your logo, maybe or incorporate something like your logo or put it in some sort of situation that you would prefer.

14:26Text synthesis is like something that we want to do, for example. So those are some areas that we want to head towards where there's like higher utility and less like you make an image and you just post it to Instagram or something like that where do you want to take uh the company and the product over the next few years like what is a long-term vision of what you're doing um if people are out there kind of like working on scaling text we're basically trying to focus on like scaling pixels in the first area that we're basically started on is just images and the reason why we're working on images instead of say something like video or 3d or something like that is one part one issue with 3d is that it tends to be better to work on 3d if you're like making the content like you're making pixar movies The tools in 3D tend to not make as much money.

15:10And then the other thing with video is videos is just extraordinarily computationally expensive to do inference or even training on. And a lot of the video models first train, like pre-trained with a billion images first anyway to have a rich semantic understanding of pixels. We just think that images is maybe the most obvious place to start because A, the utility is quite low, and B, it's like actually somewhat efficient computationally like to do. So long term, I think that we're trying to make a large vision model. There's not really like a word, I guess. Like we have LLMs, but I'm not really sure what the word is for vision or pixels.

15:49If you're trying to make like a, you know, a multitask like vision model. And so the goal would be to try to like do three areas of a large vision model would be to be able to create things, edit things, and then understand things. And so like understanding would be like GPT-4V or if you're using something open source like CogVLM or there's all these amazing vision language models that are happening. And then editing and creating are things that we've kind of talked about. But it would be really amazing at some point that if you made this like really amazing large vision model that it could do things like not just like create things like art, but like maybe like help some kind of robot like traverse like some sort of path or like maze.

16:28And then there's like things in the middle that are sort of like, you know, maybe you have like a video camera or surveillance system or something. And it's like able to understand what's going on in that. But I think right now we're just really focused on graphics. And then how do you think about the underlying architecture for what you're doing? Because, you know, traditionally a lot of the models have been diffusion model based. And then increasingly, you know, you see people now starting to use transformer-based architectures for some aspects of ImageGen and things like that. How do you think about where all this is heading from an architectural perspective and what sort of models will exist in the next year or two?

17:06My kind of like controversial take perhaps is that, you know, there's this thing called DIT, which people allegedly believe like Sora is based on. And then there are variants of DIT. There's this thing called, I think, MMDIT, which I think Stable Division 3 is supposed to be based on by that research team at Stability AI. And my overall feeling is that transformers are definitely, I think transformers are definitely the right direction, but I don't think that we're going to get a lot of, enough utility if we're not somewhat trying to figure out a way to combine the great, amazing knowledge of a language model and then just like using, you know, something like DIT, which is completely trained from some kind of video caption or image caption to an image.

17:50Because there's not enough like interpretable knowledge, I suppose. Like you're not able to interpret anything about the input, which a language model is really great at. But then there's like these models that are just trained on these captions that emit images. And it's kind of unclear like how we might marry these two things. And so it sure would be nice if like somehow how we could combine these two things. So I think the architecture is like mostly going, is most likely going to change. I don't think that DIT is like the right architecture, but transformers certainly. And just for people who are listening, DIT just stands for diffusion transformers.

18:23So in case people are wondering. One belief held by some of the large labs focused mostly on language today is in the end, we end up with like one truly multimodal general model, right? That it is like, we don't end up with a language model and a video model and an audio model and an image model. It is any modality in gigabrain knowledge, reasoning, long context, and any modality out. Do you believe in that worldview or how do you see it differently? I definitely think the models are going to be multimodal. And in fact, that's kind of what I mean about some of these models that are just strictly trained through a division transformer.

19:10A division transformer that's only taking caption image inputs just completely lacks some knowledge. And then conversely, if you look at just the language models, we know that language is at a much lower dimensionality than, say, an image, which has like all these pixels that like sort of tell us about lighting or physics or spatial relationships or size and shapes. So, you know, for example, if you were to like take a glass and like shatter it on the floor, and then I asked you to describe it and asked, and then I described it, we would both come up with like completely different descriptions if Elad had to go and like draw it.

19:46Right. So we know that like pixels have an enormous amount of high information density compared to language and language is just really between me and you. It's like a compressed way that you and I can converse with each other at somewhat of a higher bandwidth, right? Like we have an abstract view of what those words mean. So I think that the models, there has to be something, like language is really great because it's compressed information. And then like vision is really great because it's so information rich, but it's been hard to annotate until recently. It's only because vision language models exist that it's now suddenly a lot easier to like sort of label or annotate understand what's going on in an image.

20:28So I think that these two things are going to be very likely married. The only question is, does language, to me it's kind of a question of, does language, language has this wonderful trait where it's like you can use language to control things, which is pretty cool because of its low dimensionality. But my question would be like, I wonder if language will hit a ceiling, has like a lower ceiling than say vision, because it's very easy to get lots of pixel data. And that pixel data is like very, very high density. Very easy to get more pixel, like additional pixel data to the already collected data from the internet that's gone into these models.

21:05Yeah, I mean, there's an assumption maybe, one assumption I tend to question is like whether the internet data is sufficient. Like the internet is very big, but maybe there's like some kind of boat collapse even with internet data. Whereas like with vision, at least you can like, you can like make a robot that just like travels down the street and like just keeps taking pictures of everything. You can get like infinite training data with vision, but it might be trickier to like sort of filter and clean internet data, especially as like more synthetic data ends up on the internet. Well, another area that I know you spend a lot of time is music, you know, you make your own music and produce it.

21:38And, you know, there've been a number of different applications, refusions, you know, et cetera, that have sort of come up on the music side. I was just curious how you've been paying attention to that, what you think of it and, you know, where you think that whole space evolves to. Yeah, I love audio. That'd be the other kind of thing that I would go work on if it weren't for Playground. Partly I didn't work on music because the music industry is like only, the whole industry is$26 billion. So it was a little hard for me to figure out like how big like a music thing could be. But I definitely think audio is going to be enormous.

22:12Like things like 11 Labs are very interesting. But anyway, I, yeah, I mean, the way that I've been, I've been trying to find ways to figure out how to use it as a user, because that gives me like a really, that gives me a stronger sense of maybe where things are going. And so one thing that I've been waiting for for many years is that instrumentals in music are actually very easy to get or to make. You know, there's a wide variety of like quality, of course, but generally instrumentals in a song, like if you hear a song from Taylor Swift or whoever, a rap song, those beats or those instrumentals are fairly easy to make and easy.

22:51What's hard is to get lyrics and vocals. And that's always been like a difficulty of mine. Like, how do I find a singer? And then how do I get them to like write lyrics and then sing it? That's a much more scarce resource in the music world. And so for the first time with like something like Suno AI, it was really cool because it's the first time that I heard, you know, them be able to make like a rap song where the rapper has like good flow. flow is just like the swing of lyrics to a beat and um or like a you know you hear like actually really good lyrics that feel like very emotional have the right breathiness doesn't sound like um like it's all made on like auto-tune i guess and so um i have this like little flow where i like make a song in suno and then i use a different ai tool it's ai tools all the way down i guess to like split the stems and just grab the lyrics but then throw away the instrumental and then I get to like make a song with the instrumental and the vocal.

23:51Anyway, so I put some songs on my Twitter where like I basically tried to do this and it sounds, you know, so I can get to like a higher quality song, I guess, because I make the instrumental. There are still some like weird errors in the songs, but that's been like a really cool way to use AI, in my opinion. So Hale, thanks so much for sharing everything that you're working on at Playground with us. Find us on Twitter at NoPriorsPod. Subscribe to our YouTube channel if you want to see our faces. Follow the show on Apple Podcasts, Spotify, or wherever you listen. That way you get a new episode every week.

24:24And sign up for emails or find transcripts for every episode at no-fires.com.