Image Generation and Visual Intelligence with Black Forest Labs

0:01Welcome to the Practical AI Podcast, where we break down the real-world applications of artificial intelligence and how it's shaping the way we live, work, and create. Our goal is to help make AI technology practical, productive, and accessible to everyone. Whether you're a developer, business leader, or just curious about the tech behind the buzz, you're in the right place. Be sure to connect with us on LinkedIn, X, or Blue Sky to stay up to date with episode drops, behind-the-scenes content, and AI insights. You can learn more at practicalai.fm. Now, on to the show.

0:41Daniel Whitenack:Welcome to another episode of the Practical AI podcast. This is Daniel Whitenack. I am the CEO at Prediction Guard, and I'm joined as always by my co-host, Chris Benson, who is a principal AI and autonomy research engineer. How are you doing, Chris? Hey, I'm doing great. Can't wait to get into today's conversation. It's going to be fun. Yes, yes. For an audio podcast, we're going to talk about a lot of interesting visual things. Maybe before we get started, just a little teaser. Practical AI is posting some videos on YouTube now. So if you do consume podcasts that way, you might go check us out on our YouTube page.

1:24Daniel Whitenack:But speaking of images, videos, and more specifically, image generation, really excited to have with us today, Dustin Podell, who is co-founder and researcher at Black Forest Labs. Welcome, Dustin. Yeah, yeah. Thanks for having me, guys. It's really great to be here. Yeah. And I know that Black Forest Labs does more than just kind of raw image generation. There's a lot of workflow related things, hardware optimization, all sorts of cool stuff you're involved with. But as we get into some of that, I'm wondering if you can just help our audience with a bit of a state of image generation methods and workflows for the industry.

2:06Daniel Whitenack:We've talked on the show before about diffusion models, and we'll link some of those episodes in the show notes maybe. But a lot has happened, right? There's a lot of people working on a lot of interesting things. And I'd love to kind of understand, like, over the last year, what are some of those main points that might be good for people to orient themselves to where things are at now? Yeah, yeah. No, it's a good question. I mean, if I'm allowed to take it even a little bit further back, I mean, the state of... Absolutely. Yeah, the state of image gen, video gen, generative models as a whole has kind of gone crazy, so to speak, in the last three or four years.

2:50So where are we now? um where we came from about four years ago uh where where i first kind of entered into into the scene so to speak um is we were at models that were essentially just doing little blobs of color that were kind of related a little bit to where you were with the prompt you know okay oh a lighthouse on the beach or this and that and you would get something that okay that vaguely looks like it and you would show someone and it'd be um yeah i could see it i guess and and maybe it would interest a few nerdy people but and then now today we're at the point where you know if you've probably been seeing plenty of this stuff online where we're seeing whole short films made entirely with a ai generation where certain scenes are almost entirely indistinguishable from uh from reality so to speak so i would say we we've uh we've come quite far but i will also say the core of the technology hasn't actually changed that much it's been a a pretty nice like forward progress.

3:48I don't want to like dive immediately into anything technical here, but, but it's, it's certainly, I mean, for anyone who's been paying attention, I'm sure, or anyone who really hasn't been paying attention, this probably came a bit out of, out of nowhere. So, yeah. Yeah. I was going to say, I guess, you know, with the broadly attention by the general public is so much on kind of more of the LLM generative world in terms of, you know, that, that stuff and everyone's finally on apps, regardless of whether they're technical or not. And so I think a lot of people kind of miss the tremendous advancements you guys are making on that side.

4:23And so like, you know, could you take a quick moment and now that you've kind of done the highest level, maybe step through some of the things that people may remember, like we talked about stable diffusion and things, a couple of those things that have kind of led up to what we're going to dive into today with some more specifics. And as Daniel mentioned, we can offer some links for past conversations if people want to dive into those specifically. But that would really be interesting to kind of hear distinct points on that timeline as we hit to this point. Sure, sure. First, before I do that, I guess I'll ask Chris, how technical am I allowed to get here?

4:59You're allowed. We're going to say we'll like if you dive into something, we may stop you and say So that is what, like, what is that acronym? But other than that, you can go as technical as you want, and we'll make sure that everybody can stick with us. Okay. Including me. Yeah, yeah. I guess if I'm allowed, I'll do kind of at least my, the best of my knowledge overview of the last bit of time since. So obviously the field of AI and ML has been around for a while, trying to structure and learn distributions of different data. Mostly it was used for, you know, tabular prediction, all sorts of other things for a while.

5:36And then in 2017, some researchers at Google figured out this very cool thing called the Transformer, which I'm sure people have heard many, many times about right now, which I guess the best way to go down this path is to say like we found a way to train much more general models, like learn from much more general data. I'm trying to think how to essentially take this story from here. It's a dramatic pause. It works, man. It's good. Yeah, essentially the story I want to tell is the story between the kind of, I don't want to call it the battle that's been going on the last few years, but essentially the story between diffusion and autoregression that's been going on the last few years.

6:17So let me start by trying to define a little bit what diffusion autoregression is. Autoregression is anyone who's been playing with language models, oh, you know, all these, you know, GPT, CLOD, all the new stuff, everything that's been going on in the last few years. These are what you would call an autoregressive language model. And so what this is, is it's predicting data essentially one piece at a time. So it predicts one piece of data, and then it looks, okay, what did it make? Then it predicts the next piece of data. It looks at what it makes, then it predicts the next piece, and it slowly, slowly, slowly, slowly builds out.

6:52You know, at first it was just very basic language, and then it became, you know, now, I mean, we're having agents and a lot of very, very fancy things with them.

7:03On the other side of the coin has been something called diffusion modeling. and i mean i i don't want to get into it because now now we're doing more like flow matching but let's just go with the fusion as the as the base term here and with the fusion what you're doing is you're not trying to model one say like in language model it's you know you call it a token but you just assume like okay one word or at a time with the fusion what you're trying to do is essentially trying to think how to tell how to tell how to tell the story here yeah yeah no it's all good i i love i feel like i'm watching like a thriller where you're about to like reveal uh the next thing it's good keep yeah so so what we do with the fusion is i'm just going to tell practically what we do and then we can kind of break down you can ask me questions that's fine um so what we do with the fusion is we take a whole continuous medium and what i mean by continuous medium is the most famous one is images the world is continuous there's not you know it's not that there's you know one you know like a letter is discrete there's a t there's an a a b a c a d and e and an f a color shape the structure of the world the color you know the the light the all of this this is this these are continuous natural mediums and so um what we do is we take one of these continuous natural mediums that that we've now encoded into digital space so we use rgb so you know okay 256 colors we typically use uh for red green and blue and then we encode this into an image.

8:31And then what we do is instead of trying to predict, so an auto aggressive, you might try to predict like pixel by pixel by pixel. Instead, what we do is we essentially try to remove information. And the way we remove that information is by adding noise. So you imagine you like add a little bit of grain on top of an image, you can still kind of see the image. Maybe it's a little grainy. Maybe it's a little blurry. Oh, you can't quite see it. And you add a little bit more. Maybe you see some of the shapes. Maybe, okay, there's like, is that a dog? Yeah, you kind of see it. The color starts to fade.

8:59And then you keep adding. And eventually you just get, okay, this is just noise. I don't see anything in this image at all. And with the diffusion model, what we're trying to do is we're essentially adding a bit of noise. And then we're saying, now try to predict what the clean image is essentially for this. So, you know, okay. So we add a little bit of noise, predict the clean. It just has to remove a little bit of noise. You add a lot of noise. And now what it has to do is there's so little information, it has to actually like come up with essentially like how to infill this properly. And then what you do is then in what we call inference, which is when you actually run the model, you ask for an image.

9:35What you would do is then you start a fully noisy image and you say, give me a dog wearing a top hat on the beach. And then essentially what it does is it tries to remove a little bit of that noise to get closer to this idea of a dog and a top hat on the beach. And you might get some structure. It tries to figure it out. It's very coarse. It's very just trying to figure out the shape. And then it has a little bit of shape to latch onto and then it does the next step where it removes a little bit more of that oh and it starts to come into view it starts to you start to get a little bit more in focus and you do this and what you see is essentially you completely remove the noise of this image and oh my goodness now you have this this completely generated image of a dog on the beach with a top hat for a long time this was uh i don't want to say not good i mean all the like all the models have gotten a lot better but when it first started out these look like blobs of color it was like these little like you could barely see it.

10:24But fundamentally, what I just described is still the process that's been happening over the last four years from these old image generator models for anyone that maybe tried like Dolly Mini, I mean, the old stable diffusion models that we work on, you know, I don't know, many of the different models are out there now to these modern video models that are producing whole 10 second sequences. Fundamentally, it's doing the exact same process of removing this information slowly with this noise and then just slowly going the other direction, essentially creating info to infill this, if this makes sense.

10:57I'll pause there for a second just to kind of like that. So let me tell you, because as I'm listening, very, very good explanation, probably the best one that I've heard. So you are right on target if you just carry on what you're doing, because I'm enjoying this.

11:13Daniel Whitenack:And you mentioned that when this started, right, you sort of ended with these blobs, et cetera. I think there's a lot of people that have seen very impressive things like the, like you were saying, like whole parts of maybe movies or commercials or something being generated in this way. What is the current, I guess, state of the art in terms of whether it be your models or others? And I know there's more to talk through as you get in. Like there's more than just raw image generation. There's how this fits into workflows and doing certain tasks. But what is kind of the state of the art in terms of what we're able to generate, where are the bounds currently and in terms of quality and that sort of thing?

11:56Sure. So I'll tell you kind of like in the last year, I'll split it up into, let's call it three different categories, but they kind of blend a little bit, which is these continuous mediums I talked about. So I gave the example of a continuous medium with image. Video is obviously an extension of that, adding the time dimension. But then there's also audio as well, which now a lot of these video models generate audio. but then there's also things like, I don't know, maybe you've heard the, the like Suno or these music models. Some of these are also using this exact same technique of removing the information with noise, bringing it back in.

12:27It's just a different medium that they're essentially applying to. Um, I'm happy to talk about our models all day. I love talking about our models. Um, but I also want to be fair to just anyone listening and honest to kind of the state of the world on if you want to um you know go out and and try some some uh some nice things out there as well um uh i i will say that before i kind of like dive a little bit into into what i think is you know maybe the best one to recommend i i will make like one statement of of that like best quote unquote here is a little bit hard to define sometimes and this is something we're always like wrestling with is like what makes a best model for someone because you know obviously you when someone uses these models, they're typically prompting or putting in now you can reference images or videos or other sorts of things and kind of treat it a little bit like more like a creative companion.

13:18And then like what you expect out of this can be very different for different types of people. That said, I would say that if you want to see kind of the quote unquote state of the art right now for what I would say is like text to video at the moment, um it would be c dance so c dance has done some extremely impressive work with the c dance 2 model over the over the last year so they're doing uh i think it's now 4k generations up to 15 seconds um which is i mean a very nice quality it's it's focused very much on like cinematics um i will throw a shout out to anyone that uh you know if anyone from the sora team ever listens to this i think sora 2 was a quite a nice model it didn't rank so high on um some of these like leaderboards that we in the ai community there's a i'm sure you guys talk about this plenty like leaderboards rankings where models place um i think in the llm side of things this is much better covered where there's very clear metrics of how well does it program how well does it do math on the more creative side of things with video models image models audio models we typically end up falling down to this single preference uh type benchmark which is just like a general preference everyone in the world goes and can can vote on you know on these leaderboards it's a a couple of different nice companies and sites that do this.

14:34But I feel like this kind of removes some of the specificity, the granularity. That's the word I was looking for.

14:43Daniel Whitenack:Listen, I've been to an incredible amount of AI events, many of which are good, but many of which are not practical. You know I love practicality. We're on practical AI. And some are just hype-focused. Some are sales-focused. that's why I'm always eager to share about an event that I truly think is practical and useful for people. That's what I discovered last year at the Midwest AI Summit. And they're going to have another Midwest AI Summit October 15th in Indianapolis this year, 2026. One of the reasons why I love this event was there was an actual AI engineering lounge where you could sit down and talk through your use cases with actual experienced AI engineers and practitioners to really brainstorm and come back from the event with actual solutions and practicality rather than just a bunch of content and slides.

15:37Daniel Whitenack:But there were also amazing keynote speakers, speakers from even that had been on the podcast before, like Rajiv Shah was at last year's event. I would really recommend that you go to midwestaisummit.com and for our listeners, you can actually get 20 % off with the code practical AI 20. So go to midwestaisummit.com. Don't miss your chance to attend this event and get 20 % off with the code practical AI 20. I was going to ask you, are the models that you're describing, are they kind of the traditional diffusion models? Or you had mentioned in passing a few minutes ago about flow matching. And so it's just in my head, I'm trying to kind of categorize how do the ones that you're talking about right now, how do they fit in and, and like, and how, what is it to go back and pull that up as you were talking about diffusion and you made the reference to flow matching.

16:38How do those, what is that, what is that transition and how do those models that you're talking about now fit into that? I'll, I'll, I'll first keep it very simple and say all of these models are doing the same process of what I described earlier of adding noise and training the model to then remove this noise. The difference kind of, this is where it gets a bit more technical in that, in how we actually approach this. So we used to do more of this process called diffusion, which I don't, I don't, I don't necessarily know if I want to get into how, how deep and technical this is, but essentially what we've done is we've cleaned this process up to what we call flow matching.

17:17And flow matching is essentially this very simple process of still doing the same thing, still training the model to remove this noise. But fundamentally, what it's learning under the surface is anyone out there who has ever seen like a velocity map or like a flow map, what this essentially looks like is if you can envision like a, I almost want to, I don't have a whiteboard behind it. I don't have a marker, sorry. So you're going to have to. For the audio folks only. They're not going to see it. That was great. It's probably better I don't have the marker. Yeah, it was great. Dustin, if you're on video, you saw it, but Dustin was literally about to turn back to the whiteboard behind him, which I love.

17:54Like if I was in the room, I'd be like, go, man, take me there. Unfortunately, a good bit of the audience won't be able to see us. You're going to have to describe it. Yeah, yeah, no worries. Let's keep it purely in the audio space. So I'll draw a picture with my words as best as I can. I spent enough time prompting anyway, so hopefully I can do this. All good. But yeah, essentially the way I like to think about it is if you imagine like a landscape, like, I don't know, you can imagine your town or the region you're in and imagine you're looking at it from like the sky and you see like your house, you know, maybe right in the center of this map.

18:33and what you might want to do now imagine okay there's wind going all over all over the area and and the thing that you want to do is you want to be able to throw up like a paper airplane from anywhere in this in this landscape your city and you want the wind to carry it so it lands on your house and functionally what we're trying to train here is essentially that in a much much grander hyperdimensional space. Where instead of it being your house and instead of it being a wind and a paper airplane, what it is is your house in the scenario is what we would call the manifold of real images. It's essentially the place in, I'm trying not to use too many, stop me if I use too many words here.

19:16I want to say latent space. I'll ask you, but you're doing fine. Keep going. We're fine on technical. We'll just get explained it as we go. Yeah, it's the place in latent space that would be where the actual real images are. So another way I'll try to describe in a simple terms, okay, with your town here, you're in 2D space here, where you are above an XY and the paper airplane has to fly in this XY coordinate. So you land and then you have the XY. In this space, instead of it just being two coordinates, it's enough to have all of the colors of the entire image. So that's why I say it's hyperdimensional.

19:50It's the center is where literally real images are. And so what we're doing is instead of it being wind and the rest of your town, imagine the rest of your town here is every image that isn't real. And what that means is noise. If you think about noise as a real image, you can go generate a bunch of noise and save it to your computer. But that noise would exist somewhere in this space of all possible images. They're not real images, they're noise, but you can save them. They're RGB values, you have them. And so what we're doing with this, essentially this flow matching is we're training the model, Like when we say we're training it to remove noise, what's really happening under the surface is we're training this flow map so that we can land anywhere in this field of noise.

20:35And then these flows, these winds in our scenario, when you take a step, will take you closer to your house or to the manifold of real images. And that's what like as you get closer, it starts to look more and more like a real image. It's blurry. It has this. And then when you actually finally land on this manifold, boom, you have hopefully a real image, but you'll have some image, you know, and then the better the model is trained, the better you have mapped this flow to actually take you to where you want to end up.

21:03Daniel Whitenack:And I'm assuming, yeah, no, and I'm assuming because you are kind of mapping this from where you start to where you end up with this real image, maybe in a way that's just to be crude, I guess, less random. I mean, at the end of the day, all AI and machine learning, Like it's sort of that training process is very much trial and error, but we have a lot of optimizations around it. Right. So I'm I'm assuming that this then allows you maybe to, I guess, advantage wise, shorten the the training or make that more efficient. Or am I am I misconstruing that in some way? Yeah, yeah. I mean, definitely like one of the things that's taken us a lot further over the last three to four years is just figuring out a lot of optimizations, both in the actual training itself and then also in just, you know, architectures of the actual models have improved.

22:01um but i will say the like fundamental underlying process of just removing this information and finding a path back to it is still the same process it's it's just like the car you know you've been having cars go down the road since the 1950s and we've made better engines and better safety and all this but you're still driving down the road like that that's yeah yeah that that

22:21Daniel Whitenack:makes sense and now definitely i think that's a good setting in the sense that we've got to a point where, you know, the models coming out of Black Forest Labs, the models coming out from other places, the models that are even like now in my text messaging, right? I get an image, I can immediately remix it with an image model, right? In some way. So these things are becoming more embedded in our lives. But I don't know if everyone in the audience, some might have been along for this ride where it was like, oh, cool, I can generate an image of a astronaut riding a horse on, you know, you know, wherever.

23:00Daniel Whitenack:And that doesn't seem that practical to folks. So I'm wondering if you could now kind of, given that that foundation that we have, and we know sort of where we're oriented in the state of, I love how you put it on your website, visual intelligence, which I think is, I love that statement, because it gets to more like language, although it's being used a lot in terms of agents and intelligence, like you mentioned, it is very much a subset of the information that we process as humans, right? There's this visual element, there's the audio, et cetera. So now that we have that foundation, could you help the audience understand some of, I guess, the practicalities and the outworkings of the like, okay, we can do this now.

23:47Daniel Whitenack:So what? So how does that help people in the real world in ways other than maybe just pure creativity? There's certainly like the cinema, like you were saying, that side of things. Not everyone's going to be generating movies, maybe, or maybe more people will, I guess. But yeah, I think you're understanding what I'm saying. Like, where is this going to impact? Where is it impacting me now? Where is it practically going in terms of the application? Yeah, yeah. I'm very happy to talk about this. This is a, I think we're going through like a very nice transition period right now where we finally get to leverage these for some very useful things.

24:24And if I'm allowed to take a little bit of a tangent and kind of lead up to it again. Wherever you want to go is good. We're all good. Yeah. Yeah. So, I mean, I guess I'll take it back to like a little bit of a timeline of things. So, okay. So, so early on we had these nice models where most people recognize them for you put in a prompt, you get an image out, you put in a prompt, you get a video, maybe you get a song. it's just this one way okay you make something this appeals to maybe creators or ad agencies or movie you know now cinematographers and and i mean we love this stuff we love the creative side of this and and what it allows because it fundamentally to me this is like a nice potential communication tool um but then i i feel like something uh changed i don't want to say changed but uh there was definitely a a a timeline moment when we started to move into um editing so uh was it two years ago now a year ago i don't know around a year and give or take uh we released our first um in context editing model called flux context and uh on the surface you would look at this and go okay well this is this is an image editing model it's something you could take a photo you can clean it up you can add a hat you can do silly things you know you can uh whatever you want to do with it it's a general editing model that's supposed to supposed would do all these interrelational things.

25:42And on the surface, that's very cool. It seems like another creative thing. But if you think about like what's actually going on under the surface, for the model to be able to do this, it has to understand a significant amount of relationships in the world and what it means for these, like how these relationships actually interact together. So if I say, take a picture of a, like a, I don't know, we have like a water glass here on the table. And I take a picture of this water glass and I say to the model, knock the water glass over and show me what happens. The model has to understand some part of the actual world.

26:14Like it has to actually model the world in some way to know, okay, it spills over, maybe something gets wet, maybe XYZ happens. And essentially what we're trying to do is train a model that could do all of these types of relationships. So the model is learning not just like this one thing, but just how the world works fundamentally so you can do this editing. Now we can take this a step further and we can look at all the video models that are coming out right now and you can do a very similar thing. You can take, you know, in a knit image and say, you know, okay, this person now grabs a fire extinguisher and puts out a fire, and it has to actually understand these relationships to do that.

26:52So, well, maybe this wasn't our original goal way back in the day. You know, we were trying to make very cool models and figure out how to, I mean, I think in some sense we wanted to model the world, but maybe we weren't thinking this far ahead. Inherently through this whole process, we've learned to build these models that are developing this and i'm trying really trying to avoid the the use of the term world model here because i'm gonna go there if you don't i'm just telling you yeah you might notice me trying to like skirt this because i think it's a little overused these days um but we definitely talk about it um but fundamentally many people which is where i was going so yeah yeah fundamentally this this is what what these models are doing when you train them at scale and you and you really train them to be general and robust is they need to be able to simulate parts of the world to get an output.