Text to Video: The Next Leap in AI Generation

0:00When I first sampled this model, I was actually shocked that it works so well. The pure improvements in performance and text understanding of these models is a possibility to write something like a physical law from such a model. I think a really important part of this is the fact that these models have been accessible to everyone. It learns a representation of the world. Today, many people are familiar with text -to -text and text -to -image AI models. Think, chat, TVT, or mentoring. But what about text to video? Well, several companies are working to make that a reality. But for many reasons, it's a lot harder.

0:39For one, their size. Just think, you'll often find text files in the kilobytes. Images? Maybe a few megabytes. But it's not uncommon for high quality video to be in the gigabytes. Plus, video requires a much more dynamic representation of the world than incorporates the physics of movement, 3D objects, and more. Imagine the hand challenge in text image, but in this case it's hand -square. But this is not stopping the researchers behind stable video diffusion, which as of November 21st was released as a state of the art open source generative video model. And today you'll get to hear directly from two of the technical researchers behind that model, Andreas Blatman and Robin Rombach.

1:21Robin by the way is also the co -inventor of stable diffusion, one of the most popular open source text to image models. So in today's episode together with A16Z General Partner, Ajay Mehta, you'll get to your first hand what makes text to video so much harder, the challenges like selecting the right data sets that enable realistic representations of the world, applications where the technology is actually already being put to use, plus what the video editor of future might look like, and how constraints continue to spur innovation and ultimately keep this field moving. And if you like this episode, our Our infrastructure team is coming out with a lot more AI content in the new year, but in the meantime, you can go to a16z .com slash AI for our previous coverage.

2:05All right, the first voice will hear is on today, then Robin, then Andreas. Enjoy. As a reminder, the content here is for informational purposes only. Should not be taken as legal, business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any A16z fund. Please note that A16Z and its affiliates may also maintain investments in the company's discussed in this podcast. For more details including a link to our investments, please see A16Z .com slash Discoachers.

2:41This is a conversation I've been super excited about for a while and maybe we can start with just a brief overview of your team, your research lab and for listeners who are unfamiliar, maybe spend just a couple minutes talking about what stable diffusion is and what stable video diffusion is. Absolutely. Thank you for having us. What is stable diffusion? Stable diffusion is a text to image model, the generative model that means you type in a text prompt and it generates an image based on that. In particular, stable diffusion is, as the name suggests, a diffusion model. the fusion models are a type of alternative models, which has been super successful recently for image generation and it's based on a technique that we Developed while we were still at the university.

3:25So me and Andreas and Patrick and Dominic all in the same team now at stability. I we are a multimodal company and our specialty is to produce and publish models try to make them as accessible as possible. that includes publishing weights and making foundation models for all kinds of modalities, not only images, but also video available and enabling research on top of that. So we have seen that. The diffusion was super successful. I would say much more successful than we initially anticipated. And there are like hundreds, if not thousands of papers, that are building on top of that. We in particular, our group is focused on visual media.

4:10So that is images, that is videos, and stable video diffusion that you just introduced. It's kind of the next iteration. It's our first step into the video domain when we published a model that can take in an image and turn that into a short video clip. Maybe we could spend a couple minutes on a brief overview of the fusion models that might be helpful. How do diffusion models differ from other types of generative models and techniques like other aggressive models if you could just give us a little bit of context before we dive in? Fusion models are really need to go models right now for visual media images and videos.

4:44They are kind of different to order aggressive models because they don't represent data as a sequence of tokens which we know from order aggressive models. And since images and videos are composed as a pixel grid, this is really a good beneficial property. Also, they favor perceptually important details which is inherently baked into these models because their learning about objectives is tuned to favor these important aspects of images as we perceive it as humans and that is what we actually want, right? But they also have some commonalities with autoregressive models. Both autoregressive models as well as diffusion models, they are iterative in their nature, but as opposed to autoregressive models which iteratively generate token by token or word by word for language.

5:34These models gradually transform in small steps. So they gradually transform noise to data. One point to add to the difference, maybe in diffusion models, you train the model on initially you use like a thousand different hoist levels between data and like pure noise. But the interesting thing is that at sampling time, you can actually use less steps. You can use like 50 steps. You have published a distillation work a week ago that actually shows that you can go as low as one sampling step, which is, I would say, the big advantage of these diffusion models. For folks who may not be familiar with why a single step sampling breakthrough is important, could you say a little bit about what benefits that leads to for creators or users of the model?

6:20Oh yeah, absolutely. I think the most intuitive thing is that And you actually see what happens while you type in your text prompt. So think of this text image model, your type in your prompt, one and a half years back, you had to wait for a few seconds, maybe even up to a minute. Now you see what happens and the quality is even better than what we had with the first iteration of StabilityFuton. So super exciting, actually, to see that kind of trajectory, these kind of developments. When I first sampled this model, I was actually shocked that it works so well. to keep pulling on that thread for a bit.

6:52If we rewind the clock back to a year and a half ago, which is when you guys first put out stable diffusion. Between then and now, what has surprised you most about image models that you didn't expect? The pure improvements in performance and text understanding of these models in spatial compositionality of what these models can do just by typing in a single prompt you can describe a scene in really fine -grained and it gives you a highly detailed visual instantiation of it. The development has been huge, we published SDXL in June and even then it was a huge improvement in visual quality and prompt following.

7:35Also other models we see right now is like most recent Dalai free is like a huge improvement still but also as Robin said that there has been a lot of different samples proposed to make these models faster and faster and faster and And right now we're getting really close to 50 steps performance and even one step. This is a huge improvement. And I think a really important part of this is the fact that these models have been accessible to everyone. So open -sourcing, a foundation model has stable diffusion initially, that led to a whole lot of research on these models, which was in retrospect extremely important to do this.

8:12I think otherwise we wouldn't have seen the improvements we saw until now. Even before that I was surprised that text image with diffusion models works so well just before we published the model Like when I first saw this myself we had this latent diffusion approach that we developed at the university I mean we got a machine with like eight a digiabyte a 100s just after we put it on archive and then yeah immediately start working on Hey, we want to have this text image model, but not train it on one GPU Let's use all of like this is our little cluster with eight a digiabyte a 100s between this latent diffusion model on 256 - 256 pixels was the first time that we had to deal with large scale data loading and these kind of things and then using this model combining it with classifier free guidance which is a sampling technique that further improves sample quality at basically no cost.

9:01I was like really surprised that we could do this on our own and achieve a pretty good model I would say and then like two days later OpenAI published all the two and all the hype It was a pretty nice experience. Something you mentioned, Andreas, is that the fact that you guys chose to release stable diffusion as an open source model resulted in this crazy ecosystem exploding around your research, which is just something that doesn't happen as quickly or as fast with models that aren't open source. And so in the last year and a half, one of the things that's been really fun, at least for me to watch is all the really surprising things that developers and creators have done with the base model that you guys put out.

9:49You've provided folks a set of Lego blocks that they can mix in matching different ways, things like control net that give people more controllability, allowing the community to build their own front end. Out of all of that, I'm sure came a ton of requests. As you guys were prioritizing all those asks that came in from the world in the community, Why was stable video the thing that you guys decided to prioritize? Video is an awesome kind of data because to solve that task, to solve video generation, a model needs to learn much about like physical properties of the world, or the physical foundations of the world.

10:27There is so much, without knowing about, for instance, 3D scenes, you cannot generate a camera pen around an object, or you cannot make an object move. If a person turns around, the model needs to hallucinate how this person looks from behind, right? So to know so much about the word by just including that additional temporal dimension, this is what fascinated me most on working on videos. It's also really next level of computational demands because you have an additional dimensionality which makes everything much harder, I think. And yeah, I think we like challenges. That's why we probably focused on doing that.

11:01Yeah, something that's not known about you guys is by background, originally I believe you're physicists. Yeah, I'm a physicist, but I haven't done much physics in a while, unfortunately. I'm originally a mechanical engineer, but that is also really related to physics, and I was always inspired by physics and really fascinated by it. Well, both of your backgrounds academically were spent studying the physical world, and I just think it's poetic that your primary interest in generative modeling came from trying to understand and at some deeper level the physical world. And it seems like that seems to have motivated at least some of the intuition and the research around your approach to stable video.

11:39Yeah, absolutely. I fully agree with just this question. So first with the kind of video model so we have right now, having something like we are seeing in language modeling, but trying to on pixels on videos will probably give like super interesting downstream behavior. Not only like generating videos, but also understanding of the world. Like is it possible to drive something like a physical law from such a model, I don't know, or such a model is also always predictive. So you can start with an image or with a sequence of images and try to predict what happens next, of course. And then I think like also coupling this with other modalities, such as language, will maybe provide a way to ground like these models more in the physical world.

12:24I think that's a good segue into what is the main focus of today's conversation, right, which is a generative video, to folks who are early users of stable diffusion, stable video was a much awaited sort of natural progression from the original model. Just take us back a little bit to the original sort of conception of the project. How long have you guys been working on video modeling? I would say roughly half a year. And like for this model that we just put out, I think the main challenge was that we actually had to scale the data set and the data loading. So if you train a video model on a lot of GPUs, to do suddenly run into problems that you did really have before.

13:02Like loading high resolution videos is just like a difficult task if you do it at scale. Also, only decoding videos is really hard. Like a data loader has to transform like the spides that loads into a suitable representation for the model. And to do so, you have to do a lot of computational work to transform it into a suitable input sample for the generative model. And this is, the competition is really expensive. And since we have so fast GPUs right now, it was really like the CPUs, we're just like in the beginning, too slow, building an efficient data pipeline for a video was really a challenge, which we, I think, solved really well, which probably took most of the time.

13:44We spend on building and scaling these video models. Yeah, and actually, there's like interesting bugs that you couldn't encounter during training. So we had one where you have your data and then you add noise to that data that the model tries to remove, right? And if you do that on a video, you add noise to each frame of the video and then we add the back where we added like different amounts of noise to different frames in the video which just complicates the learning task unnecessarily. Things like this is just like one line of code that can go wrong. What was the biggest difference between the image model research and your video work?

14:19Because noise sampling and noise reduction. These are sort of diffusion techniques that are shared across images and video, but it would be helpful to understand what we're unique to the video challenge. First of all, the pure dimensionality of videos I mentioned that before with this like additional dimension, this introduces of course a higher GPU or memory consumption. And this was really a challenge. So for diffusion models, it's really important to have a high batch size because you can approximate the gradient, which thrives the learning much better at the batch sizes is higher, especially for diffusion models.

14:56It's like really an important thing to have a really high batch size. But I have to increase your number of GPUs, which again, introduces new challenges in terms of scaling, in terms of like redundancy in your training pipeline. and if something breaks somewhere and one GPU, it will just like throw down the entire training. And the more GPUs you add to your cluster and on the more GPUs you train, the higher the probability will be that somewhere there's just like a hardware failure even which also happens. This additional dimensionality just introduces these new scaling challenges which are really, really interesting to come by I would say.

15:39Well that's very helpful I think. One of the most valuable things that your guys' research has done for the industry is that you often share in very excruciating detail some of the infrastructure challenges that came with training and I think since scaling models at the magnitude that you guys are as a relatively new infrastructure challenge, I think it's very very helpful for the researchers to be able to hear or the sort of nuts and bolts that you have to figure out, to get these models out. Then there's the whole other set of data related challenges that aren't about the data pipeline per se, but it's about the representation of the data set, curation and the data set mixture.

16:22Could you guys just talk a little bit about how you approached picking your data set for this release? And what was your intuition and what kinds of data were most important, how you wanted to filter it? And ultimately, what ended up being your most important learnings around the data set when it came to training stable video. We actually spent a lot of time talking about this in the paper that we just put out, or we also defined in this paper is that we can divide this training process into three stages. And the first is that we actually trained in the image model. So for training video models, it's usually just helpful to reuse the structural spatial understanding from image models when there are like powerful image models.

17:00We should reuse for them training the video model. And then there's next steps. So having an image model, like stable diffusion, for example, you have to get this additional knowledge about the temporal dimensionality and about motion. So for that, we train on the large data set that we still have to create a bit. So we don't want, let's say, optical characters. We don't want like text in the video. We want nice object motion. We also want nice camera motion. So we have to filter for that. And we do this in like two regimes, we train on a lot of videos in the first stage and in the second stage we train on mock rate very high quality smaller data set to really refine the model and it's similar to image models where you train on a large data set And then refine on a high quality data set was a paper recently that met up with out That also described just this process for image models in detail one of the largest open questions in video generation for a while has been structural consistency of 3D objects.

18:01When the camera is spanning around a person or a car or any subject, to make sure that it stays and looks like the same subject from various angles has been a challenge for a generative video. How did you guys approach that? You mentioned in the paper that 3D data and multi -view data was important. Actually, I think the main point we want to make in the paper is the one that we talked about earlier, like having a a foundational video model, actually it gives us much more than just a model that can generate nice looking clips or videos, right? It learns a representation of the world and one aspect of that is that we try to demonstrate in the paper given a video model which has seen a lot of objects from different views, lots of different camera movements.

18:46It should be much more easy to turn that into a multi -view model. And that's kind of the main message. So we take the pre -trained video model, which has seen a lot of different videos, a lot of different camera movements. And we fine tune that on very specialized multi -view orbits around 3D objects, and turn the video model into a multi -view synthesis model. And that works pretty well. So one of the dominating approaches before that was that you would take like an image model, like stable diffusion, and turn that into a multi -view model. But I will show that it's actually helpful to incorporate this implicit 3D knowledge that is captured in all of the videos Into the model and then the model can live much quicker than if you start from the pure image model.

19:28So that's kind of the main message, but You're right you can also try to use this explicit multi -view data in the video training or maybe even Something that we do in the paper trained Laura's explicitly on different camera movements and then put this Laura's back into the video model. So you get control over the camera for your very general video model, which is quite cool. Yes, at this I found was one of the coolest pieces of the paper was incorporating Laura's for fine green control in the creation process. Could you maybe give us a quick overview of what Laura's even are conceptually intuitively and what led you to intuition that Laura's would be an important part of the architecture?

20:10Nora's are just like really lightweight adapters which fine -puned onto a existing base model, which adapt the attention layers and by that you can just like on a small, really highly specialized data set you can tune in a really lightweight way different properties into the model. And in this case we just like tuned different kinds of camera motion into our video model. So if we use a small data set which only contains like zooms or panning to the left or to the right, we can actually tune such a Laura as a small adapter to the attention layers of our model to just like get exactly this behavior and this is a really awesome way to just in a really lightweight fine tune these foundational models and it has shown to be like really effective and accordingly it's like really highly appreciated in the community I would say to get these kind of easy fine tools.

21:06Yeah, and I think for image models, it's like extremely popular. There's so many different dollars that people plug into these models. For video models, our goal is just to demonstrate that this is something that's possible. It's just like at the beginning and there's much more that should be possible. Like very specialized kind of motions. So I think there's a lot of creative possibilities. That's actually worth exploring for a little bit. One of the windows that you guys have into the future is by understanding where the research is going, you get the time travel and kind of get a glimpse into the future of creativity.

21:39And so having seen how effective loras are at least at a few sets of tasks like motion control, right? So in the paper, you propose using loras for camera control, banning zooming, et cetera. The history of video creation has usually required creators to have a ton of different knobs and dials in their software that they use, right? whether it's an Adobe After Effects or some other professional software, you literally have hundreds of dials and buttons that you can use to control and edit these videos. And conceptually, should people think about Laura's as mapping to these controls, in the future, will a director or creator of videos basically be relying on hundreds of different Laura's to express the control they want over the video?

22:24Or do you think fundamentally Laura's will hit some scaling sort of limit and that's the wrong analogy to use. How should creators think about these new tools that you've given them? Yeah, I think you actually said it reds the maintaining like a library of hundreds of lower this maybe not like the most scalable approach. Actually, if you look at the model that we put out now, it's just like taking an image and animating that reds and we can do some stuff like with these lowers. But what you actually want, I think, is given the image and some text prompts do exactly what I describe in a text prompt.

22:57There's already some work that explores that, but you're giving more control over what happens in the video. Be it through Laura's, but maybe through a text prompt or through spatial motion guidance, like in Brunway's motion brush. There are different ways of doing that, but you definitely want more control over this whole creation process. And then I think you're at the stage where you can really start to generate personalized individual content, especially probably for video creation. and we want something like we just, the image models, you want like very fast synthesis. Because then this will become more like, I don't know, sometimes I think about this as like a video game, right?

23:35You type your prompt and you immediately see what happens, given your input view. And I think this might be a super nice user experience, actually. So we want this additional control and we want a fast rendering, fast sampling, fast synthesis. You know, you said earlier that you're hoping that the community explores more things. Now that you've actually put the model out there, They're going to be developers and creators who listen to this podcast. What would you like them to explore first and most intensely? Oh, I think just trying out the model rendering some awesome stuff. Of course, also further exploring maybe the representation we build in the paper we mentioned that we trained this model on a whole lot of data.

24:12And this has just seen really, really much motion. And the representation is really fruitful. We showed that by our 3D fine tuning. By the way, this was completely surprising for me, seeing that model after 1 ,000 -2 ,000 iterations like already getting 3D reasoning, this was really, really nice. So as we saw that, it will be extremely interesting to see other such approaches. Models open source, people can try it and give it another couple of weeks and then we will see what happens. But I'm excited for it. You know, my personal favorite for what people did on day one was obviously animating memes.

24:49I'm sure you guys have seen all that. That was really funny, yeah. What are your guys' favorite creations so far that you've seen anything that jumps to mind? I think the one I have always forgot the name of that meme, but a man is looking after another woman. The man looking behind. Right. The man looking behind you, that one. What's that? What a visual of it up. Yeah, exactly. I think it just visualizes very nice, additional experience that a video model can provide, but people are used to like 2D memes, but then I can actually try to animate this and see what happens. Also, if you think about famous artwork or something, just bringing them to life is a really, really nice property and it's now enabled.

25:29Everyone can just like poke around a bit of Mona Lisa and see what she's looking from the side. Oh, that's cool. I haven't explored that one, but you're saying prompting the model with an image of an notable art piece. Just like making Van Gogh's Starry night, make the stars shine and flimmer and I think it's really cool. The world is pretty lucky that you guys have gifted the model to the developer and open source ecosystem. It's already such an incredible sort of step leap forward right in what you can do with images and with video. What do you think are the two or three biggest sort of open challenges that you guys want to prioritize next that are still limitations in video generation?

26:06I think a really important thing is to get these models to generate longer videos, to process longer videos in general, not only generate them, also see them, because I think eventually processing longer videos is key to understanding what we talked about earlier, fundamental aspects of this world better, physical properties. And so this is a really important part to enable these models to generate longer content, and more coherent content also with other kinds of motion. And when Robin already said, I think, making them fast, we'll just like, unlock so much more exploration. So this is really a nice thing and we're actually working on it.

26:48Yeah, there are simple things like thinking about like multimodality, adding an audio track to your generated video that is in sync with the action that is rendered. I think that is a lot of stuff to explore. So you're talking, Andrea, I said earlier about the infrastructure challenges. If you had a magic wand, what infrastructure improvements do you wish the industry could solve for you? I mean, you could ask for more GPUs, more CPUs per GPU and this would solve much of the data loading issues, also not only GPU memory is always good, but also CPU memory. But I think like hitting these limits is just a some form of natural way you always want to try to improve your efficiency.