#172 Cristóbal Valenzuela: Can AI Revolutionize How We Create Art?

0:00Particularly over the last four to three years, we've seen an explosion in quality, consistency and overall resolution of image and video models. And it's been because of a few key advancements in techniques, but also more compute and more data has become accessible and available. So around three years ago, we started working on this idea of creating new kinds of models for image generation. And we do collaborate sometimes with other research institutions. So we did a collaboration with the University of LMU Munich in Germany. And that paper that we published was called Latent Diffusion, that introduced a new technique for generating images on a latent space and conditioned those images to a variety of different inputs.

0:45Hi, I'm Craig Smith, and this is Eye on AI. In this episode, I speak with Chris Valenzuela, co-founder of Runway and one of the creators of Stable Diffusion, the open source text-to-image model. We talked about advances in text-to-video models and what that means for the future. But I had this conversation literally the day before OpenAI introduced Sora, its incredible text-to-video model, so I wasn't able to speak to Chris about that. He did talk about the advancements in techniques, increased compute power, and data accessibility that have made applications like Sora and Runway possible. Join us as we explore the intersection of art, technology, and the future of AI-driven creativity.

1:40I hope you'll find the conversation as fascinating as I did. Can you start by introducing yourself, give your background, particularly your educational background, how you started Runway, how you built Stable Diffusion. Yeah, totally. So I'm Chris Valenzuela. I'm the co-founder and the CEO of Runway. I'm from Chile originally, and I moved to New York seven years ago to study at NYU, where Runway was our research work with my co-founders. We were interested in finding ways of training and creating large models for artists and creatives. And we started the company around 2018, so almost five years ago.

2:27And since then, we've published and made great progress on making sure models are accessible and empowering for creatives around the world. Filmmakers, designers, musicians that use runway to make anything from future films to short films. This is something I've been very passionate and deeply care about for almost now a decade. I've been working on media and research for a long time. So still on the way to go, but excited to see how we're changing and how many people are using Ranoi these days. Did you study under Jan LeCun at NYU? No, I went to art school at NYU, to Tisch. Oh, really? Yeah, I have a son that graduated from Tisch.

3:13Oh, nice. So Stable to Fusion, can you talk about how that came about? There are a few projects that kind of emerged around the same time, and I've never slowed down to figure out how they're related or not related. You know, there's Midjourney and Dali and, you know, various other models. So can you talk about how that idea started and what was happening in the underlying research that allowed all of these to emerge kind of in the same time frame? Yeah, absolutely. So a lot of the research and the things we're seeing these days on AI are not overnight success. These are things that has been on the works for years and sometimes decades.

4:13Everyone is building on the shoulders of giants and building on other work and open source research that has been published for a lot of years now. I think particularly over the last four to three years, we've seen an explosion in quality, consistency and overall resolution of image and video models. And it's been because of a few key advancements in techniques, but also more compute and more data has become accessible and available. So around three years ago, we started working on this idea of creating new kinds of models for image generation. And we do collaborate sometimes with other research institutions.

4:58So we did a collaboration with the University of LMU Munich in Germany. And that paper that we published was called Latent Diffusion that introduced a new technique for generating images on a latent space and conditioned those images to a variety of different inputs. That model and and that set of research eventually became Stable Deficient, which is just a larger version of the original latent diffusion model. Stable Deficient was an open source research model that we've released, and it's been one of the most, I guess, used and widely accessible models these days for image generation. There are other techniques also generating kind of like great images and great videos.

5:40And so we're also working towards making sure that we continue to just push the boundaries of how you can create and the kind of outputs you can make it up these models. Can you talk about diffusion models first of all? I've never actually talked to anybody about what a diffusion model is and how does it relate to transformer models? It's a different research stream although sometimes they overlap on some ideas and a lot of the improvements on video and image quality this day comes from actually transformers-based video generation and image generation models. But diffusion comes originally from a few concepts from like physics and thermodynamics.

6:20And the idea is basically to train a model that's, first of all, adding noise to an image. You have an image, a set of specific like references of pixels. You start adding noise to that, and then you train a network on being able to remove the noise from that image. And you do that millions of times, and the model gets really good at understanding then how to start from random noise, just a bunch of different pixels, and convert an image or create an image or generate an image out of that. It's a process that's been becoming more, I guess, widely used these days because the results have proven to be much better than previous techniques.

7:03Previous techniques were much more like they were using things like guns that were proven to be good. But the thing with guns is they're expensive to train and to use. And they're not as customizable and easy to move around or use as the diffusion models are. But again, I think the diffusion perhaps won't be like the last researchers we use to improve the quality of models. And we already have seen this where transformers have proven to sometimes be better than diffusion for some tasks. Yeah. Is DALI a diffusion model? I think so. DALI is a model that hasn't been released and open source, so it's hard to look into underneath the hood to understand how the model was strained and created.

7:48But you can, I guess, a safe assumption is that there's a diffusion part of generating an image in something like DALI for sure. There might be also a transformer kind of architecture at some point in that model as well. And mid-journey is also a diffusion model? We don't know also either. Everyone has different techniques. Those are different products from different companies and everyone can have different approaches to how to solve it. You can have educated guesses and time to approach if it's a diffusion model, a transference model, but they might be using some technique here and there that might be diffusion inspired or diffusion based.

8:24Are there any other open source models for image generation like stable diffusion? Yeah, there are a few models that the open source community has been putting out. I think there's definitely a lot of interest in trying to improve the quality of open source models. So yeah, there's a few models from different teams and different initiatives that are trying to... Sometimes, you know what happens? There's a lot of not just pre-trained models or baseline models, but also fine-tuned models. You take an existing, for example, stable efficient model and you tweak some parameters, or you fine tune on a particular data and that model gets released.

9:03And so there's actually like marketplaces where you can use some of those pre-trained and fine-tuned models. And that's becoming a widely kind of like a successful effort in open source and community building because it doesn't require the amount of resources and compute and time that creating the first baseline or pre-trained models in the first time, in the first case, like required so it's much easier for someone with like not a lot of resources or maybe just a few gpus to train very uh impactful or great models in terms of quality stable diffusion is the largest open source community in this space is that right so stable diffusion is an open source model um it's it's uh just a research and a paper and sort of weights and code that's been released is people have taken that model and have kind of like tinkered, modified, inspired by it.

9:57No one really has any particular saying on how the community grows. It's more of like the community just evolved on its own and people are building on top of that, for sure. That's how open source works. And that's how open source works for any project these days. And that's kind of the idea. You can build on top of each other and basically find and discover new things. Are there other models out there that are open source that have licenses available? I'm just trying to understand the landscape. There are the proprietary models like DALI and Midjourney, and then there's Stable Diffusion, which is open source, and then these licensees that have fine-tuned models.

10:42Stable Diffusion is an open source project that a few researchers, including Runway, collaborated on. That's an open source model. Runway creates and builds their own models. We don't have a membership. We have nothing to do with any other company. We're just building on top of Runway's like research. And so we have our own preparatory models. We have Gen 1 and Gen 2, which are video models, for example. Those video models are not based on any previous architecture or open source models. They're based on Runway's own research. And those are like set-of-the-art video models, for example, that are able to generate up to like 18 seconds of video.

11:18And those are in the same way as you were describing DALI or Mid Journey, those are models that are running on their own. There's an open source community of people who have taken some of the research we've done, but also some of the research other companies have done and have used that to build other companies or projects. And I think that's kind of the great part of research that you can have both ends, closed source and open source. And to be honest, I don't think there's a there's a peak where you have to be 100 percent of one or the other. we've been in the middle. Sometimes we open-source stuff, sometimes we don't.

11:50But I don't think of it as a binary state where you either open-source everything or you don't open-source anything. I think for us, what has worked really well in terms of research and community building has been open-sourcing some things like we did with some versions of table diffusion and latent diffusion and not open-sourcing others like Gen 1 and Gen 2 that have not been open source. And so it's a different strategy. And I guess we've chosen to be kind of like in the middle. Right. And those, the two versions of Runway that you're talking about, those are built from scratch or Runway's original product before you got into video.

12:30Was that built on top of the open source model? No, we build, yeah, we build all of our products and research in-house. We have our research company. We have a research team that's been working on these problems now for a few years. And so we build on our own kind of like stack. We don't use third party models. We don't use like APIs. We kind of like just work on the core fundamentals of our product and our research internally. And so I think like 85 percent of our team these days is just research engineering, advancing the training of these models and also the user, the use cases and the fine tuning of those as well.

13:06Okay, okay, that's clear. So what I really wanted to talk about is video generation, these models, and you're doing it. There's a company called Pika that's doing it. Are you both using similar architectures? Are they completely different? And then, of course, Google and OpenAI, everybody is coming up with their own version of this. So can you talk about the underlying research for video generation? Is it a completely new stream of research? Is it built on top of the diffusion architecture? It's definitely building on some of the findings that people have found around diffusion models, but also transformers and other techniques for generating images.

14:03Video presents a different problem, which is temporal consistency, how you make sure the models and the videos are consistent across different seconds of video generation. And so making sure that you can close that gap by creating new architectures has been kind of like the core fundamental research challenge these days. You want to make sure these models understand how the world works, the physics, interactions of objects, how characters move in a scene, and that's critical to make good quality videos. Every company might take their different approaches, so it's hard to define or tell that everyone is using one particular thing.

14:40This is more like you know, it's like manufacturing cars. People, you want to build different cars and different cars can have different logistics and factories and the cars that might come out of those factories might also be different based on what you're trying to like create in the first place and so we have a specific manufacturing facility and we make a specific version of our car but other companies might want to try different techniques to create those cars and also output different cars and so really for us it's about improving our own models and kind of like creating the the best car experience that we want for our customers and that kind of like metaphor analogy of cap manufacturing.

15:19But there's no one single way of building a car. It can be different depending on what you're trying to do. Yeah. And right now the videos generated are very short. I've seen longer form of videos, generated videos that are very hallucinatory, you know, dreamlike, make the things sort of morph into other things. Is that what happens if you run the model that you have long enough? Or, I mean, is there, why are you only doing, you know, a few seconds? So you can create up to like 18 seconds of video these days with Runway and probably and very soon longer videos. It all depends really on the artistic like direction they want to take it.

16:09people you might want to create more surreal videos or more uh realistic videos and that's kind of like up to the creator of how you want to take it um but really like it's not limited to time you can always extend that video for longer the way you create those videos though is you can start with chunks of four seconds and then eventually get too longer so you might have seen specific versions of those videos in particular moments in time but yeah you can create way longer videos if you want. What I'm talking about, these videos that people have produced that are very sort of psychedelic because there's no consistency, is that coming from, I mean, does Runway produce those?

16:58What is going on in those videos? Yeah, it's hard to tell without like seeing, I guess, watching the videos that you are are referencing too. There's a lot of different techniques to generating video. Some of them are more surreal, more psychedelic than others. Some people are actually creating just models to create psychedelic videos and that might not be using Runway at all. So there's no one singular way of creating videos in the first place. You can think about it really, a good analogy is to think about it as a camera. You can have different cameras to create different types of videos and scenes and it's up to you how you want to like create those videos using one particular camera and cameras might work differently depending on what you want to make.

17:42But with runway you can make it so it's very flexible so you can make abstract shapes and surreal objects all the way up to like very realistic and almost live action kind of like shots. Yeah and so consistency is the issue right when in as you as you... Temporal consistency, which is, I guess, how do you make sure objects pertain and maintain their consistency across different seconds of videos. And how do you do that? And can you talk about the underlying architecture of your model? You said it's kind of a mix of diffusion and transformers. And just to give you an idea of where I'm going, you know, I talked to this company, Wave AI.

18:28I'm sure you've heard of them. They're actually an autonomous driving system. But they have a world model. And that's why I asked about Jan LeCun because he's, you know, done a lot of research with video and world models. But they have a world model that they train, and then it can plan and generate scenes in a representation space. And then they can decode that into pixels, and it's remarkably consistent. I don't know if you've seen it. But are you working with world models? Are you doing something different? Yeah, we're doing the... So we have a current effort called General World Models that it's trying to solve, I guess, something on a very similar spectrum, which is not applying to self-driving cars.

19:30So the problem with self-driving cars is how do you have systems or cars that can navigate very complex physical situations to the point where they can drive themselves to avoid objects or kind of like situations that are arising on the street. The way you train that and the way you can do that from a world model's perspective is you don't decode the rules of the system. You have a system trying to figure out how the world works and how objects can fact predicting what's going to happen next. That's kind of the key aspect of it. If I throw a ball to the air, I can kind of predict, because perhaps I've seen enough balls going to the air that the ball at some point will start going down, right, because of gravity.

20:16For a model that's really hard to know, like the model hasn't been hard-coded to know how gravity works. But our belief and thesis is if you train models large enough and you give them enough data and with the right kind of parameters, you can actually try, the model will try to learn that. And so world models are models that not only look at particular videos or images or type of data, but they're actually trying to predict what's going to happen next. So transformer models are great for language because they basically do that. They predict the next token in a sequence of words. And by doing that, they are able to generate coherent language.

20:52In world models, you want to do something similar with video frames, for example. You have a video frame, one particular frame, you want to predict the next one, the next one, the next one. And you do that sequentially until the point you can understand what's going on, occlusion, movement, gravity of other objects as well. So our architecture behind the scenes is trying to leverage that idea. A big component of it is just large, very large models in the first part, but also training them in a way where all of these things do make sense for the model itself. Can you talk about the level of compute necessary to train these models and how large they are at this point and how large you want them to be?

21:33what the roadmap is? They're very large and they're very expensive. There's a lot of compute that's needed to train these models. I think one of the biggest expenses in research is just compute because these models are just very large and training them requires vast amounts of compute power. And so we're talking about thousands of GPUs sometimes or even more, tens of thousands, to make sure that the models perform and work at the kind of level that you want them to perform. I think that's only going to be, to be honest, like continue to increase over time. More computer will become more needed.

22:08At the same time though, there's always going to be optimizations and tricks that both chip manufacturers but also software engineers and kind of like the software layer can do to optimize how these models are trained so you can squeeze even more power out of existing GPUs. And I think both efforts are happening at the same time where more GPUs are being kind of like put into the market and at the same time, more optimizations and better techniques for model training are going to be kind of developed and are developed. And so by putting both together, you're going to have larger and bigger models come up.

22:40Hi, good tech solves problems you know about. Great tech solves problems you haven't even thought about. What can the commerce platform trusted by millions of merchants do for you? It's time for Shopify, the commerce platform revolutionizing millions of businesses worldwide. Whether you're a garage entrepreneur or IPO ready, Shopify is the only tool you need to start, run, and grow your business without the struggle. Shopify puts you in control of every sales channel. So whether you're selling satin sheets from Shopify's in-person point of sales system or offering organic olive oil on Shopify's all-in-one e-commerce platform, you're covered.

23:23Shopify powers 10 % of all e-commerce in the United States, and Shopify is truly a global force, powering Allbirds, Rothy's, and Brooklyn, and millions of other entrepreneurs of every size across over 170 countries. Plus, Shopify's award-winning help is there to support your success every step of the way. Sign up for a$1 per month trial period at shopify.com slash ionai. How do you measure these models? Do you measure them by parameters the way you do in LLM? Depends what you're measuring. I think if you're measuring consistency and quality, you can rely on benchmarks and evals to make sure that models perform as you want them to perform.

24:13Although I'm less a fan of benchmarks and evaluations. and much more of a fan of quality with humans. If you want to see how good the model is, it really doesn't matter how many parameters, how many billion set of parameters the model has, or how long it's been trained. If you want to see how good the model is, just show it to a filmmaker and have them use it to see what they can make with it. And that's going to be a much better indicator of if you're making a good job or not, if they like it, at least from a visual standpoint. That's kind of the goal of making these tools for artists in the first place.

24:46I would presume that the larger the model and the more compute applied, the more data and the training, the better the produced video. Exactly. So the larger the model, both on parameter count and the larger model on the data side as well, the better the model will come and the better the model will work. So on parameter count, what is your largest model? I can't really go into too much of the details, but it's on the billions in terms of size of parameters. Yeah.

25:21It's not saying that much, Cristobal, because everything's in the billions these days. But yeah, and in terms of this, you're using GPUs, I would imagine, for the training? Yes, we're using mostly GPUs. These are H100s and A100s for the most part. Yeah. And can you give an idea how many GPUs you have involved in the training? I can go into that level specifics, but a lot. We have a lot of GPUs. Yeah. And so where is this going? You know, you've made a lot of progress. There's some other people in the market that are making progress. Where on the curve are we? I think it's early, you know, I think it's very early.

26:07We're although in the midst of a major change in media, I think we are about to see a new kind of media that we haven't seen before. I don't think we're going to use the same word to describe it. So where we're speaking about right now filmmaking and movie and movie making, we're probably not going to use the same word to describe the type of things that you can make with these models in a couple of years. And that's kind of like a very interesting point because technology will create a new art form and we've seen that in the past. And so perhaps the Tisch schools of the arts in the future might be teaching something which just don't have a name for it yet.

26:43In the same way that perhaps filmmaking for people 150 years ago would have been like unthinkable off. You can imagine that someone will study something like that and now it's like of course an institution and something that's well known. And I think that's the interesting and more compelling part for me that technology is moving so fast and so quickly and in such an affirmative way that it's not going to only change the way we see the world, but it's also going to change the way artists interact and tell stories of the world. And that's going to create on its own a new art form. And so, yeah, it's still the very beginnings of that transformation.

27:21And so right now, most of your users are commercial advertising firms. I mean, who's using Runway for video creation right now? Yeah, we have everything from directors, producers, artists, filmmakers, production teams, people in Hollywood working a lot. we have musicians, we have advertising companies, and also creatives and creators, independent creators, people who are just making videos for fun or for a living. It's really anyone who wants to make and tell stories. That's kind of the goal of building Runway in the first place. And it's a wide spectrum of creatives that spans from professionals to casual creators.

28:06Yeah, we have what we call like a streaming website where you can watch things people have made with Runway and that's open. So we can also follow up if you want to just see anything people have made. And there's also in the application itself when you want to generate, you can also watch and kind of get inspired by what other people are making. So by doing those things combined, you can basically get a lot of inspiration to not only watch, but also create because that's kind of the point. You are now in control and you can not only view a viewer, but also you can be part of the creation process of making video like that in the first place.