Building the future of VFX and video editing with Runway CEO Cristóbal Valenzuela | E1724

0:00Hey, everybody. We're back with another awesome AI interview as part of our Innovators in AI series. That's what we're calling it here. We've never seen technology move as fast as artificial intelligence is currently moving. This is not hyperbolic. This is reality. Every week, every 48 hours, somebody builds on the work of an open source project or another company or by doing prompts or by connecting things together. So we're going to dedicate a large percentage of the show to AI this year or until this thing slows down. So today I'm joined by the co-founder of a company called Runway. Runway did some of the earliest research that eventually led to the creation of Stable Diffusion.

0:42That's the project that takes text and turns it into an image, right? You type in, I'd like to make a painting in a Renaissance style of Donald Trump in a courtroom, and then all of a sudden you start getting really amazing output. So the language that we're going to use to program going forward is going to be called English. So the CEO of Runway is going to walk us through. And that's really what I'm trying to focus here on the show in 2023, actually using the tools and seeing them in action. So you don't just get to hear the opinion of some podcaster who's not using the tools. I'm using the tools, my producers are using the tools, and the people on the podcast are building the tools.

1:17You're going to actually see them at work. This is why you come here to This Week in Startups is to get inside information. And what Runway is doing is merging generative AI with visual effects, VFX. So the visual effects team that won Best Picture for Everything, Everywhere, All at Once, a really dope film, they used Runway to help build the images that you saw in that Best Picture winner. Also, Late Show with Stephen Colbert, they're also using this tool, Runway. And many other brands are doing that. So it's an amazing interview. The founder is really brilliant, shares a ton of practical examples and demos.

1:51You're going to love this one. Stick with us. This Week in Startups is brought to you by Clumio. When you're building a company, don't let backups and compliance requirements distract you. Let the data protection experts at Clumio help with immutable air gap backups that put compliance on autopilot. Visit them at clumio.com slash twist to start a free backup or sign up for a demo. Squarespace. Turn your idea into a new website. Go to squarespace.com slash twist for a free trial. When you're ready to launch, use offer code TWIST to save 10 % off your first purchase of a website or domain. And the Microsoft for Startups Founders Hub helps all founders build a better startup at a lower cost from day one.

2:37Startups get up to$150 ,000 in Azure credits, access to free OpenAI credits, free dev tools like GitHub, technical advisory, access to mentors and experts, and so much more. There is no funding requirement and it only takes minutes to join. Sign up today at aka.ms slash This Week in Startups. All right, everybody. This AI stuff is moving faster and faster. So here at This Week in Startups, we're going to have somebody on once or twice a week who's building something interesting in AI. So consider this week in startups, this week in AI for the next couple of months or until this crazy, crazy pace slows down.

3:25Today, I'm joined by Chris Valenzuela. He is the CEO and co-founder of a company called Runway. It's an AI browser based creative suite. You can check it out at RunwayML.com. there's just 30 different tools on the website photo editing video editing auto editing gener of ai instant visual effects like uh removing the background on a photo text to color grading audio silence removers all that kind of stuff he started runway back in 2018 before gener of ai even had a name chris welcome to the program thank you for having me here all right right after graduating nyu in new york city my hometown you started building this uh and then things have gotten pretty busy it's a sas based company it's a free tier but after that you pay 12 to 28 bucks a user per month and not dissimilar than i think the pricing of canva or the adobe creative suite and you've got all kinds of people using this including uh the late show with stephen colbert likes to use it, I guess, to make interesting video graphics.

4:41Explain to us what you've built and how it's going. Yeah, happy to. So Runway is, as you were saying, a company that's built a lot of products and offerings for the creative economy. But at the core, we're actually a research-driven company. We got started around 2018. and since then we've been publishing and really pushing the space on the generative AI landscape for both image and video and so very proud of being pioneers in researching that space with things like stable diffusion and latent diffusion and Gen 1 and Gen 2 which are what we consider very important models that have driven a lot of the excitement that we see nowadays and so we do that research we do the fundamental research for those models to be built those foundational models and then we build products on top of those those models ourselves um so it's been a quite a journey since we started yeah how many how many people are using this product now because you've raised a ton of money i noticed uh you recently raised wow 50 million at a 500 million dollar valuation so so how many people are using this tool how many people are paying for it we have millions of customers all over the world that range from professional post-production films and award-winning directors to ad agencies and post-production teams and small creators as well.

6:09It's everyone really who wants to tell a story with video can be and should be using Runway. So it's millions of users. Most of those would be free, I would assume, because that's a large number. You've got tens of thousands of people paying for this, something in that range? We have a large amount of paying customers. We have enterprise deals and subscription-based offerings. You can also pay for usage of the platform so you buy the credits to render video as well. So yeah. So maybe you could show us and I'll sportscast it a little bit so maybe you could show me. I know you have some new features coming out but maybe show us what it does here and then sportscast it a bit because most people listen to podcasts as you know and describe what they're seeing on the screen.

6:57I will. So here's a research work that we've been working on for the last couple of months. It's a state-of-the-art video generation model. Gen 1 and Gen 2 represent our family of video generation models. This is a way of rendering and creating video in completely new ways, in ways that haven't really been possible before. So let me walk you through what Gen 2 really does. There are eight different modes of using Gen 2 and Gen 1. One of the most perhaps interesting ones and different ones is that you can generate video using a text prompt. So you might come across or have seen like text generator, image generator models and kind of like models that are able to create images out of pure text.

7:46This is very similar, but instead of generating one single image, generate a continuous video frame, right? This can be of anything. you can basically type anything you want and you'll get you'll get a consistent video in that style so i'm just showing here uh an aerial footage of a mountain range and we basically see that that shot um a mundane with some kind of like lights there's the sun on the on the back you can see the consistency of the frames kind of applied here um and then that was generated just with that one sentence or did it take that was no just that one sentence you just input a sentence and you get a video um and so you can use sort of how long would the video be uh is it like limited to a five second or ten second video no right now we have 15 second video clips uh support um so we're working towards uh allowing our customers to do more than that uh but but but yeah 15 seconds will be available very soon in the product um and so you can you can use a sort of different input mechanism so the one i was telling you here is like text text to video is another one so i'm just uh using the prompt of um of um late afternoon sun peeking through the window of new york city loft you basically get that representation on the right but then here's interesting here's an interesting combination of things you can you can drive the video generation process with a combination of a text prompt and an image so in this case what you're seeing here is we call this character mode.

9:14You can input a video of your an image of yourself or anything at all. And then describe how you want to animate or create a video out of that single image using a language description and basically you grade that video base of that. Then there's also a variation of this one it said just so we read the text here you uploaded an image of a dapper looking gentleman with a very nice mustache and a wispy hair and it says that was also generated by the way. oh so that was a generated image and then you said of you upload that generated image and say a low angle shot of a man walking down a street illuminated by the neon signs of the bars around him and we get what looks like uh you know a very um uh diffuse background of lights hanging in the air almost like he's in hong kong or something on an alleyway and he's walking around it's a little bit weird a little jerky but it's pretty impressive for i guess is this a 1.0 of text to video generation yeah this is the first iteration that will make public uh next iterations are going to come over the next couple of months so this wouldn't replace a movie that or a tv show but it would certainly be an amazing storyboard or a piece of collateral to start with i think right now it definitely can be the the the collateral or a b-roll or some sort of a storyboarding tool but photorealism and consistency and quality of outputs it's only a matter of time and we're working towards getting that really out of the door as soon as possible again it's think about this as like early versions of gpt one or two right you're seeing you're seeing earlier iterations and early models that are able to render this and do this.

11:03These models didn't exist just a couple of months ago. Are you trying to build the next Uber or Robin Hood? I hope so. Well, here are three important tips for you. First, make sure your data is protected. Obviously, remember, just because your data is in the cloud doesn't mean it's automatically backed up. Your data is your responsibility. You got to make sure it's backed up. Second, don't let backups and compliance requirements distract you. Let your engineers focus on product innovation, not writing scripts for compliance audits. And when it comes to backups, the major players have major limitations like snapshots.

11:36If your account is compromised, your backups are compromised. But Plumio stores your backups on a different server. So if your main account is compromised, your backups are always safe the way it should be. And third, take control of your cloud costs. You'd be surprised how much of your storage costs come from backups, snapshots, versioning, and replication, or maybe you're maybe you wouldn't be surprised because you've seen these jaw dropping bills. Well, Clumio can help you with all three of these important points. They provide turnkey data protection that is air gapped, immutable and cost optimized.

12:08Clumio has saved customers over 30 % on backup costs while putting security and compliance on autopilot. Visit them at C-L-U-M-I-O dot com slash twist to start a free backup or sign up for a demo. That's Clumio.com slash twist. How long have you been working on this text to video just to give people an idea? And then how many people have been working on this? Is this based on an open source project that you're building on top of? No, this is entirely our own research. We've been pioneering research in this space from a long time. So we are the authors and the creators of a model called latent diffusion that was then turned into stable diffusion with creators of a few audio based models as well.

12:52And so really we've been pioneering... Isn't Stable Diffusion an open-source project? Yeah, an open-source project. We open-source and release with the University of LMB Munich in Germany. Oh, so you were part of that original project? Yes. So here are the authors. It's LMB Munich, Heidelberg University, and Runway. Those are the three main organizations behind Stable. When did Stable Diffusion... And you said there was a name for Stable Diffusion before it was called that? So this is the paper. So you can consider this the origins of that innovation, right? High-resolution image synthesis with latent diffusion models.

13:26And so latent diffusion was actually the short to describe this approach, which is the technique, the research behind this. Got it. What is latent diffusion models? Explain to us technically, for non-technical people, what this actually means. And when did this research paper, this was written in when, 2020 or something? 2022. oh okay this is last year yes it's uh i mean the public how fast this is moving yeah uh the paper was released and published on april uh 13 of 2022 um subsequent iterations of a model have been released since then the model has been open source and as you know has taken a life of its own uh latent diffusion introduces a different technique to generating images using a efficient-based technique in the latent space.

14:17That's basically, in short, what we've improved and introduced here. And it's a very generalizable approach that can be used for different techniques, not just image synthesis, but in painting and using semantic maps for driving those image generation processes. It was pretty much a very, very unique approach towards tackling the speed and the consistency of image generators out there. and I'm sure you've seen the kind of like wave of creativity that this model and then stable diffusion has created. So yeah, I'm very proud to be to be part of that transformation. So were you did you help write that paper or you started building the open source project on top of what was learned in that paper?

15:01No. So the paper again is the collaboration between two research organizations runway and the University of LMU Munich. Our research scientists together with the the research scientists at LMU kind of like pioneered their approach. That was driven by Runway, which was, I guess, the company we founded four years ago, exactly to drive these kind of like research outcomes out. Got it. And so in 2018, I came out to NYU, did research there, started the company right out of school, and then started building the company, raised money, started building the research efforts here. One of the many publications we've done was this, that it's, I guess, one of the most well-known.

15:40So you have this latent diffusion, then you have Stability AI. That's another company that kind of leveraged this open source project to build generative video and AI products? so once this diffusion model was released multiple i guess the model took a life of its own and we've seen all sort of different companies building open source solutions on top of it which is great i think that drives a lot of the field forward in very positive ways uh but as the kind of like original authors of the paper and the work and the research and the engineering here uh we always kind of like focus on improving and continue to push the boundary of like research and that kind of like manifests itself in our family of gen x models now so this is for us the next frontier in video generation that takes a lot of the insights from from our image generation process from before and this stuff is starting to make its way i had mentioned stephen colbert before but maybe we could pull up the image you used runway to make a music video sorry commemorating the anniversary of the american dream ice cream by stephen colbert uh and i actually have the url of it here i don't know if you have that uh but oh you probably have it in your customer stories so how does one get a customer like steven cobert how did that happen and then what what is the reception you got from hollywood uh because it does seem like these this is a very clever beachhead market people who are doing late night tv shows they love doing skits but they don't have a huge budget they're building this stuff in real time so it's not like they're going to you know be able to afford to go to industrial light and magic and make something so did you do this for them or they did it themselves or did you send some people over to their offices to teach them how to do it they did it themselves see it that'd be great yeah it's uh they did it themselves which is funny that the story about how the team behind the little show started to use runway was basically we don't we don't have marketing ourselves so everyone who comes to runway comes mostly via word of mouth um someone inside the team here started to use one of our tools we have around 35 different tools so the tools i mean telling you and showing you gen one and gen two are one of the many tools that we have we have one tool for uh rotoscoping which is a traditional like frame by frame task that editors do and it takes them like a lot of time like hours if not days of work in runway you can do it in like i'm not kidding like a second like literally one second one of the editors started to use it for the show and eventually like pick up the entire team inside inside the the Colbert show and now they've been using for a bunch of different like edits and they've taken and this is this is true in the interview that I'm showing here six hours of work combined into six minutes and that is the power of like this kind of like technology when you can automate and go ahead and what we're seeing here is a dancing a pint of ice cream with a singer in front of it who looks like um is that a real singer this is katie perry yeah oh it is in fact katie perry and then some people in shark outfits so what's real here and what's fake so the ice cream uh it's it's fake that's been that's been added in post that's the visual effects component are the sharks real the sharks are real yes but this is what's mind-blowing about this i'm looking at it with you and i'm like that looks like katy perry but because i knew this was computer generated i'm like is that a computer generated katy perry i can't know no no this is an actual shot were real and the ice cream was real and then everything behind them was fake but the sharks are not real the dancing pint of ben and jerry's isn't real and then obviously the pineapple trees behind it that's crazy so just to be clear just to be clear everything here is like real the only thing that The magic here is you combine all of those real elements into one singular shot that you believe is happening in real, right?

19:45But the Ben and Jerry shot is shot somewhere else. Katy Perry shot somewhere else. Got it. Oh, okay. So those are elements. And then you said mix them together. Right. But mixing them together takes you a lot of work because you have to go frame by frame matching. And this is the work that editors, this is the work that BFX studios are spending hours of work doing, right? This is why green screen success. You shot in a green screen and then you take the actor out of the green screen, right? Amazing. If you don't have a green screen, then you're bounded to like hours of work to recreate that effect.

20:17And that's where like our tools come in and just help you do that, right? Got it. This was from the Super Bowl. It was a Super Bowl performance. Years ago. She did something like this years ago. So this is all stitching that together. Exactly. So this is amazing that like you are getting to the point where the elements are just hard to determine what's real and what's not. And you said this takes them a couple of hours to do this when they have those elements already. Yes. So they went from six hours to six minutes of work, which for any filmmaker or editor really, really makes a lot of difference.

20:51If you can tell from the podcast lately, we've been doubling and tripling down on Founder University at launch. In fact, it's basically the future of our venture capital firm. And that's awesome because I'm working with a couple of hundred early stage founders really early and getting to see what tools they use. You know what tool they show up with most? They show up with Squarespace. They put up their first website instantly, quickly with Squarespace. And it's beautiful and it makes them look like a million bucks. The thing you may not be aware of is that Squarespace, beyond the beautiful templates that make your company look like a million bucks and that work on mobile.

21:26It's not just a pretty website. It is a powerful e-commerce platform now. and they have member areas. What's a member area? You know, people like to sell content now and premium content. It's a big business. Well, they have that built in to Squarespace and they don't take, you know, double digit percentages of your revenue like those other platforms do. And they also have appointment scheduling. So, you know, if you're doing a business where you're a consultant, you want to charge for your time. Well, you have scheduling built into it as well. And this is the brilliance of Squarespace. It's going to look beautiful, as you know.

21:57So here's what I want you to do. Just head to squarespace.com slash twist for a free trial when you're ready to launch use the offer code twist you save an extra 10 off your first purchase of a website or a domain we love you squarespace you know how it is when you're a technologist everybody in your family your friends your circle your network come to you and say hey i gotta get a website up can you find me a developer a designer a product manager and you just say you know what yes i can find you all that and more at squarespace.com slash twist you mentioned film and hollywood um and which which brings me to like actually uh our like film festival that we hosted in new york and in san francisco just a couple of weeks ago this was a celebration of films and that are already being using or have incorporated ai techniques into the process and we're not talking about like for the types we're talking about like real filmmakers who've been working on film for for years we had darren aronofsky come in to the panel to chat about darren aronofsky came yeah yeah i hosted in 19 in the 90s i hosted a screening when he was broke for a film called high which i think he shot on he shot it on eight um eight millimeter i think and he was broke and i met him at a party and i said wow i really want to see your movie and he said uh oh yeah i love your magazine jacal and i said uh i'd love to host a screen he said would you host a screening for it with like silicon alley tech people i was like sure how do you do that he's like you rent it there do you pay for it and you invite people i was like all right i'll do that for you and i spent like 1500 bucks renting it there and having drinks after and he was so kind darren aronofsky he came and he hung out with us and the star of the film and we just had drinks and we did that over on avenue b and like 12th whatever that movie theater over there was it was quite fun so he came so he's obviously into this these are short films that were made these are short films we we premiered 10 short films that were made in a combination of either entirely generated ai techniques or analog we generated these are films made by either professional filmmakers new filmmakers and really like just showing the quality really like it's just mind-blowing here the films are already accessible in our website so if you go to runmel.com you might you might get a sense and a chance to see them all which is which is definitely what i encourage you which one won did one win the yeah we had a palm door yes the grand prix that's how we call it there was a grand prix a gold a silver honorary and then merits uh the grand prix was an incredible film by ricardo fucetti who's been working on visual effects for some time now called generation the first time i saw the film it just like it got goes it's on it's on the website can we just hit the play button let's yeah let's watch it okay so we're watching this i'll sports cast it okay it's got a little warning here that could be flashing images we're watching this you don't have the audio that's uh that's yeah i don't need to have the audio it's good for us to sort of describe what's happening here but it was a close-up of a human now there's a human dancing on the floor in uh you know essentially some sports uh bra and uh shorts and then their body is turning all kinds of different crazy colors i'm assuming this is a human that's actually been shot on video or is that a computer-generated human that that's a that's a real human then now you're seeing yes and now you're seeing there's a there's a voiceover that's kind of like telling a story about evolution behind the scenes so you can hear that right now but there's a music that's uncompanying this and now you're seeing this human being turned into some sort of animation and like graphics are merging with the body of the dancer wild those are all generated and so that the the ends the the the aspects and the effects here were all driven by um by ai generation kind of like techniques got it so they found a modern dancer interpretive dancer there or just a person who's uh really high dancing like a maniac and then put all these incredible effects on it which are truly stunning and it's hard to look away and if this was made in the 80s or 90s or let's say the 90s for mtv uh this would have cost a million dollars in graphic effects it's like this outdoes you know any peter gabriel computer effect you know sledgehammer or whatever that was claymation but you know all these different things that were done in the 80s and 90s for music videos this would be the number one video of the year yeah definitely definitely and this is only the beginning i think this is this is technologies and tools that are only accessible for the last couple of years to be honest but but to get to your point on hollywood these are tools that the industry has already been embracing And we're actually working really closely with filmmakers themselves to incorporate more of this AI revolution into their processes.

26:33We have a division of Runway called Runway Studios that partners with filmmakers and with film studios and with filmmakers to really help them understand the power of GenoDVII for film and for video generation. And so we work with a few films already, with the Berlin Film Festival, with a few films in Barcelona and a few other places. This wasn't possible two years ago. oh this is really impossible like yeah actually the tagline of our companies make the impossible we're making things that were impossible just a couple of years ago uh so yeah like two years ago literally this wasn't on the table maybe even 18 months ago this all started in the last 18 months we've been working on this for the last four years i think but but more importantly i would say the last six eight months has we've seen an explosion on the quality consistency and overall understanding of the technology and i think that that really helps a lot um so definitely it's been a lot of work behind the scenes but uh something like gen 2 which is a text to video model was yeah literally not possible a couple months ago all right so some questions here i think are very important what does it cost to make a music video like that in terms of what you have to pay your company so what did runway make in fees in software fees to make that dance video we just saw which was super trippy and would have won video of the year at the mtv music video awards in 94 um so there's there's the cost of training a model and so we have a large research cluster that uh we use to basically train these models from scratch first of all invent them and then train them.

28:10Then there's the cost to me as the director. Oh, the cost really depends, like depends on like the team that you have and the scale of the ambition and the idea that you want to like execute. But let me give you an example of like, I'm just saying the software costs, if I had five people working on this, and we use your software. It's 15 bucks. So it's just 15 bucks a month. You get this is what I'm sort of getting at. There's all the amount of money you spent, I'm assuming you spent millions of dollars building this model with dozens of people uh yes we're 40 40 40 people in the team so 40 people and millions of dollars invested over four years or tens of millions let's say low tens of millions um to now make a tool that can make music videos that in 1994 would have won video of the year and the person who made it probably gave your company 20 bucks a month or if they had 10 people on it 200 bucks a month that's the power of dysphagic technologies just make anything everything more convenient and easy to use all right so at the pace this is going you happen to know a lot more than myself and the people in the audience because you're actually building this at the pace this is going certainly you're aware of what john favreau has been doing with the mandalorian with those 360 degree screens that they use as the background you know this technology i'm i don't know the name of it um virtual virtual production okay virtual production so describe what virtual production is and what has led to you know disney with star wars being able to make without sets incredibly compelling products like the mandalorian in a very short period of time at a much smaller budget and then compare where you're at and when would these two things be indecipherable for myself and my daughters how many seasons out we're on season three or for the mandalorian now how many more seasons how many more years until these two things are basically um indecipherable um i'd like the exact number of years please but you don't have to go to months with days or seconds as well i can't expect just years i think i think like it's always good to look at like the pace of of research in a scale so if you if you see gen one and Gen 2, video generation was really hard to imagine in the quality of results that we see today just eight months ago, right?

30:39So eight months ago, we went from like, oh, you can do this, to hey, now you can render photorealistic video, and you can type anything, you can just give me a prompt and I'll generate it for you. I think we're still a lot of ways to improve these models, and we'll see that over the next couple of months. I think the next 12 to 18 months, will see a big step up in quality of and controllability of these models over time um and so we're not far away from from from getting a an entire mandalorian made with general dvi for sure five years three years oh definitely less uh less than five less than three yeah for sure no really indecipherable so so here's a hint you you're not you're going to be able to generate you're going to be in the main mandalorian character right the fact that you can render and generate films on demand will allow you to generate any film that you ever wanted to generate so that's an interesting also position to be in where the type of films that you're consuming right now are going to be very different from the films you're going to start watching in a couple of years because they're going to be made very differently you are going to be the main actor in every film video scene that you ever imagined and also you can generate every film technically you will be able to generate entire films and so it's a very different i guess uh i guess position to be in because you know there's this incredible thing that occurred i don't know if you're aware of it you seem to be you're a fan of the film you're a fan of cinema of course huge fan of film huge fan of cinema you got a favorite uh favorite director or two uh huge fan of darren arponofsky i think that's yeah really you got a favorite uh was a requiem for you what's your favorite uh i think wrecking was was definitely a highlight uh uh the fountain was also like see i knew you were gonna go fountain fountain is just it's so good um and then more recently star in the fountain yeah i think it was uh yes and then he he decided to go somewhere else um that movie is just just beautiful and and then uh the well more recently i don't know if you've seen the well but oh the well was incredible it's so good yeah yeah i mean if you what's your what's your favorite well you know requiem for a dream is one of the most powerful films ever made yes um the fighter oh i'm sorry the wrestler rather the wrestler yeah is just extraordinary so you know he really can go two ways with darren uh there is the surreal and then there's the character performance right and and these if you look at something like the whale and the wrestler these are incredible performances uh in each one of those films and then if we look like something you know pie requiem and um the fountain these are very high concepts they also have great performances don't get me wrong but conceptually requiem for a dream how the storytelling occurred the fountain how the storytelling occurs or pie they're very big think kind of films right and they rely not primarily just on the performance of one exceptional person where you connect with them which is really what happened in the well with brendan frazier and um the wrestler with um mickey roar but yeah he's just an extraordinary filmmaker who's your who's number two on your list you got a number two uh it's hard it's hard to get favorites to be honest um i've tried to watch backwards from a film what are films that if you had to watch a film for the 10th time you'd have no problem watching it be excited to watch it the time yeah i mean a lot um i watched melancholia too many times uh really yeah it's a beautiful film as well um i have too many i think i mean melancholy what a pull that's a good of course yes that's a that is a another one of these like intellectually mesmerizing interesting uh films if you haven't seen it it's yeah and then the square i don't know if you watch the square uh no it was and and force majeure um beautiful movies as well yes yeah yeah fascinating look at least i got a movie review out of this i i'm a big ridley scott fan gladiator blade runner etc i like i like i'm a big ridley scott uh what's your favorite ridley scott oh yeah kira kurosawa guys huge you know for kurosawa i go both ways i love the film noir and i love the samurai genre um i love high and low and stray dog if you haven't seen the um if you haven't seen those two on the film noir side and then obviously seven samurai thrown of blood i mean it just it's all amazing yes it's just hard for people to watch those films because they're a little bit too slow and methodical and people want to look at their phones and they can't just sit it's like it's like somebody who goes to a museum and you want to take in a picture or the venus de milo it requires you to quiet your mind and absorb it for you know five but five minutes of your life which for in today's modern era to let your mind absorb something for five minutes straight let alone for 120 minutes and give yourself over your attention over to something this is very hard for humans is a really something's been lost um yeah i do think still that um a lot of the best films are yet to be made i think we're one of the things that are going to be really interesting about United AI is that you're going to start seeing stories from filmmakers that never had access to the tools that only a handful of filmmakers used to have so that from a democratization and storytelling perspective it's interesting to see from that side which is we've loved cinema with all of these directors and filmmakers who really pioneer really interesting narratives and films yet I don't think we've come like even close to the level of creativity we'll start to see once you can put bfx studio kind of like tools in the hands of everyone and that that is like a really interesting um thing to have in mind as we continue to think about just generally in general i think this is the key observation and this is something i watched happen at sundance in the late 90s in the early 2000s when i would go there you know and i grew up in new york and i was in new york in the 90s a digital camera the vx 100 or 1000 came out the sony you could record on digital cameras but blow it up to be on actual um film you blow up your digital to film this is before they had digital projectors in movie theaters so there was this path yeah where a guy named bennett miller created a film called the crew shot on this vx 1000 or whatever it was and uh darren aronofsky shot eight and then blew it up and then you know many other people uh wayne wang did center of the world and a bunch of films were made on digital cameras and it really did change who could make a film and then at sundance you started to see all these digital films come out uh because you could shoot an unlimited number of takes and you could do it in a more of a guerrilla fashion and you didn't have to develop film just at the at the end when the film was finished you would then send it to get printed up to film and now that's that's not even necessary so here you're going to have people who don't know how to use a camera and don't have the ability to access a thousand dollar or two thousand dollar camera be able to make something and that is just next level fascinating yes yes and that that for me is like the most interesting thing that we'll start to see coming out of this these models story you know and that was really the basis of Sundance was trying to find new voices to tell stories um people who hadn't had access uh and you know documentary whether it was documentaries or Napoleon Dynamite or Riding Giants or Super Size Me these films were just a different generation of filmmakers if it hadn't been for the digital camera it could never have happened no yeah it's just never been able to afford it it's a different set of affordances and creative possibilities And so you have filmmakers who really, I think a lot of filmmakers are just technical innovators.

38:55They're the ones driving the innovation forward. I mean, think about Darren. He's pioneering a lot of things in the space. And I think that combination of like art and science is what we're starting to see more specifically in Jornada BI. As more people start to get into the field and understanding the power of these tools. This is a new camera. That's the best analogy to think about this. These models would allow you to capture a reality in a set of expressive tools. in the same way that the camera did for us 150 years ago the level of output from a creative standpoint is it's like unprecedented and yet to be seen because we haven't had this technology before um but really thinking about it as a tool to like do those kind of storytelling i always go back there's um i don't know if you watch this beautiful documentary on disney plus for um about the story of ilm um i think that's in that yeah super incredible i definitely recommend that and And it's kind of like taking the position and the perspective on really understanding how the field of visual effects was born.

39:53It wasn't a thing until like a bunch of hackers and thinkers and artists just came together to make a film about Star Wars, right? And flying like starships in the space, right? Imagining that and physical was impossible until someone had to create it. And then from there on, like a whole set of like domains and films were possible. I think we're early stages of something very similar here. Yeah, I mean, it's so funny. I have a list of the films from 99 to like 2000 and that era. And it's wild to see like Harmony Corrine doing Julian Donkey Boy. That was a digital film. And right next to it, Star Wars Episode I, The Phantom Menace, one of the first major blockbuster films.

40:38Blair Witch Project was another one. Lars von Trier, The Idiots in 98. and uh the center of the world by wayne wang in 2001 uh tape by richard link letter and you're starting to see like some of these very steven soderbergh did uh full frontal i mean really like famous david lynch did rabbits these were films that could not have get could not have gotten funding and so that's and 28 days later was also that danny boyle's film wow a lot of these films would not have been given the budget you know if they had to use actual film stock and i think this is going to be the very interesting thing that you're going to do all right everybody our friends from microsoft are here tom davis a senior director at microsoft for startups and you're a former founder you are here today to talk to us about the giant leaps that Microsoft has made in the AI space.

41:34You've been giving Azure credits to startups, and that's delightful and amazing, but people really want access to the OpenAI API. Yeah, absolutely. So there's two things. First of all, we've got a benefit that we offer our startups. They can get$2 ,500 worth of OpenAI credits, so they can get access to the latest and greatest models that OpenAI are delivering. But then they get access through up to$150 ,000 worth the credits that we offer through Founders Hub to leverage the Azure OpenAI service, which has a full SLA around it. So when they want to go into production and really have that reliability that we provide with the Azure SLA, they can leverage the Azure OpenAI service APIs.

42:19And they can do things like the GPT models with codecs for coding and also for the DALI models as well for images. So it's a full service. It's not just the great APIs that you get and access to the LLMs. They can build out their own LLMs using open source, and then they can manage those with our AI tooling services as well. Amazing. Well done. And if anybody wants to sign up for that, do it now while you are in front of your computer, aka.ms slash This Week in Startups, aka.ms slash This Week in Startups. Well done, Microsoft, and well done, Tom. Tell me about who is most interested in this today.

42:56Is it independent voices and people tinkering? Or is Hollywood really sweating what you're doing? And who in Hollywood is starting to tinker? Who's tinkering? Hollywood is particularly excited about what we're doing, to be honest. I think we are working really closely with both filmmakers themselves, but also production houses and studios. I mean, from a cost perspective, if you really think about these are tools that will drive costs down. so the big promise here is that you're going to take the cost of content down to zero so making professional films where the cost of making a film will continuously go down to zero from a producer perspective from a studio perspective it's great because you can make more stuff right um and filmmakers are also understanding and embracing that it also means that you can iterate more in your ideas if you're a filmmaker and you want to like execute on a particular idea instead of like spending too much time and too much money on one singular shot imagine having a world where you can try a bunch of those and then go with the one you like the most and so from Hollywood as you were telling before like with the late show with Colbert translating hours of work into minutes of work that's the value they don't care about the AI and the the tag and the models and everything else it's like this is a these are tools that are about time and cost and so of course you're going to embrace it because it's just so liberating it's so creatively inspiring to be able to do this so fast and so hollywood on the one end i think our conversations with folks in the space and our collaborations are already driving in for with with companies and professionals there have been really like positive in in the most um little sense and then of course you have other spectrums of folks who never had the chance of having those tools at their disposal and that for us is going back to our conversation around like new stories where we'll start to see more and more over time those new stories that will start to emerge what was this all trained on what data set was this trained on because it understood uh you know a new york loft it understood mountains i get that but it also understood like a low angle shot um assuming there's some amount of training that you're doing inside your company to take what are industry terms a tracking shot a wide shot a close-up you're training it on that or and you're saying hey this is what those are here are examples of it or it's has it just scooped up every single movie ever made how did you train this it's uh it's it's it's there are a lot of different like systems and models behind the scenes of seeing something like this being possible.

45:41And so models are trained on different data sets depending on what you're trying to do and accomplish. And so we have our own internal data sets, but we're also partner with media companies and industry experts to train models on their own data sets. So think about you're a filmmaker, you're working on a new animated film, you have all these reference shots about a character you want to animate, you can give us those data, those images, those references, and will train a model just for you. So you basically now have a system, an AI system that you can prompt with images, with tags, wherever you want, and you can generate infinite amount of videos out of it.

46:15The more input data that you have, the better you will understand consistency, coherence of characters, like temporal consistency of objects moving across the shot. And so it's really about customization. If you think about it, you start with a baseline model and then from there on, you can just keep on improving to have the model learn more about it. So if you were to work with, say, The Simpsons, you could say, give us all 30 seasons of The Simpsons. And if I tried to use your product right now and said, make me Marge Simpson, you know, as a Jedi Knight, but who also has the superpower, has an Iron Man suit, would it be able to do that?

46:57or because those are very highly protected ip sources you can't train on that data therefore we're going to see some maybe the simpsons make their own ai and you could have simpsons gpt or simpsons ai where you take your software their content library and they say hey you can go to this website and you can make your own simpsons and we're going to have our simpsons film festival just type into the box what you want to happen and then uh we'll have our own contest but it has to live on simpsons ai tv and here you're exactly right and the interesting thing is that i truly believe that creation and distribution will become more intertwined right so think about video think about a movie you you create the movie you render the video the movie or the video file is baked right it's pre-made you're all everyone is watching the same film right that someone directed the moment you have the systems running real time you're going to be able to visualize and render and view a video but that video is being generated the moment you're watching it almost almost in real time right and so the notion of like taking a baked video and distributing it and streaming it in real time might change as well um and that for me is like a key change in the market in terms of like understanding how distribution and creation will start to change radically specifically again on the video side of things yeah i see this fan fiction stuff is going to get really interesting there's um a website on youtube or sort of channel on youtube star wars theory and he was like a super fan and he made um this uh fan fiction called um vader and i'll just play a little clip for it here and it's obviously real actors but i think a lot of the backgrounds were created he's he does fundraisings for i don't know 20 30 40 thousand dollars to do these uh and his fans give him the money and he goes and does it and star wars is pretty cool about if you're doing non-commercial um they'll be okay with it but like you know it's pretty impressive and i think that this is going to be the future is you'll be able to go on disney plus and then there'll be disney plus fans and they'll let anybody create stuff and then maybe they'll only let certain ones onto the platform the best of but if you go on the web maybe you'll be able to see other stuff but this is really going to be a fascinating new world has anybody i don't i know you can't talk about private conversations and stuff like that but is there somebody on the ip level of a marvel star wars simpsons south park who is currently working on this right now i think there are many companies paying attention to it for sure they know and same as you kind of like notice now and make sense of it i think a lot of companies are making an understanding that that's an option they can take and from a market perspective it's just a great opportunity to get your custom your your ip and your characters and your stories out to more people and also engage them in different ways um by the way there's i don't have you seen star wars uncut it's a similar project but every shot is made by someone else someone different star wars movie that's like that's really nice yeah yeah that was a crazy project so they and they i think they're doing it also yeah star wars uncut was they just told everybody do 10 seconds but do it in your own fun style you could do claymation you do animation you could do stop motion you could do live action you could make it whatever yes uh and uh yeah those things have been out for a while yeah i like that that level of creativity there's also the yoni cash project that was very similar it's like just just one frame everyone like all the fans designed one frame just design one frame just teach all together only one frame oh i thought it was like a couple of seconds um uh you know i think the johnny frame the johnny cash break is just one one frame i might be wrong but that was my that's wild maybe that was the raiders of the lost ark one then there were the kids i don't know if you ever saw this but there were a group of kids who started making a raiders of the lost ark film um and uh they just worked on it for like 10 years as they got older and older this is empire strikes back uncut and uh yeah this is something that got the blessing um yeah that's star wars um but here is the the take the empire strikes back one i just sent you because what's really creative about that one is you'll see everybody does a different style of empire strikes back and they get a couple of seconds each and this thing is and this is in 2012 this is 10 years old uh but uh this empire strikes back one so i put in the slack here we go uh here are puppets so somebody decided they would do puppets i mean it's crazy uh and if you put this with chat gpt the interesting thing and has this come up yet inside of your uh you know i mean it's just crazy watching this isn't it it's like so creative uh and this is 10 here's one that I think like if you think about a project like that with the technology that we have now it's so interesting to see I'm actually excited to see if someone want to take this further here I'll share with you a project I saw that in TikTok two days ago so if you want to open that that's driving video on the top just someone recording himself and just like his house just it's a piece of paper and like a bunch of other shots wow you can you can translate that with runway into any style that you want um that's wild so is this using your software or just some other yes no this is runway yes this is runway so basically he takes a shot and then tells runway hey here's some text prompts so make this in the style of a cartoon make this in the style of an impressionistic painting whatever wow so it's giving the starting prompt and then going to the next thing that's just demented um that i want to like a lot it's just like a wall with like floor and it's just like you turn into a beach or like that one i like a lot as well like a paper paper boat and just like a real boat but here's the interesting thing you know i don't know if you know about this auto gpt where people are making like agents that will operate in the background and make stuff you could literally have somebody say take today's current events in turn you know take chat gpt4 um read the top headlines translate them into a story that is similar to the marvel characters and have marvel characters intercede in these world events and make a movie about it and so it's like oh ukraine and russia invaded ukraine and then there's this tension in taiwan and all of a sudden captain marvel's going to taiwan and you know tony stark's going to ukraine but what would happen is chat gpt would be set on a mission to take the top five criminal stories in the united states or you know dramatic events match it with a character and make a short film make the dialogue and there would be no human involved it would just publish this stuff to youtube in real time yeah i think that's technically possible everything you described you could just literally just yeah you can do it you're worried about any of this stuff i mean there's a whole group of people who are worried i mean you're making fun images is there any downside here that you can that you do you have a group of people in your company worrying about safety or is it just like well this is entertainment purposes or do you worry about some you know buddy creating people saying something that would get them canceled and then publishing it on social media and that kind of stuff impersonations etc we take safety really seriously we have a quality and alignment team inside runway that makes sure the the model outputs are safe and compliant with our terms of use and of course the field is evolving really rapidly and we've been pioneering all of things in the space and so it's always been a kind of like starting point for us to think about the best possible outcomes of the best possible approaches to use this technology in a safe way.

55:20I do think about it in a perhaps, you know, we're talking about the analogy of the camera. You can point the camera to all sort of like things you want to point the camera to, right? It creates, it's an incredible tool, but it's also a tool that needs to be used in the right way, right? From our digital perspective as well, just having the camera won't make you a film director. You just need to understand where to shoot, what to shoot. This is no difference. Like you have a tool that allows you to generate something, an idea. It's about iterating on that idea and understanding what works and what doesn't and what's safe and what's not.

55:53And all of those things are going to be things that we're going to figure out as an industry in the same way that figure out systems and safety procedures for sharing videos and cameras. Right. I think we're still very early on all of it. And you might see this with language models being discussed more. It's still very early for the industry at large. But yeah, we take it. We take it very seriously. like nudity adult content and that kind of stuff obviously there's a place for adult content in the world i'm not talking about pornography necessarily but there could be a scene that is a risque scene from an r-rated movie or there could be a violent scene in a movie how do you manage that in your terms of service quentin tarantino doing inglorious bastards and literally scalping nazis would you allow that on your platform or not allowed on your platform i think we're taking a more pause approach towards uh helping people understand how to best use the technology and so right now we don't allow nudity on the platform for example uh but we're working with filmmakers themselves to understand what that does mean and like what limits can we change uh but again i'll go back to this is still very early got it i mean if If Darren Aronofsky came and said, listen, I'm making Requiem for a Dream, it has some nudity in it, it has some violence, it has drug use in it, you could make an exception for him and then just make sure that they're not abusing somebody's likeness.

57:24I mean, I think that is the really dark part of this is taking people who do, even if you're a public figure, and listen, I'm a public figure, people are making me into a superhero as we speak on Twitter or they were making Star Wars characters out of the all-in cast. and it's like okay i'm cool with it but like let's not get too crazy folks um but you can see uh with these tools that people will get crazy at some point so i think it's probably like you've probably got a pretty good concept here um which is we don't allow stuff that would be risque right now or could hurt people but we could have a conversation about it if you want to submit what you want to do and we would have the right maybe to veto or prove it since you're using our tool to do it that might be reasonable yeah i think that's that's right i think working with the communities that you're trying to serve is really really a key component here so what's the hardest part of your job right now as an ai company well i mean it's ai is becoming this insane you know people have been working on this for decades and then in the last six months all of a sudden it's like somebody lit the fuse and now the rocket is hurtling towards the edge of space and so are you just inundated with too much inbound too many people want to throw money at you too much distraction or too much competition too much change in the technology because it seems like this stuff is moving at a pace that is um incomprehensible even to people in the industry no yeah things are moving fast i i would say i guess exactly what you mentioned that there's been decades of work behind the scenes to get to this point and for someone and for for our team who've been working on this for a couple of years now it's really important that that consistency and the level of like outputs of our research team and our product teams continue to be the same There's, of course, more noise around, more people are paying attention to it, which is in some way great because you don't have to convince people that this is worth paying attention to.

59:30Everyone can just say it. I think we've, I keep saying that we've crossed the most important chasm that there is in technology, which is the mom threshold, which is my mom now uses runway and understands what I do. Which means that the technology has gotten mainstream enough where it's more accessible or more understandable for more people. That still means that we haven't gotten to its final form. there's so many things we need to do. It's so early. I think that for me is one of the key challenges is really emphasizing where we are with the technology at large, what needs to be improved and also exposing more people to how things work.