Generating the future of art & entertainment

0:05Welcome to Practical AI. If you work in artificial intelligence, aspire to, or are curious how AI-related tech is changing the world, this is the show for you. Thank you to our partners at Fly.io, the home of changelog.com. Fly transforms containers into micro VMs that run on their hardware in 30 plus regions on six continents. So you can launch your app near your users. Learn more at Fly.io.

0:42Welcome to another episode of the Practical AI Podcast. I am your co-host, Chris Benson. Usually I have our other co-host, Daniel Whitenack, with us. He is not able to join today, but we have a great show in store. We have with us a super interesting guest. You may very well, if you follow AI, have heard about this guest and this company doing some super cool stuff. So I'd like to introduce Anastasis, sorry, I'm mispronouncing, Driminidis, who is the co-founder and CTO at Runway. Sorry, I screwed up your name there. Did I get it anywhere close to right there? Yeah, all good. Thanks so much for having me.

1:23No, sorry for the stutter there. Thanks for joining us on the show. You guys are doing some really cool stuff at Runway. I wanted you to actually, before we dive fully in, kind of tell us a little bit about your own background. and then we'll kind of dive into kind of the environment that you find yourself in and the industry and what kinds of problems out there are interesting as we dive in. So first of all, you know, CTO of a hot AI company, how did you get there? How did you get to where you're at right now? Well, the first thing I would say is that I did not get here by planning for it. I think in some ways it's planning against being where I am today.

2:03So just to give you the background, So my background is kind of a hybrid of engineering and art. So I was, for the past decade or so, I've been kind of in different startups, working as an engineer, at the same time having my own art practice. And so doing kind of a variety of work in kind of media arts and interactive arts. Runway was the first time where those two kind of different worlds have converged for me. But Runway started in art school. So this is not really where companies, AI companies get started usually. So my motivation for going to art school was actually to take a break from technology to really explore the more creative and in some ways open and exploration of those technologies without any concern about making something that would make a commercial sense at some point.

2:55but it just so happened that you know i met my co-founders there and we started kind of making those small tools and like one thing led to another and we realized that this was kind of a really useful thing to build out and kind of spend our focus time on it sounds like it was a bit of a passion project you know without that commercial intent up front you know in the beginning you you kind of fell into it because it was what you love yeah and i think that's how the best things get started very usually. And that's been a general pattern, I would say, not just at the start, but just throughout the way we rebuild the company is where there is this book that we give to every employee that's called Why Greatness Cannot Be Planned.

3:39And it just talks about this idea that when you have very concrete goals in mind, it's actually very often you end up not meeting them. And sometimes going for the next stepping stone is the right approach to actually get to very interesting findings or novel insights. And so that's been part of how Runway started and that's been part of how Runway had continued to grow. But yeah, initially I would say our main goal was these machine learning models are super difficult to understand, super difficult to use, especially when we started around five years ago. But they're super interesting for artists and they can make really compelling things with it once they get to the point where they can actually use them.

4:20At that point, generative models, AI was a bit at an earlier stage in terms of both how many people cared about it and also the results of those models. But it was still, even at that point, really useful for artists the moment we gave the right tools for them to use it. And so that was kind of the inception of Runway. I'm curious, recognizing that there wasn't the master plan that you were implementing, there was a bit of serendipity to how you arrived there. I am kind of curious, you mentioned that you would kind of set aside technology before you were going back into art right there. And I'm kind of curious, did the technologies you're in prior to art school play into where you've come out here with, you know, in terms of runway being that end result?

5:06Or did you, you know, is there any connection there? Or were they just you happen to be in a different area? And we're finding AI? Were you active in AI prior to going back into art school? My interest in AI kind of goes back into like at least high school and before. So I've been, before Runway, I was working as a machine learning engineer, as a kind of distributed systems engineer at different companies. So definitely had a background in this area, was very interested in AI. My interest was specifically in neural networks, which, you know, when I was kind of decades ago, they had become kind of like ignored area of machine learning.

5:42They were kind of seen as a dead end, that they wouldn't be able to, at that point, support vector machines around there. The kinds of models were more popular. But there was still something very compelling about neural networks that made me actually start working with them from high school with some initial projects. So I'm very interested in AI throughout. The motivation for going to art school was, and just to keep more context on the kinds of art school, It's a program at NYU that was kind of exploring the intersection of art technology. Technology was still part of it, but it was less kind of technology for the sake of technology or for just like novelty for the sake of novelty.

6:24More understanding like how the technology could be used in creative ways or in ways that are maybe unconventional. As you were coming into art school and you have this background as a machine learning engineer and the passion for art, what has been you know your initial vision for that industry like within entertainment human creativity which are things that you currently are targeting how did you see them how did you expect to be able to impact the industries with AI going into the process so like things are moving so fast and we're seeing these amazing technologies which we're going to be talking about in the minutes to come but I'm really curious what your perspective was about where this was going for art and entertainment prior to actually arriving there?

7:09The perspective for us has always been that those models, those techniques are never going to be a source of ideas. They're going to be an acceleration and expression of like creators' ideas. This is the kind of mindset that we started building those tools around. And that's why from the beginning, we started working very closely with filmmakers or with designers or with artists in making those tools and getting their feedback on how to make them. The other aspect in terms of how we were seeing the trajectory of those models was when we look back at 2017 or 2018, when we just started working on this, the results of those models were pixelated, low resolution, very experimental.

7:52The composition was off. But you could see the trend very clearly that every year the resolution was doubling, the fidelity was improving in a fairly predictable way. And so it was not a matter of if, it was a matter of when this would arrive. Timing those things is always really difficult. So we didn't really know exactly when we're going to get to this breakthrough where those models really started becoming actually useful. But we knew that it was going to happen at some point in the next years. Most people who are machine learning engineers, and I work with university students a lot and people at the company I'm at now and previous companies.

8:30And that's kind of their dream job. And I find it's really interesting to me that you said, I'm going to set that aside for a little bit and go and do art school. What was the driving factor for you? Because obviously that turned out for your story, that turned out to be crucial, that juxtaposition, if you will, of those different factors. I'm just curious, what made you say, I think I'm going to put down machine learning engineering for a while and go back to art school? I was just curious what that was, because obviously that seemed to create a perfect environment for you to spring from. I would say mainly just the motivation and the need to explore the possibilities of something without a very clear expectation that it needed to result in a tool that was kind of necessarily useful or just being in an environment where I can kind of have this open-end exploration of the possibilities of this technology.

9:27it was less that I wasn't interested in machine learning or I wanted to get away from it it was more I wanted to explore it in a context where there was no kind of expectation that I needed to build something that was commercially valuable or super useful of course that took a turn and that was a way to get to something that ended up being a very good fit for a company but I would say initially I was very interested in And at some point, I think in 2015, 2016, they were just starting to emerge this kind of new movement around making art with AI. And there were some initial explorations, a lot of them in kind of the open source world.

10:07And I just started contributing to making kind of small projects around making kind of tools to make art with AI. And so really just wanted to spend more time building those things and less kind of purely in the industry, working with machine learning because I think those two things, you're working with the same underlying models and the same technologies, but the actual results are very different that you're creating with them. And just one more kind of story from Art School Act to illustrate. One of the first projects that we built with my co-founder, Chris, was this drawing tool essentially where there was this model that NVIDIA released that was meant for kind of self-driving car research.

10:52And the main idea of this model was you could give a layout of essentially a street view, so kind of indications of where pedestrians are or the roadies or other cars are, and then generate an image using that layout. It doesn't sound like the most creative model or creative use case for a tool. The context of that model is very much for as part of self-driving car research and just kind of creating synthetic data for that and so on. But we decided to build this drawing tool around it where you could define kind of the layout of a scene and then generate kind of street views based on that layout.

11:31We saw that the moment we gave it to artists, the kinds of scenes that we were creating were super different than like what the regular opposite of the model was. So they would create like giant pedestrians or like street signs flying from the sky. So there's the same insight there that you're working with the same types of models, the same types of technologies, but seeing them with a fresh set of eyes and a different perspective makes all the difference. And so this is what I came to art school to do, is to see the same underlying kind of ML, AI technologies with a new kind of set of eyes, exploring new possibilities.

12:08And this is what we hope to do also with the tool itself.

12:23What's up, friends? Is your code getting dragged down by joins and long query times? The problem might be your database. Try simplifying the complex with graphs. A graph database lets you model data the way it looks in the real world instead of forcing it into rows and columns. Stop asking relational databases to do more than what they were made for. graphs work well for use cases with lots of data connections like supply chain fraud detection real-time analytics and generative ai with neo4j you can code in your favorite programming language and against any driver plus it's easy to integrate into your tech stack people are solving some of the world's biggest problems with graphs and now it's your turn visit neo4j.com developer to get started.

13:08Again, Neo4j.com slash developer. That's Neo4j.com slash developer.

13:35so you arrived at art school for that purpose of seeing all this through a new set of eyes and you met your co-founder chris and you guys had that spark of an idea which would become Runway. Can you talk a little bit about the insight that you had there that created Runways? Before we dive fully into what Runway has done since, I'm really curious what the moment where you and Chris kind of said, we have something here. This is something we're going to go do. Was there a distinct moment? Did you just kind of gradually arrive there? What was that moment like where you decided it's time to go be an entrepreneur in this context?

14:10So I wouldn't say it was one moment that kind of was the turning point. So we were working and a lot of different projects with Chris and Alejandro, the other co-founder. And each of those projects was kind of a standalone tool around kind of helping for, let's say, a specific art project for an artist or for a specific kind of medium or specific kind of context. Over time, we realized that there was a lot of the same things that we had to do for each new project. And at that point, setting up, being able to run models was even more difficult than it is today. Even running a Google call-up notebook was too much to ask sometimes for artists without any technical background or know-how about how those models work.

14:57The initial idea was let's start from what's already out there in the open source world. There is already a wealth of different models that perform different tasks, but let's make a creative tool around them. Let's bring the interface and the kind of experience that artists are familiar with from other creative tools, but use those new models that were coming out that have all these interesting possibilities kind of on the backend. That was like the main idea of Runway initially. And also, as I mentioned before, there was kind of that vision was there from the start that as these models were becoming better and better, more the applicability of those models will go increasingly more from, you know, the more experimental use cases to something that's like actually driving production.

15:45And it's like really, really useful for a variety of creative workflows. And we saw that happen kind of very quickly after starting running. You mentioned along the way there that the difficulty of implementing some of the models and even today with a number of different choices out there, it's still something that many companies are contending with is, you know, how to address models, how to, how to train them, where they're going to train them, what the deployment, how it fits into products. There's a gazillion questions out there. You were doing this at a moment where that wasn't even as sorted as it is now.

16:19And it's still in development at this point. How did you manage that? Because when I've talked to other people, that's often been one of the biggest challenges is just getting the resources in place, especially at that time when it was still in early development. What was that like to try to bring your vision out when obviously the environment that we were doing AI in was still fairly exclusive in a lot of ways in the sense of access to expertise, resources? You're in an art school that's designed to help you do that, but that couldn't have been easy. Yeah, so we essentially had to figure out a lot of things from scratch as we were building this.

16:57so as I mentioned initially Runway was based around providing access to existing open source models but we quickly actually realized that we needed to build a new house research team in order to really get those models from something that makes a good demo or a good prototype to something that's really useful so that was actually from the first few months of Runway it became very clear that we needed to do this of course none of us three had built I researched it before. I had engineering and research in some kind of ML and research background, but the experience of how to build the team, what skills is to bring in, was that nobody on the team had it.

17:41And so a lot of the things we just had to do and figure out from scratch. One nice thing I would say is that because we started so early, we had years to figure this out. So if you're just coming into AI and as part of building a new company today, the time horizon. You need to figure those things out in a much more accelerated fashion. For us, we spent the first years figuring out what does it mean to actually build a research organization within a startup and what does it mean to build a robust kind of deployment pipeline so that you can not only serve those models but also serve them interactively because a big part of the way we build tools at Runway is the interactions subject.

18:22It's a very key aspect of really making those models useful. I think when I've talked to other entrepreneurs about this, they have a tough time. As you're kind of getting to the place where you're at now in terms of being able to, you now have the research, you're doing amazing research, but you had to kind of get from A to B in the meantime and kind of keep the company alive. How did you approach from funding, customers, things like that while you were kind of figuring all these things out? Because that strikes me as a pretty hard problem to tackle as you're moving along, but you still have to pay the bills, if you will?

18:59How did you tackle those kind of issues in terms of creating an AI startup that couldn't instantly be everything that it is today from day one? I would say the main insight is to, we wanted to make sure that Runway was useful at each stage of its evolution. So even though the generative models were not quite as powerful back when we started as they are today, they weren't as big a part of the initial kind of tool offering. And we wanted to make the tool as useful from the very beginning as possible. So the product of Runway went through many evolutions that really track how the AI models evolved and at which stage they were useful for which things.

19:41A big part of early Runway was building out a video editor that really combined some of the more traditional video editing techniques with AI-based techniques to speed up the process of a lot of beta editing workflows. And that wasn't necessarily something that had generative models powering it, but it was a really useful tool that really gave us a lot of insight about how to build tools that are really useful for creative workflows and how to really solve real pain points of beta editors. But at the same time, while we're building those tools, we're also at this kind of research that was ongoing that was still remaining at a kind of more academic level of just really demonstrating how we can improve the results of generative models.

20:21And at some point, there was that intersection point where we started bringing those generative models to production. So the overall strategy was, we knew that generative models would be really powerful given enough time, and if we invest the resource on the research side. At the same time, we knew that at the beginning, not everything is to be powered by generative models. So we're building a lot of AI-based tools that incorporated, that were really useful from the beginning, and that they were used by VFX artists, by video editors, to speed up a lot of their workflow, even far before we released things like Gen 1 or Gen 2 for text-to-video functionality.

21:02You're saying generative, but it was definitely the early days of generative. And you certainly, like right now, it's all the rage. Everyone's talking generative in every context. But you had some insights into that. You talked about the fact that you guys knew that that was going to be the case going forward. But to your credit, not everybody did. There's been a lot of people went, aha, much later than you went, aha. And I'm kind of curious, is there anything that stands out as what drove the insights that you guys had and why? Because you were really one of the very first to get these kinds of functionalities to product.

21:41That's very notable. And, you know, you might say the rest of the world didn't, you know, not that many. And so what were some of the things that gave you that confidence to say, this is clearly going to be critical to our future. This is going to drive the industry at an early stage. You were pioneering that thought process. How did you get there? From the very beginning, a big part of running was working directly with artists and building those tools. And so when we gave them even early versions of generative models, we could already see that there was really compelling aspects of working with them, even if the results were low resolution or not as high fidelity.

22:22So early forms of things like prompt engineering, like figuring out how to traverse the latent space of those models were still there at the beginning of Runway. And we saw how artists were engaging with them, like how they were finding them to be really compelling and really useful. And so really part of it has been just having this early view into how artists with kind of more early adopters, I would say, were engaging with those models and just extrapolating that once those models improve, other people will equally find them as compelling. So working with artists, I think, has been a really important part of just really understanding kind of the future of those models and extrapolating of how they would be used.

23:06And also just looking at the kind of history of art and how toolmaking was always part of, like how new tools always allowed kind of new, created new kind of art movements or allowed new kinds of kind of genres to emerge and just assuming and kind of predicting that the same would happen with those general models. Along the way, as you were going down this path, what stumbles did you have, you know, as part of putting, because it's quite remarkable because you clearly could see the future, you know, before you got there and, and with more clarity than others that might be in a similar position as you did that, what kinds of things did you were either unexpected, uh, or challenges that were bigger than you thought, you know, the things were maybe at a moment in time, you were grinding your teeth and going, or this is not exactly how I had it planned.

23:58Do you have any stories to that effect during this process? Many stories and many learnings along the way, for sure. I think the biggest requiring insight that we've had around how to build for those tools and the thing that I think is still not fully appreciated today is how important control is in terms of interacting with those models. And so every time we invested into adding more ways in which you can really control the outputs of the models that people were using inside Runway. We saw a whole new set of possibilities and whole new kinds of usage. So that has been a really consistent theme.

24:35And even at the beginning, we just saw that those models had a lot of flaws that they might not always, like if you have a very simple ways of controlling them, they might not really give you what you want and you might have to do a lot of tries with the same model, kind of generate a lot of outputs to get to kind of where you want your desired result. And so that's really what we saw with the kind of early, like when we first released Gen 2, you could only kind of control things with a text prompt. And we saw very quickly that that led to people just kind of generating like tens or hundreds of outputs in order to get to the result that they wanted.

25:13And so we invested kind of continues more and more, adding more and more ways in which you can like manipulate things essentially as a film director would think about creating a scene. so a film director would have a vision not just of like a high level description of what the scene is but how the camera moves in the scene or how do the characters interact with each other so having ways in which you can control really the camera motion or the motion the object motion the motion of the characters in the scene all those things that make total sense from a curious point of view but they're not necessarily how ML researchers would necessarily think about those models I think that has been always the insight that we never saw negative effects from adding more and more ways of controlling those models.

26:20This is a Changelog Newsbreak. Pewter is the internet OS. Pewter is an advanced open source desktop environment in the browser designed to be feature rich, exceptionally fast, and highly extensible. It can be used to build remote desktop environments or serve as an interface for cloud storage services, remote servers, web hosting platforms, and more. I've been around long enough to see a bunch of these desktop OS and a browser window demos and toys. But this is the first time I've been impressed by one enough to keep the tab open longer than 30 seconds. From the URL structure to the cloud storage integration to the developer portal, Pewter strikes me as an actually viable internet-based operating system with potentially real-world use cases.

27:07And that's saying a lot. Oh, and it's also entirely built with vanilla JavaScript and jQuery. So you know the devs haven't cargo-culted together something they can't grow and maintain. On that note, they say, For performance reasons, Pewter is built with vanilla JavaScript and jQuery. Additionally, we'd like to avoid complex abstractions and to remain in control of the entire stack as much as possible. Also partly inspired by some of our favorite projects that are not built with frameworks. VS Code, PhotoP, and OnlyOffice. You just heard one of our five top stories from Monday's Changelog News.