AI-Automated Film Making

0:03Welcome to Practical AI, the podcast that makes artificial intelligence practical, productive, productive, and accessible to all. If you like this show, you will love The Change Log. It's news on Mondays, deep technical interviews on Wednesdays, and on Fridays, an awesome talk show for your weekend enjoyment. Find us by searching for The Change Log wherever you get your podcasts. Thanks to our partners at Fly.io. Launch your AI apps in five minutes or less. Learn how at Fly.io.

0:44Welcome to another episode of the Practical AI Podcast. This is Daniel Whitenack. I am CEO at Prediction Guard. And today we're really excited to talk about AI in filmmaking and content production. As we have with us Sami Arpa, who is CEO and co-founder at Largo AI. Welcome, Sami. Thank you. Thank you for having me. Yeah, yeah. It's great to have you here. I remember specifically, of course, we're always looking for interesting folks to join us on the show and talk about how AI is being used in various verticals and industries. And I remember seeing a Variety article about this sort of Sylvester Stallone-backed team at Largo AI.

1:38and it talked about the world's first fully AI automated film company, which was very intriguing to me. I'm sure we'll get into a lot of those details, but maybe before we hop into the specifics about Largo AI, I know that you all have been in the industry for some time and been doing this work. Could you just give us maybe a high-level picture of how AI and kind of advanced technology has been evolving in recent times in the film industry. Of course, for a long time, maybe many people know about CGI and certain technology that's actually fairly advanced that's been used in filmmaking for some time.

2:24But maybe give us your kind of state of AI in filmmaking and how that's evolved in recent years. Yeah, absolutely. So the film industry has always been advanced with the technology, but the progress, the adoption with AI has not been easy. I can tell that with our journey during the past six, seven years. So, I mean, we can think the primary adopters of AI in industry as Netflix and Amazon, because they started to use AI for recommendation systems. that was already 20 years ago. And then they started to create their own content with original contents. And that also created another step for AI because they could analyze the content and order the type of content that they know already that that will work.

3:23That was also with AI because specifically, for example, for Netflix, they were having the system of micro genres which they will still have that, knowing the audience behavior and also understanding content at earlier stages by using AI tools, you could order the right content right from the start. So this way they could get larger audience with narrower catalogs. And so then if you put this as two phases, then we can also name a third phase. This is the phase that we see starting with chat repeat. Like every industry in the film industry as well, that we saw more of applications. And also that has changed the adoption of the industry.

4:11We measure adoption of the industry for usage of AI during past six, seven years. It was one, two percent when we started Largo AI. Now it is around 30 percent. And that is a great progress. And it is a progress despite the things like strikes. We had two big strikes also in Hollywood and partly that was also against AI. Yeah. And, you know, maybe just to dig in a little bit there, what some people might think of as kind of the first or the first thing that pops into their mind with AI and film is maybe what they've seen around the actual video generation side or changing the visual effects. But it sounds like you're talking a little bit more kind of wide reaching and operationally across the film industry.

5:10So could you give us a little bit of a picture of maybe the question is how kind of an overall categorization of how AI might be used in different parts of the of the AI industry? I know you're digging into certain parts of that, but maybe you could help us understand kind of more generally the different categories or ways that it could be used. Yeah, I think for that, it's important to understand the chain of development of a film. A film is a very big project and it takes many years. As audience, we just see the end results on the screen. But any film has four main states of development. The first part is development of overall story, then pre-production, the stage that we attach also people to the story and also raising the budgets.

6:05Then we have production and then post-production and distribution stages. So at any stage, there are many people that are involved and there's a lot of work and many projects cannot finish all this process. And actually, so for example, going to development to pre-production, already 90 % of the films are eliminated. Or even coming to development, that there's a big amount of projects are eliminated. Like there are scriptwriters writing screenplays that are never picked by the producers. But at each of these steps, there are a lot of works. And eventually, obviously, the goal is to bring that to the screen, but only a few projects are coming to that stage.

6:54And so here, so of course, the sexy part is text to video, the visual part. But for all other parts, there is applications of AI. And actually, there is a big benefit of usage of AI. That's something we have focused as well. we have focused more on earlier stages for the usage of AI, understanding the content, character, casting, and predicting financial results for helping to raise the budget for the project. And of course, there is strong applications for post-production, production with new text-to-image tools that we will see more promising applications of that as well that we have seen. Recently, Google VO3 has been released, which is amazing.

7:46Actually, the results look amazing. So that will change still significantly post-production and production parts as well. So we will still observe that there is not enough strong applications at that stage for especially live action movies. But the point here to summarize for every step, there is important efforts. Some parts are not visible to the audience. And for every step, there is type of AI tools that we can apply. Yeah. And just for, of course, probably most of the audience that has not been directly involved in any of these stages of film production, maybe except watching it on Netflix or wherever the venue might be.

8:36But could you give us a sense of the kind of investment and how much effort is put in kind of proportionally in each of these stages leading up to the distribution investment in both kind of time and people and scale? Yeah, biggest investment is in terms of money, it is for production and distribution stages at production, really just to produce the content on the sets. And then I include post-production budget in that as well. And then distribution is the parts for spending marketing budgets. so these are investment wise these are the biggest parts but for time wise for most of films they are not actually but for many projects development and pre-production states are taking much much more time there are projects even like having like a six seven years of development or pre-production and the biggest challenge over here to convince many people to bring around a project and while doing that, you need to make a lot of iteration.

9:49So, producer, screenwriter, even director might be involved at that stage. So, you create a story and that's, you engage people around that story and you need to find people to put money in that project. That might be studios, investors, etc. That's actually normally biggest, most difficult step for most of projects. And in general, most of projects are not good, actually. That's also a reality, which we can see in our AI system as well, that we have producers, they put their projects to get financial results. And in most of cases, it says that project will fail. Because yeah, finding good project is not an easy task.

10:42And it requires a lot of work, a lot of study. That's why AI tools at that stage can be very, very helpful, very critical. And for a producer, filmmaker, the biggest hurdle for future is to fail at the current project, because that's also a way to open the door for next projects or not. That's why making sure that that project will be successful is very critical. Yeah, and that's very interesting to me that you're sort of focused in this. I guess what I'm hearing is there's sort of noisy early phases of these projects where you've got a lot of kind of maybe good projects mixed with a lot of noise.

11:34There's difficulty in kind of parsing through that. also for those that maybe have written this story or are promoting the production of a project, it's hard maybe to stand out. How is, from your perspective, is this sort of technology, and we'll get into exactly what you all are doing, but generally in digging, bringing technology to these early stages, does that change the dynamics of, you know, smaller, you know, smaller studios or script writers or maybe lesser known folks that could maybe use technology to help them play on maybe more of a level playing field with kind of the big studios or well-known folks?

12:21How is Is that dynamic sort of shifting or is it? Absolutely, yeah. I mean, the AI and new technology create much more opportunity for smaller production companies, newcomers. I mean, if you think from studio perspective, they have, especially at early stage, they have a lot of resources to understand if a content can be successful or not. They have experience, but they can get many research done, including focus groups at very early stages. So those kind of things are not available for early stage, for newcomers or small capacity production companies. At development stage, you don't have your movie budget.

13:11You have not raised yet. So if you are lucky, still you can find some people or some institutions are investing at development stage. If not, they use their own resources to understand if the content will be successful or not. That's why using AI tools at that stage is a very cost and time effective way to understand the potential success of content. That's also for later stages. It will be the same as well, by the way, because we will see the cost of production, post-production will reduce significantly with the AI tools. If$100 million budgets can be produced for$1 million budgets, that will change whole ecosystem, right?

13:57Because$1 to$10 million budget films can be produced by independent producers, but not$100 million. others. We will see also all those changes in next years.

14:13Okay, friends, build the future of multi-agent software with Agency, A-G-N-T-C-Y. The Agency is an open source collective building the internet of agents. It is a collaboration layer where AI agents can discover, connect, and work across frameworks. For developers, this means standardized agent discovery tools, seamless protocols for interagent communication, and modular components to compose and scale multi-agent workflows. Join Crew AI, Langchain, Lambda Index, Browser Base, Cisco and dozens more. The agency is dropping code, specs and services. No strings attached. You can now build with other engineers who care about high quality multi-agent software.

15:03Visit agency.org and add your support. That's A-G-N-T-C-Y dot org.

15:15Well, Sami, we've kind of talked or referenced some of Largo AI and the way that you're digging into these early stages. Could you give us a little bit now kind of digging in specifically to what you all are doing? Could you give us a little bit of the backstory of Largo, kind of how it came about, what those initial ideas were? Obviously, you mentioned it's existed for, I think you said, six years. So this is before the latest kind of boom of AI, at least as far as the general public has perceived it. So yeah, give us a little bit of the backstory. I'd love to understand how all this came about.

15:55Yeah, sure. I will connect that with my personal background because that's how Largo started, the story of Largo started. I did my PhD at EPFL, which is a university in Lausanne, Switzerland, in the field of computational aesthetics, understanding art and generating art by using computers. And film has been also one of the subfields over there. So that's the technical part, technical scientific part. But parallel to that, I've been also director, producer on some small projects. So I've been on the creative side as well, like just working hands on. While working on my own projects with a bit of head of engineer as well, I was always curious why we don't have a way of representing films similar to music.

16:56Because music we can represent with musical sheets with the partitions, which gives a way of understanding of the certain formats or the rhythms and the type of structures for different genres. That doesn't make music less creative. In reverse, actually, it makes more creative because you can focus on a specific structure and go deeper on that. And when you say that sort of partitions or structures, you would mean like a pop song or something like that has a verse, chorus, verse, bridge or something, or even in the Musical.ly, there's bars and other things like that? Exactly, all those things.

17:40In film, there is a bit of structures, but these are too much formulated. So it's not like the same type of low-level structures. That was a bit of my curiosity. That's something I shared with my PhD advisor at that time as well. So we work a bit on that. we were thinking how we can create like similar type of structures for film as well that can be useful both for people working on that creatively but also for machine because understanding and learning from films is also very difficult for for machines because yeah like film is like a two hours of content all the frames millions of pixels or like a screenplay is like hundreds of pages of content and you need to associate this with all metadata, but there is not even enough sample of data to learn confidently on those.

18:33So if we can represent any film in a more structured way, in a smaller space, smaller vector space, that's even easier for machine to learn from that. But that was our starting point. And it was with that starting point, we created this, we call that genre recipes, emotion recipes. So this is like we put any film in nine dimensional space of genres. And then, for example, let's say drama. Drama is like one of these patterns. You find how drama is evolving over the story from start to the end. Same thing for comedy, for romance, for thriller, horror. So this way we get like on a timeline, we get a map of a film, of a TV.

19:18series or for any content. That can be from screenplay or from a direct video material of a content. So this is like a baseline representation of a film for us. That was our starting point. And with that, of course, then we engage many other points, metadata, like the actors, the budget, all the other content that becomes a representation for a film. So this is, we start to use those as a base both to provide as a feedback to creatives so they can see the release structure of the content, but also to machine. Once machine is learning from that type of data, it becomes much easier to learn. For example, easier to learn financial predictions, box office predictions, streaming predictions.

20:05Yeah, yeah, that makes sense. I can sort of imagine this graph of drama or comedy going up or down on the arc of a movie So that gives kind of an understanding level, I guess, of the movie. How then does that connect to more concretely? How can that connect to concrete kind of value for those involved in promoting or writing a story or producing a movie? Yeah, I mean, for producing a movie, there is three important elements. The first one is content itself. The second part, the people are involved, primarily the cost. And the third part, the financial part, the budget and expected return for that.

20:58And our forecast insights are also in these three main categories. So the content analysis, it provides insights related to weak, strong points of the content and audience emotional reaction to that content. And character casting part is about understanding the characters and then making casting propositions. So the AI is making propositions for the cast. It has for this character that that actor will be best fit, for example. And then the third part is the financial part. So here for that part, it makes the predictions directly. how much money the film will make with given content and then all invested money and the other metadata like attached cast, director, et cetera.

21:47And that part, of course, there's many sub part of that because that part is also relevant to understand the audience because how much money you make is relevant to understanding audience, writing the rights, marketing, all these things. So it goes deeper to get predicted demographics of the audience for specific countries. And along with that, we have also the simulated focus groups, which is like one of our most exciting tool. There you can really get quantitative and qualitative feedback from the audience. Yeah, that's great. I'm wondering, because in the earlier stages, like you talked about, of determining what type of content to make, casting all of that, I'm assuming the assets, I'm thinking more from the technical side now, the assets that you have to work off of, I guess, are the script and maybe some other things.

22:50Could you talk a little bit about kind of the inputs to this? Like what's required to really get good results out of a system like this as a starting point? Yeah, it depends on the stage. But if starting very early stages, the system will need at least a treatment. The treatment is a very short version, early version of story. Typically, it can be even like a two or three page. And a bit later stages, it will be a screenplay. It's like a full storyline of FFM. Coming to screenplay stage, together with that, the system would ask also basic packaging information. Because a screenplay, if you think about financial forecasts, a screenplay can make any money, bad or good.

23:40It can make any money because how much it makes is also very relevant to attach people to that project. and also the budgets, how much budgets that has been put. Like with the current standards, let's say, if you try to make a perfect sci-fi screenplay with$1 million budgets, we can tell that the results will be very, very bad, right? So it's not difficult to tell even for regular people, but yeah, I mean, I will give the same warning as well. So actually in that manner, that's like we always say content is the king. It is very important, but once we look at, once we put all the features of making a film and look at AI learning, we see the parameter that is impacting most the financial results is budget.

24:38And it doesn't mean having a high budget, it means the right budget, especially for a good return on investment. Sometimes some films are having too much of budgets than what they need, then it becomes very difficult to make it profitable for people who make the project. Yeah. And just practically for maybe some of the practitioners out there that are working, maybe not in the film industry, but they might be working on simulating other things in other verticals or different types of production processes or whatever that might be, maybe unrelated to film. But just for their benefit, it sounds like the system that you've kind of built with Largo, you know, works on various types of projections.

25:28There's various stages to it. I'm assuming, you know, sometimes there's this misconception now, I think, exacerbated by Gen.AI that you have kind of one model, you put one thing in and then you get everything out. I'm assuming that your system, which has been developed over years, kind of involves multiple models that maybe do different things. Like you mentioned the one around kind of detecting or mapping these genre distributions or semantics across a film. I'm assuming there's different stages of these things with different models involved. Maybe the financial forecasting model would be different than the model that's producing the genre results.

26:16Could you give us just kind of at a high level an understanding of how this kind of all fits together as a system? Yeah, absolutely. You are right that we have a lot of models. so using the models that we use for genre prediction for financial forecasts it wouldn't work especially financial forecasts typically is very different models or shallow models compared to content understanding models which are much deeper models so that's also the important thing with the current AI wave that LLMs are really great to answer for many things. But even if you go to LLMs, even chat GPT, we see that they have many models.

27:10Actually, each model is better at different type of solution. The same thing, of course, for us as well. So we have for each type of task, we have different models. And also, we have two main categories. One category is for the models that are learning from past data. And it uses this learning to make the predictions for new content. That's one way of learning. The second learning is learning audience. There what we do is basically we are creating digital twins of real people. So we don't learn anything content related. We learn people themselves. And then we show the content to these digital twins of real people.

27:55and actually the second one is having advantage of not missing outliers almost because like the one big danger for like just learning from past data is yeah, outliers in the team industry that we can often have. Okay, so for general content we can predict successfully but we can always have something completely new that we don't know well the type of audience behavior for that type of content. the model will miss that. But with the second approach, with the digital twin approach, we can even capture outliers because you are much closer to humans. You are already creating their digital twins and that digital twins are having very short lifetime, like one year.

28:41So you are very close to the current behavior of people. And it is very successful also to capture new approaches. Well, I'm really intrigued by kind of the way that you've built up this system of tools that kind of helps in various ways throughout the film creation, film production process. I'm wondering in terms of, and this is probably something on a lot of people's mind in relation to AI models and content, especially, you know, art or movies or images, that sort of thing. Obviously, you need some sort of reference data with which to train models and kind of help them produce results. You know, maybe for financial projections or something, you know how much a movie has brought in or something, and that's public information.

29:37I'm not sure actually how much of that is public information. Not fully, but yeah. Yeah, so basically my question is, how do you go about kind of creating the data sets you need in an industry where, of course, there's a lot of, you know, there's proprietary or copyrighted content, that sort of thing. What does that look like for you as a company? Yeah, I mean, there are open data that we can learn, like, for example, movie summaries. is pretty open or movie metadata, like who has been engaged with which film. So there is already a lot of open data or box office data, like how much they have done, which is for most of films is publicly announced.

30:24But there are also type of data that is not publicly available. And one of the most important of them is streaming data. Streaming platforms do not provide data. Netflix has started to publish some data recently in terms of like a viewership, but it is still very limited. And also, yeah, like not having that type of data is shaping the industry, not just going outside of like AI perspective, because we know many producers are complaining not to have that data, because the value of a film is very much related to the size of audience. And that relationship is very clear in the box office because you are just putting the film in the box office and you get the money as much as the tickets have been sold.

31:16But that relationship, at least from the producer's side, is not clear on the streaming platforms. Of course, platforms themselves, they know they can make a value on their side, but it becomes a bit one-sided. That has been a bit of a problem. We do streaming forecasts as well. And the way we approach that is analyzing social noise in the past. And we created the models to correlate social noise with the households' viewership. And from that, we even started to create like a fair value calculations. So basically, if streaming platforms were paying according to household ownership share, how much they should have been paying considering their subscription revenues.

32:10We also make this kind of fair value calculations. Of course, it is not relevant with what they are paying because they are paying according to their own calculations. that's that's the way the way we calculate we say if it was open like box office that will be the share of the film so yeah i mean the data part is like that obviously there's like a different model requires different type of data content models we look more content data financial models looks content metadata and financial results and then again yeah here our data dependency is a bit reducing with our simulated focus group, this digital twins approach, because there you don't need anyway to pass film's data because we just get people, people's digital twins, so their reaction becomes our data.