AI Just Democratized Filmmaking (w/ LTX Co-Founder)

30:29Yeah, yeah. So they're even smaller now and they're still really good.

30:32Yaron Inger:So I think that overall, like the ecosystem of open source models, we see that, you know, at some point it wasn't unclear whether like the open source models are going to close the gap with, you know, with the frontier models that are closed source. I think that, you know, we see now that it's definitely the case. Sometimes it's a bit, you know, lagging behind. But overall, I think we're in a very good place there. So I think that all of this and, you know, the thing is that it's very, very modular, right? So if you just want to run like an LLM, not locally, you can do that and then run the image generation locally as well.

31:15Yaron Inger:So you can, you know, do this mix and matching depending on your hardware, depending on the current state of the art. kind of where you're sort of workshopping your idea before you're ready to say go what kind of idea i don't know like i guess i'm thinking like you know the idea of if you're ideating around like here's this character i want in this video i want to make that like you might use an image model there or specifically image generation to kind of refine what you're looking for before you come in and say like you know bring this guy to life yeah I think that this is this is basically part of what we do in LTX desktop where you can have your own you know media library and you can see all your generations that's why it's also built for AI because for example one of the you know these are really small things but when you build your own you know your product for yourself or for a specific use case this is becoming extremely usable so for example when you have multiple generations you just have arrows where you can you know scroll for these specific generations and not have like you know million like generations like flat in the the same like media library so you need to know which went to each generation and and so on and i think this is this is why it's going to become like very cool because you know now you can build the application for yourself um but you can also use like the same models.

32:45Yaron Inger:So everything that you learned about how to integrate with LTX2, how to use it to produce the best results, you can, the packaging is something that you can very, very quickly create depending on your needs. So in contrast to how we build software, you know, I want to say like a year ago, but it's even like six months ago, right? Where you can now build this thing, you can now build this thing for, you know, almost like a one-off like project and then throw it away and build something else depending on your need. What is still going to be there is the model layer because the model layer is something that's, you know, still very, very expensive to build and require a lot of expertise that you, you cannot just say, hey, LLM, like build me a foundational model and then you get that.

33:35Yaron Inger:So we do the hard lifting for you guys. And now you can imagine and build whatever you want on top of it. That's awesome. I think the other use case that we didn't talk about would be like, can you actually hook up an agent to this and let the agent make the video edit for you? Yes. So this is definitely something that's definitely on our mind. And, you know, agent is like a big word for like, it's like, it can be like, you know, the use cases again, like are endless. But one thing that's definitely very interesting is how you can use an agent to generate clips that are, you know, multi shots.

34:16Yaron Inger:So everything that you basically said about storyboarding, so you can do it like on the fly with an agent. I think the interesting part and like the potentially like missing piece for that is that agents need to somehow like close the loop. And what I mean by that is that, okay, you've done something, right? So you've done some action. Now you need to have some score on whether you actually managed to get the good results or not. Because otherwise, okay, so you even now can say, okay, generate like 10 videos for me and, you know, just save it. And that lets your next loop evolve and the next loop evolve.

34:56Yes. Next loop evolve. Yeah.

34:58Yaron Inger:Yes. So you need to be able to say, okay, whether like the result is good or not. And that's really depends on, first of all, like the quality of LLMs, right? and how well they understand videos because these are now the best models to actually analyze videos and say, okay, it's good it isn't. But also, I think it leads to a much bigger question of whether there's even a single objective function that you're trying to optimize. And my answer is clearly no. And this is where creativity comes into play, right? So we're not saying, by the way, in contrast to many other companies who build video models, Like this is going to destroy creatives.

35:39Yaron Inger:I think it's like the other way around, right? Because what we've seen until today, like with both like with, you know, the diffusion models, so like image, audio and video models, but also with LLMs is that these models are not creative, right? These models like learn like a ton of concepts and they know how to mix and match them together in ways that we don't even like fully understand, right? But they're not going to write the next underground song that no one has ever heard that eventually is going to get adopted and becoming like part of the, I don't know, mainstream music. Right.

36:21Yaron Inger:So that's actually where humans like enter the picture. And taste is going to become very, very important. important. Okay. And I think this is going to be become like part of the IP of everything that's, that's going on. Right. So, so you are going to be the one that's actually will need to define, even if you use an agent, like what is a good output for you? And it may, may be something that looks really bad in terms of objective function to someone else, right? Because they like something that looks completely differently, but you need to be, so what we need to do, basically is to be able for you to provide like these kind of tastes right right whether it's through fine-tuning or defining to agent like what you like which is also like i think a big open question like how you even do that right um so the the productization like of this is still like an open question and no one is doing it right so that eventually we can with high probability generate content that you like and maybe do like some kind of online learning where we ask, we continuously ask you questions, right?

37:34Yaron Inger:And you answer or you say, okay, I like this. I don't like that. And maybe explain even why, right? So maybe we ask no questions or not good enough. And then we can somehow learn from that. And maybe, you know, this is only one thing that you like. So you can save a preset of that and use an agent to generate content with that specific person. where it's like, I like this for this aesthetic for this given type of content, but I want a completely different aesthetic if I'm telling a different story. Like my advertising aesthetic is different from my action movie aesthetic, which is different from, you know, if I want a drama or, you know, whatever else.

38:09This all makes me think of Rick Rubin. You know, Rick Rubin, the famous music producer. He's done the Beastie Boys, Run DMC, Tom Petty, Johnny Cash, everyone. You know, he famously doesn't play any instruments. He says he makes the big money for the confidence he has in what he likes. And the thing this makes me think of is I often think of AI as like some kind of a great enabler almost for people like that to be able to bring an idea out in some way that previously wasn't possible. But it doesn't mean it's not still a great idea, a great picture. it also replicates the hollywood the hollywood model where they're constantly doing surveys or pre-screenings to see how people like it getting feedback before they release any movie and killing good movies yeah i mean they do that quite often because it's very expensive but this i mean the coolest thing about ai video generations if you think about it from like a first principles perspective is it lets it reduces the barrier to entry for anyone who is trying to make movies it is so i have a screenwriting background just so people don't know it is so expensive to get anything made like today and and and and it has never been more expensive even though it has never been easier for us to actually go out and film things but it's just so little stuff get a permit for a beach off yeah near la somewhere and see what that costs you yeah and so it's like you don't think about the the you know the the take where you look at this from the negative is it's like, oh, well, the big studios are going to use this to, like, compress costs and create less jobs.

39:45But on the flip side, the benefit is that now anyone who wants to be a filmmaker could create with these tools, which I think is really awesome. And you making it open source makes that actually possible, where you don't have to pay Google or anyone else to do this. And sci-fi is going to be so great.

40:04Yaron Inger:That's for sure. That's for sure. A hundred percent. You know, I think that in a sense, it's, again, maybe a bit counterintuitive, but the fact that everyone can create now and we're big believers in democratization of creativity means that, you know, the content that will shine is the actual content that, you know, when you look at like the basic, you know, storytelling, the ability to convey like feeling, right? And it's funny to say it in the era of AI, where everything is maybe very mechanic. I think that people still haven't cracked it. Maybe the models are not, at this point, good enough to have these proper expressions that you can actually make you feel.

40:53Yaron Inger:But we have people creating content internally that was also shared that I think it's pretty good. and getting like, you know, I actually felt something when I watched some of these videos. So I think this is, it's actually, you know, instead of like paying million dollars for production, like it's becoming about these core values, which I think do happen and do exist in Hollywood, right? Like there is a reason why, you know, these directors, these filmmakers are actually there because they're able to do it. And I think people will just get tired of watching like a ton of AI slop like very, very quickly because it will become the norm.

41:35Yaron Inger:And models are already managing to produce really high end content. but it actually makes you, you know, really like think about the story that is being told, how it is being told, like how it is being like directed and all these things that are, you know, up until now, you know, they were very much correlated. Like if the story was great, then probably, you know, there was very good production like behind it. And now it's not necessarily correlated, but we will, you know, get through it. And I think that everyone will, you know, filter the content that they eventually want to watch. And the content that will be getting a lot of traction will be the one that is still like about good storytelling, which is something that is true before AI got very, very popular.

42:31I think it makes it, it changes the economics because if you look at it before, it was this has to make money at a large scale for it to be worth us making it, even if it's a great story. And this changes the economics where now, even filmmakers that actually are making movies in Hollywood right now can make a movie that perhaps they wouldn't be able to, you know, make or get produced in the current system. but it's a good story and they can go out and film it and they can just fill in you know actually still film with real people but but be able to use these tools to fill in the background and and fill in all the things that they would have needed the you know extra hundreds of thousands millions of dollars to to do and now they can let the story a good story can live on its own things like expensive retakes over minor continuity errors and stuff yeah yeah so it's very cool uh sorry Corey, you were going to say something.

43:23Yeah. Something I want to make sure we address while we're on here, and we haven't really brought it up yet, is realistically, what do I need system-wise to run LTX2 on my computer? And specifically at HD level, let's say.

43:44Yaron Inger:Yeah, for HD level. So we basically currently, our requirements are RTX 5090, which is like the highest end like GPU for NVIDIA. But again, as I said, like people manage, and in this way it will run like very, very quickly. Okay, so you can generate, I think I'm not mistaken, but with the fast model, which is like distilled, you can generate like a 720p video of like five seconds in like 45 or 50 seconds, something like that. So it's not, yeah, it's really not a big deal. Like, again, that's why people are so excited about it because, again, the model is like the fastest model among the open source models.

44:29Yaron Inger:I don't know about the closed source, right? But I believe also among like the closed source. So because we initially also built it for scale, for, you know, for, you know, massive apps, you know, with a lot of users. So it would make sense like from unit economics perspective, but now it pays a lot of dividends for, you know, for the community, for the ability to run it very quickly. We saw a lot of people in the community that managed to run it on like 3070 and, you know, with eight gigabytes of VRAM. Obviously, it will run, it will not run as quickly. Right. And it's also like older hardware.

45:12Yaron Inger:We also saw some people managing to run it on Apple Silicon. So running it on laptops and Mac minis. Now there's the trend of everybody buying Mac minis to run Open Claw and run LLMs locally. So I would say that I'm personally very much interested in that and what we can run on this hardware. just because it's very accessible to many people. So this is also the nice thing about open source. We have a ton of people from the community who are just trying different things. And if you produce something that's of value to them, they will also try to make it run on their hardware, will tinker with it a lot.

46:04Yaron Inger:And eventually they also contribute back, which is very, very important to us. And eventually, I think this is like the nice cycle that we have here, like this flywheel where we release something, the community makes it better, also learns how to use various things, even finds bugs sometimes. And then we manage, because of their input, to create a better model and then they enjoy it and vice versa. So this is a very nice interaction that we have. It is. I run most of what I'm doing through an RTX Pro 4000, which should run it pretty well. Grant, you're an RTX 6000, right? Is that correct? Yes, yes.

46:51On my highest performance machine. But I also have Apple Silicon, and that's what I use for my coding work that I do. So that's like 20, 24 gigabytes unified. so I wonder how that could do.

47:09Yaron Inger:So this is going to be like a borderline because the model by itself is like 20 billion parameters so if it's like let's say you need to have it quantized and then also leave some space to run it but this is the nice thing like you can just try it, see how it works probably there's someone from the community who actually managed to run it like that. And there are people who are not just, again, just vibe coding their own memory-efficient solutions without even knowing what they're doing. Sometimes it's insane, right? Because they just say, okay, just make it. They tell the agents, okay, just make it run on my GPU.

47:56Yaron Inger:And then it's doing crazy stuff, right? They just looked at their solution. Like, you know, it's insane, like, what it manages to do, and it works. Just last night, Codex asked me. I was working on a little project, and it's like, I need to install CUDA. Can I go do this thing? I need to be able to better leverage your GPU. And it's like, okay. And it just does the thing. And a lot's changed even in just the last 60 days. It's really a wild time. You mentioned open models yourself. Do you have any specific models that you like to use as a CTO, actually building the technology? Do you have any preferences for your coding, for your personal use in terms of language models?

48:45Yaron Inger:So for local models, I played with Quen right now. I think it's a really great model and runs locally extremely well and runs very fast. For images, I think the best models right now are both Flux. The Flux Klein is actually a pretty good model and it's very, very small. It's actually an interesting trend to see. Also with Quen, obviously they also released the big model, but with Flux and also with Z Image, which is another good image model, Chinese one, So these models actually became smaller. So initially the open source models were, let's say, like 20 billion parameters, if I remember correctly, more or less that size.

49:36And these models are like six, seven billion parameters.

49:42Yaron Inger:FluxKline has a version of like four billion and nine billion. But they're actually becoming better as well. Better and faster, yeah. How is that happening? What's your thesis? So that's a great question. I think that there are two main aspects here. So the first is that the data that the models are trained on is just becoming better, you know, cleaner, the captions are becoming better. So when you train, you have video and text caption together or an image and text caption. So as LLM is becoming better, so you can also use them to caption the images and videos better, which also improves prompt adherence eventually.

50:30Yaron Inger:So the prompt that you write, the model actually adheres to that better. So one is purely just better, cleaner data. And as time goes by, like the teams are becoming better at filtering the data better, understanding how to caption it better and so on. The second part is that the models that we see eventually are not the original models that were trained. So imagine like a very big model. So for example, if we take like the LLM example, I don't know if like the original model that Quinn trained was like 350 billion parameters, but let's say that this is like the original one. And then you basically have a process called distillation where you take this model and take, which is called the teacher model, and then you create a student model, which could be in this case like a smaller model.

51:26Yaron Inger:and you basically let the student mimic the teacher through like another training process, right? And in this sense, you basically create some sort of compression, which in some cases even creates better results. And the reason is that for large models, what usually happens is that they have memorization, right? So they actually, for example, if you think about it visually, they actually remember like, I don't know, specific, you know, scenes from a specific clip, specific actors. But actually what we want them to learn is how the world works, how the physics work, right? How styles, different styles work, artistic styles and so on.

52:15Yaron Inger:And by actually creating this compression where you say, okay, I'm just taking like less, like an order of magnitude, to less parameters and I'm trying to teach like a smaller model how the big model works, how the teacher model works, you sometimes even get better results because you basically take the entire knowledge of the teacher and learn how to generalize the knowledge and avoid memorization. So this is basically what we're seeing right now. And with Flux, you can also see this because you see, for example, the larger models that are like 20 billion parameters and even more. So you see like the progression there.

52:56Yaron Inger:It's not that they trained the model from scratch, probably. They just took and distilled it. That's so awesome. I have one last question, but Corey, do you have something else before? No, no, you go ahead. I was going to let him go. No, no, I have one more. I have one more. And then we'll be done. So do you have anything that you want to see in the development of video models going forward? I proposed an idea in our prediction episode where I was saying the thing I think would impress me more than HD quality. I think actually the distillation thing you're talking about is really huge. Like being able to do the same quality with much less gigabytes of VRAM.

53:34But the thing I really want to see is like almost like real time generation where you can like draw on your screen and have it like fill in and generate around you. But what do you want to see with the models that you're creating or that you'll work with in the future? Like, where do you want to see this all go?

53:52Yaron Inger:Well, there are many things. I think that one, you know, you referred to resolution. I think that resolution is key, especially as we cater more for professional use cases. So professionals really care about every frame, right? They want to look at every frame. They want it at 4K. They want HDR, right? Yeah. So you can actually do post-processing. So you need like 12, 16 bits of color, but in a real high dynamic range. And they want it to look amazingly well. And I think that we're working on it, but it's still not like 100 % there. So we basically want to have a process where you can infinitely increase the resolution of a video.

54:41Yaron Inger:This is like the ideal, like the vision there, right? And this is possible because the model by itself knows how to invent details, right? And details are, they're actually fractal, right? So if you learn details in small scale, you cannot, these actually repeat in finer scales as well. So there are like technical details that basically we need to figure out, but this is really, really important for professional workflows. we want you to be able to generate, you know, 8K videos if you want, and you have the budget for it, right? So this is an upscale, like, these videos to this resolution. So this is, like, one thing.

55:24I think in terms of, you know, controllability is one thing

55:31Yaron Inger:that is also very, very important. We also, like, work in that direction, and we want to have very, very precise controls, right? So I gave a few examples like in this podcast, but, you know, I can think of other modalities that are really, really important. You know, there are models now that offer like very precise motion control, but I think it can be better. And we also were talking about like emotions. I think this is extremely important. And as one who worked like many years on Facetune and dealt with, you know, with, you know, facial like selfies and so on. I know how delicate it is and how many neurons we have in our brain just to analyze, you know, each one's expressions to actually understand like what you're feeling right now.

56:14Yaron Inger:So every pixel here counts, right? So this is extremely, extremely important. It's just like one example of a control mechanism. And everything that you said about real time is very, very, very true. So we actually, because the model is extremely efficient, so one thing that we're also trying to push is making the model what's called auto-regressive. So having the ability not to generate, you know, five-second clip or 20-second clip, as we can do right now, but generate an infinitely long clip video that's just playing. And I think the use cases there, you know, for filmmaking, maybe it's less interesting.

56:59Yaron Inger:You know, you have a few films that are, you know, like one shot of like eight minutes, you know. But it's not extremely common. It's something that you may want to do, right? I think it's very, very useful for, you know, these real-time experiences. And I'm not talking about the world models where you can, you know, press the keyboard and move around and like the gaming style. I'm talking about like having these experiences where, for example, I can talk to a teacher, right? And, you know, LLMs are doing an amazing job today with, you know, with teaching me how to do various things. Like I post to LLM like a research paper and I barely read papers today because I just say, OK, TLDR this.

57:41Yaron Inger:And then it explains something to me. And then I ask more questions. Right. But I think this can completely change the way we educate ourselves. And, you know, my little kids are going to school. So, and I'm thinking like, whoa, they can just like use like an agent, right? That is actually, you know, a streaming video. Let's say they're going, they're studying like history, right? So you can talk to a history teacher and it's not just like talking to an avatar, like all these avatar models today. You can just talk to a teacher, right? And you see like full body and then you say, okay, I want to learn about whatever, like World War I, and then, you know, the background like switches to like a scene from World War I, and you can, and you know, the teacher moves around and show you things, and then it switches to like a map of the world, and you're having like this interactive like lesson where you can, where you use like LLMs and real-time voice models together with LTX2 to generate like this entire experience, right?

58:48Yaron Inger:And I think this is going to be the future of education, right? So to me, and I also like very much like into like education and everything that's happening with AI today is going to completely change. I think the way we learn things. And so this is something that I think we'll see very, very soon. And again, like the options here are quite endless. They are. this is something grant and i have talked about a number of times too is the idea that i was like i want to learn physics from albert einstein i want to i want to go back and talk to i don't know plato i i you know and and you bring up such a good point you could like go into black you could like go to a black hole with albert einstein and he could explain it yeah yeah i want to see the inside of a tornado without having to die i want to you know but uh that's stuff that you know your kids will do that in school i just absolutely believe that that's not 20 years away not even 10 no i could i could see it yeah we're in this awkward unfortunate position with education right now where i feel really bad for people who are like juniors in high school through like about to finish college there's this window of people who are in an education system right now that's like not really preparing them for what's on the other side of it but it's also because they don't know what it needs to look like inside of it yet and it's um but yeah the future is going to be uh really wild when it comes to what you could do in a classroom.

1:00:31It's, uh, think of how many people that would be engaging to versus, you know, a textbook and a blackboard and a, you know, kids who just get lost in that system without having that immersive experience, like to unlock all this stuff. So much. Yarn, thank you so much. I would talk to you all day. So would Grant, but we're so grateful that you came to join us today and had a wonderful conversation is, uh, what's the best way to go check out what you all are

1:01:00Yaron Inger:doing what you're building and where to go try it great so uh you can go to ltx.io uh check out the model check out the api and all the links you can go to hugging face and download the model um and enjoy and give us feedback we really like it and by the time they're watching this ltx desktop will be available too right yes the links will be on the website as well yep we'll make sure that if you're watching this, we'll make sure you have links. Trust me. Yeah, we'll link the repo and everything else. Hugging face. Yeah, excellent. Excellent. Well, everyone, that's all we have for today. Thank you so much for taking the time out to watch.

1:01:38Please take just a second to hit that subscribe button so you don't miss the next awesome conversation we have with someone who's building the future of AI. But that's all we have for today. So farewell for now, humans and lobster bots.

1:02:01Thank you.