BONUS: Learn Video Prompting for TOTAL Beginners w/ NEW LTX-2.5 + LTX Team

0:00everybody what's up welcome humans uh to the neuron live we'll give folks a couple minutes here to to trickle in but we're we're excited to be here and excited you're here uh because we have some cool guests with us today that show off really neat product you're going to want to check out yeah let me let me give a quick intro as people are piling in so we're we're joined here by the team from ltx so if you don't know ltx makes really awesome open weight video models uh with us today is daniel berkovitz who is ltx's chief product officer spent more than a decade turning complex creative tech into products people can actually use appreciate that and alan yar who is ltx's vp of product and he started as an ml researcher before moving into product giving him a strong view of both how the models work and how people use them great to have you guys yeah absolutely thank you thank you for having us yeah really excited to talk about this just just a quick second of housekeeping today's live stream is brought to you by a refs ai search is rewriting the rules of brand discovery.

1:05Aref's brand radar shows how your brand peers in AI answers, what sources influence those other recommendations, and where competitors are winning visibility. Monitor ChatGPT, Google AI overviews, Gemini, Perplexity, Copilot, and more, all from a single dashboard. Go check it out at the link in the chat. Also, as we get started here, if you haven't just yet, please click the like button, hit subscribe up above. It'll help get some legs on this video and get us out there to more people. and we really appreciate it. Now back to video. All right. Real quick, so guys, why don't you tell us a little bit about LTX and LTX 2.5, the new model you just created, for people who's never heard of it, and what does it do?

1:52Yeah, absolutely. So LTX is an open, multi-model, world model. You know, sometimes people are using the term video model. Maybe later on we can talk about like why we're calling it a world model. It's open, like you said, so it's open in the way, it's open source. This is like really something that from the very beginning we thought is a key for success and for people to actually use it. Yeah, it's a 22 billion parameters model that does like multimodal audio and video and also keyframes along the way. Yeah. Very cool, very cool. question how did you build it wow that's a very good question actually i mean um like if you take a step one step back like the company uh because in the company like um our mother company lightfix we did other things before um but when like this revolution kind of started we knew exactly um what it means for our field and we knew that like we either gonna go all at it or it's not gonna work you know it's like in the beginning like we thought maybe we can start building things over other models but very quickly we understood it like in order to be a very significant player in this field and to actually innovate around it we need to have our own I remember it's like not an easy decision to make I actually remember the one meeting where a CEO was like okay we're doing it I was like, are you sure?

3:22But you know, like two and a half years later, we're here. That's awesome. So what's the standout versus 2.3? Curious, what's new? Yeah, I mean, there's a couple of things, actually. I mean, I think like when people say, okay, it's 2.3 to 2.5. I saw some people asking, why did you skip 2.4? I'm going to spare you the I think there are discussions about how the version name is set. But essentially, a few components in the architecture were updated. Because when you're thinking about a new version of the model, we're really trying to cater for multiple needs and requests we hear from different type of customers.

4:05In this release, it's actually special because it pushed two ends of the spectrum. So 2.5 is actually even faster than 2.3. It's based on the same compressed latent space in architecture that that allows you to be so efficient. But because we really improve our distilled model, it actually allows us to run flows that only uses that and really speed up the process. So we're talking about like a 10 second video generation in less than seven seconds. I know I read that and I was like, I was shook by that. That's jarring. Yeah. Yeah, it's like when you build a product, you want it to be something you can touch and tangible.

4:44and waiting three minutes for a video is okay for some users, but it doesn't really allow you to feel like you're controlling it. So like I said, it's faster. It understands the user better because we kind of re-dead the way we do the captioning with a custom enhancer. And on the other hand, which is I think the most exciting thing about it, is that we really pushed pixel quality forward. So we introduced like in two different phases of the generation. In the core diffusion, we essentially create keyframes, like high quality keyframes along the way that allows the video to stay more coherent, like and reduces some temporal issues that we sometimes see like artifacts.

5:33It really pushes like the sharpness and the quality you get at the end. We also replace the decoder with a new one. So basically you can think about it like throughout the process, the different steps of the process, we really push quality up. And I mean, later we can talk, Alon is the one working a lot with big VFX studios and stuff like that. So you can imagine it when you're working for a really big screen. Yeah. You know, every pixel matters. Yeah. When you guys are training a model like this and it's coming together, is it ever hard to hit stop and ship? Is there ever a point where you know that just a few more days?

6:17I feel like you could just a few more days it to death if you're not careful and never have a product. The story of our life. Yeah, it's until the last minute sometimes. You can always push it forward and we have increasing quality and speed. On a daily basis, there are things that are changing. So we set like this release milestone that we all like work towards it, the marketing team, the product team, the partners, tech, of course. And in this release, like 2.5, a bit of the behind the scenes, it was like, I don't know, 15 minutes before the button was pushed. He is not joking. He is not joking.

7:01I believe it. No, no, no. I totally believe it. We pushed it like 15 minutes before the deadline in one hour because the VP of research came with a new thing we wanted to push into it. Oh, man. That is so awesome. That is. So you mentioned big video production companies are using this. How did that come to be and what is it like working with them and serving their needs and all that? So for us currently, my part in the company is like working with partners on the R &D side. So finding like the best solutions for them, making sure they integrate it into their pipeline correctly and just getting the value they should from the model.

7:44And first of all, it's fascinating because we are working. The fact that it's a world model, we are working with such diverse like industries. So from real time avatars to physical AI to VFX to color grading to social media companies. So it's pretty diverse. And for the VFX specifically, so it's like we really had to dive into their pipelines and understand how they work, how they think. We're not trying to replace anything like we are trying to replace some things, but in the pipeline itself and to integrate it into it. So replace like components maybe instead of the entire process. like we're not replacing the artists and it's extremely important it's not just a cliche like we we are actually like the fact that it's an open source model for example it allows them to customize the model to their specific needs so it either on training on their proprietary data and doing the training on their end on-prem without like sharing any ip um or fine tuning for a very specific use case that we're not just shipping as a product company that hey hey, now LTX support X.

8:52It doesn't work like this. So every VFX company and studio just fine-tuned for their specific needs. We learn a lot from it so we can share knowledge in the company and build very specific things for them, such as the quality upgrades that Daniel talked about.

9:15Are you seeing a lot of adoption among professional VFX studios and things like that. I just kind of wondered how the vibe is around that. I think it really changed in the past, I don't know, few months, I would say, both for VFX and to animation as well, actually. Awesome. Yeah, and I think that we are working very closely with Asteria, a production company, film and animation studio that we partnered with. They are like AI hybrid with traditional filmmaking. Great, that's cool. When we visited their studio, we actually saw such great use of the model for real cinema film productions that will end up on the big screen.

10:01So that's amazing. For us, it's incredible because sometimes we see AIs like this fun thing, funny thing that you can do for friends and stuff. Like a toy. Yeah, exactly. But no, it actually goes live to cinema biggest screens out there. So that's amazing. That's really cool. I feel like in Hollywood, or at least the perception from outside Hollywood, is that it's either a threat to jobs or it's slop. And it's really, we've yet to, or at least publicly, we've yet to really have the narrative take hold that, no, this is actually really useful and studios can use it without it being either of those things, which is really cool.

10:41And I'd love to hear a little bit more about how you're approaching that. Yeah, Grant, I think you're absolutely right. And I think it's also what Elan was talking about, because, you know, we've been like going to the West Coast quite a lot in the past two years. And the sentiment has definitely took a big shift, like in the beginning of this year, it was absolute fear in the beginning. And, you know, I think like, from mostly from my opinion, from like from the unknown, you know, it's something new, you're not really sure how it's going to affect you. So you're just keeping some distance from it.

11:10But I think as they saw that, like, you know, beginnings of what they can do with it at the end of the day these people are extremely talented and creative and technical and you know they I think they realize that maybe there's like opportunity for them in that and what we're seeing is that when they actually step into it as really creative people they have the best ideas and they make magic with it you know it's our job to kind of like make sure that they can actually use it for those purposes you know we talked a little bit about like what's new we're we're at least now like the native HDR support.

11:45For VFX Studio, it's a go or no go. They cannot compromise on these things. But it's been a real joy to see that and how they're using it. The best part is that when they surprise us with the thing they're making, like, oh, wow, that's really cool. You made it with LTX. That's so nice. That's awesome. So let's talk about how people are using it. So what tools do they use? How do they work it into their workflow? So like help us wrap our heads around how you actually can use this thing. So yeah, just throw a few examples. So one could be like for a 2D animation, we have like partners working with a model with like their artists are working on like the keyframes and we like LTX is doing the in betweening for them.

12:29So we created like a very specific kind of training on their proprietary data set to have like a model that actually create like very structured non-smeared in-betweens because everything in 2d like the lines are super important so we actually created a solution for them and they are now producing the in-betweens um with the model other partners like are on their way to integrate it as well um so that's on the 2d animation we also have like on the same visit the same day we visited asteria um or a day before we visited like markup robotics um so it's uh a startup from San Francisco that fine-tuned the model to move a robot and do like all sorts of actions like making a sandwich and stuff.

13:18Wow. And that was like maybe the most surprising moment for me at least. Working at Lightrix for like eight years coming from creative background and stuff. So robotics, it's not something that I am familiar with. And suddenly seeing how they just use the same model that we just saw like on the VFX studio that they use it for cinema and they use the same foundational layer the same infrastructure but fine-tune it for their needs that was like pretty insane and other use cases could be like real-time chat like two-way conversational chat with an avatar like in a video and audio that kind of interact with the user.

13:59These are just a few. Awesome. So if, for example, for the 2D keyframing one, how are they actually, what tool are they using? Are they piping the model into an animation software? Are they using a separate tool like Comfy UI? Do they have a proprietary app that they built so that they can use it? How does it, from a mechanical perspective, how does it actually get used? The partners that we are working with currently, working with Adobe Animate, most of them, and then they're building additional plugin or tool that handles all the LTX generation. Then they just work again, hybrid AI, you got the generation, you have the in-betweens, you pick the ones you like, you place it on the timeline, the traditional one.

14:52Awesome. That's cool. And other partners like in the community, as you can imagine, most people are using ConfUI that we also partner with. And we also have an API that support that. Again, for some companies, I think sometimes it's really critical for them to run it on-prem, but others might actually benefit from doing it through our API and kind of let us handle that part. So it's also available there yeah well before we uh before we dive into the demo uh find if we bounce through a couple questions from our viewers real quick uh first question comes from damian rufus and he's asking what actually makes a great first prompt here what context structure constraints examples are framing are we maybe missing that can really improve your first response I mean, I can take it and Alon can add up for sure.

15:53There we go. Yeah, I think that it's all about managing to describe what you want. And I also saw different people doing it in a completely different way. So some people start with something very simple and actually let the output of the model guide them in the right direction. So you can start very simple. We have a custom prompt answer that does the matching of like whatever prompt you put and kind of make sure that the model understand it properly and that it fits the way the model was captioned um and then basically based on the output you're getting and that's a part of it like being really fast you can actually do that they start like adding more and more details like i know yesterday i was working on a prompt of like a car spinning and i started then it was too slow so i started adding like no make it go really fast to really get to what I want.

16:46A completely different approach that you can take is really go and describe the scene in a lot of details. The model can definitely take that. And what that can give you is really how to really control and guide, direct what you want. So starting from what does scene look like, what are the actions that are happening, the order that are happening, what sounds are playing in the background, if there's a dialogue. So I think if you can do it in a very structured way, starting with the style, the camera motion, the setup, and the actions, you're very likely to get what you're asking for. I always tell people to think about their senses, too.

17:23Think about what you're like. I feel like the more information you can give it, the more you can get things dialed in quick and easy. I must say that prompting is hard, and we hear it all the time. Like this black box that you just prompt something, try your best and get something that usually you didn't expect. Like we are constantly working on finding alternative ways to guide the model. So either with a video condition or audio condition or multiple keyframes or anything that can help the model understand your intent in a way that is not textual. We see, not just us, like the people we are working with and the artists at the end they just prefer to talk visuals and not like text at this yeah so yeah what are what are some ways that they could do that like if you're an artist or you think more visually what what are some ways that you could apply your your vision to the the model so first first of all we're gonna see a demo of it but audio is a great condition sometimes you just have like a very strong audio that you want to rely on and this guides the the model like amazingly well it just like drives everything the the how expressive the person is the the environment the people around it um so audio is one thing for video uh kind of references we see a lot of um different uh variations of it so one could be like i don't know for 3d artists or people from the 3d space so having like a blocking animation or something that is very basic from the cg software to just guide like the camera trajectory or the placement of the objects in the scene and having this as the um baseline or like the the driver for the generation or taking a pose of a person like talking with his hands or any gestures and just copying it to the generation instead of prompting, he raises his hand at second two, and then it's impossible sometimes.

19:34Yeah, it's such a more efficient way to communicate if you just have a picture of a guy doing this. It is. We've got one more question here before we dive into the demo, and the rest will hold off until after. But is there an application to creating training videos for businesses? Or have you seen anybody doing that yet? Yeah, so like people using, say, like LTX models to create a training video for a business. Like training in the sense of like an explanatory video? Yeah, onboarding, you know, like tutorials, that sort of thing. Yeah, no, absolutely. So we actually worked in the last two months or so on like an avatars feature that can take like long form.

20:22I don't remember exactly how long we got it to work, but I think it was 30 minutes. Yeah, 30 minutes. So, yeah, and then, like, you can, if you can feed it, like, with your information and data from your favorite LN, probably, then you can absolutely do that. And something else that was really cool that the team was working on is a cut on action. So you can actually do this from multiple angles and keep the timing and everything intact. Yeah, so it really gives you a sense of, like, seeing someone talking to the camera. That's so awesome. I'm now terrified as a podcaster. I'm just kidding. Oh, yeah.

20:59There's so many hours of my face on YouTube. So it could totally fine tune it. It's ready for training. Yeah, exactly. Exactly. Well, go ahead, Grant. Yeah. No, no. You go. You go. How about if we take a look at it? Can you guys show us a tool? Show us around? Maybe see some videos? Sure. So first of all, it's important to note that it's not like anyone can consume LTX in different ways, like via ConfUI, using code through the API. We also have LTX Explorer, which is kind of the platform that we built for everyone to test and evaluate all the different features. So this is the platform we're gonna see today.

21:42It's great and I recommend everyone to just go and test things out. So the first feature that we wanted to show is just, oh my God, can you hear the sound? Yeah, you might want to turn the volume down a little. Wow! Wait, it just played. No, I'm not sure. Go ahead and try it now. Sorry, it's going to be a bit loud. That's okay. It's just a random generation.

22:15So I just prompted like, what was it? A cinematic shot of a rhino charging in New York City Street. So a very basic prompt. I didn't try to create something too directed or like just something to get out of the box. But here I think we have a different example already in the platform of a Dalmatian dog running fast along a sandy beach with some like description of the environment, the lighting, the camera. Where is the van, the red camper van behind it? So I just hit generate to start with something. And here on the platform, we can just choose the model that you like, the aspect ratio, the FPS.

23:03Sometimes we do see that higher FPS gives better results in terms of like, I don't know, for fast motion or very dynamic scenes where you get more kind of better motion blare looking rather than artifacts of like the model sometimes. Yeah, that makes sense. Can we zoom in on the prompt a little bit and maybe you can break down the structure of it? There we go. So it seems like you start with the subject and what they're doing, the action. That is a bit about the aesthetic, the vibe, the tone, also adding some details on the environment like the water, the forest, the vintage grim and red camper van.

23:48then about the camera, how it moves alongside the dog, about its depth of field, the lighting, and then also about the sound, because we are like multi-model models, so you generate audio and video together, and then you can also direct the sound from the prompt. And here we just get like a few versions. Oh yeah, there's a camper van, there's the dog. That's so good, you guys.

24:22Wow. Wow. It's crazy. He's just running and running. In this one, the very easy. In that one, they parked it on the water. Bold. But I think the fun way is if you give it a very specific prompt, like this one, that you have everything in it. So you can then just start iterate on it. So here, for example, I can add like until it runs into the water. Let's generate this one and see. Let's give it eight seconds instead. So the fact that I... What's the URL for LTS Explorer, by the way? I want to share it with people. So LTSIO is like the main website. And from here, you can just hit try now or straight to app.ltx.

25:20Perfect. I'm going to link that in the chat if anyone wants to follow along as we're doing this. Great. Yeah, and maybe something to know about the platform while it's generating is that this is connected directly to our API. So you can just top up as much as you want to try. There's no payment or something. You pay for your generations, but nothing else. You don't have to commit to anything. So free to sign in. Oh, wow. Yeah, that's good context. Do you just like... Oh, Corey, cut out. Sorry. Can you hear me now? Yeah, try it again. Do you just like buy some credit up front and generate away?

25:59Right, yeah. So you're getting like something to begin with, but then you can top it up directly from our console. Awesome. Alon, I believe you were about to say something before I cut you off there. Sorry. Let's see if it runs into the water. Perfect. We're rooting for it. I noticed the van got moved up the hill. Yeah, did it! Something happened to him. He immediately hit it.

26:33Yeah, there you go. That one looks great. You start to direct it, and again, it's not the same generation because we didn't use the older versions as the reference for the next one. We just used the same song, so it doesn't commit to anything from what we saw before, but still you've got the same vibe, and you can just keep on iterating until you get something with your life.

26:57So what if I wanted the Dalmatian to be the exact same spot pattern every time? How can I enforce that? Do you have any good advice there? For these kind of things, I think this kind of moves to the more advanced flows of like using a video reference and start like iterating on it. So we have like, I don't know, maybe I can find it here. Spring shots even maybe. Yeah. Or even start with an image. So if you go like that, these are all text to video. So if you start with an image, you know exactly what's going to be your starting point. Yeah. Yeah. That's probably like the way I would do that. Yeah.

27:33Yeah. That's right. Right. I think for some tasks, if you just want to use the same dog in the same environment, just iterate on it in a more precise way, let's say, or very specific way, then you do have all sorts of flows like relighting or adding things into the scene and stuff like that, or day to night changing the lighting and stuff. So you just use the same video and iterate on it. So this way, everything remains the same beside this change. that's really yeah that's awesome cool we do have a few more cool things that we wanted to share yeah that'd be great let's do it yeah yeah nice so yeah we're just we're just here for the ride the next the next is audio audio to video well which is extremely cool and can be used in very smart ways so for this we've prepared some footage in advance wait let me grab it okay so here's the audio file here's the image we're seeing the one second yeah I mean we're still seeing a switch to a tab yeah yeah oh so there's an option that says share this tab instead.

28:56Share the tab. OK. There we go. Yeah. I knew something's going to. Yes. OK. I love this guy. Wait, who is that? Who is that? OK, and we have this voice sample. Please take a moment, if you haven't yet, to subscribe to the channel so you can check out all of our interviews with the people building and impacting AI every day. All right. Let's just give it a shot. I've begun, Corey. This is amazing. Usually we don't add any prompt to audio to video, to be honest, because the audio condition is so strong. Let's just try and see what happens.

29:40Yeah. Oh, I'm so excited. It's getting personal. It is personal, but it's okay. It's okay. You mentioned the amount of content that you have on YouTube that people can use and abuse. And just how easy it is. We almost did it. We almost trained something on your voices. The instant you could duplicate a voice, my horrible, horrible troll friends went stripping videos off of YouTube. Started just sending you audio files of yourself. Audio files of me. And I was way less cute than I am as a puppet. Oh, actually, speaking of puppets, while this is loading, Amanda Barton in the chat asked, can you use a real-life background with a human-made puppet creature and create variations of that creature with LTX?

30:36If we need to break that down, we can. Yeah, wait, I mean, like... I'm trying to process it. So, like, okay, one way to interpret that is, like, can you film your own footage of yourself, let's say, and then animate that and use that as the starting point and add maybe additional characters into content that you've created yourself. Yeah, so I mean, I don't know if anything I get to show you today, but you can do in-painting with the model, so you can definitely mark areas in the videos and add elements to it. So that's probably the way I would do that. We also have, you can extend the video, so you can start with a certain footage of yourself and extend it.

31:20So it's going to maintain, it's going to be consistent in terms of your voice and the visual and the way you move, but allows you to kind of keep on. So that's probably how I would do it. I don't know, if there's maybe a follow-up chat, we can go into more specific things. For sure. I'm curious to see the video, though. Yeah, let's see this first. I'm going to see what he does and says. please take a moment if you haven't yet to subscribe to the channel so you can check out all of our interviews with the people building and impacting ai every please take a moment if you haven't yet to subscribe it's perfect it's so good can we can we just get this version of you