Step-by-Step: The State of AI Filmmaking Workflows

0:00The top three tools that are just, if you're kind of just trying to like in a chat mode, easy to use, that kind of hits the bases. is going to be runway references, chat GPT, and flux context.

0:14All right, welcome back, everyone, to Denoised. We're going to do sort of another deep dive episode today, looking at AI filmmaking workflows and what sort of the typical workflow that a lot of people figured out is happening, tools that are used along each step of the way, and also how that's changing a bit with more impressive video-to-video models. Yeah, and if you missed our last episode, we did a deep dive on Comfy UI and some of the fundamentals of image generation and image workflows. Give that one a go before this one, if you like. Yeah, yeah, check that one out. That goes into the nitty gritty of like how AI image generation works.

0:50And this one, we're going to be a little bit more in flow of... More creator driven. Yeah, and just sort of the workflow and different tools. And most of these are going to be off the quote, off the shelf tools, websites that are pretty consumer friendly to use, but sort of how you have to kind of think about the process each step of the way. And so a lot of this has adapted. Well, first off, I started building out this flow chart because I started getting asked questions. You know, people are like, I want to, you know, make an AI movie or I want to use AI to help make a film. What do I do? Where do I start?

1:20Where do I start? Then I was, I was thinking about, I'm like, well, you know, I'm like a bunch of different tools and options would pop in my head. And it's like, there are a lot of questions of like, how you want to think about this. And also there are multiple tools along the way that you sort of have to use ish tools are getting better at kind of integrating to kind of be a one stop shop. Yeah. So you don't have to hop around between a bunch of tools, but we'll cover all the tools just so you can because some are better at some things and some are better at different things. And it depends, too, of like what you need.

1:48It's like, do you need a more action based shot? You need more static shot. Do you have characters performing, characters talking, multiple characters? So there are all kind of things you got to. Yeah, and regardless of what tools you use, we have a very agnostic workflow here that you'll find beneficial just in general. As the tools change and evolve, the flowchart here that we share won't really change. Yeah, I mean, I've seen this also evolve from watching all the creators on Cinema Synthetica, the AI on the lot filmmaking competition, and they all basically use some form of this. And it's hard.

2:19It's like text image, image to video. that's sort of the gist of it and that was sort of in some fashion yeah the workflow that all of them used today today also that is still a good workflow too if you just have a computer you don't have any resources you don't have a camera you don't have like access to film some stuff you can make something visual a film oh yeah uh just using your computer with that workflow you can make a pretty convincing film if you do it right yeah and have a good story and this all goes to the root of you know good story and good and something to yeah to tell amazing visual making techniques amazing visuals won't save you from a bad story exactly exactly all right so first up starting out with you got to have a good story yeah and assuming you've got a good story you know what you want to say you want to shot design that story you could do this yourself and obviously each step of the way you could kind of figure this out yourself or you could lean into some of the ai models um lm so in here this case for the tech stuff the general lms are going to come into play so either google gemini uh claude or chat gpt Gemini 2.5 I've been hearing a lot of really good stuff and I've been using it a little bit more and more, especially with like shot design or kind of creating prompts for shots.

3:26I use Cloud a lot. I found Cloud's always been sort of the best at creative writing tasks and stuff like this. So you could start off with... Is it Higgsfield specifically designed for stuff like this? Higgsfield would be more later on of like you want, once you're like, you want to start generating images or turning images into video. So this is still in the kind of ideation planning phase. Like, so first off, you figure out, okay, I want to make an action scene of like a guy, you know, being chased through a mine tunnel on a mine car. Temple run again? What kind of shots would I need? You can help brainstorm the shots.

3:59And then from there, you could also help generate the prompt to generate your image. So go from text to image. Before that, though, you probably want to figure out a consistent style or visual of how you want all the shots to look. So you don't generate a new shot each time and it has a completely different visual look. A couple of ways you can do that. One is you just help you describe what kind of look you're going for and you create a descriptive text prompt of the style. Right. Other way, which I found comfy, is you can make a Laura. Way better. Way better. And you can guide it with an existing image.

4:33Uh-huh. So if you have like a cyberpunk 2077 type look, you know, futuristic Blade Runner-esque thing, then give the model imagery of that city. And it'll sort of start to get a good look and feel of that city from just a few models, from a few images. Would you use existing images of just like reference images from other movies that have the style you're looking for? or would you try to generate like in mid journey images that have the style and train it on those images i don't know i mean there's an ethical question there i would just uh generate stuff specifically for influencing generation if that makes sense right create an image like let's say if i create text image in stable diffusion yeah in that style 20 images or so with the cyberpunk style and feed that to train the laura yeah okay other option other way for creating style if you're working in mid-journey, mid-journey has srefs, which are these codes that trigger and call up a specific style.

5:31So cool. Yeah. And so, and you could also, similar to the lore, you can give it a bunch of images and it'll create a style that'll call it up by that. This only works if you're working in mid-journey, but it works really well. Yeah. And the technical term is style transfer. So it'll sort of take whatever the image is generating and then apply like a filter on top of a style transfer. Yeah. And then the other way is you can use reference images. And so some of the tools like runway references, chat GPT, you can or Google image, image in flux context, flux context, you can give it another image.

6:10And so you want the style of this image and not necessarily any of the details, but you just want the aesthetic look of that image. And you may have to run it a few times, but you can use that as a consistent. That's how we got to all of the uh spirited away miyazaki stuff that those were all those are all transfers right but that was uh they was it knew that style and those were just people calling up that style exactly it was they weren't giving it an image of like from spirited away and then giving it tp already did that for you knew what that style was but let's say you're trying to create a unique original style yeah you can give it an image yeah and your image of what you're trying to generate and and try to have it uh match that yeah like a like a wes anderson film has a very specific look original style right we want you to make original stuff yes i was just pulling an example but yes that is a very distinctive style okay so you want to kind of so this helps because shot to shot you want to make sure each new first frame feels consistent like it's in a consistent world with a consistent cinema look yeah and this is where you as a creator really matter right yeah Like storyboards is what's going to drive everything down the pipe.

7:24And so if you have an eye for blocking, eye for frame composition, for lighting, for cinematography, this is when all of that's going to come into play. And then the other thing that's going to be an issue with consistency is characters and making sure that your characters look the same shot to shot. And so ways to do that, similar to the style, you could either, if it's sort of a generic character, you can just have a very descriptive text prompt. this is sort of what a lot of people do are currently doing with vo3 because vo3 is currently just text you video there is no you can't give it a frame so in some of the launch videos it was like the swat the police action video that dave clark did but that's kind of an easy one to do because yeah the faces are kind of covered we're not really seeing the details of their faces and and it's something that's sort of easy to prompt and get a very consistent looking physical character right but if you're you know actually building characters where we see their face and they're talking and performing.

8:16Laura's is another option as well. You can turn it on a character. Exactly. And, uh, you know, as human beings, I, I bash this with a hammer. Every episode is like, we're so susceptible to, uh, looking at people's faces. So the minute that you have a slightly uncanny Valley type face, a face that doesn't look quite natural, like instantly your project breaks. So you want to spend a lot of time and effort here. Yeah. And then we'll talk about actually making them perform and talk later, but that's the other issue. And then also, you know, going back to the tools that we've talked about where you can just sort of give it a single shot.

8:47Yeah, you go a lot of other tools. If you just have a photo of a person that can sometimes work and you like runway, you just give it a photo of the character and you know, say I want them in the scene. Hopefully, sometimes it'll like, recreate the person pretty accurately. Another trick that you could try to do is to create a character sheet. So a single image, that would be much more of like your front face, side profiles, full body. So like a character reference sheet. Yeah. That's a single image that you can give to these models. And that is done in animation all day long, right? When they're in character development, they call them post sheets where all that is created in CG traditionally, but now you're doing it in Photoshop and then putting that up onto your Control Net model.

9:33Mick Muppets does a good job. He does, and he has a whole comfy workflow where you can give it a single image of character and then it'll create an entire post sheet. Yeah, ideally, if you have 10 characters in your film, every of those 10 characters needs their own Laura. And then they'll need their own post sheet. Even with a Laura, you'd want a post sheet? A post sheet would be like, let's say you have a hero male character. So the post sheet would be male standing like this, standing in the back side, three-quarter view, face close-up, face smiling. but i guess i'm saying would you train the laura on that post sheet or where does the two you have the character laura and you have the post sheet so how do the two come into play if you need both the laura build happens before the post sheet okay right you use the laura to create the post sheet and then the post sheet influences the generation because you're the character is moving in space okay so it'll use both the laura and the post sheet together to figure out what that human should look like at whatever angle or whatever framing.

10:36Okay, got what you're saying. Okay, so now we've got, so these are sort of the foundational things you kind of want to figure out before going and to start making your first frames of what is your character, all your character is going to look like, what's your style going to look like. And also, I didn't get it, but like if you're having a location. That's a big deal. Yeah, the scene, the location. One trick that I did with the AWS, the Amazon thing. Oh, no, no, I'm talking about that Culver Cup filmmaking comp thing. that I competed in. I had a 3D room in Unreal, but then you could export a 360 panorama.

11:11Oh, interesting. And then I style transfer the entire panorama, but then that way I could just use portions of the background. Oh, I love that. So that if it was like a conversation like this, it was like, oh, if I had the reverse shot, I would at least have a consistent background scene. Yeah, and spatial consistency in image generation is still a fourth frontier thing. Yeah. I know Flux Context has a decent way of approaching that, and so does ChatGPT. But again, you're essentially shortcutting that process by using a 3D model of a world. Yeah, because they have a world understanding. Yeah, exactly.

11:44Yeah, I mean, we're going to keep going back to it. But I think the top three tools that are just if you're kind of just trying to like in a chat mode, easy to use, that kind of hits the bases is going to be Runway References, ChatGPT, and Flux Context. Those are sort of the top three of just being like, here's an image of a thing in a world. And that's only sort of state of the art today. Yeah, as we're recording this. Yeah. And that is only for this pre-production portion of the production that we're in. And now we take that and everything we're doing. And now we start generating the first frames for each of the shots that we want.

12:20Yep. And so this is either going to be text to image, merging it with if we have Laura's or if we have reference images in our chat interface with some of these tools. And then also this is we're also using any of the LLMs comes helps come into play to help you refine and structure prompting, structure the prompt, help you write out the prompt, at least give you. It's also good to just have a write the prompt as a starting point and you can kind of come in and tweak it. But they'll write much more detailed prompts than I'll think of off the top of my head. There are some VLMs, visual language models out there.

12:54I haven't done too much research into this, but you can give your image to a VLM and then it'll caption it for you correctly and give you a suggested prompt. So if you have a reference frame, you're like, I want this kind of shot, give it that frame, turn it into a text prompt and then use that. Use that as an input. I think Gemini is pretty good at that because Gemini is multimodal and you can give it input images and videos and it'll describe or tell you what's happening. Yeah, and remember what we covered in the last video, the clip encoding? Yeah. It's bidirectional. So you can go from text to image, image to text, back and forth pretty easily.

13:29And that's what VLMs do. Okay. Okay. So yeah, that's another good trick, too, if you have a reference shot where you're like, I want to replicate this type of angle, feed it, turn it into a text prompt, and then generate your first frame. Okay, so also just to round up with sort of the common tools for generating the first frame this is going to be all of your text and image uh generators so mid-journey runway ideogram flux chat gpt image and reference you could use it with the references feature which we talked about also the fourth one is image in as well google's image in which does have pretty good you can give a reference you can give a character reference a style reference location and it does a pretty good job of having that chat conversational workflow to help modify as an ai creator that that That is the key, right?

14:13Otherwise, it's just a random thing. Yeah, the more control you can have, the better that's the thing you're going to be fighting all the time. And then we've talked about this before too, but you could also kind of do a hybrid approach with Blender or Unreal and very sort of kitbash a scene together, but at least roughly frame up a shot and use that as a base image to style transfer. Yeah, if you're pretty handy with Unreal Engine, I mean, use metahumans, pose them, right? And then use a camera to kind of block the characters and then bring that still image into one of these things as a reference.

14:44And then there might be a final stage too. Like sometimes you might get an image, but maybe it's like 90 % of the way there of what you want, but it still has some issues, but it's like not worth it to keep regenerating it. So then you might take the image and you might have to modify it either with in-painting to change specific aspects of it. And most of the image tools have some form of in-painting. So like majority. Yeah, explain to the viewers. Yeah, so in-painting is like, okay, we wanted to keep this image as is but like i want to change just this computer regional correction yeah i just want to keep everything the same but i want to change my coffee cup to like a big chalice or something and so then you would take the image and you would like paint over the coffee cup and then you would text prompt to make it a chalice yeah and it would generate the chalice but blend it theoretically hopefully blend it in with the image oh yeah it doesn't really i mean most painting models are really good photoshop has firefly built in and right that is one thing that's just pretty good at of taking, you know, and also, I mean, sometimes in some cases you might just straight up need traditional Photoshop.

15:41Yeah. Just modify exactly. Yeah. And don't, don't be shy to go back and forward between traditional tools and AI. Yeah. Yeah. It doesn't have to be AI all the way. So like a lot of times, uh, this stuff still needs a lot of massaging. Yeah. And a lot of times even just traditional compositing or trying to think of building out your element separately might be the better option. We won't get into that here because that gets a little bit more complicated. and this is kind of more of a vibe filmmaking flow we're talking about. I find that with multiple characters, it's probably easier for you to generate a consistent character and then composite them in Photoshop and then take that reference frame as your image generation, video generation frame.

16:16So generate the characters as separate elements and build it all as one frame inside Photoshop. Yeah, yeah. Like it doesn't have to be exactly, you know, geometrically correct and stuff like that. The generation will take care of it, but at least you'll get, you know, your three hero characters in the same shot on that reference frame if you try to just generate that in comfy ui you're looking at a pretty crazy workflow like three different lauras three different character sheets yeah i mean i have seen some hacks now with flux context out of like combining images into a single image and giving that to flux context and that working pretty well and comfy you've you've showed that in one of the episodes i think yeah yeah uh right it was like a yeah, it was a hack or something.

16:57Yeah, you get five people in there. Yeah. Yeah. And then also, I mean, I've seen runway demos where you could just kind of, I think they built in a sketch tool into runway because people are doing this so much, but you could just kind of roughly sketch out how you want the scene blocked. And it sometimes does a pretty good understanding of replicating that in the image. Yeah. Okay. And so then you've got your first frame and then you want to turn that into a video. And now there are some, most of the tools will support image to video, first frame image. And some tools have more control where you could give it both first frame and last frame or first middle last frame but the the general broad overview of like all the tools that will support image to video got your runway vo3 uh mid journey just launched video and that's been pretty impressive so people are impressed by it playing uh high luo the chinese models that we talked about that also that uh get really good reviews uh luma ai yeah you can also run this on your own so one two point one which is by alibaba is it's a very popular comfy ui video generator model oh seed dance seed dance new one too yes yeah from uh by dance yeah if you have like a rtx 3090 4090 like last year's gpus i would say like a 720p video for 30 seconds you're looking at like an hour or so like it's not that terrible it's like a old school cg render timelines and then i mean this part is always still feels a bit weird to me because it's like you do all this work to get the image the first frame but then it's just like well i'll just give the first frame to the model and like go for it to its thing so some of these you some of the models you'll be able to give it a text prompt to try to describe what you're looking for yes and some of them are a lot of them now have sort of pre-built camera moves to which can also help get what you want yeah you want to resolve ambiguity as much as yeah runway and cling and their older models they had a kind of a camera control movement where you could sort of map out how you want the camera to move and runway that was only in gen 3 it's not in gen 4 yet okay so gen force doesn't have have as many features as gen 3 does but i'm sure yeah eventually that will get added uh we didn't mention um the one you mentioned higgs field yeah higgs field is specifically built for cinematic shots yeah i would say it's a bit more for like advertising or like very dynamic high motion shots for like an advertising spot sure but try them all yeah i mean a lot of these is just like yeah whatever tool gets you the shot the pro here is all these tools are really easy to use and they're relatively cheap right if you're looking at like how many cents how many dollars per video it's not that much it's not like gonna break your bank the con here is the control is still something that we're all trying to figure out yeah yeah getting the control of what you want which there are some good work uh examples a little bit later but other thing that might come into play too if you have characters and you need them to talk then now you gotta introduce another tool yes uh so that could be something like uh hedra uh which has really good character performance i mean it can give it an image of the first frame and give it direction and the audio So maybe Hey Gen.

19:49Hey Gen's another one. Yeah. Runway Act 1. Yeah. Which also has not been updated for Gen 4. So it's still Gen 3. Okay. And then there might be elements of compositing these things. Because it's like you might have elements of like where the character talks and you need the lips to move. But then also you need camera movement. So this might come into play where you have to start compositing. Yeah. Or treating like the performance separate from the camera move. I would say VO3 probably does the best job of animating faces for speech and matching speech. but VO3 is only text to video. So you're going to have to have a very descriptive prompt.

20:24That's why it worked really well in that commercial. The PGA did. The Liquid Death one. Not the Liquid Death. It was a gambling. Cal Chic. Clashy. Clashy. Clashy. Something like that. Yeah. That was a good one. It was good, but it was like every shot was a single scene shot with their own characters in that shot. So you didn't have to worry about a character looking the same shot to shot. And there wasn't dialogue, right? There was no dialogue. No, there was dialogue. It was the news interviewer. like kind of interviewing people but every shot was like a different person different interviewer so uh it didn't yeah it didn't matter that's why you see a lot of sci-fi stuff with like masks on no faces yeah it's still hard yeah easier to to deal with the performance but stuff's getting better i mean hedra hedra's new version is really good we're testing that with a project so we'll yeah report back yeah so just another thing to think about you know i mean joey what about this Like, can the human performances be actual humans with a real camera?

21:17And then you cut to your AI generated shot. Right. And that's the thing too. You'd have a hybrid workflow of real people in front of a green screen or totally using another workflow like Lightcraft Jet Set. And then you nuke composite that human's face onto your generation. Oh, I see. I actually just do the face onto the generation. Yeah. That's interesting. If you're good enough at nuke. I was going to say, how complicated is that in nuke? Well, you got to match move the camera. and then take that match move camera into your generation. So the generation is from the same angle. And if it's a static shot, it's fairly easy.

21:49But if it's moving, then you're warping that face. Yeah. I was thinking of more just like if you shoot a person in front of a green screen with something like Lightcraft with a lo-fi 3D background. Right. And then you take the background separate and process the background. Exactly. Separately. That totally works too. And then run it through Bebel for relighting. So the whole scene matches. is that's another workflow that a lot of people are exploring. Yeah, I think AI faces and AI speech is still not there yet. Yeah, I mean, we've seen AI is very good at these like one shot action shot generations, but having that consistency shot to shot.

22:24Imagine a 90 minute film where there's like hundreds of lines of dialogue, right? You want a real human. Yeah, it gets tricky there. Yep. Yeah, so that's high level overview gist. That's why there are a lot of steps and tools. There are tools that are sort of combining this all so you can kind of do it in one stop. So like FreePick has a lot, like kind of one-stop shop, even Hedra. A lot of them are building, connecting to the APIs with other tools. So you don't have to bounce around a lot. But another thing to think about that's sort of changing a lot is since Luma AI released the restyle video feature.

22:55Yeah, that's great. That's been blowing up. And that's been kind of shift, like thinking about this completely, where you can just shoot a very lo-fi object blocking video. I mean, with John Fingers, his demos are crazy. The one where he's just like swimming on a carpet, but it's an astronaut in space. Yeah. Or it's like someone with the sheet moving and then it's like a stormy ship. He's really imaginative. Yeah. Or he had like two shoes, like pulling the strings of the shoes and then turned into like a race car scene. It's like a race car. Yeah. Yeah. So this, and I mean, you know, we are quote old.

23:28Like I just can't imagine what, you know, some like kids who take this and run away with it. Cause it's like, it could just kind of re, you can just re-imagine. you don't even have to have the same rules of like matching a shot. Like if I think of a race car shot, I would still probably be like, oh, I need some toy cars and I'll turn the toy cars into race cars. Yeah. And it doesn't have to be video to video. Think about this way. This will blow your mind. You can go from video to video to video to video. Like you can keep iterating on it in Luma to get the quality and the stuff that you need.

23:58Yeah. And I'm sure Luma would like that because that's like generation. So, yeah. So this type of workflow, I think, could just totally throw a lot of the stuff we just talked about out the window. Well, not necessarily because you still need the solid image references. But now you're shifting to a more of a video to video workflow. Yeah. It's like, oh, what if I just take my phone and I could just shoot with a few people or I could just shoot with some toys on a table. And now we've got like an epic action film. But also, I like, you know, when I'm like, oh, it's crazy that you just give the first frame to the image and then you like let it do its thing.

24:29Yeah. But now, you know, Luma is mimicking the actual movement that you do. Photography, everything. Luma does take a first frame. You can either not give it a first frame or you can give it a first frame. So you could combine the elements up to making that first frame. And then that camera control could be, you know, Luma, or it could also be a Blender or an Unreal Engine recorded camera movement. Exactly. You restyle the first frame, and then now you're really driving what the image should look like and how the camera should move in that space. Yeah, the biggest benefits to physical production and principal photography is character performance, direction from the director on the performance, and then cinematography.

25:09Like, you want to preserve those aspects of filmmaking as much as possible. And I think Luma allows you to do that. And yeah, I mean, video to video has been pretty slim on who's offered it. I mean, Runway was the only one that did it for a while. And it's still, they haven't updated it. It's still only for Gen 3. I mean, I'm sure it's only a matter of time before Gen 4 has video to video. crystal ball let's go announce it on the podcast crystal ball come on on yeah so also once that comes out too then it's gonna really be because gen 4 itself right now is pretty crazy yeah it is so once once the video to video comes out yeah it's gonna be another and it's such an exciting time we live in like all of these tools we're talking about they're all great yeah right not one of them is especially in how sure like if you look at the stuff like people posting things like from two years ago yeah and it was like you know just abstract bizarro yeah like vincent van go style transfer that's all we got yeah and so like how fast this has come in just two years i know it's crazy and it's already pretty usable but we're gonna get to a point where all this stuff is effortlessly usable yeah yeah and there won't be a massive um flow chart workflow yeah you won't need joey and addy anymore but hey man we'll move you'll have ai generated joey and addy yeah but i think i'm a good place to wrap it up yeah so uh yeah i mean this was this is a lot of of the tools and workflow that I've been seeing, but if you've got your own workflow or other kind of things that you've been using that we didn't cover, please let us know in the comments.

26:28I would love to hear it. Let us know if this is the type of content you want to see. We'd love to make more of this stuff, but it's up to you, the viewer. So give us a like and give us a comment. The power is yours. All right. See you next time. Bye.