How To: Hybrid AI & Live Action Filmmaking Tutorial (Nano Banana Pro, Kling, Wan, Resolve)

0:00Hey everyone, welcome back to the Denoise podcast. Joey is out. So today we'll be sharing another fun short film that I made. and here it is what do you bring to the table that we already don't have here

0:38Awesome. So let me show you the source behind it and how I set up the multi-camera. So this is a three-quarter view of me in this very table that I'm shooting this video from. A reverse into the creature, back to me for some paper shuffling. Notice the handoff of the paper there, asking a question. And then a reverse over the shoulder shot with both of us on the same frame. side by side, a seamless transition, and then the final shot. But now that you've seen where the source material originated, a couple of tricks and tips for those of you who are trying to make a hybrid live action, including an animated creature into a live action short.

1:25The first tip is ground yourself in reality. And what I mean by that is use a real set, use something tangible and tactile that then you can ground the cameras to. In this case, this round table was my device for that. This image here where we have both of us on the same shot, I actually shot it with myself and without myself. And this table is going to ensure that the cameras have the same scale as it goes through generation. And there's something physical and tactile and the reflections from the table itself is going to help dictate the lighting in those generations, which we're going to get into.

2:02So I'm going to walk you through the multi-camera consistency setup that I used and then the acid generation part specifically generating the background environment as well as the different camera angles with lizard in them. And then finally we're going to go through the generations themselves. I'll show you a couple of failed generations just for fun. And we're going to correct all of this in DaVinci Resolve. Now I'm doing a lot more than color correction. I'm actually correcting for lighting mistakes and trying to have seamless, consistent look and feel throughout the whole sequence. So we're going to get into that as well.

2:36Before I ended up on the lizard creature, I actually tested a few other creatures. You can see an alien here. It just wasn't the right vibe. And I think it would have been a little too distracting. So I went with something more like this. and so I started out with an empty chair as you can see here and then after a few generations I was able to get the lizard in there it's a nana banana generation and once I was happy with the size and placement the arm registration on the table I then started to work on the environment now for the environment I went through a few different iterations I first tried a futuristic boardroom here as you could see it was just a little too much i wanted to emulate a sitcom setup with multiple cameras and not really go for a cinematic film look so i didn't intentionally add any grain or any vignetting or any halation and all of those things i wanted to have a very standard sort of broadcast look and feel i tried a more corporate boardroom here more traditional one as you would say and it and this is an example of a problem that i find with a lot of image generation editing even with nano banana pro it tends to not recognize spatially where you should be and it just arbitrarily places you into different environments after a lot of tries i finally ended up with this really moody boardroom type thing a lot of wood paneling which i love and a very cozy atmosphere there was just a couple of mistakes around the light so i corrected some of that and then this was the final one that I ended up with yeah I was happy with this now the challenge was to take this very environment build the rest of it out just from this one shot I then had to recreate the opposite view so after a few shots now I had the opposite view where the monster would be sitting for that three-quarter shot once I had that then I started to place the table in there and the table just sort of landed arbitrarily quite a few times after I got this scale wise this This felt correct.

4:40So this is a four foot table that feels like a 15 or so feet wide room. So once I was happy with that, I then took this into Photoshop to do the cropping. So let me show you the trick to that. So in Photoshop, you could see that I overlaid the actual table from the actual camera into the generation once I placed the monster. And the reason I'm doing this is just to get the scale and the camera perspective correct. So how high the camera should be and how much it should be pointed down. These things are not guaranteed in image generation. So I had to do this manual step. So then I tried a couple of generations from that side.

5:21Here's one here. This is from VO. Just him breathing heavily just to test what it would feel like and if I could actually cut back and forth. So the next challenge was to actually do an actual piece of paper handoff. So I would pick up the paper that the lizard then hands me. And this will sort of allude to us being in the same space and kind of, again, ground ourself in reality. And so for that, I actually did a first frame generation from Kling 01. I tried Kling. I tried Kling 2.6, VO 3.1, just to kind of get it correct. What ended up working for me actually was one 2.2 animate. It just happened to give me the most photoreal look without completely distorting my face and body.

6:08So again, my camera, I'm not too worried about. I know I could nail it, but can I nail the creature side from a fully synthetic generation? So let's take a look at that. So the first thing I had to do in order for me to generate that piece of paper handoff was actually place a synthetic piece of white paper, a resume of the lizard onto the table next to him. so after I placed that in there the next thing was to then generate him handing off a piece of paper you could see there just the random hand appears after many generations I just still couldn't get rid of it and also the motion is just really slow so I sped it up in post and I cropped it to get rid of that arbitrary random hand okay so once I had those back and forth shots all figured out the next one was over the shoulder shot.

7:00Now this was a little bit challenging because it's both of us in the same frame and so the expectation for photorealism is high. The creature can't look too synthetic while I'm in the frame and I can't look too real if the creature is synthetic. So it's a play between the two styles, AI generation and the real plate. So let's take a look at how I achieved that. I had to have the right perspective. I actually had to match the exact camera perspective as the one from the real world and so i just took a screenshot from here that piece of paper is real and i'm going to be using it as a prop as you saw in the short so once i had that it was then uh quite a few iterations until i got to this where the creature was in the frame and you could see here the creature is just a little too big so i had to play with scale but he's seated correctly he's in that same moody room that i am in the table looks quite real again grounding in reality using the table for all of the shots to kind of come back to the origin point one quick thing to note is before all of the video generations once i was done with the image itself i pushed it through a skin enhancer so this only not only helped my skin tone just have more detail it actually helped the lizards scales quite a bit as you can see here we go from sort of a blocky scaly thing to now the scales themselves just having a little bit more texture now am i happy with this not really i wish there was a little bit more detail here but i'll take it i think this is a great starting point and a good test so this goes right into the different video models that I talked about.

8:44I tested with Kling 01, Kling 2.6, VO 3.1, as well as WAN 2.5. So now that you saw the shot of the creature just sort of sitting on the table opposite to me, it's time to now place my shoulder and the left side of my body into the frame. So let's take a look at Resolve. And this was done using the AI Magic Mask and Resolve, which as you can see is literally one click and once i had myself masked out correctly i then placed myself into the frame as a layer in front of this creature layer and then with some color grading to match the two plates a little bit of blurring on me to defocus myself i was able to achieve this shot awesome and then finally we got into the most challenging shot for me which was the side by side shot so again we start with an image from the real world and then once I removed the background from there this is just me talking to an empty chair I then had to place this creature or this lizard in there so after a few tries I got the lizard in there as well as the environment so this is what I ended up with and then I pushed this also through the skin enhancer to get more lizard detail on that scale again I'm not 100 % happy with this the room feels a little bit smaller a little bit more claustrophobic however after having done about 20 or so generations i was happy with this enough to then proceed to the next step so i was ready to generate the video lighting wise there's quite a few mistakes here this region is just too bright here again this wall is just too close there's too much light on the legs here as well and then behind me is also too bright so the lighting is just not working in general and there is no consistency across the board and this tends to be a general problem with image and video generation today is there's very little control over spatial perspective so moving cameras around as well as lighting as your camera changes so if the lighting remains static as you change your camera it actually doesn't really respect where the light previously was.

11:01So once I was happy with that, then it was pushed through some generations. I tried a few different models and the one that I was super happy with was actually Kling 2.6. It's really hard to control performance through prompting. I wish I had done another pass and just sort of mocap myself on the other side as a lizard. I just didn't have time for that. So I went with a full synthetic approach. I tried, you know, the lizard tapping on the table and just wanted to try a few things. Basically, I was trying to allude to the final moment of him eating the resume. And I just wanted I just needed a transition shot, really.

11:40So this was the one I was happy with where he leans in and I lean in with him. Again, the paper is not exactly in the right place, but it's good enough. If you look at my face and the background, everything is just really soft, but it's going to go by quick. And I actually reduced it to just one second because I was just not super proud of it. Okay, so now that I did that, the next step was to then incorporate that final shot of him eating the piece of paper. Again, this is the same formula as the last shot of over the shoulder. So this is the plate that I generated. great ending to a short i think so now i just have to add my shoulder to it and transition to this shot i want to try something just a little bit different just to spice things up i wanted to do like a bullet time transition from the side by side to the over the shoulder so the camera was just moving at a 45 degree pace and in order to do that all you need is a star frame and an end frame give it to cling 01 and it'll figure out a smooth transition i've seen some examples of it on youtube so i just wanted to try it and throw it in there so for the first frame it was obviously the last frame of this shot was the beginning of the transition and then the end was the first frame of the next shot right so you're just basically putting this thing in the middle of those two shots and this is what we ended up with here and this is a five second clip it's it's really slow it has a weird ramp up and ramp down the motion is not smooth so i ended up putting this into resolve as you can see here as speeding it up adding a lot of motion blur to it and then just making sure it just is as seamless as possible from the last shot to the next shot okay so that is all of the generations now it's all put together in resolve and edited together and after that now it's not just the color grading but what i call lighting fixes and the way i solve for the lighting mistakes is using magic windows i'm not a hundred percent expert on this stuff so i just use the really basic shapes like rectangles and gradients and I'm just turning lights down on regions of the frame and then I'm isolating the lizard when necessary and isolating myself when necessary to just get the right contrast and the right color.

14:08The yellow on the lizard's sort of torso on the front of his chest, it was just a little bit too yellow for me. It just had that AI tinge that we're all just too familiar with. I wanted to move away from the AI look and feel as much as possible. so I shifted to a cooler temperature I brought the yellow down using a color selector here and now you could see that this is much closer so here is the original frame without correction and then here is the frame with correction I did this across the board for all of the shots even on the over the shoulder shot here quite a bit of adjustment when it's necessary the one that needed the most adjustment was this side-by-side shot here you could see here under of the table where our legs are that entire region needs to be dark because no light is really hitting there behind both of us it needs to be dark so i darkened all of those areas with magic windows finally i used the depth map feature in resolve to just kind of knock the background out of focus a little bit again there's only so much you can do if you don't have good quality imagery to begin with so you can only push it so much and unfortunately with video generations the resolution is just so limited.

15:23Some of my generations were 1080p, some of them are 720p, and overall there was just a level of softness that I just could not get rid of. Having said that, this is still a really incredible glimpse at what the future of visual effects could be with AI Power Tools. I hope you enjoyed it. Let's take a look at the clip one last time.

15:59What do you bring to the table that we already don't have here?

16:15Awesome. If you have any comments or questions about this workflow, or if you want to see more, just comment below and we'll see you on the next one.