Animating Characters with Wan 2.2 AND Wan 2.5

0:00Let me take you through my local desktop version of Comfy. I took a screen grab of me, and I wanted to do two things. I wanted to do a background replacement and a costume replacement, and then relight myself.

0:15Welcome back to Denoised. Addy, good to see you. Good to see you, Joey. All right, so you've got Comfy, a big spaghetti, Comfy UI workflow shared. What have you been working on? So we did a Comfy Basics episode a long time ago, a few months ago. and a lot of our viewers asked for more and we are now going to deliver so i have a few comfy workflows that i want to show you i'll start with the videos first the first thing i wanted to test was one 2.5 which as we talked about is a multi-modal model so you can not only input a frame reference of the video you want but if you give it an audio performance then it'll animate to that audio performance.

0:59So a long time ago I worked on a character called Space Cat, which was all fully built in Unreal Engine. So I just took a screenshot of that show and then put it through WAN 2.5. And that's what you're seeing here. This is fully animated with AI. There is no Unreal Engine whatsoever. I want to show you that WAN 2.5 output to the show that we actually made about four years ago fully in Unreal Engine using motion capture, using audio, OBS, you name it. It was quite the extravagant tech setup, and this is what it looked like back then. Shout out to Jeff Sloniker, who was the character and was the chief behind this.

1:42So I'm going to mute it and just talk through the tech setup here. This is puppeteered in Unreal Engine using a Rococo suit, a motion capture suit and it's it's being rendered in real time the fur and everything all of the reflections right versus today using just a screenshot from that show we get this from one 2.5 what's up it's space cataruru and i got a massive load i need to deliver from this planet what are your thoughts i mean just look at the screenshots yeah they look similar my question is because like yes i know we could get out of the box like something that kind of looks similar but how much control are you getting over one 2.5 with its performance yes fantastic segue into my next show yeah that's exactly it you none to answer your question i mean one two five it's like yes you have the mocap suit you can move right it can follow exactly what you want your performer to do what can we do in 2.5 absolutely or any or any of the one models yeah okay so the question you asked how much performance control do you have over the hand gesture the head gesture and stuff i would i would imagine a little bit maybe through prompting you can perhaps prompt for exaggerated or subdued but not one-to-one not what you would get with motion capture and a human performer with that intent i'm going to show you what I did with one 2.2 animate to answer your questions about matching performance human performance so I'm going to play back this video here hey Jeff this is space cat man this time I'm in AI no more unreal and you could see here that in the video my performance is pretty spot-on right I mean it retains it one-to-one yeah the hands the hands look good yeah your face looks good exactly and you can even see that when my hand touches my face like that hand registration part which is generally difficult to do in computer graphics is now taken care of by ai so the only thing and this is a big thing the thing that it's messing up on is the proportion of the character uh to that of the original character the big note with one 2.2 animate is that because it's trying to match the character's performance to my performance and using me as the reference and i have human proportions right like long arms and long fingers and you know relatively small uh torso compared to space cat who is a short stubby sort of cartoon character so it's stretching that model out to fit my proportions which is um which is wrong on the on animation level and that would not be considered ready.

4:33Right. Okay. Let's go to the workflows. The thing that I'm trying to understand, you used 1.2.5, which you can't give it a motion performance, correct? You can only give it an image and you can give it an audio file to drive the performance. Also, it does not run locally. It only works in the cloud, correct? For me, 1.2.5 was all done with Comfy Cloud, which I'll show you here. Okay, so you're using the cloud, their new beta platform, so you can run Comfy in the cloud. Yeah, so this is Comfy Cloud, and if you go through all of their templates, and there is a 1.2.5 template, which I'm going to open up.

5:12Let me just search for it. 2.5. There it is. So you double-click that, and now you get the full node tree, and it's a really small node tree. And yeah, you can see in the top right, it's got the 50 cent sign. So it's a API node. So it's not running locally, even though this doesn't matter because you're in Comfy Cloud anyways. But if you're running Comfy on your system, this is a API node that you pay for. You can't 2.5. You can't run locally. That's right. Yeah, I think it's just too big. So here, the audio input node is optional. So what I'm going to do is tie the audio input node to the audio input on the one image to video node here.

5:5850 cents a run. Very clear here. Does it give you any other setting options like duration or output resolution? Yes, it does. It's all down here. Yeah, so I have the option of 48720 or 1080p. Duration, I think it could go up to 10. I just did 5 just for testing. And of course, you have C control as well. And then typically on the saving side of things, I hard set it to MP4 and H.264 as the codec. Otherwise, I see it having some issues with playback. So that's it. It's a very simple node tree for 1.2.5 on ComfyCloud. None of this is running locally. What direction did you give it in the prompt?

6:50And did you experiment with the prompt? Did you kind of find the different prompts could give you some more of the hand gesture performance that you got with the other tests? I actually didn't give it any prompting because I wanted to just have the audio drive the whole thing. Yeah. Okay. you want to run it out? I'm curious give it a prompt like the cat speaks and gestures with its hands let's see if we can get it to move its hands more character has exaggerated gestures as he talks points at the camera a lot maybe change it back to 5 seconds so this runs a little faster 5 seconds let's go down to 480 here we go quarter there's all right 25 cents so i'm gonna hit run here and right away it's gonna go out and grab a free gpu it's already executing so if you look at the queue here it's running and even has history of my past renders the last one took 155 seconds was that a 2.2 10 seconds this was 2.5 that was 2.5 okay yeah so i think within i don't know half that time 75 seconds or so a couple of minutes we should see a result all right joey so that came back super fast here we go we're gonna play this back for you well it's gesturing i can't i can't hear it but uh it has more of the i mean it's kind of doing finger guns shoot him a cabin so yeah i mean trying to control trying to control physical performance through prompting is obviously not the right way to do it yeah that is a very tough task yes but it's kind of there i guess if i were to get really granular i would go prompt structuring you know one second mark make this hand two second mark make that yeah you would go the json the json route yeah yeah yeah see how that works out so i just wanted to quickly show you what one 2.5 is capable of on comfy cloud this is something anybody can access today you don't need a nvidia gpu you can do this on your Mac.

9:02Yeah. Yeah. One, 2.5. I did find it was the best cause I was working on a project and we had audio and I wanted to drive the performance of an image and I tested everything that could, you could give it an audio input and an image. And, um, what were the, what's the other main one? Kling? Does Kling have that? Kling? I think you could do. Well, you could put audio in into Kling. I think you could do audio to drive like a character performance. Um, I want to say Kling was the other one that was like a close competitor, but 1.2.5 was really good at something that felt natural and that also as natural as it could be given all of the limitations and restrictions of being just an audio driven model.

9:44And did not because a lot of these other models are feel very like designed for UGC videos. So they have like very weird over the top performances. so 1.2.5 felt quote the most realistic of the available options right now so it was really good for that yeah this is the Kling this is Kling's node tree and I don't see an audio input model it's not a comfy thing it would be like a Kling I don't even know it's an API I think it would be like Kling on their website because they have more options on their website so now I want to get into 1.2.2 animate this is a much more complex node structure here and i'm going to walk you through what each of these are doing just just know that you don't have to build any of this yourself in this case what you're looking at here is on comfy cloud it's fully built up for you and then i'll show you the one that i actually ran which i ran locally on my desktop with comfy the application and we'll get into each one of those blocks so this is on comfy cloud unfortunately i think because this is in beta there's some issues with it, I couldn't get it to actually output anything.

10:56It was a good learning moment for me to just learn the node tree. Okay. And so also to clarify, so this workflow they're looking at right now is from their 1.2.2 animate template that's built into comfy. Correct. Okay. Can you show where that is? Yes. So you can go to browse the templates. And what I do is just typically just type in animate and it will have one 2.2 animate character animation template double click on that it takes you to a whole thing which actually this has a a sidecar youtube video um with this gentleman driving it he doesn't tell me his name but yeah he's he's the guy that basically drives it yeah he's also the face of like the comfy youtube channel Yeah, he's the get comfy with comfy guy.

11:45Shout out. Shout out to, we can't remember your name right now. Sorry. Yeah, please introduce yourself on the next video. So this is the node structure. I know they have it sort of built up into sections. And also they do have really good documentation as well. So they have nodes everywhere and things identified for what you would need to change for whatever you want to do. because there's really only three or four nodes in here that you really need to mess with to change, to do what you want. Everything else is already set up and it's just like, don't mess with it. Exactly, yeah. So if you don't need to mess with anything, just don't disconnect a node tree or anything like that.

12:24I think the most cumbersome part to any workflow is finding the models, loading the models, and being ComfyCloud, this kind of just takes care of it for you. So this is saving you a ton of time here. So these are all the models that it's looking at. For the actual model itself, it's 1.2.2 Animate, 14 billion parameters, floating 0.8 with E4M3FN. And I think this is very important because this is how the GPU actually computes precision. So it could be an E5 model or an E4 model, depending on your GPU. I just ran with what they have. actually okay because it's on cloud i don't even care what gpu it is yeah so when you load this up on cloud is everything already quote downloaded like and ready to go yeah for sure so i just opened the template here and you could already see that um well this is the two lauras that are needed are loaded this is the actual model itself this is the heavyweight big boy model it's already in there.

13:33If you click on it, it kind of takes you to load other models, but it's all in there. Oh, I haven't even seen that. I haven't seen that new UI yet. Oh, that's interesting. Because yeah, before it used to just be a little dropdown and you had to like decipher with the long text name what the model was that you're looking for. That's a nice new interface. Okay, cool. Yeah. And then the clip encoder that complements that one model is also here. And again, this is in that same enumeration, E4, M3, as well as floating point eight. These things have to match. The clip encoder definitely has to match the model.

14:07Okay. And, but by default, these things are already set up where it has the models that you would need to run. I would also say if you are running this locally, you know, if you have a machine that could handle it, a PC and stuff, when you load this workflow up, if you don't have the models, it'll pop up with a big window that says models missing. And then there's just like a little button you just say download. And And so it's very easy on a local machine to set up the models that they need. It'll just, they'll automatically download them, install them for you. So it's pretty straightforward.

14:34And I'll go through that when I show you my local instance of Comfy, where I actually did the manual work. Perfect. So once you have your models loaded, it says step two is your prompting. You know, very simple prompt here. And then there is some global parameters, the video size. here you can define if you want a wide aspect ratio you could put 1280 720 1920 1080 what have you and this will propagate into both the incoming image as well as the output video right did you try full hd like 1920 by 1080 or 720p for most of my generations i did 722 i haven't i don't know if it can do full hd i'm curious i should try it we should try it i mean i'm sure it could do 1080p I don't know if it was trained on it.

15:21I remember looking up the specs and I think 720 was the max. I'll try in the future. I'm curious. All right. So 1.2.2 Animate has two different types of solve. And it kind of has the note here with it. So if you read the note here, it has a mix mode and a move mode. so the mix is basically retaining the background image with the character being replaced so and i'll show you an example of that yeah and this is what we sort of talked about in the last episode when we were talking about this at a high level and i experimented and it turned me into an elf because i was trying to drive the performance of an elf yeah so this video that you're seeing of Goku being animated.

16:11That's also done with 1.2.2 animate, but it's using the mix mode where the background is retained and just the foreground, the character is kind of comped in. So we're not going to go with that. We're going to go with the move mode, which replaces the entire frame. In this example here, I'm going to just upload my image. So give it a second to upload. So this is your reference image. So this is what you want your starting frame to be, roughly. Yeah. And it's not going to animate the background, obviously. It's just going to animate the character. But it's replacing the background from my performance.

16:52And if you were doing a mix mode, you would take a screenshot of you or whatever you wanted. And then like Nano Banana modify your character or you. And then give that as your starting frame. Totally could do that. Yeah, exactly. So once you have the mode that you select, then the next thing, if you are doing mix mode, then you have to dial in which part of the character you want to mask and which part you want it to leave alone. All right. So I'm going to go ahead and give it the video performance. Takes a second to upload. So your video performance is a vertical video? Yeah. Yeah. Okay, you found that that still worked and stuff?

17:36Yeah. Okay, interesting. So this is my video performance, frame by frame. And then once this loads up, this will actually switch to an image of me where I can move these dots around, add more of these dots if needed. Yeah, and I think it says it in the notation. So if you hit run, it will run the cycle, grab the first frame of your performance video, and then load that frame here because what you're clicking on right now is not going to match up. You could either hit run, it'll grab the first frame from Addy's performance and then load that into that box that he was clicking the green checks in, or you could just upload your own reference image.

18:22Right. So let me take you through my local desktop version of Comfy. instead of that SpaceCat example, I ran another example of what I would call live action. So I took a screen grab of me on this very table, right? Just wearing a black t-shirt. And I wanted to do two things. I wanted to do a background replacement and a costume replacement and then relight myself. So I actually went into FreePick and I did that with Nana Banana. And here are just some of my notes. I had a poster, removed this. I had a bunch of stuff on my table that I wanted to remove, remove my watch and so on. Right. And then I ended up with something like this.

19:06And then Joey was like, Hey, you look like you're in breaking back. You're making meth. Yeah. Cooking meth. So I was like, okay, maybe a different look, but I love the, like the apocalyptic bunker and like the whole the whole vibe with the with the light on top above your eye i don't think i noticed that before yes so i ended up with this still image here so it slightly distorted me like it added i don't know a few pounds and also like you're eating well in your bunker it's not it's not my face 100 it's like 99 my face but the nice thing is added some dog tags which I wanted a distress sweater with holes in it.

19:48And then I wanted a gun rack in the back. The background to me feels very like Unreal Engine Marketplace. It feels synthetic, but it was a good enough test to try, right? And so here is the video of me. So you could see there that my dog tag is moving slightly. There's physics on my dog tag, which is so cool and so unintentional. And then my performance is there, right? My hand and my, you know, me touching my face and exaggerating. It did mess up my face slightly, which I think I can address going next time. And I'll get to that. But look, overall, I'm super happy with the results here that you're looking at.

20:29And I did this with one 2.2 anime. So this is exactly the workflow that I used, right? So you've got, so basically everything we covered before with this workflow on ComfyCloud, your reference image is the image that you modified with nano banana yeah so this is the reference image with nano banana and then you're driving video performance okay so with this one did you do the pose transfer or did you do the complete um complete just okay and so let's show everyone how you would modify the default workflow to switch the mode it's really easy i want to give a shout out to whose workflow this is real quick.

21:06This is a YouTuber named MDMZ whose workflow that I downloaded. Oh, so you're not using the default 1.2.2 workflow in Comfy? No. The default 1.2.2 workflow is just a little too excessive and too crazy. So both MDMZ and another YouTuber that I followed, they took a lot of the extraneous stuff out and just made it much simpler. Okay. So what What is different about this workflow? Okay, I thought you were on the default one. So what's different about this one? So this workflow is specifically dialed in for the entire frame, which is not the mask mode that I showed you earlier. So it removes a lot of that feature out.

21:45And the way it does it is actually really simple. Let me show you. So is this workflow just built for a full transfer? Like, is there an option on this to do the pose transfer? or no, this workflow is just you want your video to drive the performance of a completely new frame. That, exactly what you said. And the way to do that is really easy. You can take the default workflow or this workflow, and if you just remove the connected objects to the background video node and the character mask, then the model defaults to animating the frame. Yeah. It no longer does masking. Yeah, if you are in the default workflow, what Addy's showing, the background video, the character mask, it's a little not the most intuitive because you have to kind of trace the nodes.

22:32But you basically have to delete those connections to the node in order for it to bypass it. You can't just, and it says in the nodes, you can't just bypass the nodes. It doesn't work. You have to delete the actual connection. So it's a little, it's a little bit annoying to switch modes. It's a little hacky. Yeah, it's not the most. I wish there was like a nice switch for it, but there isn't. But this workflow that you downloaded by default, it's already set up to do the full character performance driving a new frame. Correct. Yeah. And actually, I can run this locally and just kind of show you.

23:08It does some really neat things. So one of the things that it does is it takes my performance as a video frame. it'll do some sizing upscaling downscaling stuff which is all built in here and then i have the pose estimator node here which is detecting my hand body and face you can of course turn off your hand if you want to right turn off your face like you pick the part of your body that you want tracked one 2.2 animates magic node is really this which is essentially running a mocap on you to determine what your body pose is what your face pose is and then put that into the diffusion model so it could generate the character in that pose so i'm going to quickly just run it here and now you could see it it's going through the nodes whichever node it has that green box around it that's the one it's running in real time yeah so you could see the pose estimator is running completely and then once it does that it it'll do something really cool which is that it'll create a frame-by-frame estimation of your facial performance that you see here as well as your body performance that you see here and it's using this to then animate the character in the diffusion model yeah that's cool yeah you can hear my GPU spinning up in this room and you could look at the the green taskbar on top and also if you look up here my GPU is at a hundred percent the temperature of the The GPU is going up.

24:41V-RAM. What resolution were you running this at? Great question. So this, just like the last model, this has a global resolution dial. So I set it to 128720. Okay. And this then propagates into the entire workflow from these parameters that I set. And how long has it been taking to make one, and what are you running on on your system? Yeah. Great question. So I am running an RTX 3090. Not the best GPU, but not too shabby either. And this is like a two to three second clip, very short clip. I think we're looking at maybe five to seven minutes or so. Okay. I'll ask you other questions. Did you keep the frame rate?

25:30I think the default it leaves at is 16 frames per second. Yes. The frame rate is... No, actually, I matched the frame rate to the incoming frame rate. Okay. So it's going to go track at 30. Did you change that? Yeah, so let me show you. On this particular workflow, where is it? It might be on the output where it's, like, before save video. Oh, right here. Okay. So there's a node here called get original frame rate. This will go into the input video that I gave it, And I shot that at 30 FPS. This show is shot at 30 FPS. That's the standard thing that I use. And that went into the create video node.

26:14So that is going to override the 16 frames per second that I had before. That's a nice improvement because the default WAN 2.2 workflow, it's just a text field and it defaults to 16 frames per second. And then you just have to manually change it. So that's a nice improvement that it grabs the frame rate of your source video because obviously you'd want it to match. Because if you run the default comfy, if you run the default 1.2.2 animate, and then you play the video and you're like, why does it look like it's plain choppy or slow? It's probably because the frame rate is 16 frames per second and it's going to look a little choppier.

26:50The main reason I wanted to do that is because this side-by-side video that we're going to show you, and I wanted that to be frame-by-frame accurate. Because I was just putting my VFX artist hat on and I was like, you know, I want to do background replacement, costume replacement, and then have it feed back into my VFX pipeline. So of course it has to be the same resolution and the same frame rate as the incoming video. With this workflow, because I'm curious what else this workflow improves offers. What's all the stuff on the bottom that's turned off? Yeah. So this is the video extend workflow.

27:24So in comfy purple means bypass. So none of this is actually in play right now. Actually, this should be more like this. And what this just means is 1.2.2 animate node that's built in. And shout out to someone named Kijai who built all of this. And Kijai's hugging face repository is what all of this is based on. So the way Kijai built the Comfy UI plumbing is it's at most 81 frames long. And if it's 30 frames a second, then you know however many seconds you have. In order for you to get past 81 seconds, you have to chain multiple inference together. And the way you do that is using these nodes, you can actually cut and paste and have more.

28:13So it'll generate the first 81 frames, take that last frame, generate the next 81 frames, take the last frame, and so on. Would this pretty much basically be determined if the source clip that you're trying to modify is longer than 81 frames, you would turn these on? Yes, exactly. Okay, got it. That makes sense. Yep. So that is the Comfi local workflow in a nutshell. Yeah, I'm really happy with the results. Look, I'll tell you, it's not usable for current day feature film TV production. Having said that, if I was to build a short film or a YouTube channel or any sort of short form content around putting myself in a post-apocalyptic world and then doing a podcast or a broadcast from this world, this is totally usable for that.

29:04Yeah, and I think just the performance recognition and pose transfer, it's been the best that I've seen out of all the options out there to really kind of capture the source fidelity. Yeah. And a quick note, I know some of our viewers are very advanced Comfy UI users. So maybe for them, this is perhaps not the most advanced video. But for most of you who are just coming into Comfy, you've done some basic image generation stuff. And now you want to get into video, you want to do animation. Well, I think this is a great resource for you. So again, watch this video, then go to the Comfy video that we'll link to, as well as MDMZ's video.

29:47And I think it's a great way for you to get acclimated with Comfy in general. Yeah, and like we said at the beginning, you might load the workflow up and see a bunch of nodes and lines, and it might look intimidating. But there's really, with a lot of these things, only three or four things you need to change, and everything else is pretty much set up for you. I'm curious if you experimented with different angles with WAN 2.2, because one issue I had with WAN 2.2 Animate was I had the reference image I was trying to give it was a character, but sort of like a character profile shot. And the driving performance was something like this where I'm looking straight to camera.

30:27And the outputs, it would always keep shifting the character reference to face forward to match what my driving video was. Did you mess at all with different angles at all or anything? I have not done it in 1.2.2. I've been trying it with Nano Banana now that they have the camera control thing. Is that a Nano Banana thing or is that a FreePick thing? Yes, it's in FreePick, but it is powered by Nano Banana. So let me quickly show you. And it's actually a picture of you, Joey. So this is what I generated the other day, shared with you. So right away, to me, that doesn't feel 100 % like Joey. like it automatically messes some things up like uh buffed up so i'm i prove it the reason i want to try it here is because like that that's not my face you know it's it's it's kind of like my face but not really so i wanted to give it a front performance with a 45 three-quarter view of me as a reference image and try to generate that so this would be next this is something i try next I took a screen grab of our last episode where I was wearing this t-shirt.

31:39Okay. And you rotated the camera or the character. So yeah, that's what I'm like, if you gave it that image as the source at the first frame reference image and your character performance video, what would happen? I'm wondering, like the thing I haven't experimented with is like, if you record your character performance from the same perspective of the image you're trying to give it if that solves the issue. So yeah, if anyone has done that or experimented with different angles where stuff isn't just straight onto camera, I would love to know. So let us know in the comments. Yeah, and in general, in the comments, if you have been playing around with one 2.5 or one 2.2 Animate, let us know what your thoughts are, what the strengths, what some of the weaknesses are.

32:21We'd love to hear that. So thanks everyone for watching. Like we said, let us know in the comments what you thought. And if you've got any other questions or anything else you want to see with Comfy, we'll try to figure it out and post about it. Links in for everything we talked about as usual down in the show notes or over at denoisepodcast.com. Yeah. So if you want to see another Comfy episode, this is the best time to let us know because we're doing this. So let us know what the next subject that you want in Comfy, and we'll dive right into it. a quick shout out to mariana at promise who i forgot to shout out when we were talking about promise leadership the other day she's the head of product and she actually invited us to the event that joey and i went to um fun fact her son is sick of our voice because she listens to us in the car when she picks him up from school mariana number one fan thanks mariana keep playing us for your son yes uh all right thanks everyone we'll catch you in the next episode