ComfyUI Proves AI Art Is Not Zero Effort

0:00Comfy UI allows you to do is really tweak the actual model pipeline, like how the model is inferenced. And that lets you do potentially a lot of very interesting things and get a lot of control over the model. Comfy UI is just the more advanced way to use some of these models. GPU, like right now, we spend a lot of effort optimizing the memory. You don't need that much memory anymore. Video is definitely not topped out yet. There's a lot of things to do there. We're going to see more efficient and better models because I feel there's a lot of optimizations you can do it with video. Welcome humans to the Neuron AI Explained.

0:46I'm Corey Knowles and I'm joined as always by Grant Harvey, the only co-host I trust to bring both nuance and a backup analogy. How are you, Grant?

0:55Corey Noles:Oh, gosh. I have the nuance, but I don't know about the backup analogy. It's like pizza. You throw a pizza in the air and maybe, you know, you put something together. Depends on the day. Well, I'm really excited, Corey, because today we are talking to Yannick Merrick, original creator and co-founder of ComfyUI, which is the open source node-based workflow engine that started as a power user tool and is increasingly becoming a real production infrastructure tool for visual AI. We'll get into what Comfy is and, you know, what they're building and how it works with both open source and cloud-based models.

1:32Corey Noles:And actually how professionals are using Comfy UI in the real world. And hopefully what the next wave of model development might look like. Excellent. Excellent. I'm really excited. I've used Comfy for a little more than a year with images and creating video. And with that, Yannick, welcome to the Neuron. We're so excited to have you. Well, thank you for having me. Absolutely. I guess for starters, for listeners who have only used tools like ChatGPT, MidJourney, or something like that, how would you explain what ComfyUI is in plain English? ComfyUI is just the more advanced way to use some of these models.

2:15Like if the simple way is to just like spin up, go on chat GPT, Gemini, and just type, generate me an image of this. But that doesn't give you any control over the actual model pipeline. It just outputs an image and then that's like you can't really, it's a bit difficult because what Comfy UI allows you to do is really, tweak the actual model pipeline, like how the model is inferenced, and that lets you do potentially a lot of very interesting things and get a lot of control over the model. It's like it actually gives you control over the model itself, which depending on how creative you are, you can do some very creative things with that.

3:13Corey Noles:Maybe it'd be helpful to establish for people who aren't as familiar with image models, right? Like maybe they're used to working with language models and text. What do you mean by, you know, inference in this case, like with an image model? And, you know, why does the pipeline matter? Like why should someone care about that? Yeah. So yeah, basically, pretty much all of the good image models and video models right now are diffusion models, which means they start, how they work is they are models which are trained to denoise an image using a text prompt or other input. So how you actually inference the model is you start from a full noise, a full noisy image, and then the model will predict, try to predict the noise in the image so then you can remove the noise and then get an image which hopefully follows your prompt.

4:25What that allows you to do is you can run the model multiple steps, and each step will remove a percentage of the noise from the image. How exactly that happens, there's different algorithms to calculate that. Those are called samplers. If you use Comfy UI, you'll see on the main sampler node, you can select a bunch of samplers, which are basically math equations to solve for the zero noise using multiple steps from the model. So you have a lot of control over that. You have control over... You can do interesting things. like for example you can one of the very popular techniques is image to image which is you take an image add 50 % noise and then give that to a model and denoise that and what that might do is depending how you use it you might get cleaner image it can be used maybe clean an image like you upscale an image and and then you want to clean it up, add some creativity to it, and you add noise, use the model, and get something else which is slightly different from the original input, depending on how much noise you actually added.

5:57It's really, this is very different from just like the simple like ChatGPT, Gemini, you just give it and say it, and then it gives you an image. You have very, like, you can also, there's so much stuff you can do, which depending on what you want, how creative you are, you get, like, people love to just tweak stuff and change. Like, okay, like, oh, I use, I have this model, which sampler works best with this model? Like, how do I, like, what's, what gives me the most creative images that I want? It's just tweaking, gives you a lot of different parameters to tweak to just get something interesting.

6:49And just an infinite number of different things that could happen. One of the things that's really interesting is that there's a balancing act to it when you start playing with your noise and your number of steps and the different things in the equation and changing like which denoiser you're using. there's even from model to model workflow to workflow there are so many different things i feel like that that can really influence what you get and how you can tweak it yeah there's yeah and also gomfi lets you chain these different models together like and there's a few different models that share the same latent space for example so you can like a few different models use the Flux 1 VAE.

7:33Now there's some coming out that use the Flux 2 VAE. So you could do some things like denoise half the image with one model and do the rest with another model. So you can get some very interesting combinations to just try to get something that, like if you're, like for example, if your goal is generating high-resolution images of a specific type, then maybe using... You can set up a complex pipeline to just try to get the best, like, super high-resolution images or... It's... There's really, like, it's not... It's... Basically, it lets you do whatever you want and tweak whatever you want. So the limit is only how creative you are and how much you play with the tool and see what it does.

8:35Most people are only using about 5 % of what AI tools can actually do. They open chat GPT, ask a few questions, maybe write an email, and that's it. Meanwhile, other people are out there building workflows, automations, research systems, custom agents, and saving hours every single week.

8:52Corey Noles:AI is moving so fast right now that most people don't even know which tools are worth learning anymore. ChatGPT, Claude, Gemini, Codex, AI agents, automations, 5Coding, it's a lot. And that's exactly why we built the Neuron Academy. We built a practical, self-paced AI learning platform that helps working professionals like you understand the tools that actually matter and how to use them in real work. Built by us, the team behind the Neuron newsletter and the podcast designed to help busy professionals actually level up their AI skills. Inside, you'll learn prompting, AI workflows, research systems, automations, productivity strategies, and practical use cases across the entire AI ecosystem.

9:34Everything is hands-on, beginner-friendly, and built around real-world use cases you can apply immediately.

9:41Corey Noles:These are real skills that help you work faster, think better, and stay ahead in AI. Right now, you can join the Neuron Academy for just$499 a year, unlocking 50 plus lessons from full AI courses to quick hit micro trainings you can apply immediately. So if you're serious about leveling up your AI skills this year, this is where you need to start. Join the Neuron Academy today. And the first time I ever used it, so originally, you'll know more about this than I do, there wasn't a GUI element that made it really easy. And for people who don't know, GUI is a graphical user interface. It's like how when we're looking at a computer, there's like an interface that we can click buttons and do things with.

10:21Corey Noles:And, you know, at least to start, you had to install a lot of stuff and there wasn't this really easy, you know, what I would recognize, you know, as a normie, as like a regular application. Right. And then eventually you did launch that. And what makes this really cool for people who maybe like don't get all of the denoising and why they really care. They're like, I'm fine with ChatGPT. What makes this awesome is that you can actually use it and run images and run the image models on your own computer. You can also use cloud-based models, which is something that you all added. I don't know. You can tell us exactly when you added it.

10:57Corey Noles:But you can use cloud-based models as well, so with an API. Or you could just run the image model directly on your own computer. And depending on the size of your graphics card, you can actually do the whole process. Like if you were calling ChatGPT's image models. On your own computer. Yeah. I've run a few things, but I'm probably bigger than I should have run. Yeah, and it depends on the workflow, right? Like depending on how many steps you do, you can run it on different machines and things of that nature. But the best stuff I've ever gotten has come out of Comfy, hands down. Yeah, we put a lot of effort into making sure the open source models can run on pretty much a pretty wide variety of different computers.

11:43Like we have spent a lot of effort on optimizations, especially memory optimizations. So, yeah, if you've been following the last few months, we pushed a lot of different memory updates, which basically we revamped how the entire memory system worked, which, well, some people complained about it because it broke their custom extensions. but we're trying like this makes things a lot better for most people if you don't use those specific extensions. Not everyone has a$12 ,000 desktop. Yeah, so we try to make it as accessible as possible because our core has always been to be able to run these open source models as well as possible.

12:38like we do support the like API closed models, but those are very easy to add because they're just an API call. So there's not that much effort. Like the reason we support them is because there's some people that want them and it's not that much effort to properly support those. I can see there being value in like,

13:02Corey Noles:let's say you do it locally and then you could like send something off to seed. dance, right? Like, that could be good. Yeah, it makes, like, it's not, like, it again, we're trying to be open source, but we also are realistic that there's a lot of people that want this, so we should provide it, because it's valuable for a lot of workflows. Right. Something I love is that you all have done a lot for accessibility with these, like, the ability to have so many pre-built workflows that you can one-click download and just get up and running with. Like, that was really big for me starting because, like everybody else, I came from ChatGPT.

13:49And I was like, oh, there's this other way. And I got all excited. And I downloaded, trying to remember, Flux1 Cria. I think, I can't remember, but it feels like it was Flux. and that's kind of where I started playing with it and working through things. And the nice thing about being able to pull down those pre-built workflows was that I could see how it worked. Like I didn't have to build it from scratch. I could see it and then I know what happens if I move this over here? What happens if I raise this dial all the way to the top?

14:22Corey Noles:It makes it so you can experiment a little bit more because you're like, okay, I see there's this little field over here. So maybe this is what it's set at as the presets. Maybe if I change it a little bit, I can see what happens as opposed to trying to set all that up yourself the first time you come in there. Yeah. It's good. Yeah, I think that's the way you're supposed to use it because you have all these samplers, but you don't need to actually understand what they do to just try them out and say, oh, this one gives this type of image on this model or, oh, this one is broken on this model.

14:57Because if you know the math, even if you know the math, like you're still not going to know what like what the actual effect is going to be depending on the specific model. So I think really the best way to learn is just just change things and see what happens. And if you if you break something, you can always load an earlier workflow. Yeah.

15:20Corey Noles:Well, you mentioned the extensions, too. And I think one of the interesting things about Comfy is how like there's a huge community around it, or at least huge relatively. Yeah. Speaking to me, there's so many people building stuff on it and creating these workflows. It's really awesome. Yeah, I think that's because we have like I always kept things. It's very simple to make a custom node for Confu UI. Sometimes maybe a bit too simple because some people like they make custom nodes that they don't really behave the way they're supposed to. and then they break things, but I think that's just normal when you have such an open platform.

16:05You have to sacrifice a bit of potential user experience of someone, oh, I download this workflow, and then, oh, something is broken because this custom node was built for the specific version of Comfy UI, and then it doesn't work on the latest one. That's something we're slowly trying to solve, but it's a difficult problem because we don't want to put restrictions because I think if we put too many restrictions on the custom nodes, that's bad. Yeah. But so by default, we try to keep it pretty free. And like physically people can do whatever they want and just because that goes more with like the spirit of the of the project.

17:02Yeah. But, you know, you guys have always had kind of a if it can be open, it should be open stance around these things. Now, with having waded into projects like cloud and API and enterprise, I was wondering, how do you decide what belongs to an open source? What's going to be our commercial product? How does that balancing act work? I think the balancing act is that we, like, ConfuEye has always, like, since the beginning, we've always been, okay, this, you should, it should be the best platform to run things on your own computer. So we've never, like, I've never, like, the project has never been, okay, this is the best platform to run on your, like, enterprise, expensive enterprise GPUs.

17:52So you'll see most of our, like, there's some people that do that, but I think, like, I think how we differentiate things is that, like, our cloud, for example, has a orchestration layer that I don't think we're going to open source because that's not beneficial for the person running ComfyUI locally on their own computer.

18:16Corey Noles:Right. So that's how I see things. If it's beneficial for the user running it on their own computer, then it should be open sourced. If the user doesn't, like, it won't have any beneficial effect, then I don't really care about that. I actually love that. I think that's the right way to approach it. It is. Excellent. What about, so for example, when you have a project like the orchestrator, how does that work or what is that used for? Basically, our goal with the cloud is that we have a bunch of GPUs that we rent. And the goal is just to someone running a workflow on cloud that the workflow should be ran in the most efficient way possible.

19:16and that means like a quick example of an optimization is that, okay, if there's a machine with a model already loaded and there's some workflows that use that model, well, they should be executed on that machine. But that's a very basic example, and we can have like more advanced cloud-specific optimizations. like we could go and opt, like depending which GPUs we get, we could do optimization specific, very specific for those GPUs and there's potentially some which like optimize for these GPUs that wouldn't affect anything for the local users so that like, that might be like cloud only or we can just have like, okay, this machine runs this model, this other machine runs this model and then split the workflow across machines.

20:17If the workflow has different branches, we can potentially run different branches on different machines. So just make it go faster than someone would if they just ran it on their local machine.

20:36Corey Noles:Does that also make you a better cloud provider for image models than if you were going to some generic cloud provider that does mostly language models? I think the Comfy is that you can have all this control, which a generic cloud provider, you have the API, you don't really have that much control. You can't do all of the things that you do in Comfy just using the API. I think that's our value, and it's also going to be, like, if you spend time making a workflow on local, and you can just put it on cloud, and hopefully it works. If you don't use any custom nodes or extensions, it's going to work.

21:28But we're working, like, cloud is also supposed to be the easier path. We have some pre-installed custom nodes and extensions on there that lots of people use. It's basically supposed to be like the value is because local hardware is expensive.

21:52I want it to be a choice. If you want, you can buy hardware, like a couple thousand dollars, and that's get a decent machine when it comes to UI. Or you pay like$20 a month for cloud, like for people that maybe if you don't use it eight hours a day, that's the cheaper option. Yeah, right. Yeah.

22:18Corey Noles:No, that's fair. Yeah. That's so that's the value of cloud. and there's also the enterprise. On the enterprise side, there's a lot of potential value which we can, like cloud can provide, like manage stuff, like enterprise. They want, for example, there might be an enterprise. They say, OK, we want to use Comfy, but we only have the license for these models, like not for these other ones but that's simple stuff like that where you can you can provide to enterprises and so employees they go on comfy and they see oh but all the models you can use in this comfy you're allowed to use just yeah that's all have there ever been any any you know I know I know probably primary use is specifically image and video.

23:21But have there ever been any, like, surprising uses of Comfy that really kind of shocked you? Or anything that's been made with it that was really impressive? Wow. I realize I'm putting you on the spot there, so we could... Yeah. Yeah, well, shocking things, I'd say. Well, there's a few interesting people using Comfy UI for not like like like I think I heard some people using it for like finance stuff which I a bit confused but I think yeah yeah and then there's yeah lots of interesting people using comfy UI and some because yeah when I one of the good things with open source code is that it's very private so you might have like some people uh basically do like there's a lot of people doing like like a lot of the the big big movie companies

24:27Corey Noles:they use it but i was wondering about this yeah and some well some of them don't want don't want us to mention that they use it so i think kind of can you mention the types of workflows that the big studios are potentially using if you know even if you don't name them i'm assuming they're using more than my to my than my dell desktop probably too yeah there's uh yeah like it's they use it for a lot of things like uh like it's very popular with vfx people that makes sense yeah yeah i guess i would wonder like with vfx i'm trying to imagine how this would work so maybe they're re-rendering stuff with it are they adding background elements into stuff that they shot like what types of not anyone in particular more just a how would someone vfx use it yeah because so i'll give you an example so when i use it right i'm generating everything from scratch but i know that there's a lot of workflows where you can bring in your own images your own videos that you've um you know generated yourself and you can add additional elements into that so you could add like a background uh you know you can totally swap out the background um maybe you're using it to like upscale um image it like i i'm curious what what you know what's the best use case for it in that industry i think uh like like video in painting which means like removing elements changing elements i think that's uh that's one of the most popular uses.

26:04Like you can, like for example, well, right now there's a new model that recently came out that lets you like a video editing model, just like image editing, but with video. So there's that, but before this model, like people, like one popular workflow was just you get a model to generate a mask of object or something, and using this mask, you can in-paint out, or in-paint the subject or whatever is masked out in the video. Like that type of, I think this type of workflow is, yeah. there's a lot of people doing that. And then, yeah, it's, yeah, with Comfy, you can, like, use high resolution, so it's maybe get a decent quality.

27:10I even think about, like, what are people using it in the gaming space in particular? And, you know, and I even wonder, is there a, you know, if we reach a point where there are open-source world models, a play there as well that could be a cool future in Comfy. Yeah, but you should... The thing with... Yeah, there are a lot of people using it for video game, like video game assets, for example. It's just that they... I don't think many... Well, some of them probably don't want us to talk about it.

27:48Corey Noles:Well, I think what we're dancing around here is the fact that for a long period of time, maybe up until now and still for a while yet, there's a backlash against this in the creative community, where there's still a certain subset of creatives who hate AI because of the way that it was trained and that it potentially stole from all of these artists, right? And so I think it's been really hard to convince creatives that it actually is useful for them. And my counterargument is always, well, open source is the greatest use case for this. And that's why I promote you guys and Comfy is because actually if the models are open, then everyone has access to it.

28:32Corey Noles:Everyone can use it, especially if you can run it on your own computer. But I'm wondering, you know, from your point of view, how you've seen the opinions about, you know, generative AI has changed over time as you've been building this tool and in the creative community specifically. Yeah, I think the fact that we are such a complex tool means people haven't seen that much backlash for ConfuI itself because we're open source, we're complex. And I think a lot of the people that are anti-AI are mainly, they think, oh, there's no effort. yeah a lot of the times it's like oh like if you can there's zero effort and with it like because like oh a game that uses ai like you're gonna get like people are only gonna notice the crappy games that are low effort yeah and people won't put no effort in them so they use ai and it's obvious just like they say about writing too yeah all slop you know if you can catch the slop it's it's slop if you can't catch it then it's great yeah oh it's like cgi people only complain about cgi when it's bad when it's good they don't see it so they all complain i think ai is very similar and for like I think we just need like one very good like game or anything that uses AI but doesn't advertise that they use AI because and then it stands on its own because I see a lot of these problems I see a lot of these like oh this is a video or a movie or a game this was made entirely with AI and it's crap.

30:33So of course people are going to have a negative reaction about that. Oh, there's a cut every two seconds. Go figure. Like if you make something that's actually good, that stands on its own and you don't mention anything about AI until like six months later and say, oh, by the way, this was made with AI. Then people will get it. People might start getting it. And especially if you show this was a lot of effort, like this maybe was less effort than doing it manually but was still like a lot of effort because you have to like if your quality bar is high you still need to put effort into getting there yeah yeah like I would argue using comfy as a skill unique to itself almost and and I think where it'll really be noticeable is not so much in companies that have been lazy or developers that haven't taken a good approach but where people have done things that weren't possible before like when you think about you know like i think it was sony last year adding real-time interactive npcs that that could generate dialogue as you go things like that i think when you start getting into the things that just weren't possible in a game before this, I think people will come around.