We talked to the CEO of ComfyUI

0:00So node-based interface became this very natural existence in here to allow you to do that. And that is pretty much what ended up allowing people to adopt Comfy because it gives you as much control as possible into these AI models. And it's to this day why Comfy is really unique and different and being adopted as the kind of central AI tools in a lot of these studios.

0:25Yoland Yan:all right yolen comfy anonymous thanks for thanks for coming on it's great to meet you actually yeah this is the confusing part right off of that maybe i didn't give you guys enough i am not the comfy anonymous my co-founder is yeah yeah i i'm the ceo okay yes yeah because comfy anonymous that is a good lead into our first question the though like what is can you give us kind of your background and the origin story of Comp2UI. Yeah, yeah. It's actually funny that it confuses a lot of people because Yannick go by Yann and my last name is Yann. So it's like, you know, things start to get very confusing.

1:03And Yannick usually isn't very, you know, communicative with some of the open source players. And I am. And we had this incident where folks on hogging face side was like wow this guy's bipolar uh half of the time he's like not responsive and uh you know sometimes uh somewhat aggressive and half of the time he's so nice and so clear in terms of his communication what what is going on i was uh yeah like that it does confuse a lot of people let me just go back in terms of uh the history of the the product or project. Initially, this started as a pure open source project. This was beginning of 2023.

1:48Actually, the background story of how Yannick started all of that itself is also another rabbit hole and fascinating stories. But just to give you the high level, he was, prior to this, never wrote a single line of ML code, worked for 10 years as his dad's dental implant factory as a C++ programmer on automation in Quebec City. So like, you know, pretty much the furthest you can imagine from, you know, a Silicon Valley AI tech company and wrote it in two weeks, put it out there. And at this point, you know, or pretty much a year later, he became probably the best inference person in the visual space.

2:32And the project after he launched at the beginning of 2023 start to gradually pick up. If you look at the history of Comfy's journey, it has always been this very, very steady growth. Never this huge jump, people hyped over it, you know, as we probably are seeing from OpenClaw. It takes a lot of effort for people to onboard to this product and it gradually grew and the ecosystem start to flourish over time. About a year into this journey, that's when I came in a bit earlier than that, but that's officially when we kind of got started. I, on my end of my journey, built my own product on top of Comfy and then started to see the value of this entire ecosystem.

3:21At the same time, we were also really curious in terms of where the project is going over the long run and ideally, if possible, help guide the future towards more beneficial on our end to be able to build on top of Pumfy. And got kickstarted the conversation, started contributing to the space, got connected with Yannick and other open source contributor and came to the realization that it is a incredibly grassroots process. No one's, you know, there's no Jesus taking the wheel type of scenario. Everyone in here is just, you know, focus on a very niche area. I think Comfy Anonymous, Yannick, at the time was just doing a huge amount of model improvement, inference optimization, and other folks in the ecosystem might manage the extension and someone else might do the front end and a very uncoordinated effort.

4:19And that's when we realized the problem and the opportunity to kind of centralize the effort under a single organization and start a pitch towards them. And about, you know, at the beginning of 2024, that's when things officially started working, us working together and brought in quite a few open source contributors in the ecosystem. So that's kind of the origin of starting point, I guess, both in terms of the tool and the company. Happy to kind of go into more the past, you know, ever since we got started, what are some of the progress we've made in here But yeah, I don't want to blabber on for too long.

4:58Yoland Yan:Do you know what inspired him to want to make this tool in the first place to kind of start building out early versions of Gen AI imagery? Yeah, actually, there are three key characteristics of Comfy or what it enables people to do. One is control, the other is quality, and then the third one is scalable and scalability. What started this project at the first place is still what remains to be true today in terms of why Comfy is this central adoption in the studio VFX animation world overall. When he first started, he wanted to actually have a system that renders the initial part of the image using one diffusion model and using another diffusion model for the higher resolution generation.

5:52If you guys played around with a lot of open source model, you might know that some models are very good in terms of their aesthetics, but they might have limitations, for example, in terms of the resolution that they generate. And sometimes if you ask those models to generate higher resolution, what they give you is, you know, four quadrant of images, because that's what those are the resolution they were trained on. So he at the time wanted a model that is really good with aesthetics on the initial 50 percent of the steps in terms of generation and then using another model to finish up when the foundation of the image is already laid out.

6:29And there's just no product out there. There's nothing that gives him the control like that unless he just rewrite a code himself. And even when you write the code, there isn't a good visual feedback on a screen that tells you like, oh, OK, yeah, you change these parameters. This is the result you get. So node based interface became this very natural existence in here to allow you to do that. And that is pretty much what ended up allowing people to adopt Comfy because it gives you as much control as possible into these AI models. And it's to this day why Comfy is really unique and different and being adopted as the kind of central AI tools in a lot of these studios.

7:13Amazing story. I love it. Do you have the, does Yannick or you, do you guys have the original image? Because that can go up on the AI History Museum, you know, 100 years from now. we do uh man uh this company has gone through a lot of uh you know in terms of its brand uh position one thing i sometimes show people is uh i'm gonna pull up over here in terms of the comfy org website we really leaned in hard maybe a little bit too hard at the beginning in terms of this uh grass fruity energy this is the initial website of the company uh this is the very first image generated with comfy ui and that's literally like you know the home page as you can see it's yeah the home page itself is also very just like make me cringe a little bit at this point looking back um but also back then that was uh you know uh what we're leaning into in terms of kind of wanting to be this open source uh and you know very grassroots energy type of organization And gradually over time, start to become more mature, both as a product and as a brand.

8:25And now the website looks different. And in the future, the website is going to even be more professional and, you know, very much speaking the voice towards the professional creatives.

8:36Yoland Yan:Yeah. I mean, can you tell us about kind of what the current landscape looks like with Comfy, sort of the kind of current products and where you're looking to go with where it develops? Right now, there's some of the high-level examples I can give in terms of companies' popularity and adoption. There's even more in terms of high-level roadmap, what we're looking at over the long run, given the current where we are. In terms of the local product, it's actually fairly prevalent. We now get close to around 50 ,000 downloads per day in terms of the local installation. There are approximately total about 4 million users out there.

9:20And given the professional creative ecosystem, it's actually a non-trivial amount of number in here. I think the total estimate on our end of the entirety of professional creative in the world is about 50 million overall. and already approaching about 10 % of those group of users. And then ideally over the long run, we'll keep on expanding in this category overall. And largely these are kind of folks that want exactly those three characteristics, control, quality, and skillability in the end. When you have a pipeline, you can kind of push that out into a production environment and keep on running it against different assets, different input.

10:04But over the long run, at least from our end, we want to build Comfy into this, what we call operating system for visual AI. Essentially, what in effect it means is that in the future, let's say five to 10 years timeframe, more than 50 % of generative content goes through this Comfy workflow layer. It doesn't have to be through our interface or product. It can be running on local, it can be running on our cloud, it can be on mobile SDK, it can be on anywhere. But as long as Comfy is the engine or workflow orchestration system that empowers a lot of the creativity, I think essentially we've changed the entire landscape in a way that the tool is owned by the creatives as opposed to, you know, historically, I'm sure everyone knows that largely the creative tooling space is usually locked in in a close ecosystem.

11:08And, you know, and one might even argue that it's fairly toxic in terms of the relationship between the tool builders and the users out there. And that's the paradigm we want to shift in the new AI era. Yeah, I think it's really admirable that you have a grassroots origin story. And, you know, to some extent, Comfy still feels very much like an open source independent company. You know, if you compare it to like the big tech giants in the visual space, like your, you know, your Google, your OpenAI or whatnot. And I think that grassroots story, it really resonates with the creatives because nobody loves, you know, I think everybody loves a grassroots story about a creative origin story like George Lucas and Star Wars, right?

11:55Like that started out of his garage and things like that. So I think if that is in your DNA now, that is definitely working for you and it'll continue to work for you in the future. Yeah, we actually, the one thing that, you know, people probably don't realize enough is the amount of success Comfy has enjoyed. A huge portion of that actually came from the open source ecosystem. These are custom node extensions within the Confu ecosystem, the builders around that. You'll be surprised every day. They're just a new node people created, vibe coded these days, that put out there that solves part of the problem and open that to other people that can adopt into it.

12:43There are more than 60 ,000 nodes in our ecosystem right now, which is, you know, if you compare it in terms of extensions to other platforms outside of creative world, it's probably one of the top type of platforms out there to have that many open source developers. Same thing for workflow builders that people were, you know, combining these nodes together for the visual creativity. And pretty much everyone in this entire circle of ecosystem are contributing model builders as well. So, yeah, that's one of the major reasons why Comfy ended up being such a huge success in comparison to any close systems out there.

13:28Yoland Yan:This was sort of an ongoing question that I always have brought up, but ComfyUI is free. It's open source. You can download it, install it. Only recently you've had Comfy Cloud, but before then, how did it make money? How was it being developed and flourishing and continuously improving? I would say the initial journey of Comfy as a personal project was completely zero funding. There's not even sponsorship for Yannick in this process. Obviously, he was partially employed during some of his time as stability to continuously maintain the open source project. But ever since kind of we came together as a team, it's mostly initially through venture funding that we found folks that are very aligned in this vision.

14:23And my God, there are we talked to a lot of people in this ecosystem and we filter out a lot of, you know, from outside view, very solid investors. But fundamentally, we just knew they were not aligned with the vision or come in with the perspective. Hey, Comfy is a good brand. Maybe now is the time to shift towards a Comfy Pro version. So yeah, at least at the early part, it was finding the right investors and fund the team to keep on moving forward. In terms of more recently, our cloud launch and other commercial kind of product launch in here. And this is generally describing why it's actually an incredibly powerful model to empower the growth of creativity and creativity tooling over the long run.

15:15Comfy at the core is built as a, you know, the open source product is built as the one person, one machine type of experience. a huge amount of optimization and work we've put into. And this is pretty much a state of the art for the past two years is around, how do you enable people on a single system that just run these models and get as much out of it using your own local computer. At the end of the day, as an individual creative, you're not gonna buy 20 RTX 6000 and put that on the cloud or run that in your basement. It's financially irrational to actually do that regardless, even if you are able to run it at all time.

15:58So from those kind of perspective, a very natural pricing or business model around it would be providing a very powerful backend that we are able to optimize for on the cloud in a very distributed fashion that no longer requires you to kind of maintain this kind of complex logic of running things in the cloud. Hence why a lot of people adopted ComfyCloud as a product in here. In that world, we actually didn't need to do any feature gating as typical open source product. A lot of these developer products will have some of the features out in the open. Let's say, oh, I will provide 50 % of the nodes in the open source community version.

16:43And then the remaining nodes all happen on the cloud. Really not necessary because the speed and performance of the product is a feature, is one of the major feature. Are you going to wait for two days in terms of your rendering pipeline? Or would you be willing to kind of pay for the value of the orchestration on the powerful cloud backend that can finish the entire run in 20 minutes? Those are the kind of key differentiation that enables people to naturally flow towards using ComfyCloud. And we actually even see people who are both using local when they have some decent machine. And concurrently, they will run things on the cloud just because, you know, both systems have some amount of wait time.

17:26And right now, your creativity is going faster than the compute or rendering time any machines can provide you. So in that world, the more you provide out there, the more people will use it. So it's a very healthy kind of usage-based model for us.

17:40Yoland Yan:What are some of the applications that you've been seeing, you know, either in studios or larger companies? And any of those applications been surprising to you or unexpected? Oh, man. Surprising. There are a few, some of which I don't know how much I can talk about in terms of the surprising part. But in terms of general application, a large portion of our users are in the studio, marketing, VFX, animation world. And that's primarily where we focused a lot. And for those group of users, the pipeline is usually, you know, a high level pipeline is very similar. Like individual workflows might differ a lot inside this company.

18:23They might have a pipeline that swap out, you know, autonomously swapping out, you know, logos that are not their own brand as part of their video footage and make the video look more natural. Another company might have a voice replacement model that allows the footage or the voice generation to be more natural or translate to a different language. But in general, the three key criteria is still the same for all of these users. The application being, you know, giving as much control to the creatives, you know, having the highest quality and then also in the end, a pipeline that's scalable over the long run.

19:07So, yeah, like there's a huge long tail of of these applications across different studios. But the fundamental nature is very similar. I saw in a recent post on Comfy's LinkedIn website that Comfy UI is now a job description requirement on a lot of AI facing jobs. How do you feel about that? And how does that help guide you in future product roadmap and things like that? The fact that you know, this grassroots thing is turned into a requirement in a lot of cases. That's definitely super exciting on our end. And this has actually historically been the case that there's a Netflix post almost two years ago or a year and a half ago when we just got started.

19:55Said, hey, inside Netflix is looking for comfy creatives. And at this point, there's actually a lot more throughout the entire ecosystem. I think Netflix is fairly ahead of their time. One thing we want to do in here is, and historically, I would say quite transparently, Comfy didn't do that great of a job to empower kind of the on-ramp or learning process from individual creatives onto the platform. And going into the future, this is definitely an area we want to address. We want more people to actually come in with the experience to say like, oh, Comfy, I learned that in this course. I learned that in college.

20:35I successfully ran through the process of onboarding within the Comfy product and already know the basics. So these are areas where we do want to help address. And from a product side, there's also a lot of planned features that will enable people to go through a smooth on-ramp process. AppMode is one of them. app mode enables people to escape the sort of workflow layer and then just operating at a higher level that you know you will see in your day-to-day or normal type of creative ai tools yeah i think

21:12Yoland Yan:that's kind of how we view it yeah well kind of speaking about the onboarding and sort of building out the workflows i kind of had a question about how you're thinking uh how you're thinking about a combo of like the rise of vibe coding and agentic AI and sort of if you see building features into the product. For my personal use case, I've like tried CloudCode to build out a comfy workflow because at the end of the day, it's like JSON files and it didn't do so. It didn't quite get it. So I didn't know. Yeah, I'm curious if you're kind of thinking of, you know, the workflow and future ways to make it more friendly, whether it's like an MCP server or something that can interact from just plain language like i want to build a workflow that does xyz and then you get a starting point with a comfy workflow that exists and they can modify yeah yeah it's a great point a lot of people have brought this up to us so far i would say like it's definitely a direction that we will we are pursuing over the long run just you know the general theme of making a product easier and enable lm systems to you know even as part of the on-ramp make it easier for people to learn and adopt the tool and also in the future edit the tool.

22:27If you think Comfy has somewhat of a visual programming language, I would say we're still in the, you know, precursor or the VS Code days. Even from that perspective, there's some product level of uplifting we need to keep on pushing on that actually get to the VS Code level of quality of a product. But yeah, one nice thing about Comfy being open is that other people can do this. Just yesterday, actually, a client, this AI coding tool, organized a hackathon on their own version of Comfy MCP. Alibaba created a tool called Comfy Co-Pilot that does exactly this. And a lot try. But yeah, if you actually fully, you know, kind of build a agentic system to generate these workflows, it's going to be a lot more powerful and more accurate.

23:25Right now, at least the nice thing of being open source is other players in the ecosystem that will come in to help. And that is just generally a repeating pattern you see from the comfy ecosystem. Someone will come in to deliver these features and help the rest of the ecosystem. And eventually on our end, it becomes a critical point or path for us to deliver these features from the core Confi experience, then we will definitely execute on that as well. Just on that note, there are now quite a few node-based AI workflow orchestrators, right? Some of them are, you know, like, what was the company that got acquired, Joey, by the Foundry?

24:05Well, there's Grip Tape. Yeah, there's Grip Tape, there's Invoke, there's Weavey.

24:12Yoland Yan:FreePix has nodes. Yeah, nodes are very Invoke right now. And how do you see Comfy sort of maneuvering this in the future? Do you see the product getting easier to use or the opposite, getting more and more defined by professional use cases? And in some cases, it's getting harder to use. Or do you ride the line in the middle like you have been? On the one thing we keep on, you know, emphasizing within the team and also to the outside user and rest of the ecosystem is Comfy never really want to sacrifice the power, the core power inside the system. Because that's actually why people at the highest end of quality and control adopt Comfy is because it has this core power.

24:58If tomorrow I all of a sudden dump down Comfy in a form of, hey, let's just do image to video to text to 3D and to audio. That's it. You know, this kind of general process in here. You will probably bring on a lot of the new users, but you also ended up completely losing the opportunity for, you know, making Comfy this highest grade of professional user experience. That being said, I do think user interface is a accessibility problem. And that's something that we actually can both maintain the power of Comfi while addressing some of these easy to use issues. I actually think it's interesting that a lot of times, you know, from our understanding, when you use other node based systems or products that came out over the past half a year or so, it's still a complex system.

25:57It's still something that requires onboarding and people need to understand, oh, this is how you connect things together. So even from that lens, there's complexity all around. From our perspective, we just want to cater towards the professional users and also the group that connects towards this professional group. AppMode is actually the prime example in here where it didn't originate from us thinking, oh, let's, you know, kind of build Comfy into a consumer product. It actually originated from professional creatives that want to be able to have this type of feature so they can enable their workflows to other creatives in their team that doesn't use Comfy but do have ideas that want to test out against these workflows.

26:43so those are or even in some case scenario they built a successful pipeline that just continues run and they don't really want to go into the you know node graph layer to make more modification so they want a simpler version of this to engage with the workflow they've built not it doesn't mean they're no longer you know professional creatives it just simply means it's the accessibility layer that you enable for their needs that's the lens we're looking at tell me

27:09Yoland Yan:about AppMode and sort of the overview of the new product that you're launching? Yeah, AppMode basically, if you imagine all of the workflows as some kind of templates, AppMode is basically this layer that allows these templates to be easy to use. What you can do is on the workflow graph, you can select a subset of parameters you want to expose on within app mode. And once you complete with that, it's actually a fairly simple process of just a click and select a bunch of parameters in here. And you can swap this workflow into the app mode. And within the app mode, only those parameters are surfaced and you have a more of a pure, simpler interface to directly interact with the workflow.

28:00And if you just change the prompt, then the result will run through the workflow, but in the end, only the prompt is getting changed. So is Atmod a method to getting a local executable? Or it's, I mean, how does a user use it who doesn't need to touch Comfy? Would enable people to keep on spreading AppMode type of workflow in the same JSON format, in the same experience. You can send it to other people and they can install this app either on their local instances or on cloud as a shared link. And then through that kind of mechanism, people can use it everywhere or share with other people that they can just install the nodes they need or models they need and in the end execute that on their local instances.

28:48So it's a cloud-based workflow abstraction layer, would that be accurate to say? It doesn't necessarily link towards cloud. Anyone local can also establish that as an app, and that will be registered as part of your workflow file to say, hey, by the way, this is a workflow that's app mode preferred or at least opened up in the app mode experience. And then you can send the same workflow to other people, or you can put it on the cloud and then give people a URL, a link. That's the kind of general experience we would imagine people to do either directly do the typical process of share the workflow file itself or share a link.