#231 Paras Jain: The Future of AI Video Generation with Genmo

0:00We were really focused on driving two things forward, being the best of these in the market. Number one is motion, having the best motion, because we feel many of the products in the market today, they don't feel like real video. They feel more like live photos, almost off your phone or GIFs. And our model, Mochi One, has the state of the art motion in the industry. And number two is prompted humans. It does what you ask, and that's really important. We have the fastest time to press pixel in the market for our model. We will start to see videos rendering off of the GPU within 10 seconds. So Paris, lovely to have you here.

0:33Why don't you introduce yourself to listeners? Tell us a little bit about the company, and then I'll start asking questions after that. It's really great to be here, to connect with the wonderful audience here. I'm Horace. I'm one of the co-founders and the CEO of Genmo. We are a company that is working to advance the state of video generation market. We are building some of the frontier models for this. We recently released Mochi One, which is one of the state-of-the-art models for video generation of the market. It provides some of the best motion we've seen out of technology today across the field.

1:15My background is I started my journey doing a PhD at UC Berkeley, working on large language models. It's insanely exciting technology. and I started the company with my co-founder and very remarkably my brother Ajay who was one of the pioneers working on diffusion specifically he is one of the co-inventors of the one of the most important architectures in image generation and so it was kind of a confluence of a perfect set of factors for us to start a company to work on video because we feel video is represents the ultimate medium for communication. Two questions. Runway and Pika have kind of dominated the public imagination in this space.

1:56How did you approach starting a company? When did you start it? And in the face of that kind of competition, and how do you differentiate yourselves? So when we started the company, video generation just was not possible. We started in December 2022. to. Ajit and I both were doing our PhDs at Berkeley. And, you know, he had worked on image generation. He had done the foundations of 3D generation. He had done this project called Dream Fusion while he was at Google Brain. It was the first model that could create 3D objects from text. It was remarkable. And it got so much excitement that like, I remember it was number one on Hacker News for like two days or something like it just massive traction.

2:39I think we hosted the website on our own academic servers ourselves and we racked up like tens like i think a 10 or 20 000 cloud bill in just two days just for the the landing page it was insane because there was so much excitement this was the moment when we were like video or just like creativity was really exciting but first time people were willing to you know um pay for it ultimately something that we just thought was like not possible because the ai technology was was growing so fast um and so So we've shipped three generations of this technology. We actually shipped the first image to video model on the market in January 2023.

3:15And yeah, I mean, it is a competitive field, but the way we see it, we are less than 1 % of the way there to the generative video future. The models today are rough. They make short videos and they're not perfect by no means. And they're hard to control, right? It's hard for artists to actually get good results out of them. There's a lot of lottery pulling is what we hear from our users about just the sea of technology and we were really focused on driving two things forward being the best at these in the market number one is motion having the best motion because we feel many of the products in the market today they they don't feel like real video they feel more like live live photos almost off your phone you know gifs and our model mochi one has the state of the art motion in the industry better than both the companies you mentioned right and number two is prompt adherence it does what you ask and really important because it's very hard to use these models in production or actual use cases if you can't control them right and emoji one is one of the first models that really sets the bar for prompt adherence uh prompt adherence and then the other big issue with these models is consistency um is that how how do you guys do on that has that been solved because i've been seeing some generated videos where the facial consistency is much better than in the earlier models.

4:40It's getting a lot better. We made the decision to open source this model. That was a huge step. I mean, there has been no good open model available on the market. And honestly, frankly, coming from academic roots in UC Berkeley, if there's no open model, it's just a game with a big incumbents. There's no potential for kind of startups, let alone academics to succeed. And coming back to your question about consistency, one of the things that open source enables is people to fine tune the model. They can take the video model and they can say, produce a Craig or a Parse model. Like if I wanted to make a Parse avatar or digital twin, I could do that.

5:19And so people are beginning to take this technology, fine tune the model, and then it gains character consistency. It gains the stability. like it will produce stable and consistent identities ultimately across many clips and that's something that's possible for the first time because of open source that hasn't really been possible in the prior era where you only could describe things with prompts right it's very difficult to get consistency there yeah uh well i have a lot of questions but let's start with uh building the model i mean that's if you're building it from scratch you're not building on top of an open source model.

5:59Where did you get the capital to train a model? How much capital did you need? And how long did it take to train? Yeah, so we spent almost close to a year working on our core motion engine in our model. It's a step breakthrough for the field. It's actually a fundamentally different architecture from how these models have been trained many times in the past in order to unlock motion. We talk about this in our open source release, but one of the key innovations was a new architecture. We call this like this somewhat technical term, ASYMDIT, or it's essentially an asymmetric architecture for video generation.

6:36It actually makes it more cost efficient to train and run. It enables us to actually reach that better quality motion. You know, I did my PhD in large language model scaling, but a lot of my PhD was actually focused on efficiency of training these expensive models. Because they are really expensive. That is abundantly clear. What we feel is it's really important to find steeper scaling laws is what we call this. Essentially what we say here is, you know, for the same amount of compute, just get more amount of oomph and scaling and parameters effectively within that same hardware footprint. You know, in terms of capital, we recently announced our$30 million Series A.

7:13And so that was, you know, that is a that is a large round. But I think compared to what many other companies have done in VideoGen, it's actually fairly conservative. And we are currently at the close to the top of the leaderboard. We're the number two model in the entire market on the artificial analysis leaderboard, which is, you know, third party evaluation by hundreds of thousands of independent raters, effectively, the general public. And we did that with way less capital than many of our peers. Well, can you talk about the initial model? you said you spent a year getting motion down, but when was the official model launched or made open source?

8:01How large is it and how did you train it? Yeah. So we just launched this October 22nd. And we released a 10 billion parameter model. So That's a relatively small model, but for video, that is one of the largest models on the market available today, especially in the movement. But it remains comfortable enough that I think what's really remarkable is we shipped it initially to run on a full DGX server. That's like$250 ,000 of hardware. And within four days, people compressed the model by a factor of 25x. So now it actually runs on a consumer graphics card, something you could go and buy$1 ,500.

8:43dollars. Yeah. And so it's really remarkable. Like everything here goes back to open source, but that's the power of open is that the community is taking this and they're finding ways to make it smaller, make it more customizable, make it faster, just improve its quality. You know, in terms of time, like, yes, it took about a year to develop this, but, you know, ultimately we think of our innovations in three key pillars. Number one is data. You know, ultimately these models are garbage in, garbage out. And one of the things we've been really remarkably proud of to work on is synthetic data pipelines for models.

9:16It's something new. This is really going to be very important for video generation in the future. But we actually find that it's often a faster, more efficient way to learn the rules of physics and nature, ultimately, and how the world operates through just simulation rather than just doing this directly through real data. It's really early. We're still innovating on this, but I think this is going to be hugely important for this field. Something that was never possible for language models, but now is something we can do with video. What do you use for the synthetic data generation? We've been exploring ways to kind of interplay between 3D generation and video generation, specifically because my co-founder has deep roots in the 3D generation field.

10:01And so, I think there's some things that are hard for video models to learn today. One of those is 3D consistency. You know, a test case here is if you walked around in a circle, you end up at the same place, right? And you'll see with video models today, there's actually a gap there. And so it's something that's a specific test case we can monitor. And because it's so specific, we actually find ways to augment it with synthetic data. Just learning also just like 3D structure for objects is also really interesting. To be clear, this is really early. It had a relatively small part to play in mochi one but it's something that's becoming more and more important for us as we go on yeah i mean the reason i ask about the synthetic data is a colleague of mine from the new york times named dale kim built a company was acquired by meta and i i'm not going to be able to remember it now but it's had the word dream in it um and then of course there is unreal engine.

11:02I mean, do you use third-party systems like that to generate the data to train Mochi1? We're testing this internally. It's really early. I think, you know, before I did my PhD, I was in self-driving. And I think what's remarkable there is the state of the art in self-driving is simulation. People train these things literally in 3D engines. Like I remember at our startup, like setting up a massive server cluster and running a bunch of instances of GTA, literally and having our self-driving agent drive around GTA for hundreds of thousands of hours. And remarkably, that actually helped to transfer to the real world.

11:41We're searching for what the right analog here is. I mean, in some ways, our models actually go beyond 3D rendering of the traditional 3D graphics pipeline. For example, one of those areas is like, I remember this was in June or May this year. I remember in an earlier version of the model, we had made a video of a dog swimming underwater. And so when we looked at this video, you could see the sun refracting through the surface of the water as it kind of reflected underneath and simulating how light refracts through like liquids underwater like that. That's actually really complex. Like 3D engines struggle to do this accurately in a physically consistent way.

12:20And it was something our model could do in less than a minute of GPU time. Like this would take thousands of hours of rendering to produce. So I think it's really interesting to see where video generation models have actually deeper physics understanding than 3D engines versus where 3D engines still have additional capabilities over video generation. Yeah. So you built the model from scratch. You trained it. I don't know if you don't want to answer, but I don't yet understand how much it costs to train. and then you open source it and people start iterating. Where are you now in the process? Yeah, so we are continuing to grow the open source community, but we're also continuing to develop our next model.

13:09I mean, in terms of resources to train this, to train Mochi One, it took something roughly on the order of a thousand GPUs, right? So it's a fairly substantial footprint of resources. But, you know, I've heard estimates that, you know, there was a blog I'd seen that estimated Sora took about 10 ,000 equivalent GPUs to train. And so if you think about that, we are order of magnitude more efficient for something beginning approach, you know, that level of Sora quality, I think. You know, we're not quite there, but, you know, I think as I've shared on the leaderboard, we're at the top of the market, right?

13:43And so I think efficiency is going to be a huge, huge thing here, because if you think about these video engines, They're just incredibly resource intensive to train. Right. I mean, what's also interesting, though, is serving these models, which is a huge challenge. One of the things we spent a ton of effort on was reducing the latency to generate these videos. I think it's a really common experience using VideoGen as people queue up a video and then it takes 10, 20, 30 minutes to sometimes finish. Right. It's not an easy experience. And one of the things we did is that we have the fastest time to first pixel in the market for a model.

14:17you will start to see videos rendering off of the GPU within 10 seconds. And so I think not only is quality important, but also just sheer speed. It's very hard to use creative tools if there's high latency. And if we can reduce that iteration speed, that makes the creative process so much faster. Yeah. And the model is available to people through the cloud or are people downloading the weights and building their own instances? How does that work? People are using it in three key ways. We have a hosted product which we make available. This is freemium. Anybody can use it. But open source is really interesting.

14:59With the open model, people are hosting APIs for people to build access to this technology. We work closely with our inference partners to provide high-quality APIs for people to run it. But what was never possible with any video generation company was for you to be able to take the model and run it locally and really like touch and feel it, like understand how it works, pull it apart and look at the internal mechanisms. What open source means is people are downloading this thing. They're finding ways to shove it into the smaller, smaller hardware. I saw somebody working on a port of the models were running on a MacBook, literally, right?

15:33It's just incredible to see the progress there. And alongside that, people are finding ways to kind of hack on it and find new capabilities that you just could never do with an API or with a product. One example here is, you know, I had seen a community member launch video to video editing on our open source model, something that we never we never trained it to do this. But they had found by taking the open model and hacking on it, they could take a video and they could take a video of a person and say, add a hat and it would actually edit in the full hat on their head. And it looks realistic.

16:04It looks like they literally had that in their life. And it was remarkable to see that happen in the open source community with just no additional fine tuning or anything needed. This was just something that somebody could do with one person and, you know, small set of computational resources. Right. But it is available through API or not? Yes, yes. We have third party API partners who are working closely with are offering reference implementation models. So you can call in and build products. And we've seen some really interesting early product use cases built around the API as well. Yeah. Well, you mentioned third-party or inference partners.

16:46What do you mean inference partners? So one of the providers on Thal.ai, RunPod, and Modo Labs are three providers who have been hosting OSI One for their customers. And so people can go there and they provide the GPUs and the computational resources to integrate with and they serve the model. Yeah. In this whole video generation space, so you've got a handful of leaders. Do you think there will be a consolidation around one model or do you think that there are a diverse enough range of use cases that the models will become specialized in different areas. I'm just curious how you see that moving forward.

17:41I think there will be millions of models for video. I think video is just too diverse for there to be one model. You know, with language models, there's only a few reference models here at the leading edge. And I think what's interesting is,

17:58you know, actually, backing up, The way I see this is foundation models are raw materials. In their raw form, it's like crude oil. Like if you or I went and we started drilling and we found crude oil, like you could shower in it, you could brush your teeth. Like, I don't know, like it's not inherently useful. You have to refine it and customize it and distill it to make rocket fuel. And our model is very much the same thing. The raw model itself is powerful. It is valuable. But the best use case are going to be people who take it and find some interesting use case. Like it might be animation or like anime, or it could be like marketing e-commerce.

18:35It could be like, you know, some kind of new storytelling application, right? It might be some style or aesthetic and they fine tune it and they produce the best model in the world for that specific use case. And I think that's how this technology comes to market. Video is just too diverse. Like creativity is unlimited, I think. And in this way, like you can't just have one model ultimately that just is the best of the world in every single day, right? Yeah. and i were as i recall looking at your site you guys are getting close to six seconds uh for a video right yeah so we support 5.4 seconds today that's right and six sex and six seconds is kind of a a benchmark in filmmaking i mean a lot of scenes are six seconds long and you You just, you know, and then you have a longer unbroken shot.

19:33But a lot of it's six seconds or that area. How are people using it today? And what has to happen for the generation to get longer? I mean, you know, up to a minute or two minutes or something. Yeah. Yeah. You know, what's interesting about this is actually the average shot length in film has been decreasing over time. It's this really interesting property that's been happening. And I think that doesn't mean that models should be restricted to shorter and shorter windows. Like at the end of the day, today, we do single shot. Moji One is a single shot model. I think there is a moment where we go across and I think that's really the next big evolution.

20:18And so there you ask the model to produce, let's say, a 20 second end produced, let's say, commercial or something. And there the shot transitions are just decided by the AI based on where he thinks the best place to put them is going to be. The biggest challenge with length, though, is cost. Like the cost of these longer video generations actually goes up what we call super linearly. So doubling it costs more than two times the cost running it. And so, you know, I think I see this technology very much as like crawl, walk, run. Today, this is the stage where, you know, we actually chose, it's funny enough, five points to four seconds because it was just over the minimum length of video you could upload to TikTok.

21:03So for many of our peers, like you couldn't even upload their videos to TikTok because it was just below that threshold. And so we but we wanted to make it big enough that it was useful, but not too big that it wouldn't run on real consumer and hardware. If we made it 10 seconds or 20 seconds or 30 seconds, you would need these big distributed GPU clusters to run it. But 5.4 seconds is the sweet spot that it fits, again, on a graphics card that a consumer can buy. Yeah. And what about consistency between shots? I mean, I understand that you've solved sort of facial consistency as someone's moving or turning or something.

21:45But if you have the model, and maybe you can explain the workflow. If I wanted to generate a model, I mean a shot of me laughing, and then I wanted to have a second shot of me frowning,

22:07what's the process first of all to get the model to generate an avatar that looks like me and then to ensure that the second shot is matching the first shot yeah I mean since it's open source I think the step here is you would collect you know some reference images and video teach the model what is the concept you're trying to teach like what what does what is parse you know who's correct You would represent that and you would fine tune the model. It would become really adept at creating that. And at that point, you would be able to consistently produce your identity. I think this character consistency is really important for creative workflows.

22:48Ultimately, what is a story without actors? What is a movie without people? We were really focused on the first step again for motion was actually about producing realistic looking humans that don't look like claymation dolls. I think this was the first big challenge. A lot of AI video, people kind of just look like puppets almost. They don't feel real. So that was the first threshold. We crossed that with Mochi 1. I think today already it's possible to do this fine tuning in order to produce consistent characters, but it takes resources costly. You need some technical know-how to fine tune the model today.

23:23But it wasn't even possible to do this with closed source models and platforms before. You know, I think in the future, like my hope is that you can get to like few shot prompting. So you can say just like an LLM, you can give it an example and say, here's this like text. I want you to write in the style. You know, it should be possible. I think eventually for these models to say, here's here's this person and this is the character, right? Like this, this, this is the character I want you to create. And here's their backstory. And then it just can consistently produce shots of that person. We're not there yet.

23:54the technology needs this fine-tuning step still to produce consistency. But I think this is where the technology has to head to make it really usable and provide good iteration for creatives who aren't really going to be able to fine-tune this technology easily. Yeah. What's the difference between what you guys are doing and companies like

24:22Synthasia, is that it? Synthasia. Yeah. Where you can provide a lot of reference video of yourself and they can create an avatar and then you can write scripts and the avatar will speak the scripts, which sounds like it's the same technology in a way. But what's going on there? What's the difference? So these are actually two fundamentally different approaches to video generation. I would say there is a category of avatar generation, like you're describing, where they generally take an image or they'll take like a short video. And they'll manipulate like the lips and the eyes and your facial features to make it actually say something and match the audio.

25:09And I think in limited scenarios, this can feel realistic. But if you think about it, that is as far as the technology will go. it will. And now if you want to say, have the person pick up an apple and take a bite out of it, right? Like technology is not going to be capable of doing that fundamentally. It's much more cost effective and you get much more control. But I think with an architecture like ours, what we've chosen to do is we've chosen to build what we call almost like a world simulator. It's a physics engine that simulates, you know, inertia, mass, you know, it simulates optics, viscosity, it simulates optics to viscous liquids, right?

25:46Like it's everything, the cross product of all the physical relations with the world. And so we started with really simple scenarios, but at this point, like it can produce humans. What it won't do is it won't produce dialogue yet. Humans, like they'll kind of look around, maybe they'll speak, but there's no audio, right? And it's not going to be matched to dialogue. But, you know, I believe as these models continue to scale, they'll gain the capability to have better ability to actually, synthesize consistent dialogue. If you want the person to say what it's going to say, it's going to produce it, but it won't just be able to say that.

26:18It'll also be able to do things like pick up an apple, eat it, drive a car, do anything a human would be able to do. And I think this is fundamentally just more powerful and will produce more interesting storytelling in the long term. Yeah. I haven't spoken to Jan LeCun for over a year, but I was following his work pretty closely, and I spoke to him several times when he was working on world models and using video. And his experiments were pretty crude compared to the video generation models today. But how foundational was his work on what you guys do? Or was he going down one alley and you guys went down another?