Video generation with realistic motion

0:03Welcome to Practical AI, the podcast that makes artificial intelligence practical, productive, productive, and accessible to all. If you like this show, you will love The Change Log. It's news on Mondays, deep technical interviews on Wednesdays, and on Fridays, an awesome talk show for your weekend enjoyment. Find us by searching for The Change Log wherever you get your podcasts. Thanks to our partners at Fly.io. Launch your AI apps in five minutes or less. Learn how at Fly.io.

0:45Welcome to another episode of the Practical AI Podcast. This is Daniel Whitenack. I am CEO at Prediction Guard and I'm joined as always by my co-host Chris Benson who is a principal AI research engineer at Lockheed Martin. How are you doing Chris? Doing great. Happy New Year. This is our first show of 2025. Happy New Year. Yeah, this is the first one we're recording for the year. First time jumping back on the mics to talk about AI and definitely something that I think will be a theme in 2025, which will be, of course, multimodal AI in general, but I think something that a lot of people are wondering where it's going to go, I guess, in 2025, which is video generation.

1:37So we're very pleased to have Paras Jain with us today, who's CEO at Genmo. How are you doing? I'm doing great. Happy New Year, everyone. It's really wonderful to be here. Yeah, welcome. I know we've been trying to make this one happen for a little while, and I think the timing worked out well because, like I say, people are thinking a lot about video generation and how that will evolve in 2025. Maybe from someone that's working in this area and has been thinking about it deeply, maybe, you know, a lot of people, listeners have just started thinking about this topic recently, but you've been thinking about it deeply for some time.

2:19Could you give us a little bit of a sense of kind of what happened, what led up to video generation and kind of where it is in 2024. And then kind of as we're entering this new year, what's the current state of video generation, I guess more generally in terms of what people can access in actually released systems or released models? Yeah, absolutely. There's been a long path to kind of where we are today. I think, you know, given so much excitement in what you might call the left brain of AI, that is like language models, reasoning, your O series of models. You know, I think the right brain has kind of lagged progress for quite a while, right?

2:58Like people didn't really widely use kind of creative AI at a huge scale. And I think video is the ultimate creative modality here, right? If you think about it, so much of how we communicate as humans is through visual mediums and specifically just video through motion. And so I think it's incredibly exciting in video being this ultimate form of creative multimodal synthesis. It was always really exciting, but the technology really kind of was far behind, I think, what people really wanted from it. And so it's interesting. My co-founder worked on some of the earliest image generation models and then 3D generation.

3:30And then video was always this kind of big modality we wanted to target. What I think was really interesting in 2023 and 2024 was first the development of the image generation, which is kind of a precursor to video. But even then, the gap from image generation models to video generation models was always really big. Because if you think about it, an image might have thousands of pixels, or even a million pixels, but a video would have hundreds of millions or even billion pixels in just a short clip. And so there was a huge gap to cross. I mean, so compute has scaled a lot and that has enabled larger models.

4:03And so I think, you know, top of mind since we, you know, rescheduled this podcast, Sora came to market, right? And I think that was really exciting. That was a watershed moment for a lot of people to kind of see what was possible with video generation. And to me, I think this is a really early bellwether of like, you know, what is to come. I think we're still really early here. Yeah. And for those that don't know, Sora is from OpenAI, right? Yeah, correct. Yeah. And I mean, you talk about some of the challenges with video generation being a kind of different animal. I know that some people might, you know, if they've been longtime listeners of the show, we've had episodes talking about kind of stable diffusion and these sorts of models for image generation.

4:50What are the main, I guess, if you want to be a video generation model builder, what do you have to think of differently, both in terms of kind of the type of model that you would use and also kind of, you know, the process that you'd have to go through in terms of curating data and that sort of thing? Mm hmm. Yeah, I mean, I think first and foremost, video data is really data intensive, right? Like you just think about it compared to even images or text, like text is tiny images were more expensive, but video is like 100x more in terms of data volume in a short clip than you might have for images.

5:28And so when you think about training these models, that's really the most important challenge is how do you build architectures, and then systems that can scale to process large data sets. That was a big ball neck for the community. I mean, again, at Genmo, we've been innovating heavily to actually make that possible. But I think this was why progress took a little bit longer than, say, for images or language to come to market. For companies that are training this, though, like, again, they're having to curate massive scale data sets easily in the, you know, petabytes of data, essentially, just to pre-train these models.

5:58And that's really intensive. Many practitioners are beginning to fine-tune these models, too. But even that remains more challenging than your stable diffusion, for example. That's got to make it hard for new entrants to come into the field, just the sheer volume of what you have to get set up, you know, ahead of time to handle that is probably be, I would imagine beyond what most organizations are really able to do unless they're, you know, have specific expertise or experience in the area or something. Yeah, absolutely. I mean, it took us a long time to get ready to pre-train models. We'll talk about this more, but we open source to one of the state of VR video generation models.

6:33And part of the goal here was to kind of let other people have a chance, right? Let them pick up a model and begin to fine tune it. And they're kind of skipping past like a massive volume of technical infrastructure they otherwise would have needed to have built. And maybe talk a little bit about that data side, because I know that this is one of the things that's been, I mean, it has been a struggle on the tech side, but I think especially on the image and video side where there's a lot of questions about, hey, well, where can you actually source all of this video and imagery and, you know, what are the rights associated with that?

7:11But I also imagine there's definite curation that's needed in terms of like all these prompts that I've seen people do with like, oh, generate this, you know, it's shot with a Canon DSLR or whatever, like all of that sort of thing has to be curated on the prompt side as well. So yeah, could you talk a little bit about that data curation and what the source of that, kind of where you could even get videos and then the curation process? Yeah, I mean, I think pre-training in general, and that's true for image models, text models, audio models, and videos, they rely on like, you know, large volumes of internet scale data.

7:56But I think what's uniquely challenging with video is it's just, you know, it's easy to kind of get drowned out in the noise, right? And one angle here that I think is really interesting that we, for example, zeroed in on was how do we learn high quality motion with a video model? And it turns out the vast majority of video you find on the internet doesn't move. It's like a static object or it's someone talking. And if you think about that, that doesn't actually teach a generative model about the world. It doesn't teach it about physics. It doesn't teach it about how objects interact. And so it's not going to learn strong reasoning.

8:25And so the way we think about it, it's really the goal with the video model is to learn physics and realism and the laws that govern our world. And so you might think about inertia, mass, optics, you know, fluid dynamics, all these kind of base properties and how they all interact. That's really the goal with video generations to learn an engine that can simulate this because the output is a video and we can consume and it's creative and it's beautiful. But the hard step here is finding data that can really help you learn these base rules of kind of of the world. And this was one of the most fundamental gaps we had across.

8:58It's kind of trivial. I'm kind of curious as you know, as you're describing that I would imagine that some things that you're training for are harder for the model to learn than other areas. And that, I mean, if you think about just, you know, narrow it down to just, you know, animals and mammals and humans, and they move differently. And the, you know, the, the physiology and the anatomy is a bit different across those. And that all has to somehow be inferred by the model. If they're going to make a video that's realistic. What have you, in your experience, as you, you know, you talked about motion being so important stuff, what are some of the harder things it's been to get right over time, not just in where you're at today, but like for the industry, what is the industry and maybe early on, what have you struggled with?

9:41Yeah, I think it's a really, it's kind of funny that one of the test cases people use now to test different video generators is gymnastics. And I think the reason for this, there's hilarious videos online you'll see of of Sora or other video generators doing gymnastics and I think one of the answers is video generation models just can't do it now and it's really complex human motion it's really rare so you talk about data curation for example like there isn't that much complex motion we see people like doing twists and twirls and backflips and stuff in nature right and so it's kind of interesting is like that is that requires fundamental understanding of how human human kinematics behave for you to simulate that properly without it feeling disturbing and so this has been one of the challenges for people.

10:22I mean, for example, early on when we were training, I mean, we've gone through three fundamental pre-training foundational models in our history of our company. And what's interesting with Mochi, which is our latest model in the prior one, Replay, was walking was actually a really basic thing that was really hard to nail. It turned out most video generators early and kind of early to mid-2023, they would make humans kind of hover as if they're hovercrafts. Like their feet would not move, they would just kind of levitate off the ground and move. And so the models were not capable of synthesizing, forget gymnastics, just walking.

10:55And so that was one of the critical watershed moments we had to cross, for example, as a company. I would suggest that might've been some folks that had a little too much bourbon in their eggnog over the holidays right there, kind of that floating thing. I've definitely seen the like Jedi vibe, which is kind of cool in one respect, but not that awesome if you don't want it. One thing here is I think like we've invested heavily in evaluation infrastructure at Genmo. And as part of it's like, how do you benchmark these capabilities? Like one of the test cases we have is, you know, you might have a woman drinking a glass of water with ice.

11:29And you want to look that, hey, there's the ice move realistically. There's a water flow. But also like, you know, what's interesting is once in a blue moon, like the character will try to drink the water through the side of the glass, which is just not physically consistent. And you'll actually see this with some of our competitor models. It's something we had been trying to develop. And I think just that isolated test case alone communicates a lot about the video generation models capability to just understand the laws of reality, right? Like it's kind of, yeah, it's a Jedi mind trick. Like you just cannot, you should not be able to do that, right?

11:59How, like when you're using those test cases that you've developed, is that a lot of human review or how does that, like, how do you create kind of the tooling around that? because I know there's sort of like comparisons between this image and that image, right? Or this frame and that frame and you can compare closeness and all of that. But there could be a lot of sort of closeness in the overall image. But if the woman is drinking from the side of the glass, there's kind of a major failure moment, even though maybe everything around is really good. Yeah, I mean, look, there's a lack of external publicly available quantitative benchmarks.

12:44I think one of the ones that are publicly available are these leaderboards or artificial analysis has a video generation leaderboard. I mean, we are the number one open source model and kind of neck and neck with closed models there. And that's just human preferences. Hundreds of thousands of people look at two videos side by side and they say, this one's better or this one's better. And you kind of get like a chest style ELO rating. And I think this has been one of the best or better public benchmarks. You know, internally, one of the ways we think about this, though, is as we're measuring these capabilities, such as world understanding and physics, it's very hard for a human actually to rate by that.

13:14It turns out like when we as humans look at two videos side by side, and you're saying which one do you prefer, you often prefer the one that might have slightly higher resolution or more detail. But if you actually think about if I'm going to use this in actual production application, like film production or gaming or something else, like I probably actually care more about the motion. And so we actually have to override the human intuition, your first order intuition to say select for detail and use these test cases as sort of like functional testing of how we can measure these capabilities you know i in my career i started out actually in self-driving i worked at one of the early companies applying deep learning to self-driving perception and you know i took a lot i took a lot of inspiration to how we built functional safety testing for example for deep learning systems right and in that way you're going to enumerate these test cases and use cases and you can actually say yes or no do you kind of pass that that test case scenario, right?

14:02And so, you know, whether it's a human has to do that review, and we're starting to develop more automated metrics, I mean, it's just producing more structured forms of valuation, I think are really important, because otherwise, the world is just too intricate for us to test everything, right? So we have to kind of go use case by use case and just measure progress. And it turns out, as you scale the models and scale the data sets, we begin to see percentage completion rates improve. And this gives us a semi-quantitative benchmark of progress.

14:37Well, friends, AI is transforming how we do business, but we need AI solutions that are not only ambitious, but practical and adaptable too. That's where Domo's AI and data products platform comes into play. It's built for the challenges of today's AI landscape. With Domo, you and your team can channel AI and data into innovative uses that deliver measurable impact. While many companies focus on narrow applications or single model solutions, Domo's all-in-one platform is more robust with trustworthy AI results without having to overhaul your entire data infrastructure, secure AI agents that connect, prepare, and automate your workflows, helping you and your team to gain insights, receive alerts, and act with ease through guided apps tailored to your role and the flexibility to choose which AI models you want to use.

15:26So Domo goes beyond productivity. It's designed to transform your processes, helping you make smarter and faster decisions that drive real growth. And it's all powered by Domo's trust, flexibility, and years of expertise in data and AI innovation. And of course, the best companies rely on Domo to make smarter decisions. See how Domo can unlock your data's full potential. Learn more at ai.domo.com. That's ai.domo.com.

15:58so paras i'm wondering you mentioned this kind of history of pre-training at at ginmo and the most recent model which of course we want to talk about but i'm sure that that most recent model is informed by things that you tried in the past and kind of your history there so could you give a little bit of a snapshot of kind of the history of your team and how they how they approached this problem, how you all approach this problem and the kind of generations that you went through with that? Absolutely. So we're just about two years old at this point. We actually started working on the company Christmas 2022.

16:39So it was a holiday and there's a Jay and my I were both the co-founders of the company. First and foremost, we're brothers. I think that's really unique. That's awesome. So, you know, and we didn't really plan to start a company with brothers. I mean, it's a little weird. I mean, it's like, you know, normally you have sibling rivalries and things like that. I don't know. We didn't have much of that. But it turned out our skill sets were super complementary. Both of us were doing our PhDs in UC Berkeley. I was working on large scale distributed systems, you know, in the UC Berkeley AMP lab and RISE lab.

17:07And this is the same lab that created Apache Spark and, you know, Ray and the AnyScale project. And so really hardcore machine learning systems for scaling large language models. That was what my dissertation topic was on. And concurrent to that, Ajay was working on the foundations of modern image generation. So, you know, he had joined Berkeley to work on early image generation models. This is kind of like in your Gann era. And I think for him, one deeply unsatisfying thing was a generative adversarial network was like a mirage. It wasn't actually like a grounded loss objective that was learning real motion or dynamics.

17:42It kind of was like this game, but you got image generation and side artifacts. So I think his story was really interesting in that he ended up kind of writing his paper, DDPM, or the Denoising Diffusion Propolistic Models paper, which is one of the foundations for how we think about image generation with diffusion today. It's one of the most highly cited papers in this area. And that came from, I think, early inclination that how do we build video image models that understand physics and realism instead of just kind of like artificially playing this, you know, game that results in image generation, grounding it in real generative pre-training.

18:14And so that's some of the early academic history of the company. But starting the company, we decided to do video because it seemed impossible back in 2022. It was just completely outside the frontier. And we said, fundamentally, we need a new architecture to solve this. And so let us discover both from a systems perspective and distributed systems perspective, but also machine learning perspective, what the right approach to do that is. I mean, so yeah, it's been about two years since founding. We've gone through three large pre-training runs. And in each time we learn something new about the world and integrate that into our approach and our framework and architecture for how we train these models.

18:47I think the single underpinning thing, though, is motion. We always joke like Genmo doesn't really have an explanation, but we kind of retroactively apply this idea of generative motion. Right. Genmo is like we care so much about motion and video that that that's really a core element of our founding history and our framework for how we approach video generation. I'm wondering, you got me, I had a question for a moment or two as you were talking through it. You kind of talked about that evolution and, you know, kind of starting with GANs, the generative adversarial networks, and finding your way across kind of the architectural progression that you guys have found.

19:22Could you talk a little bit about that in terms of like, you know, if you were coming into it during, you know, the age of GANs right there, and that was the thing. But I'm kind of curious, at high level, what was the problem with that? Why did that not work for you? What did you look to next? Could you kind of give us kind of a highlight, skip over the top of a couple of different major architectural twists and turns to give us a sense of what your journey might have been like? Yeah. So I think the earliest form of image generation models that I think started to work well were autoregressive image generation models.

19:56So this is very similar to a large language model. you kind of take a you know you might take an image and you make it a single vector a line so if it's like you know 28 by 28 image now you have 784 pixels in a straight line and you just go one by one by one and decode the next one so that was the earliest form of image generation your models like pixel rnn or pixel cnn or image gpt which were from open ai which were the earliest works here that worked well but the problem is like images have millions of pixels this would never scale to produce high resolution images. I think what's interesting from when Ajay was working on this like early on in 2018-2019 was I think I remember he trained in autogestive image generator modeling.

20:37The first models he trained were trained on like L-Sun which is a data set of bedrooms basically. But what was so interesting is that it would be like a little 5x5 or 10x10 pixel region it would start to put artwork on the background of people's bedrooms. Why? Because that's just like what nature looks like. That's what real estate listings look like. But it was the first indication of AI generated art in some sense with an early image generation model. The problem is this would not scale, right? Because you're kind of going pixel by pixel by pixel. So it would take hours to make a small image.

21:06I mean, so GANs were the next kind of major approach. I think that really worked well for this. And GANs are trained again with this like generative adversarial objective. It's kind of this like dueling game between a generator and a discriminator, but they were really hard to train. It turned out they were getting these bad, you know, states where, you know, you might get like mode collapse, for example, was one of the biggest issues. It would mean you could produce images of like a single domain, but you couldn't like produce everything in the world with a GAN. So you could get a really good model for making faces or a really good model for making, you know, bedroom pictures or making a really good model of like tigers.

21:39But it was really hard to say train a model on all of ImageNet, meaning cover thousands or thousands of different categories, right? And so diffusion models was a really exciting approach that Jay began to work on because it had the potential to provide that kind of mode coverage. You could learn diverse representation of the world that generalized beyond just a single domain like faces or animals to everything. And so that was what kind of resulted in DDPM. And I think since then, I mean, you've had latent diffusion, the stable diffusion approach, and then video generation is, I think, the next major evolution.

22:11But the learning paradigm has mostly remained similar to this, like learning to this, like, you know, diffusion setup or kind of iterative denoising, right? This is the formulation of this diffusion problem. But like, it's remarkable to see how far that has scaled, like literally 10 ,000 X in pixel scale from the earliest diffusion models to kind of where we are with video generation. I guess on that front, how did you decide, I guess, because I know that part of what you've done, and I'm assuming what the intention was with the models that you've created, is to open source them in one way or another.

22:52and as you mentioned earlier, release things into a community where people could experiment and try things and fine tune. How did you think about kind of size of the model and, and that sort of, was that kind of purely driven by what was needed to produce kind of a certain size of video or certain resolution, um, a certain kind of performance metric that you were after? How did you make some of those, I guess, trade-off decisions, maybe also the compute that you had access to? Yeah, I mean, first of all, pre-training is incredibly GPU intensive. I mean, we have access to more than 1 ,000 of H100 grade GPUs.

23:33And so, I mean, that is incredibly GPU intensive, but I think it's also a question of how you utilize that hardware effectively. And one of the critical challenges with video is they have really long sequence lengths. Like training a video generation model is equivalent to kind of training a million token length kind of context window for a new language model. And so this introduces a huge set of challenges that are kind of orthogonal to kind of parameter scaling as you might see typically with large language models. What I think is interesting is though certain capabilities only emerge at certain parameter scales.

24:03So like I talked about walking, like it's very difficult to get walking to work with like a, you know, a one or two billion parameter model or something smaller than that. It just like, you won't learn that capability. So you do need a certain amount of scale for it to work. But at the same time, you're not seeing models that are like 100 billion or trillion parameter scale, as you see kind of with the frontier grade language model. So we open source Mochi1. It's a 10 or 11 billion parameter scale model. So it's a lot bigger than your conventional grade, older grade of video generation models.