The Next Frontier of AI Video Is Control

14:05Jennifer Li:This happens at Fall once in every couple of months where like the whole company gets hold of something and the creativity just explodes and everyone is just working on a new little app or a different optimization, Laura, whatever it might. Like the whole company gathered around this model and like some front-end engineers started working on like interesting applications. We can talk about our like world model accelerator team, which is brand new. They started working on the live experience. Yeah, the RTC. The web RTC live experience. So like there were five, six different parallel little projects within the company.

14:53Jennifer Li:and like I think we broke a record on Slack that day how many messages were sent in the company because like and like we have a distributed team. We have people all around the world like mostly in San Francisco but it's like incredible when you see like the 24-hour development like when people like work 16, 17 hours and then someone else wakes up and picks up that and that went on for like three, four days and that's when we released all these projects.

15:26Gorkem Yurtseven:Take me into that. It's so interesting because you imagine a model or product launch being planned out, having all these eval vendors being ready, lined up and ship something out and then you let the world or the external users take it and then experiment and build experiences put online. Yeah, it seems like people internally who are very creative just took this job, everything they were doing, like launched experience that got really popular on Twitter. Do you want to tell us about that one?

15:58Jennifer Li:Yeah, of course. One of our engineers, Rehan, just by himself completely started streaming a live stream of continuous generations of H3 Macs from his laptop. Like he was... His computer. His computer, exactly. He was doing some like prompt tricks, trying to keep like a coherent story. And then like he started live streaming that on Twitch. In parallel, Levels.io, a famous Twitter influencer at this point, had a similar idea. And he reached out to us that he has a website ready already. He wants to like host the streaming himself and have a website that does infinite streaming. Internally also, we had another team who was working on a continuous version of HTT Max.

16:47Jennifer Li:So HTT Max is like the rehoused version and levels.io version were independent clips. It's still very fast, but the clip starts, it ends, and then you take the last frame of the clip, try to put it in the next one and try to create a continuous. You need to put some work into like the last frame. There's no memory, like the second clip doesn't really remember anything from the first clip other than the last frame. But internally, the ML team was working on a version where the transition is more seamless. There's like two minutes of memory. So like you're in a scene and when you direct the model or someone else enters the room, it actually like everyone looks at that person entering and the scene is continuous.

17:33Jennifer Li:So internally, we were working on that. And then another team was working on an experience we called File Live for the continuous version. So we had three parallel efforts going on that were all independently going viral on Twitter.

17:48Gorkem Yurtseven:And these were all like spontaneous, like you didn't plan for it at all.

17:52Jennifer Li:You didn't plan for any of them. Yes, exactly.

17:54Gorkem Yurtseven:And they just became products and experiences in the following days.

17:59Jennifer Li:But Tom, let's talk about how we made the model more continuous. That was very surprising to me because I've never seen that actually work on a video model before. Sure.

18:11Batuhan Taskaya:So going back, we have been like very, very focused towards role models and essentially like action controlled or like, you know, action driven real time continuous streams of video. And the problem till like, you know, something like HTT Max was quality was not good enough at all. It was just like, you know, it degraded a lot. It didn't remember the past before, but we built the infrastructure. We built the infrastructure that we can go stream video, have people control it in real time, being able to multiplex it to multiple people, very low latency. And at the same time, our ML team was essentially trying to take every single video model and try to apply this set of optimizations and tricks to, okay, how can we make this generate instead of a five-second video, 15-second video, 30-second video.

18:55Batuhan Taskaya:But you were always below the real-time factor where you were always like, you know, you never could generate like five seconds under five seconds. Once HTT Max unlocked it, the ML team was like, this is insane. Which are like separate teams internally. We have a research team, we have an inference team, we have an ML team. They're like, they saw this and like, this is insane. We can apply all these like set of learnings that we had in previous models where we attempted to do this, where instead of trying to generate a five second chunk, let's try to generate, you know, like a 50, like 10 second video.

19:26Batuhan Taskaya:And then the five seconds from previous one is still attended. we still remember it and like as the video goes up we can like extend that memory up to two minutes and you need to do extremely clever optimizations because attending to a two minute video is just extremely extremely compute intensive and just like it goes up exponentially because uh from like a compute standpoint so like we we did like lots of optimizations there but at the end of the state we were able to okay we can remember back to two minutes which is like generally good enough from a memory perspective and then obviously with like prompt tracks you can still like continuously see, remember more finer grain details above the two minute mark.

20:04Batuhan Taskaya:And you can essentially stream infinitely. We captured it an hour from that perspective. And then that team just like released that model under H3 Max director, which is public for people to use. And I think it's the only model that can generate like, you know, up to 60 minutes, continuous videos that is action control. You can like, you know, start with a prompt, say like there's like an office setting and someone is like, you know, working. and then like 30 seconds later, it just imagines by itself. 30 seconds later, you can like say, a woman walks in through the door. Like it can take the prompt and reflect it immediately, which is the most fun part.

20:36Jennifer Li:And the office is still the same office. The camera can pan back to the original person and the original person is still there in the same state. Yeah.

20:45Batuhan Taskaya:So, you know, we released that and it got like, we did this like fall live website to just like demonstrate it because it's like, people need to see how cool this is, right? This is a new technology. I don't think people are like really aware. And it got also like very viral immediately because we also let people vote on what the next section is. It was like, you know, like a form of - Crowd source. Crowd source. Like the chat was controlling whatever was happening, which is fun. But obviously, you know, we limited on like the options and then they could pick, oh, like a banana enters the office instead of a moon.

21:16Batuhan Taskaya:And it's like more fun. And like people start like, you know, having these. And we start adding more channels and like every channel had a concept. There's like a channel where it's like full chaos. There's a channel where it's like cartoons from like 80s. And like the model is like extremely capable and it just like remembers like so many different concepts and there's like, it has like a big, big memory from like a styles and like, you know, concepts perspective. So it just became like a very fun experience underneath.

21:41Gorkem Yurtseven:Again, like there's so many really incredible experiences coming out of this. Like H3 Max director was just another huge surprise to me. It's like, I found it interesting in the, in the gem media market that you, it's not like, you know, like language model, you have like this linear graph of like just continuously compounding on like, you know, intelligence capability and so on. Like feels like in the field you're operating in, it's always like a few months of like sort of quiet time. But like a lot of things are bubbling. But like in a very short period of time, like everything bursts, like all these things in combination come together of like the base model being good enough.

22:24Gorkem Yurtseven:like you can get the latency down to the point where you can like references yeah yeah um like get the real-time experience but also like apply controllability on top of that real-time experience like this just opens so many you know opportunities of like live experiences where like end user can control what's happening on the screen which is incredible like uh we have imagined a lot of these experiences but never been able to like really play around with it maybe just like tell us more about what you're seeing from the market of like, how are people using like the director capability? Like what are you seeing creators are creating that you haven't seen before?

23:05Gorkem Yurtseven:And what do you think that unlocks as far as, you know, what people can do with this medium?

23:10Jennifer Li:Yeah, it's been like almost three weeks since we released H3 Max and already it is the most popular video model on the platform, on the file platform by like double almost, like a little more than double in terms of like volume. So in a lot of other platforms, it's also becoming the default model that people interact with because it's so fast, so cheap. It just makes sense. If you come to a platform, this is the experience that you want to see. So in terms of like popularity and volume, it's taking over at least from our vantage point. And for Max Director, again, there has been, I don't know, tens of different versions of these live streams.

23:59Jennifer Li:Some of them are still going on and becoming more and more popular. We are trying to work with some AI IP holders, people who have like AI shows on Instagram and TikTok and train a Laura on their style and do a live version of their show. So we have a couple lined up already. So that's going to be very exciting. And like the way people, like if you talk to a creative technologist, prompting with voice has already become something that like they use all the time, like using Whisperflow or the chat GPT voice mode. And now like you can keep talking to the model and it's like almost as if it's a real director in a real movie set directing like the camera, directing people where to go.

24:53Jennifer Li:You can do that. And like our creative engineers started using these models like that. So we'll see like a lot of interesting experiences are built as we speak.

25:03Gorkem Yurtseven:Very interesting. As in like the video is playing.

25:05Jennifer Li:The video is playing, you are like talking to the video and what's being displayed changes accordingly.

25:12Gorkem Yurtseven:That's incredible. And talking about like how the memory piece holds now, like, again, this may be a technical detail, like the capability of remembering what happened in the last scene or in the last couple minutes of scene. Like, are you remembering that through like the frames, the images, or is it like through text?

25:33Batuhan Taskaya:it essentially no it's essentially like it remembers the raw video obviously very very compressed because you can't attend the fall video but it essentially knows like most of the details happened in the past two minutes from its own generations and above the two minute mark it has like think of it as like it has an evolving system prompt on top of the two minute mark from two to 60 minutes where it knows like the overall structure overall detail so it remembers like the last few scenes if you think a scene is like 15-30 seconds then it remembers like the last four to eight scenes. And then on top of that, there's like a continuously evolving, gradually evolving system prompt that like keeps remembering the overall coherence of the world.

26:16Jennifer Li:Everyone's waiting for a large consumer moment in AI. Now it's like good enough and cheap enough that like a truly novel social AI experience can be built on top of it.

26:31Gorkem Yurtseven:Maybe let's talk more about the economic side of this Like what is the I guess one just like talking about serving cost For like same minutes of video With H3 Max And how has it changed your thinking around Like your footprint of like inventory of chips Like how do you want to have like different steps of experiences Serving to the end user

26:59Jennifer Li:But I mentioned this a little bit like everyone talks about how complex the next generation LLMs are, but video models are actually very complex as well because the pipeline has different components and sometimes they require different hardware configuration for efficiency, things like that. So if you were to do this even more efficient, let's call it maybe even cheaper, we would probably run different parts of the pipeline in different types of hardware. another interesting thing would be to run it on consumer hardware for people to run it in their own machines at home like optimizations don't translate 100 % but translate somewhat close to that and then we can do extra work to translate more of it so doing these optimizations in different types of hardware and combining the pipeline in a way that it's even more efficient.

27:59Jennifer Li:I think that's what we are going to do in the next coming weeks.

28:04Gorkem Yurtseven:Amazing. So you will have people like Rohan that can stream partially of the experience from this computer, but also having like the director and the control plane more living on the...

28:16Jennifer Li:Exactly, yeah.

28:17Gorkem Yurtseven:On the cloud. Makes sense. So we talk about all the consumer experiences this model could unlock. And it seems like Botwan is happy with all the efficiency, like, squeeze out of the GPUs. Now we're talking more about how do we, like, improve quality and controllability of these models so that, like, you know, the high end of the market, the Hollywood creators, directors, can take this to the next level. I saw some demos. Coincidentally, like, you know, this model came out the same week prior to Astra. People were combining the Blender experience with H3 Max from FAL. like talk about how it's going to impact the Hollywood world.

29:00Jennifer Li:Using Blender with one of these AI models together is an extremely popular workflow for professional work. Basically, you render a low resolution of your scene, what you want to do using Blender, previous like non-AI technology. And then once you add that video as a reference to an AI model, you basically get close to 100 % controllability. And this is an incredibly popular workflow for VFX artists, people who are doing this professionally, because they want to get exactly what they put into the model. And as you mentioned, a week after we launched H3 Max, people starting creating generating these beautiful scenes using an llm model gpt astra in in blender and all of a sudden it unlocked a whole new pipeline using an llm to create a blender scene and then passing that to the h3 max model or or any video model but it works very well with h3 max because it's extremely fast and you can like try many things all at once in parallel and that unlocked the whole new workflow for Hollywood and professional people and it gets you to like close to 100 % controllability.

30:23Batuhan Taskaya:As I said, we have been very, very focused towards speed, performance, quality and now we have a really good base model. I think the next month or two is going to be fully focused on, okay, how much controllability we can add to these models so that professionals at studios, professionals who want to actually produce content that fits their use cases perfectly can leverage these models. The team has been working on an amazing lip synchronization model where you can just supply the audio, you can supply a video or an image reference, and then it can synchronize the lips perfectly. Same with motion controls.

31:03Batuhan Taskaya:You can just take a motion of someone dancing and apply it to your AI-generated character, and it fits perfectly. And this is like, you can like get these results with like basic prompting and you're going to get like 80%, 90 % reliability. What we are targeting is like 99.9 % reliability in the outputs so that you can actually trust the model did every single aspect of this generation perfectly. And that's like what we've been pushing. One big launch that we had last week was the camera controls, which is essentially you can direct where the camera is going within the video perfectly to the degree.

31:40Gorkem Yurtseven:And this is by like describing in the prompt or like generating the...

31:44Batuhan Taskaya:You just essentially like underneath you give a JSON of like, I want camera at like 0, 0, 0 at T0. I want camera at like 90 degrees angle at T1. Like you essentially supply a structured description of where your camera needs to be at any point in time. And then the model is like perfectly conditioned to regard it as like the only source of truth and it doesn't like hallucinate like where the camera should go. And it's just like, you can essentially reconstruct 3D scenes from a single input, like because the model itself is a very good video model, but at the same time, you know, it's like perfectly adhered to the camera itself.

32:21Gorkem Yurtseven:And this is because the base model itself already has the understanding of the camera angle that you can...

32:27Batuhan Taskaya:It doesn't respect it. It just under, like, you need to tune them all. You need to tune the model to a significant degree. And this is like what enables like at large scale, post-training infrastructure. We now have the infrastructure to take H3 Max, add any capability to it. Same applies for any new model. There's a new video model. We essentially spend most of the time building it as an infrastructure than just one-off training runs so that we can build services around this for not just open-source models, but for frontier close-source models as well. Because we see in the market this is the biggest gap, is just how controllable these models are.

33:03Batuhan Taskaya:first we start with text to video where you put a prompt you get a video back, it was good but you never could describe the perfect character for you, and we had image to video where you used an image editing model and then generated the first scene and then the model was obviously much more fitting, but you still couldn't say, oh I want this new character appear at second tree, you need to put it to the first frame, or you can prompt it, but it was never perfect, and then we added reference to video where you can provide an initial starting frame and you can also provide, I want these characters with these voices.

33:34Batuhan Taskaya:Like, you know, that's also like a big unlock where you can essentially say, this is the voice for this character. And now like, you know, we are adding, oh, within the scene, I want camera to look at this degree at like T0, I want camera to look at this degree at like T3. And then we are adding lighting controls where you essentially say where the light is coming from. These are all compounding on top of each other. Like we just have the unified infrastructure to just apply this to any model at this point.

33:57Jennifer Li:That's incredible. Hollywood is our fastest growing segment. And there's a lot of noise about how AI might disrupt Hollywood, but Hollywood usage was non-existent a year ago. And in the past year, it grew and now it's the fastest growing segment. like Amazon MGM Studios in their conference, they released their NARA tool. It's mostly backed by file infrastructure behind the scenes. And we are seeing incredible, incredible pull coming from Hollywood. And exactly what they need, these like small point solutions rather than generating everything from scratch. They want to be able to extend the video a little bit.

34:40Jennifer Li:They want to be able to change the camera controls. They want to change the lighting and someone has to build these solutions for them. What Hollywood needs and what the creators actually need and what the research labs are working on, there's a little bit of a disconnect there. And we believe we can come in and do these little post-training projects to close that gap. Because we work with all the Hollywood studios and we hear from them what they need. And these are exactly the things they need, these small point solutions that actually make them more efficient, push out more video, and AI can actually close that gap very nicely.

35:22Gorkem Yurtseven:Maybe say in a little bit different way. Like we have been staring at this problem for the last three years as well. Like we see like companies trying to like build a, you know, movie director, like video model, like by either pre-trained or post-trained on the video side. But what I'm hearing is like different people expressing the way they want the output to come out very differently. Consumers talk about it and then like write the prompt and generate the results. very differently from a Hollywood director, which is obvious, right? Like professionals want to talk about like, you know, these camera angles.

35:58Gorkem Yurtseven:They want to talk about the lighting. Like you sort of have built a library or like a collection of post-training, like I would call it data and toolkits that can apply these, any model that you can like grab the weights on so that they are adapted to like a different audience where they can express their creativity in a bit different fashion to control the model when it unlocks a lot of capability underneath.

36:29Jennifer Li:And half the problem was capabilities of these models. We are solving that. The other half of the problem was legal and data residency, things like that. We made a ton of progress there as well. We now have a system of people can apply with their own IP and we unlock their own IP in the models, we are going to grow that and that's going to be a very powerful thing we do with Hollywood Studios. Also, we now have Seadance US hosted as well. We already had previously other Chinese models. Seadance was the missing part. Every Hollywood studio wanted us to have it US hosted. Now that's available. So there are no obstacles in front of these Hollywood studios now, everything is ready.

37:19Jennifer Li:And we believe they are going to 10x, 100x their AI usage in the coming months.

37:24Gorkem Yurtseven:It's such an exciting world for movie lovers, consumers, people who consume a lot of video and creative content.

37:33Jennifer Li:And we have our conference, Generative Media Conference next week. This is our second time we are doing it. Last year, it was mostly consumer AI. There were like maybe a couple Hollywood executives here and there just curious about it. And now it's dominated by studios. New AI studios who are like offshoots of the bigger studios trying to do like only AI shows. But also like the biggest of the Hollywood studios are also there because now they have big plans integrating AI into their workflows, into their existing systems. so you can see the change in the attendance of the conference as well.

38:14Gorkem Yurtseven:That's awesome. Well, for the audience, check out the content coming out of the Gen Media conference. It's going to be very, very exciting. And thank you so much for Kim and Batuan coming on to our show. It's a super exciting time for Gen Media.

38:27Jennifer Li:Thank you. Thanks for listening to this episode of the A16Z podcast. If you liked this episode, be sure to like, comment, subscribe, leave us a rating or review, and share it with your friends and family. For more episodes, go to YouTube, Apple Podcasts, and Spotify. Follow us on X at A16Z and subscribe to our Substack at a16z.substack.com. Thanks again for listening, and I'll see you in the next episode. As a reminder, the content here is for informational purposes only. It should not be taken as legal, business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any A16Z fund.

39:09Jennifer Li:Please note that A16Z and its affiliates may also maintain investments in the companies discussed in this podcast. For more details, including a link to our investments, please see A16Z.com forward slash disclosures.