The Rise of Generative Media: fal's Bet on Video, Infrastructure, and Speed

0:00We recently had our first Generative Media Conference, and Jeffrey Katzenberg, former CEO of DreamWorks was there and he made a comparison. he said this is exactly playing out how animation when it first came out people people revolted against it it was all hand drawn before that and computer graphics it was new and there was a lot of rebellion against computer driven animation and something very similar is is happening with ai right now but there's there's no way of stopping technology it's just going to happen you're either going to be part of it or not

0:53In this episode, we sit down with a team from Fall, the developer platform and infrastructure powering generative video at scale. Fall is a place that developers can go to access more than 600 generative media models simultaneously, from OpenAI, Sora, and Google Veo to open-weight models like Kling. We'll discuss why video models present fundamentally different optimization challenges than LLMs, why the open-source ecosystem for video has a thriving long tail in ways that text models never did, and why the top video models have a half-life of just 30 days. The team also shares insights from the demand side of the video model equation.

1:27We discuss what's happening in the app layer, from AI Native Studios to personalized education, what's happening in Hollywood, and more. Enjoy the show. Borkai, Gorkam, Batuan, thank you so much for joining us today. I want to start with the problem space that you decided to tackle. So FAL is a developer API and platform for generative video and image models. Video is massive, obviously. It is more than 80 % of the internet's bandwidth, and it follows that generative video is going to be similarly massive. But there's not that many companies that are focused on this problem. Why do you think that is?

2:01Yeah, in a way, generative image and then video was an overlooked market in this current phase of AI. In my opinion, for two reasons. Number one, there wasn't a very clear industry use case that people were going after. There wasn't wipe coding that automates software engineering or there wasn't search, which seems LLM market is going after or customer support, anything like that. also number number two the investment on the research side wasn't wasn't as big three years ago and then that ramped up a little bit slower than lms but still considerably since then and now the the models are much more capable much more useful and real industry use cases compared to what it was three years ago it felt like a toy use case this was just going to be for fun on the side and it's going to be a small market at the end.

3:03And now we can see that it's going to be a massive market with very unique use cases and customers compared to the LLM market. Like if you actually go back to like, as we were experiencing it, I think that was an interesting time. We were working on some like Python compute infrastructure and then these models like DALI 2 had just come out. And then soon after that, ChatGPT had come out And then Lama had come out and we were just like, we were initially, we didn't know that, you know, image and video market was going to get that big. We were actually just curious about like running image models much faster.

3:46That was like our initial entry point. And then we saw like the initial growth. We had a few customers and they were growing really fast. We were like, what the heck is going on? And then, you know, a few customers later, we actually thought, hey, we should double down here. And around that time, also, the other thing that was happening was people were over-indexed on language models. This, like, story of AGI was being told. And, you know, that attracted all the dollars. That attracted all the talent. So everyone was just, like, working on that where we thought, like, we had something niche, like, growing fast.

4:22You know, don't tell anyone. And then we just like started focusing on that. And soon after, like as we got more familiar with the models, we thought the... I remember, I think we changed our website copy to say generative media, like generative media platform. And then it was only like two or three months after that, Sora was announced. So we were definitely like ahead, but we really saw the whole like future kind of coming with like better image models, video models, et cetera. So, yeah, we made this early bet. I mean, you guys have a front row seat to the sorts of new experiences people are building.

4:59I think the market's only going to expand from the media market that we know today. Yeah, absolutely. I think like Alkwara Karpati tweet, you know, no good podcast without it. He did one like recently where he was talking about like why he's excited about the, you know, media models. And one of the things he said was that, like he also mentioned that, right? people are visual and we're gonna we have more so much more video than than text like wall of text and he was saying like um he was making a point around like education and a lot of the like content you consume just to like like learn things i think right now the model quality is just like like relatively it's just so much like it's so much worse than what it can be where like you could actually like have you know i do a lot of learning on chat gpt but it's like through text but if it actually rendered a video where it could compress like a concept right instead of you know 10 000 characters if it could do it in in like 15 seconds uh it'd be so much better i think like there's there's sort of like uh the quality bar where if like it's it's gonna go up and and if you have once we have that we're gonna have even more more penetration so it's a it's really a function of the quality right now and we're just like in the very early beginnings totally education market almost untouched right now with video generation and there's so much potential there and it's just waiting the quality the predictability to get there and i think it's gonna have a lot of potential totally i mean you guys sent me that generative video bible app i think it's a much better way to learn some of the lessons from the bible and it's it's you know if you're capturing consumers attention right where they are you know i think it's i agree with you We're just at the beginning.

6:47So Fall is an infrastructure company, and so we're going to structure today's interview. I love infrastructure companies in terms of the technical layer cake. So we're going to start from the core inference engine, compilers, kernels that you've built. We're going to go up to the model layer and then the workflows and then end with some observations on the markets and what people are building. Sound good? Let's do it. Sounds exciting. Okay, let's do it. The inference engine. Batuan, how old are you? 22. You're 22 years old. Okay, say around your background, I think it's super badass and makes complete sense why this company is so hardcore.

7:21I started working on compilers when I was 14. So in a way, I have a lot of experience on that front. You know, it's not just that. But I started working on open source projects. So my first contributions were around tooling around the Python language. And then I started to slowly contribute back to the Python language, core compiler, core parser, and the core interpreter itself. And became one of the core maintainers of it. uh i think at the time i was like the youngest core maintainer of the language and this kind of gave me like a unique appreciation of compilers and how flexible they are so when we first started like working on serving these image models at fall the the main idea was okay there is like these like three different image models three different architectures but this is surely gonna explode there's upscalers there's like you know there's gonna be video models we were predicting that and we didn't want to go optimize a single model, put our eggs into a single basket and then became invalidated when the next model comes.

8:19So we started building this inference engine, which is a tracing compiler that traces the execution and essentially tries to find common patterns that are fitting within the templated kernels that we do. So our bread and butter is like spending, we have a 10 % performance team that's spending all their efforts into writing kernels that are like 95 % there, but like generalized with templates. So we trace the execution of a model and find common patterns that could replace these templated semi-generic kernels to specialized kernels at runtime and optimize the performance of these models. And we found this technique to yield pretty much superior results from anything that's out there in the market.

8:56And this led us to claim number one spot on performance on all the benchmarks. And another big thing about this is we specialize in doing this sort of kernel-level mathematically correct sound abstractions that let us maintain the same quality of these models, which is a very high bar when you're in this media industry and when you really care about the output that you're getting. What's different between optimizing a diffusion model versus an other regressive LLM? In autoregressive LLMs, your bottleneck is how fast you can move all those giant weights from memory to SRAM because you have a$600 billion parameter model and you're trying to predict the next token.

9:34You're doing the attention for all the tokens, like a couple of tokens before that. in diffusion models you're trying to denoise like thousands tens of thousands of tokens for a video at the same time doing attention of it so you're essentially saturating all the compute bandwidth of these GPUs you're not necessarily bound on memory bandwidth but like the computational operations that you do are like fully saturated so you're trying to find better ways to execute around the GPU this could be like writing more efficient kernels or this could be overlapping you know softmax with gems that you do like it's essentially like you're trying to use all of the power of the GPU, leverage it in a way that like, you know, that gets you all the capabilities.

10:12So it's a different binding constraint. It's on the compute versus the memory. And what's the intuition for why LLMs are relatively memory constrained and why video models are by comparison relatively compute constrained, but not as large in terms of just sheer number of parameters? I think it's scaling issue, right? Like in terms of like, if you scale these video models, 600 billion parameters with the same dense architecture, you're going to have to do attention with all those full like 100 like let's say a single video is 100 000 tokens and you do this attention step or like you do this like denoising step 50 times and every 50 times you do like attention over all these like 100 000 tokens it's insanely insanely expensive so i think the constraint there is like just how fast you can do the inference and the same applies with lms at like larger batch sizes but like at like the traffic patterns that people do like you know the batch size are not that much and you're mainly constrained by memory bandwidth so people do optimizations like speculative decoding and other factors to like reduce that overall yeah what exactly goes into being at the top of the leaderboard in terms of performance because i would imagine there's other teams that are also very smart people and you know this is this is my olympics and so what exactly goes into i imagine people have like very similar ideas on the techniques and different optimizations they can do so i don't think anyone cares about it as much as us we are literally obsessed with jarrington media we're literally obsessed with these models we have a team that's like just focusing on this like so far like it seems like and from nvidia to other inference players everyone is like super obsessed with language models everyone's trying to get like one more tokens per second on like deep seek benchmarks whatever and like we're like on a on a different lane we have like competitors but like no one close to us because i think we assembled one of the best teams we found out like the best way to optimize these general models and we just focus on this this is like a purely focused thing right like at the end of the day you're constrained by the hardware.

11:56There's nothing unique about it. But like, we're just like three months ahead, six months ahead. Like when we benchmark Torch, like the latest version of Torch against like, you know, our inference engine from a year ago, we're clearly underperforming because like Torch caught up. The same thing is going to happen with other players. You're going to always like, the lead that you can maintain is three months, six months ahead at most. The thing that matters is just focus. If you focus on it, if you purely put all your energy into it, I think that's like there's, it's very hard to get outcompeted by others.

12:25Because models are slightly changing each month, each release. So it's still the same general architecture, but there are slight differences where we can go in and optimize where that's different. And no one else is paying that much attention to it. Also, hardware is changing as well. So we were able to adapt to B200 earlier than anyone else. And we were able to run video models much faster basically throughout the year because of that obsession with running video models with the latest hardware. Yeah, got it. What are the hardest technical problems that you think you're solving? So one thing people don't appreciate it as much is we are running 600 different models at the same time.

13:06We have to be running them. We have to be so good at running them that we should be running a single one of them better than as if someone else is running a single model. because when a foundational lab is running models, maybe they have a single version of the model, maybe they have a couple other different versions, and that's what all they care about. We have to be better than them at running those models, and we have to be doing all 600 at the same time. So on top of the inference optimizations that happens in the GPU, a lot of optimizations on the infrastructure level needs to happen. We need to manage the GPU cluster in a way that's efficient to load and load these models at the right times.

13:45We need to route traffic to the right GPUs who have the warm cache of these models. We need to be smart about choosing the right kind of machines, which kind of chips are running what kind of models. And the customer traffic is changing all the time and we need to adapt towards that. So on top of the inference engine, the overall infrastructure is also really, really hard beast to manage. And so far, we've done an incredible job at that. Would you add anything to the... I think that's a pretty fair explanation of what we do. I call this distributed supercomputing. I don't know why people don't like that name, but...

14:23I like it. But the idea is we are at 28, 30... This was a month ago. Now we are probably at 35 different data centers. And you have these heterogeneous groups of compute that split across with their own different specs, different networking, whatever. And you're trying to schedule workloads as if it's like a homogeneous cluster that you got from hyperscaler. It doesn't work like that. So we built, like we spent the last three years building that tractions over it from our own orchestrator to building our own CDN. Like we go back to like, you know, fundamentals of web and we built our own CDN service, deploying racks to call us, like, you know, just like routing traffic.

15:01So we built all these technologies to essentially make sure that we can tap into capacity wherever it is and schedule our workloads, which is like very different than like a traditional enterprise LLM usage pattern, you know, like the use case that we have are like so much more spread across so much more you know like uh more consumer facing and like when you consider that like there is so much investment going into making sure we can tap into this like scarce capacity of gpus yeah you mentioned hyperscalers and you know i hear distributed computes and i hear managing giant clusters and i naturally think that's somewhere where hyperscalers should have the incumbent advantage um why do you think that you've been able to so far execute them on the core engine?

15:41There's two things about the core engine, right? There's the inference part where none of the hyperscalers have any expertise. This is a net new field. I think this has been only happening for the past three years, inference optimization. So it's like a brand new lane that we have been outcompeting anyone in our field. I think that's like a pretty much answer of its own. And the second one is infrastructure. I think right now hyperscalers are very busy with their traditional pattern of, oh, we have this data center capacity, we'll just deploy GPUs and we don't care about trust. This has been changing recently.

16:11Even Microsoft is going buying from NeoClouds. There's an interesting pattern happening because the GPUs and the demand and the growth of GPUs doesn't fit to the patterns of these hyperscalers, the growth patterns they expect. So I think at this age, not even hyperscalers have that big of an advantage of scale because they're going buying GPUs from NeoClouds. The tables have turned a bit. Yeah, it almost helps to also be like slightly earlier, like in the company journey, right? Like if you're a public company, you also have to kind of abide by like what the market's expecting of you. So like the other thing is that there's a huge price discrepancy with hyperscalers and neoclouds, right?

16:54So like it's maybe sometimes 2x, 3x more expensive to use things, you know, through hyperscalers. What's driving that? Well, I think one is like market pressure, right? And also like there's added kind of operational expenses that hyperscalers have for like having, you know, better, they just have a better service, right? Better uptime and better SLAs and all of these things add up. And then on top of that, there's kind of an established like cloud margin, right? And, you know, the market expects the cloud margin to be a certain level. Whereas if you have a three-year-old neocloud, you're a private company, maybe you don't have as much pressure.

17:38And there's like, assuming infinite demand and limited capacity, you can actually, hyperscalers can keep their prices high and they will fill out the capacity and also get slightly better economics. Whereas neoclouds compete over the whole infinite demand and that pushes the prices down. Perfect price competition. What does it take to run image versus video models? Well, like you guys started the company around stable diffusion moment. It was the field was mostly image at the time. How does running video models compare to image? Let's actually do text image video. Even let's compare all three of them.

18:18And so for let's say a SOTA LLM, like we don't know, let's say DeepSeq or something like that, where we know the numbers. Running a single prompt, like 200 tokens. let's say it takes 1x of teraflops. I think it's tens of teraflops, but let's call that unit 1. One image is around 100x of that. And if you are doing a 5 second video, 24 FPS, that is around 120 frames. So 100x from one image. So you are already 100x of 100x. So you are at 10 ,000x for a low standard definition video. And if you want to do 4K, that's another 10x. So 10 ,000x compared to a single 200 token LLM input. So it is a lot more compute intensive in terms of amount of flops you are doing.

19:22Yeah. in general like when we started with image the infrastructure was relatively easier to do because it's like it takes three sec or it took 15 seconds back in the days it takes 15 seconds to generate an image you don't need to necessarily shave like that 50ms 100ms you have overall in the system and then when we went to video it's like even like easier because it takes like 20 seconds 30 seconds to generate a video the way that has been happening in the past like couple months is real-time video where you need to stream 24 FPS videos over like a network link from these GPUs. That's where we actually spend like some of our time.

19:57We started this progress with like speech-to-speech models a year ago. We started optimizing them where we were able to like reduce the latency of our system with like globally distributed GPU fleet. When you send a request, we route to the closest GPU, minimize our own overhead and then do stuff like, you know, we pick the best runner, whatever, stuff like that. So we are now applying those same optimizations we did to real-time video. And we actually see like really good, interesting demand there where people want to experience this stuff like as they type, as they prompt. And that's where like some of the infrastructure technical challenges differ from traditionally running image and video models because image and video are similar-ish, you know, like just more compute expensive.

20:35But like you actually need to care about infrastructure stuff when you go from like less than a second generation time for some of these models. Yeah. Another interesting thing is like image models especially, you were able to run them on a single GPU. Like the parameter counts is actually much smaller. So that actually makes it like a little bit easier for us as opposed to LLMs. And then with video, parameter count is going up. Right now, I think we're around like for the open source ones, I don't know, 30 billion parameter. Whereas, you know, we hear rumors about, you know, GPT-4 being like in the trillions, GPT-5, maybe more.

21:15So that's another, that's like, you know, on the flip side, it's a little bit easier, but it doesn't mean video models are not going to grow, right? There's, you know, rumors around numbers for VO, numbers around Sora. So like, there's also a increase in parameter count. So that you're going to have to kind of, you know, use more distributed computing. But if you're just, you know, eight nodes, or one node or eight nodes, you kind of have a slight advantage. Yeah, totally. Okay, let's pop one layer up the stack to the models. Let's do it. So one thing I think people don't fully appreciate at the media space, and you mentioned this, you alluded to this before, is that there's a very, very long tail of models that are actually used in practice.

22:00And so I was hoping you'd give people a sense of on your platform, how many models are people actively using? How is it distributed? And like, why do you think there's such a long tail of models being used? compared to the alarm space? This is actually one of the things I would say three years ago, people got it wrong. I mean, jury's still out, but people, right after the chat GPT, people start talking about omni models. There's going to be these giant models that they're going to be able to generate video, audio, image, and code, text, every type of token. This might still happen, I think, but it's more clear that you are better off if you optimize for a certain type of output.

22:41Even this is true for code generation, definitely true for image or video output. So that's one thing when we were pitching three years ago, everyone, like that's one feedback we got. Oh, there's going to be omni models and there's going to be a single way of running these. It's going to be hard to create an edge on the modality. but turns out it's not true and it actually makes sense to have a technical edge on the modality. And this is one of the reasons why there's also a variety of models because still the best upscaling model is just doing upscaling and the best image editing model, even the best text-to-image model is different from the image editing model.

23:25So all these special tasks require their own model. It might be the similar model family or similar architecture, but at the end of the day, it has its own weights that needs to be deployed independently, and that creates the variety in the ecosystem. I think this also applies to language models, where even in the same modality, there is different families of models with different tastes, different characteristics, different personas. and this happens with like language models still like the code that cloud writes is very different than the code gpt5 does right like and like we see this happening but the good thing about here is like there's like these three four different personas on top of like different like categories upscaling editing video text video whatever stuff like that so it gets you like you know close to 50 models that are active at any point in time and then you have a very long tail of models that people still choose because they might like the person of that better.

24:23Yeah, totally. Speaking of model personalities, what are some of the most popular models on your platform? What do you think are the personalities of them? So one thing that's been true since the beginning, the popular models change all the time. So there's always new releases from different labs that take over the other and it's always a moving target. But that being said, there's two types of models usually preferred by our customers. Usually there's one big expensive model that has the best quality on video generation. This could be Vio, this could be Kling, this could be Sora. And then there's usually a workhorse model, which is cheaper, smaller, but good enough.

25:08And people usually use that at higher volumes. I would say this has been true for the past almost two years is that there's an expensive high quality model that keeps changing. There's a cheaper, good enough model that keeps changing. But overall, this has been constant. And is the workhorse model for prototyping and then you run it through the big expensive model for the final product? Or what do people use the workhorse versus the last? It's for higher volume use cases. And depending on the application you are building, you might encourage different, like lots of variations of the same output maybe, but it's very application specific, I would say.

25:48Yeah. There's also another dimension, I think, that's like kind of happening in real time right now, which is based on like different use case you want to use the model for. So like when OpenAI released GPT image editing, that model had just like superior text editing, text generation and editing capabilities. and for things that require like a lot of text, people started going and choosing that model versus the other models. So it also tends to correlate with like different capabilities models are bringing and also like what they're good at, right? So like Kling, for example, people really like it for visual effects types of workflows because they had that kind of data in their dataset.

26:39as opposed to some other models. For example, C-Dance is very good at detailed textures and artistic diversity, things like that. So it's really a matter of also this sort of use case dimension that models excel at. An interesting metric that we saw on Q2 and Q3 was the half-life of a top five model was 30 days. It's very, very interesting to me. where like, you know, these models are continuously shifting, like the top five of the models are continuously shifting. Tough depreciation schedule for the model providers. Hopefully they are building on top of the work that they already done. So it's, you know, additive to the end, but yeah.

27:26Yeah, I'm teasing. And the model probably is in a more turbulent state right now than what the end state will probably be. What do you guys think is the most underrated model? Like what's your personal favorite? it i i usually like cling models for video um but this kind of has been changing because they don't have sound uh for sound we have vo3 and sora they they are the only ones a lot of people are working on it so would love to have more more variety there as well image models i like revs model and like flux still holds like a very nostalgic even though it's been a year value for me you know like I still go back to Flux.

28:08There's like variations of Flux models now that I like. I'll go with mid-journey, which is not on file. It's not available on API. I just like how they navigated the space, I think is very interesting. Like they kind of brought this like photorealism, which was, you know, that was like a very big deal at the time. You know, no model could do it. And then now they're more like this artsy model, right? Like it's no longer, photorealism is kind of cracked and like no one cares about it. So now they have this like niche, very artistic like visuals, which is very cool. Yeah. I'd love to chat about the marketplace dynamics a little bit.

28:47So I understand your business as a little bit of a marketplace where you aggregate developers on one side of the market, that's the demand side, and you aggregate model vendors on the other side of the market, that's the supply side. And the model vendors are both proprietary APIs, model labs that view you as a distribution partner and then also open models that you host and run yourselves. And so maybe talk a little bit about for the closed model providers, you have partnerships with OpenAI Sora, with DeepMind on Veo. What's in it for them? Why do they choose to partner with you? We were one of the first platforms that accumulated the developer love.

29:29And following from that, these developers work at big companies. So they started working with us and we really built the platform for simplicity and being able to get going really fast. And because the thing Batuan mentioned, the half-life of these models is really short, people usually work with many different models at the same time. So we were able to claim that we have this big developer base that love the platform and not tied into any single model and here for the platform. And model research labs see this and they use the platform as a distribution channel and tap into the developer ecosystem that we built.