Why Diffusion Will Win AI Inference with Inception Co-Founder and CEO Stefano Ermon

0:05Hi, listeners. Welcome back to No Priors. Today, I'm here with Stefano Ermon, who is a longtime Stanford professor and now co-founder and CEO of Inception. Stefano has an extraordinarily broad body of work around generative modeling, but is especially well known as one of the fathers of diffusion. We talk about his company, challenging the large labs, and why speed and efficiency are going to be the name of the game in AI over the next few years. Welcome, Stefano. Stefano, thanks so much for being here. Great to be here. I would love for us to just start with a little bit of your research background and how you ended up starting your company.

0:42For sure, yeah. I've been doing research in generative models for like basically my entire career. I started at Stanford in 2014 as an assistant professor. and I was working on building generative models. Back then, the research area was not particularly hot. You know, the models were not quite working well. We were still, like, building little generative models over MNIST. And it was, like, a big success if you could generate these grainy images of digits. And, you know, it was even hard to publish papers back then on that topic. And you had to kind of, like, justify training a generative model as a way to learn features from unlabeled data that then could maybe help you do better at supervised learning because that was the thing that everybody cared about.

1:28But then, you know, things took over, of course. And so it was like I was at the right place at the right time working on the right thing. And so I've been doing research in that space since the beginning, basically. Did you have like a, besides curiosity in the area, a personal hope for what the models would do back in 2014 and 15? Yeah, I mean, I always felt like that was the right way to think about learning from unlabeled data, that building a generative model is really the right way to make sure you understand the structure in the data. That was kind of the way I was getting at. I was not even dreaming about the kind of capabilities that these LLMs that we have today could do that.

2:11But I was thinking more from, I think, world models perspective. Like I was, I was working a lot on images and so thinking about, okay, like I have a world model. I can imagine what's going to happen if I were to stand up and walk out the door. Like I can kind of like picture that in my mind. And that's important to make decisions and kind of like model predictive control when having this kind of model of the world requires some generative capabilities. And so I always felt like, okay, that's the right direction to work on. I felt like this is going to be very hard as a problem. It's like, it's going to keep me busy for my whole career.

2:43And so it's a good problem to work on. And then, of course, I was very wrong and things evolved much faster than I was expecting. Yeah, I think that's kind of universally true, though. And sort of walk me through the state of your research and how that led you to start the company. Yeah, so I was working on generative models of images, initially working on autoregressive models, which were very slow and kind of very blurry. And then VAEs and then GANs took over. Yes. And back then we were very unhappy with the state of generative models for images. Like the GANs, they worked, but they were very unstable to train, very hard to reproduce results.

3:23And so we were trying to see, is there a way to build something that is as good as a GAN, but it's more principled. And so we started working on score-based generative models, which are basically what eventually became diffusion models back in 2019 with my PhD student, Yang Song. And so we kind of like came up with this idea of let's train a neural network to denoise images. And if you can denoise an image, then you really are understanding enough about the structure of the image that it should be possible to build like a generative procedure based on these denoisers. And that basically became the kind of like underlying technology of modern fusion models, where instead of generating images, you know, left to right one pixel at a time, you kind of like start from pure noise and then you gradually refine the object until you get like a clean picture at the end.

4:12and that started back in 2019 in my lab and then it kind of like took over the space and even today the best models for image generation, video generation, music to some extent, a lot of the protein stuff, they are based on diffusion and so my group has worked a lot on various kinds of diffusion models, technique for accelerating them, to generate samples very quickly, to improve the quality of these models and so since we were able to get them to work on images, I started thinking about, hey, how do we get diffusion models to work on text or code generation and discrete objects? Like, is there a way to move beyond autoregressive models to something that it's more parallel, more with built-in error correction?

4:56And so I've been doing a bunch of research at Stanford on getting diffusion models to work on text and code generation. We had a breakthrough in 2024. We published a paper basically showing that for the first time it was possible to match the quality of an autoregressive model at the GPT-2 scale. So less than a billion parameters, still fairly academic, but we were able to train basically still a transformer model. As a diffusion model on the same data, we were able to match the quality, like the same perplexity, and you were fitting the data just as well as an autoregressive model with the same number of parameters.

5:33But the diffusion model was significantly faster. Because it's diffusion, because you're outputting many tokens at the same time, we were able to generate text like 10x faster compared to the autoregressive model. And so that felt very, very exciting. And I really wanted to see what happens if you scale up, if you train bigger models. And so I started Inception, a company to basically scale up the technology and try to build commercial scale diffusion-based language models. Everyone has now seen the outputs of diffusion models in particular images. And I would argue that it's like increasingly a dominant form of like generated short form video from diffusion models is like a dominant form of entertainment in other parts of the world.

6:18And it will likely become so here. It's kind of unbelievable, at least to me, even having followed the field for, you know, the last decade plus the quality that is possible today. So I think that is kind of obvious. Right. And it's such a big use case in images and video generation that folks are even creating hardware to support better performance here. It's not intuitive that would work for other fields or that this is an interesting competitive direction to the current full transformer focus like AGI Labs. Can you offer some intuition on that? Yeah. So it's a very interesting kind of like state of the world right now from a researcher perspective, because like there is like two main paradigms, two ways of building generative models.

7:13There is autoregressive, where you kind of like have a model that predicts the next token or the next pixel, and then you generate left to right one token at a time. And then there's diffusion, which is a course to find generation, kind of like iterative denoising kind of generation. and as you said, like we have continuous modalities where diffusion dominate, there is discrete modalities, text and code, where primarily all the big labs are kind of like betting on the same architecture, autoregressive models. And as we move towards more and more like multimodal models and kind of like there is this idea that maybe we'll have a model that can handle all modalities and we'll know everything about the world, what architecture would that be?

7:55Like will it be an autoregressive model? Will it be a diffusion model? Nobody knows. I think the jury is still out there. At inception, we're betting on diffusion models because we believe that what matters eventually will be inference time scaling. And there are fundamental reasons for why diffusion models are better than autoregressive models at inference time. So even if you think about the story of autoregressive models, there was an inflection point in 2017 when people switched from RNNs to transformers, right? And what was that? The problem was that RNNs had to essentially process tokens sequentially, one at a time.

8:34And training was very slow. And so people came up with this idea of let's have an architecture that allows you to process many tokens at the same time in parallel. And that was a transformer. And that was the thing that scaled better for training and that enabled a lot of the successes behind LLMs. But if you think about inference, now not training, inference generation, autoregressive models are still sequential. The computation is one left to right, one token at a time. You cannot generate the 10th token until you've generated everything that comes before it. That kind of workload does not map well to GPUs.

9:11That kind of workload is extremely memory bound. You're spending most of your time moving around weights across the memory hierarchy, and you're doing very little arithmetic. And that's a fundamental problem of autoregressive models. And so what's the equivalent? If you think RNNs, transformers, autoregressive models, the equivalent at inference time is a diffusion model. Because a diffusion model is built to have at inference time a workload where you process many tokens at the same time. And so the workload that we have at inference time in a diffusion model, it's basically very, very similar.

9:49to the workload you have for training, where you're processing many tokens at the same time in parallel. And so it's built to essentially have an inference workload that maps really, really well to map malls, the kind of things GPUs do really, really well. And so we bet on trying to build the architecture and trying to build the kind of models that will scale best at inference time because, you know, economics are dominated by, you know, the kind of intelligence per watt, the intelligence per dollar that you're able to get from the models. If you think about a lot of the advances with reasoning models, a lot of it is scaling test and compute, right?

10:29And so being able to scale better along that axis will also matter. And even if you think about RL post-training, a lot of the bottleneck is generating rollouts, like letting the model explore, you know, and then scoring the trajectories and then improving the model based on the kind of things it finds. And so inference is, again, a key bottleneck for RALPoS training. And so if you have a model that scales better at inference time, then automatically you're going to get better scaling during RALPoS training. And so that's why we decided to bet on a diffusion-based LLM because it's inherently more parallel.

11:06And the bitter lesson is that the more parallel solution is the one that is eventually going to win. How did you think about applicability or what experiments did you run in terms of cracking the nut on discrete versus continuous modalities? Because I think people have also shaped the existing, you know, dominant paradigm through new tokenization efforts or methods to make video and voice work, for example. It's, you know, this is you're not in the same token oriented paradigm. paradigm, how do you make it work here? Yeah. So there was a lot of research that went into figuring out how to apply a technology that was inherently very tied to kind of like continuous structure in the data.

11:53So if you think about a diffusion model, it's learning how to denoise images and it kind of like makes sense for continuous data. Because if you think about even two pixel colors, you can kind of like interpolate between them and it will still make sense. But if you think about two words, there is not necessarily something in between them, right? It's all discrete. And so it required a lot of R &D and new science that had to be developed to figure out how to extend those kind of ideas to discrete spaces. What can you claim about how well it works today? We think it works really well. So we've been able to train diffusion-based LLMs that are comparable in quality with the speed-optimized models for Frontier Labs.

12:33So our Mercury models are on par with the Haiku models, Flash models, mini nano models from OpenAI, if you look at benchmarks, while being significantly faster. So we've crossed, I think, we went from pure research prototypes to things that are actually used. Like we are serving these models in production today. We did all the work of figuring out how to even just build a serving engine, right? You cannot run this diffusion-based LLMs on VLLM or SG-Lang. We have to build our own serving engine, and we can handle a lot of the complexity of real production workloads. And we've solved all these challenges, and we can deliver this kind of new experience end-to-end to real customers today.

13:20Actually, a great time to just talk about where Inception is as a company. Like, how many people? What are you guys actually serving? Sort of state of research? Yeah, so we are about two years old, around 50 people, spending a lot of time still on R &D, kind of like figuring out what's the right way to train these models, how to accelerate inference. It's not obvious how you, even if you think about an autoregressive model, it's pretty clear that there's not a lot of things you can do there in terms of like, okay, you generate one token at a time and that's it. In a diffusion-based model, we know that there is a lot of different possibilities for trading compute for quality at inference time.

13:59Like even if you think about image diffusion models or video diffusion models, there is a lot of techniques that you can use to kind of like accelerate sampling, distillation or like fancy differential equation solving techniques that allow you to sample very, very quickly from these models. And so there is a lot of research on the training, on the inference and then engineering, like just like thinking about data mixes, evals, RL, post-training infrastructure. There is a lot of work that needs to happen to figure out how to build recipes that work for this new model. And we try to leverage existing things as much as possible.

14:37For example, it's still a transformer-based model, so you don't have to throw away a lot of the work that has been done on good architectures. We still use attention. We still use a lot of the public data sets that people have created and evils and benchmarks. So, you know, we're a startup, we try to be scrappy, we try to use existing things as much as possible and kind of focus on the aspects where we can have the highest impact and then where we can be the most differentiated. And right now it's speed. In the future, who knows? Like it's possible that a diffusion-based language model will be maybe significantly more intelligent than an autoregressive one.

15:15Like nobody knows. So that's why I think this is very exciting because we're developing these really powerful AI systems, but it's all very fresh. It's all very new. I doubt we've discovered the best way of building these systems. There's got to be alternatives. There's got to be other ways of building these models. And eventually, yeah, efficiency will be very important. Like if you think about the AI factory, like how is that going to work? I think nobody really knows. And just being able to play in that space and thinking about alternative ways of creating intelligence, I think it's exciting.

15:49Absolutely. And I also think that in a increasingly, like fundamentally compute and supply constrained environment, the, you know, for some, I wouldn't say that there wasn't very focused research on efficiency of models, but it was a second order sort of consideration for many of the industrial research efforts versus like pure capability scaling on what we've got. Right. And new methods in that. But I just think if you if you fundamentally believe that we can use all of the power we have and there will be economic reasons to do so, then I think the focus on efficiency is going to go dramatically up.

16:32Right. You know, I think many people now argue like one of the most important decisions for a company in the AI space is like, how do we use the power we have between, you know, training and the most valuable use cases for any watt. On that topic, like here in, you know, September of 26, where does speed win? Like, where do people care enough about this already? Yeah, it is basically applications where latency matters. I mean, generally speaking, I think everyone cares about speed in the sense that if you can give me the same quality, but faster, people will always pick the faster solution. And we're seeing it with this like faster versions of even the models from Frontier Labs, people are willing to pay more to get access to faster models, right?

17:20And I think once you get used to a fast model, it's hard to go back. It's kind of like broadband, right? And then you get faster and faster. And then if you were able to try, people cannot go back once you get the fast model. Are there customers that you can talk about publicly that care about this today? There are a few that we can mention. Like in the voice space, for example, OpenCall is an example. They are building like voice agents. They, you know, speed, of course. The pipeline is like you have an ASR model, you have an LLM that it's kind of like doing all the tool calls and figuring out what to say next.

17:55It has to be a reasoning LLM typically to have the highest level of quality. and then there is a text-to-speech component at the end. Speed matters a lot to them. They were previously serving their LLMs on Cerebras. So they were using custom chips to get to the kind of speed that they need to deliver the best experience to their customers. And then they switched over to our diffusion-based LLMs because they can essentially get the same speed as what you would get if you were to run an autoregressive model on custom hardware, if you have a diffusion-based LLM that it's built to be parallel, it's accelerating at the software level, then you can get the same speed on NVIDIA GPUs, which means much more availability.

18:41I mean, GPUs are scarce, but there's more of that than custom chips and lower costs, higher quality. So that's an example in the voice space. Yeah, I was actually going to ask you how you think this interacts with the hardware landscape as well, given we've seen enough now demand from use cases who are like, yes, I want a big expensive chip with a lot of SRAM and people will pay for the outputs of that in coding and other use cases. Yeah, for sure, for sure. And I think hardware is one way to accelerate things. And software might be better if we can use the existing hardware. Exactly. And especially they are complementary.

19:18That's the exciting piece, is that to some extent, the gains that you get from the software, they are multiplicative with the gains you get from the hardware. And maybe someday people develop system, you know, hardware that fits even better the models that you're building. For sure. For sure. Yeah. Yeah. If we just like zoom out to the, you know, inception in the broader industry, I think there is a vein of concern and correct me if I'm wrong, that it's very hard to invest in new architectures today because some, you know, if there are advances in architecture or methods, it will simply be absorbed by large players with the resources to scale compute.

20:01Talk to me about how you think about going and competing as David in this situation. Yeah, it's a very valid point and something that is also like top of mind for us. I think initially for us, for sure, like the mode is sort of like the IP, the trade secrets, like the ideas that we and our researchers have to build these models and make them better. As we mature as a company, and one of the reasons we are not just doing pure research, we're also like, you know, we've developed a product and we have real customers and we are getting feedback on the models from the real world is that by doing that, we are also like developing components that are also very important to deploy these models.

20:46For example, like a serving engine. You know, if you don't have the serving engine, you can't really serve these models in production. And so by forcing ourselves from the very beginning to go out and deploy something end-to-end, we're learning a lot about how to serve these models and how to build software that it's kind of like needed to run these models. And that, again, becomes IP. Like even if you train the diffusion-based LLM, if you don't have the serving engine, if you don't have the VLLM equivalent to serve it, you're still stuck and you still cannot use it. Along the same lines, we are working with real customers and we're getting feedback on the models.

21:26We figure out what works, what doesn't. We collect data sometimes from them. We create evils based on what they're seeing. And so that, again, becomes part of the technical mode because, you know, of course, those things are a little bit harder to replicate. You can tell me if this doesn't make sense as a question to ask technically. But one of the things that diffusion models benefit from structurally in images or video generation is, you know, you're replicating something where there should be some consistent structure in the world. Voice as well, right? It is whatever is really possible and most likely.

22:05There are like some of the fields where AI has been most valuable to date. I'd say like, you know, a lot of the input data you use to train like code data, for example, it's very messy, right? And, you know, one could argue that a lot of it doesn't actually have the, like, correct real structure you're looking for. How do you think about that when it's, like, human-generated input data versus, you know, images, video, voice? Yeah, it's a good question. And fundamentally, if you think about whenever you train a generative model, what you're doing is, whether it's an autoregressive model or to some extent even a diffusion model, is you are trying to identify structure in the data by essentially building a compression scheme.

22:50That might not be obvious, but whenever you train these models, you're effectively trying to identify common structure by trying to find an efficient way of compressing the data. And so the more you can compress the data, the more structure, the more patterns you're identifying. And that's how these models work, which is the amazing thing. It's just like by predicting the next word, you are learning something about the structure of the data. And that's the same, whether you're using a diffusion model or you're using an autoregressive model, both methods are essentially trying to learn a compression scheme.

23:22And when I mentioned the original 2024 paper, when we showed that we are achieving parity with autoregressive models, the metric that we're using is basically perplexity, which is a notion of how much structure have you identified in the data. And so even though it might not seem obvious, we were actually able to identify at the GPT-2 scale the same amount of structure as an autoregressive model. Yes, I think like that empirical result is there, but the intuition would be like, well, this code is not, the dataset that you are working on is not like grounded in physics, right? There's a lot of noise in there and it sounds like you believe that's a manageable problem.

24:05Yeah, there is noise in everything. They're everything. And so to the extent the numbers don't lie, to the extent that you are able to drive the perplexity down, then it means that you can actually build a compression scheme that will get you that level of compression. And so the structure must be there, and the model must have been able to uncover it. And then it's more a question of an inductive bias. Like, is a transformer a better way of identifying those patterns or something else? Is next token prediction the right modeling framework or is it more like denoising? And that's very much an empirical question that I think at the moment we don't have tools even to understand.

24:47Can I ask a question just because you used a voice customer as the example here? One of the benefits that some people building these AI products have identified of having an LLM in the middle of this voice pipeline is they understand how to do alignment a little bit better there or controllability. I imagine that has to look different for a diffusion-based model. Can you talk about that? Yes, that's a key value proposition. And one of the things that they always look into is, you know, yeah, to what extent a lot of the value they provide is like the harness and making sure that the models indeed are doing the right thing.

25:24And the interesting thing about a diffusion-based LLM is that we've built everything to be backwards compatible. So it's still like the API is the same. It's still OpenAI compatible, text in, text out. And it so happens that the models we've trained are good at following instructions. They're good at outputting, you know, if you're using JSONs, like structure outputs, they can handle all of those things. And it was good enough, it was better, in fact, than the models they were using before. And so they are still able to provide the kind of like level of service to their customers by using Mercury.

25:57Well, very simple if the interfaces are the same and you can just use your same stack, yeah. But it could be that. I think that's actually a very interesting point, is that we know that diffusion models are typically easier to control compared to autoregressive models. And the reason is that if you think about an autoregressive model, you kind of have to wait until you've generated the full object to know whether or not it satisfies, let's say, a constraint, or whether or not it's aligned, or whether or not whatever, you know, it's on brand, whatever it is that the objective function that you care about.

26:27Maybe you're generating a molecule and you care about solubility, and then you kind of have to wait until you have the full object to be able to score it with some reward function. But a diffusion model, it's more course-to-find generation. So you can progressively do it, yeah. From the very beginning, you know, kind of like, is this object the kind of thing I want or not? And you can steer the generation in the direction provided by an external reward function or some set of constraints. And so there is a lot of evidence in the academic literature, at least, that diffusion models are easier to control and there are different ways of steering them that are just not possible with autoregressive models.