#311 Stefano Ermon: Why Diffusion Language Models Will Define the Next Generation of LLMs

0:00When you think about most existing LLMs, chat GPT is the Gemini's they're also called autoregressive and what that means is that when you ask a question they will give you an answer and it will generate the answer one word or one token as we call it at a time and that generation is sequential and that's kind of like a structural bottleneck that cannot be overcome because it's kind of like built in into the autoregressive formulation a diffusion language model is also using a neural network under the hood and in our case, it's also a transformer neural network. But the task that the model is trained on is quite different.

0:35The model is not trained to predict the next token. Instead, it's trained to remove noise. Build the future of multi-agent software with Agency. That's A-G-N-T-C-Y. Now an open source Linux foundation project, Agency is building the internet of agents, a collaborative layer where AI agents can discover, connect, and work across any framework. All the pieces engineers need to deploy multi-agent systems now belong to everyone who builds on agency, including robust identity and access management that ensures every agent is authenticated and trusted before interacting. Agency also provides open standardized tools for agent discovery, seamless protocols for agent-to-agent communication, and modular components for scalable workflows.

1:37collaborate with developers from cisco dell technologies google cloud oracle red hat and more than 75 other supporting companies to build next generation ai infrastructure together agency is dropping code specs and services no strings attached visit agency.org to contribute that's A-G-N-T-C-Y dot O-R-G. Hi, everyone. My name is Stefano Hermon. I'm one of the founders and the CEO of Inception. Before starting Inception, I was a professor of computer science at Stanford for about a decade. I've been working on generative models, what's now called generative AI, for about 10 years. I did very early work on diffusion models, the technology that is now powering a lot of the state-of-the-art image generation systems, video generation systems.

2:46I worked on a variety of techniques like flash attention, DPO, that are now widely used in industry to build large-scale other labs and other kinds of generative AI solutions. I've been working on diffusion language models for a while. in my lab at Stanford. So kind of trying to figure out how to get diffusion models to work not just on images and videos, but also on text and code generation. Had some breakthroughs in my lab last year where we showed that for the first time, this technology was able to compete with the leading autoregressive models. And then that prompted me to start the company to scale up the models.

3:28And that's where I've been for the last year or so. And yeah, we've been pretty successful at training the first commercial scale diffusion language models that we call Mercury. Yeah. And I'm curious. I mean, it's fascinating that diffusion language models work. Maybe we'll talk about that first. But I want to ask, is there an advantage to a diffusion language model over an autoregressive transformer-based LLM? Yes, for sure. And the reason is that when you think about most existing LLMs, like the ChatGPT, the Gemini, the Cloud, they're also called autoregressive. And what that means is that when you ask a question, they will give you an answer, and it will generate the answer one word or one token, as we call it, at a time.

4:25And that generation is sequential, and that's kind of like a structural bottleneck that cannot be overcome because it's kind of like built in into the autoregressive formulation. The neural network is trained to predict the next token, and that's how you use it at the inference time when you ask a question to it. And sequential computations are generally very hard to accelerate because there's not many things you can do in parallel if you have to go through a token spiral at a time. On the other hand, a diffusion language model is generating the answer all at once. And so the neural network is trained to refine its answers, to kind of fix mistakes.

5:09And then at the inference time, you kind of use this procedure where each neural network evaluation is able to essentially modify multiple tokens at the same time instead of just one. And so what we see is that these models can be much, much faster compared to ultra-aggressive models of similar quality. And that's kind of the key benefit. We're able to drastically speed up the generation, speed the time it takes basically to generate answers with an LLM and reduce costs. That's the other big benefit. the computation is more parallel, we can make use of parallel compute of GPUs more effectively, and we can effectively reduce costs for our users.

5:50Yeah, the other, I don't know that this is an advantage, but one of the problems with LLMs, and I've been talking to people about sort of post-transformer architectures, But one of the problems with LLMs is context, the compute required for context, for the context window scales quadratically. So it very quickly becomes too expensive. And then there are all kinds of problems with context windows that are too large. the LLM gets confused or lost. Is that something that diffusion LLMs would address? Yeah, that's a great question. And yeah, I'm very familiar with these issues. I was one of the original authors of the flash attention paper that is widely used to accelerate the very expensive quadratic computation that is required to do attention in a transformer.

7:01We have shared in the past, and I'm happy to confirm, that our diffusion language models are also based on transformers under the hood. So the architecture that we use is still a transformer with self-attention. And what this means is that we got the same pros and cons with respect to the context length that you would expect from an autoregressive LLM. And so the current models that we serve in production have about 130K tokens of context. We think we can expand it using the techniques that people have used for autoregressive LLMs. But in terms of pros and cons, there is not really any benefit or any downside compared to autoregressive models.

7:49We've also used different architectures. So just like people in the ultra-aggressive lab world that they've been experimenting with state-space models and Mambas and alternative architectures, that's also possible in the context of diffusion language models. You can use any neural network architecture you want inside the diffusion language model. And we have some prototypes where we've replaced transformers with these alternative architectures. and you get the benefits and the downsides of those architectures. So fusion versus architecture, those are kind of like two orthogonal axes that you can use to accelerate the computation or reduce the memory footprint or kind of like trade-off different kinds of performance characteristics you might hear about.

8:38Yeah, well, for listeners and for myself, frankly, can you explain how the fusion works? And how transformers play a role. Yeah, just, I mean, I know that there's, you inject noise during training and then it sort of reverses the noise down to the output. But yeah, if you could explain it for a layman, that would be terrific. Sure. So both autoregressive LLMs and diffusion LLMs rely on neural networks under the hood. So there is a neural network that takes as input a sentence and then outputs something. And in the context of autoregressive LLM, this neural network, which is often a transformer, takes as input a sentence and then it predicts what is the next token, what is the next word that I should append to that sentence to make sense.

9:43and you can train this neural network to do a pretty good job at it by just feeding a lot of text or a lot of code to these models and then eventually, after lots of training iterations, they become pretty good at understanding what kind of tokens make sense and how you should complete sentences. A diffusion language model is also using a neural network under the hood, and in our case, it's also a transformer neural network, but the task that the model is trained on is quite different. The model is not trained to predict the next token. Instead, it's trained to remove noise. So again, you start with sentences of code that can be maybe taken from the internet.

10:28And then what you do is you artificially add noise. Perhaps you mask some tokens or you change the value of some of the tokens. And then you train the neural network to take as input this corrupted sentence and try to fix the mistakes, try to give you back the original sentence. And so it's a different training objective. It's not next token prediction, but it's more like a global transformation of the sentence where you try to fix the mistakes. And at the inference time, these two models are used in a very different way. The traditional autoregressive model is just going to predict the next token, then you fit it in, and then you predict the next token, and then you fit it in, and you predict the next token, and so forth.

11:10But a diffusion language model, instead, it starts with a guess of what the answer should be. And in theory, it can be anything. And then you pass this whole guess through the neural network. The neural network will fix some of the mistakes. And crucially, it will fix the mistakes across all tokens, across all words that form the sentence. So it's much more parallel. And then you repeat this denoising process a bunch of times. And then at the end, you get an answer that we serve to our users. And as long as you don't need too many denoising steps, you don't need too many diffusion steps, this can be way more efficient.

11:51Because again, each neural network evaluation is able to modify multiple tokens as opposed to just one. Yeah. In the autoregressive model, I understand the token prediction and the attention mechanism. So it's really looking for the most probable next token. in diffusion, I don't understand, because it's not predicting the next token. So how does the model know the answer? I mean, know, is it looking across a probability distribution of the input? Yeah, I just don't understand that at all. Yeah, so it's, you know, like at the first time, yeah, the model never knows what's the correct answer. But during training, you have, from truth data, you have text and code that you've collected on the internet, for example.

13:02And so you do know what is the correct answer. So you do know what was the right token that came after a given context, a given sentence. Similarly, in the context of a diffusion language model, we start with ground truth text, we artificially add noise, we introduce some mistakes, we mask out some of the tokens, and then we feed this corrupted sentence to the neural network, and we ask, we train the model to reconstruct the original signal. And so during training, we do have access to ground truth, just like in the context of a lot of aggressive models. It's just a different kind of task that the neural network is trained on.

13:46It's trained to predict and modify multiple tokens at the same time. and under the hood, it does this by using attention. And so what it does is it takes as input this corrected sentence. It does attention and so it kind of like tries to compare every token with every other token. It tries to figure out what is the meaning. It tries to figure out what should have been the right words that go in every spot. And then it outputs its best guess of what the fix should be. And yeah, they're not perfect, but if you train them for a sufficiently long amount of time on a sufficiently large amount of data, they actually become pretty good at correcting mistakes, at fixing mistakes.

14:30And that's why they actually work quite well during Fresno. Yeah. And I'm going to break this down and drill down a little bit to clear up my understanding. So in the training, you're taking existing language, you know, diffusing it or, you know, whatever the term is so that it's full of noise and denoising it back down to the original sentence. That's right. But where does the prediction come from to answer the question? Is that purely from the attention mechanism or is there some next token prediction happening in the background before the denoising? Yeah, so it's not just the max token in the sense that that's the other advantage of the model, is that it's not necessarily just sequentially going through one token at a time.

15:46It's actually able to use context to the left and to the right to fix mistakes. So if the first token is missing and everything else is known in the sentence, then the diffusion language model can try to predict what should the first token be, given that I know everything else. So it's kind of like a generalization of the task that you would normally do in an autoregressive model where you always just predict one token left to right. In a diffusion language model, you have a richer set of tasks that you need to be able to solve during training, which could be just predicting the next token, but maybe it's predicting the first token, given that you know everything to the right of it.

16:30And so it's more like right to left prediction. And it's actually even more complicated because there's all kinds of mistakes that could be there in the sentences, and the model needs to figure out how to fix all of them at the same time. But in some sense, they're kind of similar in the sense that in both cases, you try to reconstruct part of the input. And in the context of autoregressive language models, you always reconstruct left to right, one token at a time. In the context of a diffusion language model, you reconstruct the sentence all at once globally. And so it's a more complex kind of reconstruction, denoising task.

17:15But we can still train neural networks to solve it. Yeah. On image diffusion models, they're sort of notoriously hard to control. they'll certainly give you an output but it it it may not be what you intended even if it fits the prompt how how does that I mean and there's been a lot of progress on that but but how does that problem relate to language yeah in my opinion a diffusion model is actually much more controllable than a typical autoregressive model. And the reason is that in a diffusion model, you kind of have access to the whole object from the very beginning. And so you kind of know from the very beginning whether or not it's consistent with the prompt, whether or not it satisfies some safety constraints or whatever is the control signal that you want to use to steer the generation.

18:23Throughout the generative process, you always know whether or not you're going in the right direction, whether or not the constraints are satisfied. And that's why actually diffusion models are relatively easy to control. And there's a lot of techniques that people have come up with to steer the generation process in a certain direction. Because again, you can kind of like check whether or not you're satisfying the constraints and you can kind of push the generation in the direction that you want. And in fact, instead, if you think about an autoregressive model, you kind of need to wait until you get to the very end, to the last pixel, to know whether or not this is satisfying the constraints.

19:03And so that's why, in my opinion, diffusion models are more controllable than autoregressive models. And we're starting to explore some of these benefits also in the context of diffusion language models, where there is still quite a bit of... It's a problem that comes up sometimes, that you want the model to behave in a certain way. Of course, you want the model to be safe, but maybe you want the model to be on brand. There are more complex ways of defining what you want the model to do, and we're starting to explore ways to use our diffusion language technology to allow our customers to control the language models in a more fine-grained way.

19:48Yeah. Now, the, okay, let me gather my thoughts here.

20:00So, well, I've got a bunch of different questions. One is about training. How do you train these? The other is about guiding them to, or, you know, in autoregressive models, they use reinforcement learning with human feedback. Are they the same kinds? Because I'm going to cut my stumbling around here out. How do you train these? Is it the same process as training an autoregressive model? Yeah, so at a high level, it's kind of like what I discussed. We're training this neural networks to denoise and the inputs to remove noise from sentences. We haven't disclosed a lot of the details in terms of the exact training objective or the exact size of the models or the model training data that we use.

21:09There are certain papers that are publicly available or in the open where people have studied the scaling laws and the performance that you see if you train diffusion language models versus autoregressive models. And there is some evidence in the academic literature that diffusion language models are much more data efficient in the sense that kind of the intuition is that because you're training the model to solve many different kinds of tasks, not just predict the next token, but let's say predict in any order, then they tend to be more data efficient. So you need less training data to achieve a certain level of quality.

21:52And of course, everything depends on the devil is in the detail and there is all kinds of things that you can do, but we have our own recipe and we've seen that it does really, really well. On top of that, it's also possible to do reinforcement learning from human feedback. It's also possible to do reinforcement learning using verifiable words. If you are in the context of, let's say, coding or places or math, where there is a verifiable word, there is a uniquely correct or wrong answer that you can use to check the quality of the results. And internally, we've been able to use all of these techniques.

22:28and adapt them to the different kinds of generative models that we're dealing at this action. Yeah, and so for evaluation, do you use other diffusion models for evaluation or do you use autoregressive LLMs in your evaluation? So it's, I mean, evaluating LLMs is notoriously difficult and the way we track performance internally is by using benchmarks. So there are a variety of benchmarks that the community has developed that are trying to measure how good the models are at different tasks that people care about, like how good they are at math, how good they are at following instructions, how good they are at doing question answering, how good are they at writing code in different kinds of languages.

23:25And so internally, what we do is we monitor the performance of our models across a variety of benchmarks, and then we keep adjusting our training methods and the architectures and the inference schemes to try to get as good a performance as we can while keeping the computation costs down, trying to also make the models as efficient as possible. So on the evaluation side, it's actually exactly the same. It's all very, very similar to what you would do with a traditional other lab. Yeah. And what are the sizes, parameter sizes of the models that you have to date? Yeah. Unfortunately, we cannot disclose the exact size of the models.

24:11We are in a very competitive kind of space and that's information that we're not able to disclose at this time. Well, that's interesting. a competitive space. Are there other people working on diffusion models, diffusion LLMs? Diffusion LLMs, yes. So there is a fairly active community in academia that is training diffusion language models, relatively small scale models, including from my lab at Stanford, I mean, early work at the GPT-2 kind of size, where we trained models with less than a billion parameters, let's say. And that is very valuable kind of research, because it allows you to compare apples to apples, like a small autoregressive model, a small diffusion language model, and see which one scales better.

25:04But these models that are sort of like either open source or that have been developed in academia, they don't perform particularly well because the scale is relatively small and the kind of data that was used to print them is not exactly the best that is available out there. There have been attempts at building diffusion language models in other industry labs. There's been a Gemini Diffusion paper and release, blog posts at least, where researchers from Google, from DeepMind, have talked about their internal efforts around building a diffusion language model. So far, it's not been deployed in production.

25:51It's a demo, but it's not available, as far as I know, to customers. It's not serving production traffic. So as far as I know, Inception is the only player right now with a working model that is used to sort of production traffic, who were the first ones to release a commercial scale model. And we're still way ahead of the competition in terms of like how mature our models are and our cervix engine and kind of like inference stack compared to anybody else. But there is certainly research attempts in different places and people are actively exploring this direction. Yeah, and the scaling of the transformer-based models, that's part of what's made them attractive.

26:40It's linear scaling, not in the context window, but is it the same? Do you benefit from the same scaling laws in these DLLMs? Yeah, there's certainly certain where we're observing is predictable sort of like behaviors that we can use to guide our experiments and decide what kind of architecture and design choices to prioritize. And again, there have been papers in the literature published showing that diffusion language models can scale better than ultra-aggressive lamps. And that's part of the reason we're so excited about this. It's not only extremely efficient at inference time, so the scaling at inference time is strictly better than what's possible with an ultra-aggressive model, which in my opinion is the thing that matters because ultimately it's all going to be an inference game.

27:39People are building these massive data centers. It's not for training. It's because they expect everyone will be using AI, generative AIs, LLMs. They're going to be everywhere. There's going to be a massive need to serve these models efficiently, quickly, cheaply. And the technology that scales better is the one that is going to win. And that's why we're so excited about this. But also on the training side, we've been able to quickly come up with models that are competitive with state-of-the-art, speed-optimized models from Frontier Labs. So we're making rapid progress in terms of increasing the capabilities of the models.

28:17Build the future of multi-agent software with Agency. That's A-G-N-T-C-Y. Now an open-source Linux foundation project, agency is building the Internet of Agents, a collaborative layer where AI agents can discover, connect, and work across any framework. All the pieces engineers need to deploy multi-agent systems now belong to everyone who builds on agency, including robust identity and access management that ensures every agent is authenticated and trusted before interacting. Agency also provides open, standardized tools for agent discovery, seamless protocols for agent-to-agent communication, and modular components for scalable workflows.

29:15Collaborate with developers from Cisco, Dell Technologies, Google Cloud, Oracle, Red Hat, and more than 75 other supporting companies to build next generation AI infrastructure together. Agency is dropping code, specs and services, no strings attached. Visit agency.org to contribute. That's A-G-N-T-C-Y dot O-R-G. Yeah. And you're talking about Mercury is the publicly released model. Right. You're talking about it as a code assistant. Is that right, primarily? Is there a reason why you're focused on writing code? Yeah. So we started with code because it felt like, well, first of all, we're all computer scientists.

30:14And so code is pretty close to our heart and we understand if the model is doing well or not. It's also a pretty interesting application for the Fusion Language models, because it's a little bit less autoregressive, in the sense that this bias of going left to right kind of makes sense if you're thinking about writing a document. It's kind of like how we write, although I would argue that maybe there is still a little bit of course-to-find generation where we start with an outline and then we fill in the details. but at least I would say if you think about code, there is a lot less left to write bias.

30:56Like, if you think about different files that are not necessarily in a particular order, there are many more tasks where you really want to be able to look at a whole code base and infill a particular function or functionality that you're trying to develop. And that's where the future models can shine because they are not constrained to be the goal. So this idea of going left to right is not baked in. So that's why we kind of focused on code initially because, yeah, it's an area where we were expecting to see big improvements. And also it's very commercially, and it's super important from a commercial point of view.

31:36That's where we're seeing a lot of the usage of LLMs where people are getting most of the value, I would say today, from using LLens to write code faster and find the coding and all those kind of things that people are so excited about. And so it was a pretty natural kind of like first step for us. Yeah. And can these models be fine-tuned in the same way that Autoregressed? Absolutely, yes. We can fine-tune them for specific tasks. And in fact, that's one of the modes when we engage with enterprise customers. We sometimes are able to, if they're able to provide internal data or specific data that they care about, we're actually able to fine-tune our models on their data and deliver even higher performance on the tasks they care about.

32:32Yeah. And is it the same as with, I mean, you said you can't disclose parameter counts, but is it the same sort of progression that the larger the number of parameters, the more accurate the model gets? Yes, typically yes. The bigger the model, the higher. I mean, as long as you trained with a proportionally larger amount of training data, so there are some caveats, but typically you can expect larger models to perform better, but they also become more expensive and slower. So that's kind of like the trade-off that we're facing. Yeah. And from your point of view, the advantage is speed and cost or power efficiency or not accuracy or, yeah.