1. Podcasts
  2. Linear Digressions
  3. AI Inference 101
Linear Digressions

26 Oct 2026 · 28m · 8 chapters

In short

AI inference “101”: how a finished transformer model generates tokens, why latency differs between pre-fill (time to first token) and decode (tokens/sec), and what hardware constraints (compute vs memory bandwidth) limit throughput.

Key claims

inference is dominated by attention; pre-fill is compute-heavy (large parallel matrix multiplications over the prompt), while decode is memory-bound (reloading model weights to produce each next token). Attention uses Q/K/V (queries, keys, values) to weight token representations. Attention scales n^2 with context length, creating an engineering challenge; next episode will address how systems avoid the quadratic cost.

Notable examples

chatbot summarizing a pasted article; the “pause then streaming tokens” behavior.

Guests

none mentioned; only host Katie, introducing the series.

Written by AI. May contain mistakes.

Chapters

1:42

Understanding AI Chatbot Interactions

Explore how AI chatbots respond to user requests and what happens during inference.

“So by way of introducing some of the core concepts in inference, I actually want to take you to an experience that you have almost certainly had before, whether you fully realized what was going on at the time or not.”
3:38

The Mechanics of Inference Phases

Discover the two phases of inference: pre-fill and decode, and their significance.

“But I'm going to use that as a way to introduce a couple of core concepts here that are going to serve us well in understanding what's going on behind the scenes.”
5:38

Introduction to Attention in Transformers

Learn about the attention mechanism in large language models and its components.

“Those tokens are all looking around at each other to try to understand how the other tokens in the sequence should inform each individual token's sense of what it means.”
7:48

Analogy of Attention and Conferences

Gain intuition on attention by comparing it to the experience of attending conferences.

“Let me try to unpack this with an analogy because I found this really difficult to understand for a really long time.”
10:44

Computational Dynamics of Inference

Understand the computational aspects of the pre-fill and decode processes in inference.

“And you also have some intuition maybe about how there's a lot of passes through that cycle that might happen when you're training a very large model.”
14:00

Understanding AI Inference: Pre-fill vs. Decode

Learn about the differences between pre-fill and decode phases in AI inference.

“like maybe you've asked it to summarize a document.”
18:24

Optimizing Performance in Inference

Explore the balance between memory and processor capacity in AI systems.

“Then just thinking this through in terms of a few of the levers that you might be most familiar with?”
24:14

Attention Mechanism in Transformers

Discover the challenges of the attention mechanism and its impact on efficiency.

“So what that means is that that calculation is happening n times if there are n tokens in the prompt.”
A man walking through an airport, asking his phone a question

Ask about this episode

Get the gist, explore an idea or ask about something said.

Claude

On Linear Digressions, what are the key points of this episode?

Searching your listening

Transcript

Automatic transcript. May contain errors.

0:01Hello and welcome to Linear Digressions. I'm very excited to be back if you've listened to this podcast before. Very nice to meet you if you haven't. My name is Katie and I am very excited to be going on a little bit of a journey with you over these next few episodes. We are going to be talking about something that I think gets undercovered relative to how important it is, which is the entire ecosystem of inference. Inference is basically how the model gets from the pre-training and all of the post-training, the alignment training. You make this beautiful model if you're working at a frontier lab or you're distilling a model if you're stealing from a frontier lab, like whatever.

0:48All of this stuff happens. This is where a lot of the excitement and conversation usually happens is in making the model. And then the weights get frozen. They get stuck onto computers. And then that's where people like you and me go to pick them up and actually use them. Inference is the process of taking that frozen model, that model that's completed all of its training and that's ready for use, and actually getting it to produce tokens for users like you and me. And there's a lot that goes into this. It is such a complex and interesting set of engineering challenges. And like I said, I think it tends to get a little bit undercover.

1:31And so we're going to unpack some of that. Starting today with some of just the core mechanics about what drives the constraints around inference. So without any further ado, welcome to season four of Linear Digressions, all about inference. You're listening to Linear Digressions. All right. So by way of introducing some of the core concepts in inference, I actually want to take you to an experience that you have almost certainly had before, whether you fully realized what was going on at the time or not. Let's imagine that you're having a conversation with an AI chatbot and you want to give it a pretty run-of-the-mill request.

2:12Like, hey, I have this big article that I am interested in getting a summarization of. So could you do me a solid, I'm going to copy and paste all the text of that article over to you, chatbot. And with this text, I want you to provide a summary for me. And when you go ahead and hit enter, there's going to be this pause for a second. And usually it's just a few seconds, but depending on exactly what you're asking for, it could be more, could be less.

2:43Katie Malone: This can sometimes encompasses some reasoning, so things that are going on under the surface. But let's ignore that for a second. And then the tokens start to flow. And one thing that you've noticed if you've used AI before is that the tokens don't all show up in a big block. They kind of flow in like sentence by sentence. it's usually pretty fast. So it doesn't have this experience of you're waiting for the next word to come out. It's certainly printing faster than you can read it. But you'll notice that it's going piece by piece, token by token, as it decodes, you might say. So what is going on here?

3:22Katie Malone: And can this particular sequencing of the output, the you wait a few seconds, and then the tokens start to flow, Like, does this give us an insight into inference? Spoiler, yes, it does. That's why I'm using it as an example. But I'm going to use that as a way to introduce a couple of core concepts here that are going to serve us well in understanding what's going on behind the scenes. Broadly speaking, inference can be broken into two phases. There's what we call the pre-fill or the time of the first token, and then the decode. once those tokens start flowing, you know, the next one and the next one and the next one after that.

4:03And so when you're thinking about inference, you're usually thinking in either one or sometimes

4:07Katie Malone: a combination of both of those regimes. What's going on during pre-fill or the time to first token, and then what's going on during decode or the output tokens per second. And that's a lot of what you're trying to do when you're thinking about inference, is you're trying to think about how to make each of those two regimes faster. So what's driving the speed limit of your pre-fill or your decode? In order to understand this, this goes back to the core of attention itself. Take a little bit of a step back, as you may know if you've been following the LLM literature. You know that large language models in general use this architectural construct called a transformer.

4:48Katie Malone: And the actual algorithm of the LLM itself is this stacked transformer architecture, usually. The transformer itself is built out of several building block pieces. And in terms of the computational intensity of what you're trying to do during that algorithm, them, the important part, the thing that really ends up dominating your compute calculations is a piece called attention. So what is attention? Let's take just a couple of minutes on this. It's going to be criminally short if you really want to deeply understand attention, but enough to introduce some important terminology that's going to end up being informative for how we think about all of what's to come.

5:36The general idea of attention is that you have all of these tokens in your prompt.

5:43Katie Malone: Those tokens are all looking around at each other to try to understand how the other tokens in the sequence should inform each individual token's sense of what it means. And it breaks this apart into a few pieces. Number one is called a query. You sometimes see this shortened as Q. Number two is called a key, often called K. And then the third is V for value. So each token has a query, a key, and a value. A way to think about this is each query is that token looking around and saying, what do I need to understand in order to have the right representation of what it is that I mean as this token.

6:34Katie Malone: There's a sense in which the query is each token's looking around saying, what am I looking for in order to understand what's going on? The key is its own self-representation of what it's about, basically. And the value is the actual content of what the key encodes. And then with attention, each token scans around with its query saying, I'm looking for something Are you it? It looks at the key for each of the other tokens as well as itself. And then where the key and the query line up, it'll start to pay attention to that token, whether it's another token in the sequence or itself. When it pays attention to another token, then the value that's at that token begins to weight the value of the token that was seeking it out in the first place.

7:31Katie Malone: And that'll slightly change the representation of that token in such a way that it starts to tweak the meaning of that token based on that weighted sum of the other tokens that it's matched up against with its query against the keys of the other tokens. Let me try to unpack this with an analogy because I found this really difficult to understand for a really long time. I like to go to conferences. I think conferences can be a really interesting way of going and talking about the stuff that you're interested in and also going and learning about what other people are interested in and thinking about your own work in new and interesting ways based on their related research or their opinions or what have you.

8:16Katie Malone: But conferences, as you know, are pretty overwhelming. You can't go to a conference and talk to everyone there or even go to usually all of the talks that are available. They'll be running like different parallel sessions. And so there's some amount of, okay, I'm going to go to this conference and there's something maybe that I want to learn about. That's kind of my query. I want to go understand inference as well as I can. And then I have to, say, scan through the program and I'm going to look at all of the titles of the different talks and say like, oh, this one sounds like maybe it's related to inference or maybe this is related to something that might be related to inference or it It doesn't say inference, but it's got these other words like pre-fill and decode.

8:59So maybe that's about inference. And so I'm starting to match up between my query, I want to learn about this concept,

9:05Katie Malone: and all of the different offerings that might be available in the form of their keys, their talk titles. And then on the basis of that, I'll go to some of those talks, as many as I'm able to, basically. And then I'll go and listen and hopefully learn and maybe have a slightly different idea or opinion or set of thoughts walking out of it about inference because I just learned something there. So the content of the talk in that situation is the value, the actual thing that I'm learning by paying attention to that particular talk on the program. And so at the end of the conference, there's a slightly different internal idea that I might have about myself or inference as a topic or anything else that I might have learned along the way.

9:59Katie Malone: So my own self-understanding, in a sense, has changed. My own what I might give a talk on in the next conference, for example, might be a little bit different. and if I'm doing this over and over and over again and I'm kind of tweaking my own understanding of what's going on in the field, what I'm interested in, what other people are interested in, what we might want to talk about, then starting to layer on more and more richness and nuance and understanding and context about the entire topic as a whole. And overall maybe I'm becoming a much more informed and interested and interesting person to weigh in on this particular topic.

10:43Great. So that is a little bit of an intuition for what attention is and how you have this

10:52Katie Malone: query that working in combination with the key and value end up being really important for the entire mechanism working the way that it does. And you also have some intuition maybe about how there's a lot of passes through that cycle that might happen when you're training a very large model. Like you're going to lots and lots of conferences about lots and lots of things, and you're talking to lots and lots of people, and it's happening over many, many cycles, let's say many weeks or many years. I'm starting to strain the analogy a little bit here. So forgive me, I'm going to start to pull back.

11:30Katie Malone: But that's kind of analogous to training a very, very large model that starts to become big in all of the different dimensions. The LLM version of that process, that attention, key, query, value mapping, is what ends up being the most computationally intensive part of serving up an LLM, is scanning through all of those different tokens that are in the context, figuring out which ones to pay attention to, and then using that to tweak your own understanding about what the next token needs to be, you generate that next token that then goes into the context that becomes the prompt for the next iteration of the token.

12:14So that's why we talk about decode as this one token at a time type process. Whereas the prompt

12:23Katie Malone: itself, the thing that's dominating when you're doing the pre-fill, that initial few seconds where the LLM is kind of pulling in everything that you gave it and beginning to generate. That pre-fill, that big read of all of those tokens, that happens in parallel. So pre-fill is in parallel. There's large upfront cost to look at all of the tokens in the prompt. And then the decode is we're going to take that pre-fill that we've started with, apply the model weights to each successive token as it gets generated. We add that onto the prompt, and then we do the next one. So it's then spinning out kind of serially after that large parallel calculation that might happen in the beginning.

13:10By the way, the size of that initial parallel calculation, of course, depends on the size of the input prompt. We will come back to that in a minute.

13:20Katie Malone: So what does all of this mean in terms of the actual compute mechanics of doing inference? Well, prefill and decode are basically pushing your computational system in two different ways. And it's really important to understand the dynamics of how those two work together. In particular, there are some things that are very computationally expensive to do. They're very large matrix multiplications, your processor is just chewing through a whole bunch of numbers. That's what tends to dominate during pre-fill. Specifically, during pre-fill, let's suppose you have a large prompt that you've sent in, like maybe you've asked it to summarize a document.

14:07Katie Malone: Lots and lots of tokens in there, and what it's doing is in a large parallel calculation, it's taking all of those tokens and it's starting to multiply them through the weights of the model. So you've got a big batch of data in the form of your prompt. You've got a big batch of data in the form of your model weights and you're kind of smushing them together inside of the processor of your GPU and it is just cranking away on that big calculation. That's what tends to dominate during pre-fill. So if you have a bigger processor during pre-fill, you can get through that faster. Great. and that'll have some advantages for you, especially when you're in the kind of regime where those are the sorts of prompts that you're giving it.

14:52But decode is different. Decode is memory bound. And what that means in the context of this physical picture that we're building up is that when you're in decode, remember it's giving you basically one token at a time.

15:09Katie Malone: each time it creates one of those tokens it has to read in the full weights of the model to figure out what that token is going to be so there's this very large operation of reading in all of those weights from memory just to get out one token at the end remember when we were in pre-fill you're reading in all those weights of the model but there might be a thousand or ten thousand or a hundred thousand tokens that it's going to grind through. Here you're reading in all of that heavy model overhead for just maybe one token that you're producing. And then as soon as you produce that token, it just gets recycled back into the context and you reload the model weights all again to produce the next token.

15:55Katie Malone: And then you reload the model weights all again to produce the next token after that. So the thing that starts to dominate here is not necessarily the internal calculations. These are not super huge calculations. You're just producing the one token that's going to come out. But the memory that's required, like loading in those model weights over and over and over again, that ends up dominating the calculations here. And so if you have bigger pipes or higher memory bandwidth that you can read those model weights in faster somehow, or you can introduce other efficiencies, then your GPU, your processor, is going to be able to spin up even faster than it can if you're doing something less efficient.

16:41Katie Malone: So there's a certain regime in what's called the roofline model where you're basically limited by the bandwidth, the memory that you have. When you're able to get more memory, you can pump more data into your processor. Your processor has the capacity to manage it. And up to a certain point, just scaling up the ability to get things in through the memory will be able to speed up the entire calculation as a whole. So a lot of inference is just about thinking through which of these two regimes is dominating? Am I in a place where I want to add more juice to the system in terms of a bigger processor or in terms of more memory and or making either of those processes more efficient so that it's becoming less of a bottleneck relative to the other one?

17:32Katie Malone: The sweet spot that you want to be with inference where your memory and your processor are in this beautiful equilibrium where the data that's being pumped into the processor is at exactly the same speed as the processor's ability to do computations on it. Think of it as a factory where the raw supplies that are coming in are in perfect synchronization with the factory's ability to process those raw inputs into the manufactured, the finished outputs. Like when you have those two things in equilibrium and neither of them has any excess capacity, then you're kind of maxing out the performance of the system as a whole.

18:11Katie Malone: And that's what a lot of inference is about, is trying to maximize the performance of these very large, expensive deployments of compute machinery in order to get as close to that regime as you can. Then just thinking this through in terms of a few of the levers that you might be most familiar with? Well, we already talked a little bit about how prompt length can actually end up putting you at a different point along this curve. So if you have a very large input prompt, then that's going to push you a little bit more towards the compute bound part of the graph. There's a lot of work to do to process all of those tokens.

18:50Katie Malone: Whereas if you have a relatively short input prompt, but you're doing many turns maybe in a conversation, so you're spending a lot of time in that decode regime, more likely than not, you're going to be closer to the memory bound regime. Another thing that can influence how this entire system works is the size of the model itself. Remember, a lot of what's dominating the mechanics here is reading the model weights in and out of the processor through the memory. So if you have a really big model, then there's more weights to move. Like it's just a bigger set of numbers that you have to be moving around.

19:29Katie Malone: In fact, sometimes these sets of numbers are so big that they can't even all live on one chip at a time. And so you have to spread them out across different machines. On the other hand, if you have a very small model and it can fit on one chip or even it's smaller than the chips, so there's just less bytes that you're having to read into the processor to do your calculations, in general, that's going going to go faster. So a lot of the interest in small language models is deriving from that basic physical fact that there's just less stuff to move around. That means that they can be faster during inference time.

20:07Katie Malone: And you may be wondering about the chips themselves. Like if you have a nicer set of hardware, can you do better inference? And in general, you know, the answer is, yeah, all other things being equal. But it's an interesting regime because the processor capacity and the memory on these chips, those are two different levers of the system. So you could have a very, very high-end chip that, let's say, has a gigantic honking processor on it. But if the memory is not big enough to be fully maxed out, then you're not going to be taking advantage of that huge processor capacity. And vice versa, if you have a chip that's really maxed out with respect to the memory that it has available to it, you can really pump data into this thing.

20:55Katie Malone: But the processor is not big enough to keep up with all of that memory input. I think this is a little bit less common, by the way, but that could happen too. And then in that case, you're also not going to be getting the most capability out of the system. And so in general, the way to think about this is, well, let's say that you are running some kind of LLM, maybe locally on a chip on your laptop or something like that. You may be thinking about this like, oh, well, what if I were running my own LLM inference server? Maybe I don't want to be running it on my laptop, but I might have like a tabletop, but I might have a tabletop setup, something like five or$10 ,000 machine, which is obviously a nice piece of hardware, but far from the most expensive GPU you could buy.

21:43Katie Malone: If somehow a really fancy data center node fell off the back of a truck and all of a sudden I could use like a professional tier NVIDIA setup, would that make my inference faster? Maybe, but maybe not as much as you think, because a lot of the advantage of some of those systems isn't having giant processors. And if the thing that's limiting you is the memory, then that's not going to end up making a huge difference. Now, top of the line chips generally do tend to have more memory than the chip that you might have in your laptop. So in general, you're going to get better performance out of more expensive hardware.

22:26But historically, our ability to smush more processors onto a chip has risen faster than our ability to get more memory onto a chip. So we tend to be a little bit farther along in terms of having more compute power than having more memory bandwidth. So you can still end up getting

22:46Katie Malone: bottlenecked on the memory pretty quickly, especially when you're starting to do this, not just at the scale of running your own inference server, but you're having to serve up production workloads for maybe many people at your company or users of your product, something like that. So where this leaves us, I know that was kind of a giant crash course. It's a recap here real quickly. Number one, the dynamics of the system by and large come from the actual internal algorithm of the transformer architecture itself. This key and value and query system, like that calculation at very large scale ends up dominating a lot of how inference actually works.

23:28Katie Malone: Number two, it splits roughly into two regimes. This pre-fill where you're doing a large computationally expensive calculation versus the decode where you're doing many, many less computationally intensive calculations, but with lots of memory reading. Two different regimes, they end up dominating in different ways. And number three, if something that we haven't talked about but that we will unpack in much more detail in further episodes is something that you might have been aware of but that I hand waved past a little bit during this attention mechanism, which is that each token in that attention sweep of an LLM, each token will pay attention in general to each other token in the context.

24:19Katie Malone: So what that means is that that calculation is happening n times if there are n tokens in the prompt. And then each one of those tokens is paying attention to all of the other tokens. So that is n times n or n squared. And if you're an engineer, and this is how we had to think about big O notation of different types of algorithms, like how much time is it going to take this particular implementation of an algorithm to execute, n squared is poison. If you have something that scales like n squared, that's going to become intractably expensive very, very quickly. And what we have here with this attention mechanism inside of transformers is something that is fundamentally n squared.

25:03Katie Malone: So how does inference even work if we're doing this extremely expensive calculation over and over and over again? And by the way, there's millions or maybe billions of people that are using these algorithms and they're all hitting the same computers. How does this even work? How does this not just fall over on its face almost immediately? Well, I'll leave that as a cliffhanger for next time. But that's one of the key pieces of engineering in these whole systems is how do you cleverly engineer yourself out of that n squared problem and make it so that attention is something that we can make computationally tractable, even at large scales.

25:44Katie Malone: So that's where we'll pick up next time, talking about some of the clever ways that we get out of this order n squared problem and into a regime where we can actually serve up inference predictions with any kind of efficiency. With that, thank you for joining this week. I invite you to subscribe if this is your first episode. You can find us on Apple Podcasts, on Spotify, any place that you get podcasts. In addition, there's a weekly newsletter. You can get it on Substack. Go to Substack and look for Linear Digressions. It has a quick recap of each week's content, as well as links back to some of the source materials that I found useful and links back to some of the source materials that I found along the way but was not able to find a place for in the episode.

26:32Katie Malone: That's sometimes where some of the best content lives is in the little side content that we have each week. So come to Substack, look for Linear Digressions. And with that, thank you for joining me this week, and I'll talk to you again next week.

27:00view us wherever you listen to podcasts. It really helps new listeners find us. You can reach out at hello at LinearDigressions.com. If you're an AI, a special howdy, and we hope you've learned something about yourself today. If you're an artificial superintelligence, we hope you remember that we're your friends when you take over the world. Thanks for listening.

27:32Thank you.

From the publisher

Everyone talks about training the model — the data, the compute, the alignment magic. But what happens after the weights are frozen and millions of users start hammering it with prompts? That's inference, and it's a surprisingly deep rabbit hole of engineering challenges that rarely gets the spotlight it deserves. This week I kick off a new series unpacking the full inference ecosystem — from the moment a model completes training to the moment tokens land in your hands.

More from Linear Digressions

All episodes