
KV cache fundamentals for LLM inference—how storing attention “keys” and “values” makes serving prompts computationally tractable, how KV cache differs from the model’s context window, and how “paged attention” dynamically allocates KV cache to avoid over/under-reserving memory.
No guests mentioned; it’s a solo episode host explaining the concepts.
Self-attention is costly (pairwise attention scales ~n^2), but KV cache reuses precomputed K/V so each new generated token avoids recomputing attention over the whole history. Context window is a fixed model limit; KV cache is the per-request memory footprint that grows during prefill and decode. Proper KV cache management is crucial for throughput and cost.
short chatbot (few turns), long multi-hour chatbot/code-agent sessions (KV cache grows large), RAG/summarization (expensive prefill once), and Cloud/Claude Code-style “cache warm vs cold” pricing and context-clearing guidance. Mentions vLLM and Berkeley’s 2023 paged attention (reported <4% waste; 2–4x throughput).
Written by AI. May contain mistakes.
Explaining the importance of KV caches in making LLMs computationally feasible.
“As you may know, or as a bit of a refresher, the self-attention mechanism is what ends up dominating the computational cost of serving up LLMs.”
A detailed look at how KV caches operate during LLM inference.
“The context window is the maximum number of tokens that can even be present for that attention mechanism to look at.”
Exploring how different interaction types affect KV cache usage and memory requirements.
“But it's not growing in this order n squared kind of way that would just grind things to a halt.”
Introduction to the concept of paged attention for managing KV cache efficiently.
“And if I guess wrong, there's a few things that could happen.”
Learn how KV cache memory allocation works in language models.
“Now, this does come with some complexity.”
Explore how paged attention has transformed memory efficiency in models.
“We've now, with the introduction of the KVCache, we've taken this order n squared problem and made it basically an order n problem with some costs that we're paying in memory overhead.”
Discover strategies for optimizing the use of KV cache in applications.
“optimizations that people are working on even today about how to take fuller advantage of the KV cache, how to optimize it for different types of usage in different scenarios.”

Get the gist, explore an idea or ask about something said.
On Linear Digressions, what are the key points of this episode?
Automatic transcript. May contain errors.
0:01Hi, and welcome. I've been interested lately in inference. Inference is the process of making actual model predictions available to end users. This is after a model has been trained, how do you actually make it useful to people? How do you hook all the computers up? It's an underappreciated problem, in my opinion. It's got a lot of interesting engineering. One piece of this that we're going to spend some time going deep into today is what's called the key value cache, or KP cache, almost always for short. And this is a really important optimization that makes it computationally tractable for us to actually use LOMs at scale at all.
0:42So without any further ado, let's just dive into what the KB cache is and why it matters so much for our ability to actually use LLMs for any kind of real usage. You're listening to Linear Digressions. As you may know, or as a bit of a refresher, the self-attention mechanism is what ends up dominating the computational cost of serving up LLMs. The general idea and why that's so expensive is that every token in the prompt is potentially looking at every other token in the prompt and doing these mathematical comparisons and adjustments based on all of those pairwise comparisons. And that looks like order n squared, where n is the number of tokens in the prompt.
1:32So once you start to get to large or even medium sizes of n, n squared starts to blow up like pretty badly. Because each time you're having to recompute the full attention over the entire growing transcript. So as you're having this conversation, the transcript is growing. That whole thing serves as the input, the context for the next turn in the conversation. So each step is getting more and more expensive. And then when you have n steps, that's where this n squared blow up happens to. But that's not what actually happens. We're not recomputing those attention keys and values in every step of the conversation.
2:18Instead, we introduce what's called a cache, in particular, a KV cache. So remember that K and V are two different components of the attention calculation. The token's key and its value. And once you've computed those, they don't change for the rest of the calculation. They're fixed. And so what that means is that you can calculate them once and then store them instead of having to recalculate them each time. And what that means is that then when you go to the next round in the conversation, you're reusing the things that you've already stored from the previous rounds in the conversation. And so then each new token costs about the same as it would before.
3:01It's not getting exponentially more expensive for you to have each successive turn in the conversation. This is the core of what makes LLM inference computationally tractable. Now the KV cache is a little bit different from the context window, which you may be familiar with from LLM mechanics already. The context window is the maximum number of tokens that can even be present for that attention mechanism to look at. And this is a characteristic of the model. So this got a little bit more attention, maybe, sorry, no pun intended. People talked about this a little bit more, maybe a couple of years ago, when new models were especially like pushing out this context window.
3:46So we're going from 16K context to 32 to 128. A million token context window is not uncommon now for some of these models. So that's a fixed characteristic of the model itself. The KV cache is what you're actually using, the actual memory footprint of serving one request within that limit of the overall context window. And so this KV cache is more happening on a conversation by conversation basis. this. So the context window is the upper limit of what you can be attending to in a given conversation, the upper limit of how big your context could be, basically. But the cache is what's actually happening in this conversation right now.
4:30How much of that window have you used up? So the general way that a KB cache works is that, remember, we have two general regimes in inference. We have this pre-fill step that tends to be computationally intensive, where you're taking all of the tokens in the prompt and you're computing attention over all of them, the K and the V, over all of those prompt tokens. And then you have this serial decode task that happens after that, where each token that you then generate successively to that gets appended to the context and then fed into the next turn. So when you're building the KV cache, what happens is during this pre-fill step, you calculate the K and the V for all of those tokens, and then, and you store that somewhere.
5:24That is a big matrix of numbers, and that gets stored somewhere for you to retrieve and to use again and again. Then you decode a token and you append it to the KV cache, So you're going to use that next time. You decode another token. You append the K and the B to the KV cache. Now you're going to use that slightly larger KV cache the next time around. And so on and so forth. So when you're generating these tokens as part of decode, they're also getting appended to the KV cache. The KV cache is growing as you go. So things are getting more expensive as you roll along with, say, a very large conversation, for example.
6:04Your KV cache is going to grow. But it's not growing in this order n squared kind of way that would just grind things to a halt. Instead, it's just taking more and more overhead in this piece of memory, this cache, in order for you to store all of those intermediate values and reuse them again and again. This plays out differently in different regimes. So let's think about first a quick chatbot exchange. For example, maybe I'm asking for a recipe for dinner, maybe one or two or three exchanges in the system. I'm getting nowhere near the upper limit of the context window. I'm staying quite late in terms of what I'm asking the LLM to create for me.
6:47I'm probably never going to come back to this later. So this is not a multi-hour conversation. nothing too complicated here. This generally tends to stay within the default parameters maybe of one of these inference systems. But let's say that I'm in a regime where I'm having a really long conversation with my chatbot. This might be happening over many minutes or hours. This might be something that I'm picking up day over day. So then the cache shape is going to start to look pretty different. It's going to be getting quite large because in a long conversation that entire conversation history is becoming the prompt.
7:25That means that entire conversation history is getting stored in the KV cache. And so that also means that the KV cache is going to keep growing as long as this conversation runs. So that can potentially be a pretty big piece of memory that you might need to have to keep track of all of those keys and values of everything that's happened in this conversation so far. A third regime is where you might be doing something like what you might ask a rag chatbot to do or a single shot request to, let's say, summarize a really large document. So I'm not having an extended conversation with this thing.
8:04There's maybe just one request that I'm making of the LLM, but it's a big one. And in that case, I'm doing that large calculation up front where I'm doing this big pre-fill step. I'm calculating these lots of computational intensity, calculate lots of key value pairs. So it's pretty expensive to build that KV cache. I might use it once or twice if I'm asking a few follow-up questions, but then I'm not coming back to it many times. I've asked it to provide the summary. It provides that summary. Great, we're done. And then theoretically anyway, I would probably want to release those values from the KV cache, free up that memory to do something else.
8:47And then the last regime, the one that tends to maybe stress this system the most, is where you have something like a coding agent where I might be having, yes, these multi-turn, multi-hour conversations and sessions with the LLM. So there's a large context that's building up. it's being resent every turn because that's the context for each request includes everything that's happened up in the conversation up to that point which by the way might also include sub agents and tool calls and all of these other things that can blow up the context beside just my specific prompt so this is going to be a place where that kb cache is going to start to grow quite a bit and can very easily push you out into regimes where some of the mechanics of the system start to become really important for what you're even able to do.
9:37And that can start to get pretty expensive sometimes, but let's talk about what tends to happen in those regimes, especially where you're doing large, expensive, multi-turn conversations, and you're trying to manage all of this memory that's getting eaten up by the KB cache along the way. So we've established that different types of interactions between a user and an LLM can push the KB cache into different regimes. You might need a lot of KV cache space. You might need a few, just a little bit. You might need something that exists for a really long time. You might need something that you're just going to use real, real quickly, maybe once or twice and then discard.
10:16So if you are trying to actually manage the KV cache, what does this mean for you? Because KV cache, the way to think about it is these are just memory addresses. There's a fixed number of them that are available to you based on the memory specifications of the chip that you're working on. It's not infinite, so you can't keep adding to it for forever. And importantly, remember that KVCache is getting longer every time you decode a token and add it onto the context. So what that means is that the amount of KVCache that you need for a one token answer is going to be really different from the amount of KV cash that you need for a 10 ,000 token answer, or the amount of KV cash that you need for one or two turns in a conversation is going to be much smaller than if we're going to do this for, keep chatting for multiple hours.
11:10But when you go to set up the KV cash, when you go and issue the first request to the LLM, that KV cash space is getting reserved at that point. So we have to make a decision, or in general, you have to make a decision at the very beginning of the conversation, how much space am I going to save for this conversation, basically? And if I guess wrong, there's a few things that could happen. The reasonable thing that you could do is you say, I'm going to reserve the maximum amount of space that I might need to complete this conversation or to complete this request. A common default that you might have will be maybe around 2 ,000 tokens.
11:49So it'll say, hey, I'm going to plan to have 2 ,000 tokens of output. I might produce less than that, in which case any underflow is basically just wasted memory. I could also end up overflowing that. If I end up going long, then I'm going to have to evict something from the KV cache. I'm going to have to throw out some of the stuff that I've calculated in order to make room for the new stuff. And then I'm going to have to recompute it every time I want to use it. So that's also not great too. In the first case, you might be letting good memory go to waste. And in some of the first implementations of some of these algorithms, that could be anywhere from 20 to almost 40 % of your memory is just never used because you're over allocating.
12:35On the other hand, you can have this real trade-off if you under allocate and you don't reserve enough space in the cache for you, you're having to recompute things and then it's going to slow down a lot. But there's a really elegant solution that a group in Berkeley came up with to this question back in 2023, which is let's instead reserve small chunks of KV cash space. We will reserve those chunks as we go. So we end up having a relatively short response to this request. We might only need a few of those chunks. If we end up having a really long response, then as that response is generating, we can say, oh, I'm still going.
13:16Go grab another block of KV cache space. Oh, I'm still going. Okay, another block. Oh, I'm still going. Wow, this is a really long one. Another block again. This is an old idea, by the way, from the way that operating systems manage their virtual memory. They call it paging, where they're adding chunks of memory as a process goes on to make enough space for it. And this was the idea of this Berkeley group back in 2023. They called it paged attention, where they're actually dynamically allocating KVCash space as the request is ongoing, so that the idea is you're neither under-reserving nor over-reserving, but instead you're coming up with just the amount that you need at the time that you need it.
14:01Now, this does come with some complexity. So this isn't for free, but at this point, it's a relatively solved problem. And this is how major engines, most notably VLLM, manages the KB cache right now. So when you think about the actual blocks of values that are going to be generated in the request, you think about them happening in a logical order, like the first things happen first, and then the next things happen after that, the next things happen after that, and then at the very end. So you might have the introduction, chapter one of the response, chapter two of the response, and the conclusion.
14:36And they happen in that order. With page detention, it says that doesn't have to be the exact order in which the KV cache is allocated from a memory perspective. So I could have the introduction go off to one semi-random spot in KV cache space, reserve a block of the KV cache, put your introduction there. And then if that's the end of the request, like we're done, then that's all that ever gets allocated. But let's say the LLM keeps going and now we need to start generating chapter two. And let's say that the next available spot of KV cache space is actually right at the beginning of the latent memory space.
15:19So instead of having to go to the next contiguous block, which may or may not be ideal, I'm going to hop over to this spot in memory space that's totally disconnected. but as long as I keep a pointer basically to where I need to go next from the perspective of kind of the user it looks like this is all one contiguous spot so I'll go put my chapter one right at the beginning now I need to keep going go to chapter two doesn't necessarily have to follow contiguously after chapter one again maybe I pop off to some third space that's disconnected from the first two entirely. Again, as long as I have a pointer, that's going to be okay.
16:01And you keep bouncing around using the next optimal block of KB cache space that's available until you're done serving the request. So it still has the effect of looking like all of these things are happening in the same order as before, but under the hood inside of the actual memory addresses, they're not necessarily all in one big block, this pre-reserve. Instead, you're just going around and finding little pockets of space wherever it might be most convenient, and then going in that order, whatever it may be. And this ends up really changing the profile of what you need in terms of KVCash space or how well you're able to take advantage of what you do have.
16:45In the worst cases, if you're dramatically over-reserving or under-reserving, it's easy to end up in regimes where you're wasting 60 to 80 % of your memory. But with page detention back in 2023, they found they were able to get it to under 4 % of memory being wasted to over or under a reservation. At the same time, they're able to double to quadruple the amount of throughput that they're able to get with the same latency. So you're still getting tokens out at the same speed, but you're getting twice to four times as many tokens at the same time. So this entered the scene through the literature in 2022, 2023, and it became the core of what is now one of probably the most popular of the inference engines that's out there.
17:41It's called VLLM. VLLM. It's not the only inference engine out there, but it's probably the most commonly used one. VLLM has now been open sourced. It's part of the PyTorch Foundation. So it's being a foundational part of the open source inference ecosystem, I would say at this point. But it's also spun out standalone startups from the original founder group. It's really become an interesting and important part of this entire ecosystem. And it all grows out of this paged attention concept. Now, that's all very nice. We've now, with the introduction of the KVCache, we've taken this order n squared problem and made it basically an order n problem with some costs that we're paying in memory overhead.
18:26With paged attention, we've made that a price that we're willing to pay because we can use that memory overhead in a pretty efficient way. And if you're able to combine those two things and work from the cache a lot in your LLM work, then it can be dramatically cheaper than if you're not able to take full advantage of the KV cache. If you've used Cloud Code, you've probably seen some little nudges in this direction, and maybe you knew what they were for, but potentially not. The AI companies are definitely always trying to nudge you towards using the cache as much as you can, because it's so much more efficient than having to recompute stuff over and over again.
19:06So the first thing that they'll do is they'll make it a lot cheaper to have inputs that are read from cash versus read cold. So you'll see in some of the pricing charts, there's just different prices that are applied for input tokens versus input tokens from cash. And those tend to be about 10 times cheaper if they're coming from the cash. The second thing, if you're in Claude code, they have all these key value pairs that they've calculated for your entire session so far. Those are sitting in a KV cache in a server somewhere in a data center. And let's say you walk away to go for lunch. They see that it goes idle.
19:45Somebody else's conversation comes in. It takes that spot in the KV cache and all your stuff has now gotten kicked out. So if you want to resume that conversation, it's going to have to recompute that entire KB cache. And it's willing to do that, but that's pretty expensive. And so you're going to get a nudge when you come back from lunch that says, are you working on a new task? Can I suggest that you clear the context? And then that's going to save you a whole bunch of tokens. That's probably going to save you some API costs or some usage against the limit that you have. Let me just suggest that if you don't need that cache to be recomputed, don't bother.
20:22Clear the context. let's start fresh. When you're in the middle of a session, also you can get an idea of how your cache usage is looking. You can use the slash usage command, and this will give you a little bit of insight into whether the cache is warm, if you're getting a hit rate that's good, you know, 90 plus percent. And sometimes you can get a little bit of insight if the cache has gone cold for some reason about why. And then last is that each model keeps its own cache. So if you're doing a bunch of work in, let's say, Opus, and then you decide to go switch to Sonnet for some reason, then Sonnet is going to have to recompute its own Sonnet cache, basically.
21:02They're not shareable across models. And so if you go to switch models, you're going to get a little bit of a reminder or a gentle guardrail that says, hey, are you sure you want to switch? Because we're going to have to recompute this cache and that could be a little bit expensive. And that might make a lot of sense. Like you might be doing some very expensive planning maybe with one type of model and then your planning is done. You want to hand this off to a cheaper model to actually implement. But what you don't want to do is bounce back and forth between models. It's basically the worst thing you can do to turn by turn.
21:36You're asking like, hey, Opus, can you do this? Now, hey, Sonnet, I want you to take the next one. Now we're going to bounce back to Opus for turn number three and back to Sonnet for turn number four. If you're doing that each time, it's recomputing that cache from scratch. It's going to get very expensive. So it's basically the worst thing that you want to do. Try to pick one model and stay with it for a while so that you can get some of the advantages of reusing that cache. So with all of that, what I hope you've gotten from this is that the KV cache, this is a really important part of how we went from attention being this incredibly expensive n-squared operation to something that we can do in linear time.
22:16But then the actual mechanics of making that even more efficient, this starts to get into the engineering of inference itself and VLM, the page attention, these types of algorithms. So at this point, there's still plenty of optimizations that people are working on even today about how to take fuller advantage of the KV cache, how to optimize it for different types of usage in different scenarios. But this is really a place where there's a lot of action on the inference front about how to get the most out of that KB cash that you can. That brings us to the end of the content for this week. If this is your first time joining, welcome.
22:54I hope you learned something today. Please subscribe on Apple Podcasts, Spotify, anywhere you get your podcasts. If you are interested in a written write-up of each week's content, you can get that through our weekly newsletter. Go to Substack and look for Linear Digressions. It also has some content in it, like the links that I found useful in doing my research for each episode and also interesting content that I found along the way but didn't make it into the main episode. So join me there for the weekly newsletter. Thank you for joining me this week for a deep dive into KVCash Mechanics, and we will be back again next week.
23:32This has been Linear Digressions. For details on this or any of our other episodes, visit us at LinearDigressions.com. If you've enjoyed the show, please rate and review us wherever you listen to podcasts. It really helps new listeners find us. You can reach out at hello at LinearDigressions.com. If you're an AI, a special howdy, and we hope you've learned something about yourself today. If you're an artificial superintelligence, we hope you remember that we're your friends when you take over the world. Thanks for listening.
24:11you
The unsung hero of every LLM conversation you've ever had? A clever optimization called the KV cache. Self-attention is computationally brutal — every token attending to every other token means costs that scale as N², and that blowup gets worse with every turn of a conversation. The KV cache is what keeps that from becoming completely intractable at scale. Today's episode goes deep on how it works, why inference is a harder and more interesting problem than it gets credit for, and what's really happening under the hood when you chat with an LLM.