Language Models Can Control Their Own Attention

0:00Okay, let's unpack this. I mean, imagine if you had to reread an entire 500 page textbook from page 1 every single time you wanted to write a single word of your essay. Oh, man, that sounds completely miserable. Right. Like you sit down, you read all 500 pages and you type the word the and then you go back to the title page, read the entire book again just to figure out that the next word should be mitochondria. Yeah, it's absurd. Nobody could possibly work that. Exactly. Nobody. But that is actually the precise mechanical reality of what standard AI models are doing right now. Yeah, we tend to anthropomorphize these massive AI models as these ultra-efficient, lightning-fast digital brains.

0:41Sure, because they output text so fast. Right. But under the hood, though, their reading process is completely exhausting. It represents pretty much the exact opposite of how human cognition actually processes continuous information. Which is crazy to think about. And that brings us to our mission for today. So welcome to the Deep Drive, everyone. Today we are exploring how artificial intelligence handles just massive amounts of data, and we're looking at a groundbreaking protocol called declarative attention that fundamentally rewrites this reading process. It's a huge shift. It really is. So our goal today is to break down how getting an AI to explicitly think out loud about its own attention span can drastically cut computational costs.

1:23We are talking about making the tools you use every day vastly faster and structurally much closer to human reasoning. Yeah, that underlying mechanics shift is going to affect, honestly, anyone who relies on these systems for heavy lifting. For sure. So whether you're asking an AI to generate summaries of massive, sprawling financial reports, or having it analyze whole software code bases, or I mean, if you're just incredibly tired of sitting there watching a loading spinner when you upload a big PDF, understanding this specific memory bottleneck is the key to the next generation of AI efficiency.

1:56Because if we connect this to the bigger picture, The AI industry is currently hitting a very real wall. A wall of intelligence, right. That's what people think. Exactly. People assume we just need smarter models. But in reality, it is a wall of memory retrieval. So to understand why declarative attention is such a massive breakthrough, we kind of have to look at the breaking point of the current hardware architecture. Yeah, I want to dig into that textbook reading problem because it feels like there is a specific technical hurdle causing it. So in the field, this is referred to as the KV cash bottleneck.

2:31Let's establish what that actually means. KV stands for key value, correct? Yeah, that is the foundation of it. So when a language model generates a response, it utilizes what's called an attention mechanism. Right. For every new word, or, well, more accurately, every token it generates, the neural network needs to calculate how much mathematical attention to pay to every single token that came before it in the conversation. So it's constantly looking backwards. Always. And to do this without recalculating the vector math for the entire conversation from scratch every microsecond, it stores a mathematical representation of all those previous words in memory.

3:06Okay, got it. And that stored representation is the key value, or KV cache. But it doesn't just store it and leave it there, right? It has to physically move that data around. Like it has to load that entire cache into its active processing core for every single decoding step. Yes, every single step. Let's quantify that so the scale actually makes sense. Please do, because I think people underestimate how big this is. Right. So if you have a one million token conversation, which is actually becoming increasingly common with these massive long context models analyzing books or entire code repositories, the AI must load roughly 15 gigabytes of KVCache memory into its active chip.

3:47Wow, 15 gigabytes. Just to generate one single token. One token. That's unbelievable. And then to generate the very next token, it loads those 15 gigabytes all over again, plus the new token it just created. So mathematically, it's what you would call an order n computation time per step. Like as the context gets longer, the computational cost of generating every single subsequent word scales linearly with it. It just gets heavier and heavier until the system slows to an absolute crawl. Yeah. And this fundamentally shifts where the true bottleneck in AI hardware lives. because we often think AI is constrained by how fast it can do math, you know, the raw compute power of the GPU.

4:23Right, the processing speed. But on modern hardware, this exhaustive KV cache mechanism actually saturates the memory bandwidth long before the compute power runs out. Oh, wow. Yeah, the AI isn't struggling to think. It's choking on its own memory retrieval. It is spending the vast majority of its time just moving data back and forth across the silicon. That sounds so inefficient. So previous fixes to this problem, trying to optimize this bottleneck, I imagine them like having an assistant frantically scan my entire overflowing library to guess what book I need next. That's a good analogy, yeah.

4:58Because there were systems that used fixed rules, right? Like only remember the most recent thousand words where they use these lightweight mathematical scans to guess what the AI might need. Yeah. They tried a lot of things like that. But that still seems incredibly inefficient because this assistant is still scanning the whole context every time just in a slightly faster way. Why are we guessing? Like, why doesn't the AI just tell the hardware what it's looking for? Well, for a long time, the assumption was just that we couldn't ask it. Really? We just assumed it couldn't tell us. Pretty much.

5:27We thought an AI's attention was just this chaotic swirl of deep neural network activations that was completely unpredictable until the exact moment the math was calculated. The prevailing theory was that the AI couldn't possibly know what it needed until it was already in the middle of processing it. But it turns out it can. And that is exactly what tees up declarative attention. The AI absolutely can tell the hardware what it needs if we just force it to articulate its thought process first. Exactly. Declarative attention, or DA, is a protocol that totally changes the paradigm by forcing the AI to emit parsable text tags right there in its chain of thought.

6:06Like leaving a note for the system. Yes. It explicitly states where it wants to look in the document before it actually attempts to look there. I want to break down the mechanics of how it does this because it partitions the AI's generation into three distinct modes. First, we have what is called the global mode. Right. The global mode. It's written out literally as a global tag in the generated text with the little angle brackets. OK. Like HTML tags. Exactly like that. And this serves as the default mode for navigation. So in this mode, the AI is surveying the full context, paying that heavy memory cost we just talked about, but it's only doing it to find relevance.

6:43It's figuring out where to zoom in. Yeah, it's essentially scanning the table of contents rather than reading the actual chapters. Okay, so once it identifies what it needs, it switches to the second mode, which is focus. It outputs a focus tag. And in this mode, the AI attends only to a named specific region of the text to extract exact facts. Right. And it just drops the rest of the document from its memory load entirely. Completely gores it. And then finally, we have the local mode. In local tag, the AI tends to none of the source context. Wait, none of it? None of it. It drops the source document entirely from its active memory.

7:18It only looks at its own recent output to synthesize the final answer. Hold on. You lost me there. If it drops the source document entirely in local mode, how does it write an accurate summary? isn't that just a recipe for hallucination? like if it can't see the text what is it basing its answer on? I get why that sounds risky but it relies entirely on the facts it just extracted during the focus mode think of it like taking notes so in global mode you find the right page in focus mode you read the paragraph and drop down the three key stats on your notepad then in local mode you close the massive textbook put it away and write your final essay paragraph using only the stats on your notepad Okay, that makes total sense.

8:00The AI is doing exactly that. It's synthesizing an answer based on its own immediate chain of thought rather than the raw data. Okay, I see how that drastically reduces the memory load. It's literally thinking out loud about its own attention span. I mean, we could compare it to putting blinders on a racehorse so it focuses only on the track ahead. I like that analogy a lot, but we kind of have to push it a step further. Because in this case, the horse is smart enough to know when to put the blinders on itself and when to take them off to survey the field. That is a very smart horse. It really is.

8:31So the AI leaves this trail of text breadcrumbs. And the software actually running the model, the inference engine, like VLLM, which acts as sort of the traffic cop for the AI's operations. It reads these text tags almost like software tool calls. So when the engine sees a focus tag, it dynamically updates the attention mask at that exact decoding step. It simply skips reading the irrelevant parts of the KV cache entirely from the physical hardware. Like it doesn't load them. It doesn't process them. It's saving an immense amount of memory bandwidth by doing that. Yeah. Massive amount. OK. But if the AI is declaring where to look using the focus tag, how does it know how to ask for specific parts of a massive document?

9:14Ah, that's the tricky part. Because if I upload a hundred thousand word legal contract, the AI can't just type out, you know, focus on the part about the indemnification clause. the inference engine needs a specific hard-coded address to know what memory to load right. Yes, it does. And that is where the concept of magic chunks comes into play. Magic chunks. I love that name. It's very catchy. To make focus mode work, the software sitting between the user and the AI takes the massive input text and splits it up into addressable segments of about 2 ,048 tokens each. Are these just arbitrary slices, like cutting a sentence in half just because it hit the token limit?

9:54No, they actually avoid doing that because breaking a sentence in half destroys the semantic meaning. The system uses natural boundaries. Like what? It looks for paragraph breaks or if those aren't available, sentence ends or clause breaks. So the chunks are semantically neat and self-contained. But how does the neural network inside the AI actually know these chunks exist? It's not like the AI has a built-in file explorer, you know. Right. It doesn't. And the way this is delivered to the model is through a simulated tool use transcript, which I just think is incredibly clever. It's so cool. It is a phenomenal formatting trick.

10:29Before the AI even starts generating a response to your prompt, the system formats the huge document to look like a chat history. It pretends that the AI has already used a digital tool called get underscore magic underscore chunk. So the AI reads its prompt and basically sees a fake history. It reads something like, user asked me to analyze this contract. I use my tool, get magic chunk one, and here's what it returns. Yep, exactly. Then it sees chunk two, chunk three, and so on. We're basically tricking the AI into thinking it used a search tool to retrieve the document piece by piece in the past, even though the whole document is just sitting right there in the prompt right now.

11:07Yeah, we absolutely are. And it leverages the very specific strength of modern models. Which is. These AI are already heavily trained to understand user assistant dialogue and to format tool use requests. Right. They do that all the time. Exactly. So by wrapping the document in this simulated tool call history, the AI naturally understands how to refer back to Magic Chunk 5 because it thinks it retrieved that chunk itself earlier in the conversation. That is just a brilliant psychological hack for neural network. What stands out to me here, though, is the cost implication, because this entire protocol works zero shot on off the shelf models.

11:45Yes, it does. And we need to clarify what zero shot means in this context, because the savings are staggering. We are not talking about taking a model offline, spinning up a cluster of GPUs for a month, and spending millions of dollars to retrain its neural pathways to understand declarative attention. No zero fine-tuning is required. You don't update a single weight in the model. You just use standard text prompts to guide existing models to organize their reasoning using these tags. This raises an important question, though, which is how this entirely software-level text-based prompting trick integrates with the hardcore low-level hardware systems that run AI.

12:21My physical layer. Because usually if you mess with how an AI reads text at the top level, you break the highly optimized math running on the GPU at the bottom level. You'd think so, but the integration is surprisingly frictionless. Really? Yeah, because the system applies this memory mask at a block level, meaning it chunks the memory into predictable 2048 token sizes. Existing highly optimized compute kernels like flash attention can run completely unmodified. Let's define flash attention for the listener, just so we are clear on why that matters. Sure. Flash attention is essentially the underlying mathematical shortcut that runs on the GPU.

12:59It is the engine block that makes fast AI possible right now. It's the standard. It is. So if a new protocol requires rewriting flash attention, it faces a massive uphill battle for adoption. But because declarative attention just tells the existing system, hey, ignore these specific blocks, flash attention doesn't need to be rewritten. It just suddenly has less memory work to do at each step. Exactly. It just gets a break. Okay. That sounds incredibly clever in theory. It's a very elegant software trick. But if the AI is literally ignoring half of a legal document to save time, surely it's missing crucial context and getting the answers wrong.

13:35Right. You would definitely worry about that. Like if you put blinders on a horse, it might run faster, but does it still know how to navigate the track? Let's look at the actual data because this was tested across 15 different long context tasks. They really threw everything at it. document QA, analyzing massive code repositories, synthesizing financial reports, and the numbers are striking. Walk us through them. Let's look at the Gemma 431B model, which is a very capable 31 billion parameter model. By using declarative attention, it reduced the total attended tokens by 52%. 52 %? That is more than half of the memory reading work completely eliminated.

14:15And the impact on accuracy, because there has to be a trade-off. A tiny 1.27 percentage point drop. Wait, really? You cut the workload in half and basically lose zero performance? Pretty much. It's essentially noise at that level. Did this hold up on other architectures? It did. On the Quinn 3.627B model, it saved 31.1 % of the tokens with a modest 2.75 percentage point drop in accuracy. Still totally acceptable for that much memory saving. Yeah, absolutely. But the way this gets truly exciting is how it scales. When you look at massive context-length tasks involving over 128 ,000 tokens, the absolute numbers become astronomical.

14:57Oh, I bet. At that scale, declarative attention saves up to 21 million tokens per response. Just to visualize that for a second. 21 million tokens it does not have to load into active memory, pass through the GPU, and calculate attention for? Yeah, it's a massive load. And when you project that massive reduction out onto optimized serving hardware, the estimated decode wall clock time, meaning the actual real world time you spend sitting at your desk waiting for the answer, shrinks to 0.71x of standard runs on the Gemma model. Wow. You're getting nearly 30 % speed up in actual wait time just by changing the text prompt to make the AI think out loud about its memory.

15:31That is incredible. I am looking at the data for the smaller models, though, and it seems like there is a significant catch here. There is a catch, yeah. The model has to actually possess the base intelligence to follow this very structured, rigid protocol. Right. What's fascinating here is a quirk about scaling capability. This zero-shot protocol only works if the AI is smart enough to handle the overhead of self-management. So what happens if it isn't? Well, they tested this on a much smaller model, the Gemma 4E4B model, and it suffered a catastrophic accuracy collapse. Ouch. Did it just fail to find the right information, or did the mechanics actually break down?

16:10The mechanics broke down. It only managed to format the focus tags correctly 58 % of the time. Oh, so it was just making syntax errors. Exactly. It would try to call for a magic chunk but mess up the basic syntax, or it would open a tag and completely forget to close it. It's a complex protocol. Yeah, it sounds like a lot to balance. It is. The AI has to balance managing this rigid memory system while simultaneously trying to answer a difficult question. And the smaller model just lacked the underlying reasoning capacity to juggle both tasks at once. The big models could handle it. Oh, yeah. The 31 billion parameter models executed the tags perfectly almost 100 % of the time.

16:50So it's a bit like giving a highly complex structured filing system to a toddler. They don't know what to do with the folders. They lose their notes. The whole thing just falls apart. That's a perfect way to look at it. But if you hand that exact same filing system to a college student, the 31 billion parameter model, they instantly optimize their workflow. Right. They know how to use the tools. Yeah, they know exactly how to say, I'm going to look at the index, find chapter four, read only chapter four and write my summary. The smarter the model, the better it is at actively ignoring what it doesn't need.

17:21So what does this all mean for the future of scaling AI? Because historically, the assumption in the industry has been that bigger models are inherently slower and much more expensive to run simply because they have more parameters to calculate. Right. Bigger means slower. But declarative attention challenges that assumption entirely. It shows us that smarter, larger models can actually be prompted to be vastly more efficient than we originally thought precisely because they possess the intelligence to manage their own memory overhead. That is such a fascinating paradigm shift. Let's recap the core mechanics we've unpacked today.

17:55We explored how declarative attention prevents an AI from exhaustively rereading every single piece of data for every single word it generates. By partitioning its thought process into a global navigation mode for skimming a focus extraction mode for reading and a local synthesis mode for writing, the AI learns to manage its own attention span. Exactly. And by using simulated tool transcripts to prompt off-the-shelf models to declare their attention with those magic chunks, we can bypass the massive hardware bottleneck of the KV cache. It's brilliant. We cut computational overhead by up to 52 % with minimal accuracy loss.

18:32And we do it without having to rewrite the underlying GPU kernels like flash attention. Bringing this back to your daily life, you know the person listening right now. As we move into an era of agentic workflows, where AI isn't just generating a quick email for you, but is actively fetching tool calls, reading entire code repositories, keeping conversations going for days and holding context perpetually, this exact type of memory efficiency is what will keep wait times down. It's absolutely essential. It is the structural breakthrough that will allow these massive complex AI tasks to run locally and affordably on standard hardware instead of requiring billion-dollar supercomputers for every single query.

19:13Yeah, it paves the way for AI to exist perpetually in the background, analyzing immense streams of data, because it finally possesses a mechanism to only pay attention to what actually matters in the moment. Which leaves us with a final lingering thought for you to chew on. We started by talking about the mechanical insanity of rereading a 500-page textbook from page one just to write a single word. Which we agreed was miserable. Terribly miserable. And declarative attention fixes that by teaching the AI to consciously put on blinders and manage its focus. But if an AI can be prompted to consciously control its own attention span, what other human-like metacognitive skills could we unlock just by asking it to think out loud?

19:50That is the million-dollar question. Right. Like, could we teach an AI to actively doubt the reliability of its own memory or decide to permanently forget irrelevant or traumatic data points from its context window to maintain its operational stability? If it can control its attention, maybe it can control its own state of mind. It's a truly fascinating frontier for the next generation of models. Definitely something to mull over the next time you watch an AI generating text on your screen. Thanks for joining us on this deep dive.