Beyond the Transformer: Titans, MIRAS, and the Future of Infinite Context

0:00Welcome to the Deep Dive. This is where we give you the essential shortcut to understanding the most fundamental shifts happening in AI research right now. And today we are tackling something that is, well, it's arguably the defining technical challenge for the future of AI. We're talking about memory or more precisely, the lack of truly effective, massive scale, long term memory in our models. Exactly. If you follow large language models at all, you know, they're they're phenomenal at short term context. No, absolutely. You know, what they just read in the last 10 ,000 or maybe even 100 ,000 tokens.

0:35But the second you need to go bigger. That's where it all breaks down. Right. When the context window needs to expand to handle, say, an entire library of documents or a massive genomic sequence or a data stream that spans years. All the mechanics that powered this AI revolution, they just they start to break down spectacularly. And that's what we're here to talk about. We are. We're diving into some really groundbreaking sources today. It's a collection of research papers and reports from Google Research, and they introduced two interconnected breakthroughs. First, there's the Titan's architecture.

1:07And second, the Mycorrhiz framework. And this isn't just, you know, a minor optimization. This is a foundational reimagining of how an AI remembers, how it learns, and how it processes these incredibly long sequences of data. That's absolutely right. Right. I think our mission in this deep dive is to really clearly lay out how this research gets over that seemingly insurmountable computational law. The scaling problem. The scaling problem. How do you process these enormous, messy, real-world data streams? The goal here isn't just about making it faster. It's not just efficiency. No. It's about combining that efficiency with the kind of deep recall and the expressive power that you expect from a world-class model, but doing it at an absolutely exponential scale.

1:52So these two things, titans and mirais, they're tightly coupled. Very tightly. You can think of it this way. The titans architecture is the specific functional tool. It's the design, the biologically inspired model built on deep neural networks. And miras. Miras is the overarching theoretical blueprint. It's a unifying theory for sequence modeling itself. And crucially, both of them are completely focused on advancing this one core concept. Test time memorization. Tip time memorization. That phrase is, it's really the fundamental concept here. These systems allow an AI model to fundamentally update and integrate new knowledge into its core structure while it's actively running.

2:34Right, while it's processing new data during inference. And this is the key. It does it without needing those traditional, incredibly expensive, dedicated offline retraining cycles that we all rely on right now. It's real time. It's dynamic adaptation. It's a game changer. Okay, let's unpack this. To really get the solution, I think we have to deeply understand the problem. So let's start with the status quo, the mighty transformer. The transformer. So the core of that architecture, the thing that made everything else possible, is the attention mechanism. Right. It completely revolutionized sequence modeling because unlike older models that had to process data one piece at a time sequentially, attention lets the model look back at all the earlier inputs in a given context.

3:15All at once. All at once. And it decides which of those pieces are the most important for making the current prediction. It calculates dependencies across, you know, vast distances within that context window. And that grasp of non-local relationships, that's why LLM sounds so intelligent, right? They're not just predicting the next word based on the last five words. No, not at all. They're synthesizing the meaning of the current sentence based on something that was said three paragraphs ago. It's the mechanism of superior in-context learning. Exactly. But, and this is the big but, that phenomenal power comes with an immediate and frankly catastrophic computational bottleneck as soon as you try to scale it.

3:54Okay, let's get into that. Because attention has to calculate the relationship between every single token and every single preceding token in the sequence, the resource consumption just explodes. We're talking about computation time, FLOPs, and memory. It increases drastically with the sequence length. Can you walk us through what that increase actually looks like? Because I think people hear the term ON2, complexity or quadratic scaling, but it's hard to really grasp the severity of that in practice. Sure. Think of it this way. N is just the number of tokens you're feeding into the model. The length of the input.

4:28The length of the input. Right. If you're using standard transformer attention, the computation grows with N squared. So if N is, say, 10 ,000 tokens, N squared is 100 million computations. Which is manageable. Manageable for modern hardware, sure. But what if N is 100 ,000 tokens, which is still a relatively short document? N squared jumps to 10 billion computations. And if you try to feed the model a million tokens, which you absolutely need for a full legal document or scientific paper, you're now talking about 1 trillion computations. A trillion, except for the attention layer. Just for the attention layer.

5:03And that right there is the ultimate blocker. Quadratic scaling means if you double the length of your input sequence, the computation doesn't just double. Not even close. It doesn't triple quadruples. And that just makes scaling for these massive data streams economically and technically. Yeah. Well, it's just unfeasible for a standard, dense attention transformer. And this is so frustrating because the need for long context is absolutely immense. We need full document understanding where your input sequences are in the millions of tokens. We need long-term time series forecasting for, say, financial modeling or climate modeling.

5:41And maybe the most demanding application of all is complex genomic analysis. Modeling DNA requires handling these unbelievably long, incredibly detailed sequences with perfect accuracy. And ON2 just makes all of those tasks a non-starter. A complete non-starter. So the research community, of course, they didn't just sit still. We've seen a lot of alternatives trying to solve this tradeoff between expressive power and efficiency. The move toward linear recurrent models was very popular, wasn't it? It was, and it produced some really exciting models. The key idea was to move away from that dense all-to-all attention and go toward more efficient linear recurrent neural networks, or RNNs, and also state space models, or SSMs.

6:23And the most notable recent breakthrough there would be Mamba 2. Mamba 2, exactly. And these models are incredibly fast. They achieve linear scaling, ON. So if you double the input length, the computation just doubles. Well, it just doubles, which is a massive, massive efficiency win. Okay, so here's a question then. If Mamba is linear, which is so much faster, why couldn't the researchers just use a much, much larger fixed vector in Mamba to capture more detail? What's the fundamental constraint that stops a huge fixed vector from having the same expressivity as a transformer's attention. That is the critical and maybe non-obvious point.

6:59It isn't just about the size of the memory. It's about how the information is encoded. Okay. The incredible efficiency of these linear models is achieved by compressing the entire history of the input sequence into a single fixed-size hidden state or vector. A single vector for everything that's come before. Everything. And even if you make that vector enormous, the model is still fundamentally linear in how it relates the past to the present. So the argument in the source material is that linear models, it doesn't matter what the vector size is, they fundamentally struggle to capture the rich, detailed, and highly non-linear abstracted relationships that define these complex, long sequences.

7:38So the fixed size compression is the fatal flaw. It's the fatal flaw. It's like trying to summarize a library full of dense, multi-layered philosophical books onto a single, giant, but ultimately linear note card. You get the speed, but you lose the expressive power you need for that deep, long context recall and synthesis. So for years, we've been stuck in this impossible tradeoff. You have transformers, which are expressive but scale quadratically and are just too slow for millions of tokens. Right. And then you have RNMs and SSMs, which are fast. They have that beautiful linear scaling. But they lack the memory richness and expressive power needed for really deep, really complex, long context.

8:18Precisely. And that, that is the genesis of the Titans of Marius approach. The researchers sought to combine the speed of recurrence, that linear ON scaling, with the deep abstraction and accuracy of attention-based models. And they did it by completely changing the architecture of memory itself. By moving beyond that single fixed size vector. Okay, let's dive into that. The core structure of Titans, the design philosophy here is just brilliant because it's rooted in biology and psychology. It is. It mirrors how the human brain works with its distinct interconnected memory modules. Right. You have short-term memory, working memory, long-term memory.

8:58The model doesn't just treat memory as one big monolithic system. And the research really meticulously breaks down the system into these functional components. This is especially clear in the MSC variant, which stands for memory as a context, and it proved to be highly effective. So what's the first component? First, you have the short-term memory. This is handled by the model's core processing stream. In the hybrid Titans variants, this is usually implemented with attention mechanisms. But not the full expense of attention. No, no. It's often constrained to a local sliding window attention, SWA, or a very limited context.

9:31This part of the system is really designed for that precise momentary recall. what just happened in the immediate vicinity of the current token. It's the model's working memory. And then we get to the really revolutionary piece, the long-term memory module. And this is the core innovation. This is what breaks that fixed-size state bottleneck we were just talking about. So instead of compressing history into a simple fixed vector or a matrix. The Titan's long-term memory is implemented as a deep neural network, specifically a multilayer perceptron, or MLP. So it's not just a storage unit? It is not just a storage unit.

10:05It is a processing unit. Why is that depth, that multilayer perceptron, so crucial compared to just having a giant vector? What does that get you? Well, the source material gives a really compelling theoretical argument for this. Deep memory modules are mathematically and theoretically just more expressive than linear or shallow models. A traditional fixed vector or a matrix-valued memory, it, it basically assumes that all the underlying dependencies in the historical data are linear. And that is a massive, massive simplification. Especially for something complex like DNA or a legal document. Exactly.

10:41Deep neural networks, on the other hand, are capable of universal approximation. They are strictly more expressive than linear models because they can encode the highly nonlinear abstraction and synthesis of past history. So it's about storing conceptual summaries, connecting distant ideas, synthesizing patterns. Not just storing raw keys and values. So a shallow memory might remember the dates of two separate events, but a deep memory could synthesize the relationship and the implications of those two events, even if they're a thousand tokens apart. Precisely. And the research really validated this through detailed ablation studies.

11:16They compared long-term memory modules that had the same capacity but different depths. What did they find? The modules with greater memory depth, specifically two or more layers, showed significantly lower perplexity in language modeling. And they demonstrated much better scaling properties as the sequence link moved into that extreme long context regime. So the depth isn't a luxury. It's not a luxury. It's essential for long-term competence. Okay, then there's the third component, persistent memory. This one feels a little different. It feels more like the model's skill set, separate from the actual running context of the sequence.

11:52That is a great way to put it. The persistent memory is a set of learnable parameters, but they are data independent. Meaning they're fixed during inference or test time? Exactly. This component is dedicated to encoding general task knowledge, the underlying rules, the grammar, the meta information about how a task should be performed. So if the long-term neural memory is fluid and contextual, it's always updating based on the input stream. The persistent memory provides the stable, input-independent foundation of the task itself. It allows the model to leverage these general learned skills without that knowledge being constantly overwritten or lost by the massive influx of new sequence data.

12:34Okay, so we have the three systems. We've got short-term attention, the deep long-term MLP, and the fixed persistent knowledge. Now we get to the dynamic part, the active learning. How does the model efficiently decide what is worth committing to that precious long-term storage, especially when it's processing millions of tokens a second? This part is arguably the most clever piece of the whole puzzle. It basically translates a human psychological phenomenon, how we prioritize what we remember, into rigorous mathematics. Right. The system is designed not to just passively log every single token.

13:08It has to actively decide what constitutes important information. And the research terms this the surprise metric. I love this concept. We remember the unexpected and we forget the routine. If my morning commute is exactly the same every single day, I don't retain those specific details. But if I see an ostrich crossing the road, that memory is locked in instantly. Exactly. The AI needs a trigger for that. And that's the perfect analogy. The AI's trigger for surprise is a mathematical measure of anomaly. The surprise metric is calculated by detecting a large difference between what the long-term memory currently expects based on its history.

13:43And what the new input is actually telling it. Precisely. And mathematically, this anomaly is measured by the gradient. It's the internal error signal of the neural network with respect to the input in what's called the associative memory loss function. So a large gradient means high surprise. A large gradient means the model's current expectation failed spectacularly, which signals a very high surprise. Okay, let's make that concrete. So if I'm feeding the model a thousand-page legal contract, and it expects the next input to be another legal clause, and it is. Then the memory's expectation aligns with reality.

14:17You get a low gradient, low surprise. So it skips permanent storage. Right. Why waste storage space on something that's routine continuation? But imagine that contract is being analyzed for sentiment and risk. Suddenly, in the middle of a really dense section about asset distribution, a new input token refers to an extreme unexpected change in market conditions. A sudden deep economic collapse, for example. That input dramatically violates the memory's current expectation for how the sequence should flow. The gradient, the surprise, is very, very high. And that anomaly gets prioritized. It's immediately prioritized for deep permanent storage because that high gradient is a signal that says this breaks the context, this is unexpected, and therefore this is vital information for future recall.

15:05And the selective update process has to be essential for efficiency. You can't just store millions of tokens in a dynamic high expressivity module without some kind of really rigorous gatekeeping. Absolutely not. And that selective process is refined even further by two crucial mechanisms that they borrowed from advanced optimization theory. They integrated them right into the recurrent update rule. And those are? Momentum and forgetting. Okay, let's start with momentum. So the update rule for the memory state, let's call it MT, is analogous to gradient descent with momentum. It doesn't just look at the momentary surprise, which is the current inputs gradient, GT.

15:41It also incorporates the past surprise or the momentum history, ST-1. Why is incorporating that past surprise so important? I mean, if the model just registered a high surprise event, shouldn't it ignore subsequent events that are related but maybe less surprising on their own? This is exactly where momentum shines. Let's go back to your banana peel anomaly from earlier. OK, in the middle of a financial report. Right. That's the high surprise event. The immediate next tokens might be something like warning, stop or hazard. Now, individually, those are relatively common words. So they'd have low individual surprise.

16:16Exactly. However, because the memory system just experienced that massive shock from the banana peel, the momentum turn carries the residual effect of that initial high gradient. This ensures that the relevant subsequent tokens that follow the big event also get captured and recorded into long-term memory. Ah, so it allows the model to memorize the entire related time frame of an event. Right. It prevents it from just immediately relaxing and forgetting the context that surrounded the anomaly. It's incredibly clever. The momentum term basically acts as a dynamic memory of surprise across time, and it's controlled by a data-dependent decay term.

16:53So if the context changes abruptly, say the document shifts from finance to poetry, that decay term can drop to near zero, and it effectively kills the prior momentum and prepares the memory for a whole new context flow. But if the tokens are still highly relevant, it stays high and maintains the focus. Exactly. That data dependency is what makes the whole mechanism adaptive. So the system isn't just reacting to a single moment. It's remembering the whole chain of events around the anomaly. Yeah. Okay, so what about the other mechanism, forgetting? I assume managing finite capacity means the model has to be pretty ruthless about discarding old, irrelevant information.

17:32It has to be. The model employs an adaptive weight decay mechanism, which acts as the forgetting gate. In any long recurrent memory system, especially one dealing with millions and millions of tokens, uncontrolled storage, just leads to saturation and degradation. The memory has to have a way to actively prune or clear out old information that's no longer useful. So this is the cleanup crew. The cleanup crew, exactly. This weight decay, or forgetting gate, manages the limited capacity of the neural memory. If a piece of information has proven to be irrelevant over a long sequence, the adaptive decay allows those memory parameters to shrink or clear out entirely.

18:10Making space for new, more surprising information. Right. And this concept is a generalization of the forget gates that you find in a lot of modern, efficient, recurrent models like Momba. The ablation studies confirm that the performance gains they got from this forgetting gate are just as important as the gains they got from the surprise-based remembering. Okay, but the technical efficiency point here is really important. Gradient calculations, especially for recurrence, sound computationally intensive. How do they make sure that integrating this deep, nonlinear, momentum-driven memory update stays fast and parallelizable?

18:44How do they keep that linear scaling advantage? They use some highly sophisticated techniques to tensorize the process. The source notes that by reformulating how the weights are calculated in the inner loop, they could use only massive matrix multiplication or mat-mull operations and sums. These are things GPUs are really good at. Things GPUs are incredibly good at, and crucially, they treated that momentum term, STT, as a linear recurrence that can be computed very quickly using parallel associative scan operations. That's the key. That is the key. By transforming what is an inherently sequential recurrent calculation into a parallelized matrix operation, they completely overcome the speed barrier.

19:24They maintain fast linear inference speeds, effectively getting the ON speed of an RNN while enabling the deep expressivity of that MLP memory. Okay, so now we need to make a theoretical leap. If Titans is the specific machine that's built to solve this memory problem. Then Miras is the theoretical foundation. The unified theory. The thing that proves that all major sequence models, past and present, are really just different ways of building a complex associative memory module. And this is a radical, really powerful paradigm shift. Mirus gives us this unified framework where we can stop seeing transformers and RNNs and SSMs as these totally disparate architectures.

20:04Right. Instead, we see them as just variations on the same fundamental optimization problem. Which is? How to efficiently integrate new information with old memories while preserving the essential concepts. And within this new framework, Mirus defines any sequence model through four sort of concrete design choices. Can you detail what those are? Sure. They're the fundamental knobs that you can turn on any memory-based system. First is the memory architecture. What is the actual structure that's storing the information? Is it a fixed vector, which is shallow? Is it a matrix? Or is it the deep nonlinear MLP that they use in Titans?

20:39Okay, that's number one. Number two is the attentional bias. This is the internal learning objective the model is optimizing for. It determines what the model prioritizes when it's integrating new data. Third. Third is the retention gate. This is the memory regularizer. Merace formally reinterprets mechanisms like weight decay or those forgetting gates we talked about as specific forms of regularization. They're just balancing the need for fast new learning against the stability of retaining past knowledge. And the last one. The last one is the memory algorithm. The specific optimization algorithm that's used to update the memory.

21:17Is it gradient based? Is it using specialized recurrent equations? That's the fourth choice. The power of this unified view, then, is that once you categorize all models this way, you immediately spot a massive shared limitation in, well, virtually all successful existing architectures. And that limitation is their over-reliance on mean squared error, MSE, or just simple dot product similarity for both the attentional bias and the retention objective. The Euclidean approach. A completely Euclidean approach to measuring memory loss and similarity. And why is relying on that Euclidean distance, or MSE, a restriction on the model's expressive power?

21:56Well, MSE applies a really steep squared penalty for any error. This means that if you have a massive data set and you get a single major outlier, a black swan data point or a big anomaly like our banana peel example, the MSE penalty is so high that the model is forced to overreact. It has to excessively skew its entire memory update just to accommodate that one single outlier. Which in massive, messy, real-world data sets must make the models really brittle. It makes them brittle and very sensitive to noise. It prevents the memory from forming stable, high-level abstractions. So Meris transcends this limitation.

22:32It provides a blueprint for designing memory systems that use non-Euclidean objectives and regularization, things that are informed by decades of advanced statistical and optimization research. Exactly. And the researchers used the Marist framework to design and test new variants that explicitly move beyond that standard MSE paradigm. They created three specific attention-free models just to demonstrate the flexibility of this theoretical leap. Okay, let's start with the first one, YAD. This one was designed to make the memory more robust to those messy data streams, right? That's right. YAD uses what's called Huber loss instead of the steep squared penalty of MSE.

23:08So mathematically, MSE penalizes errors quadratically. If the error is 10, the penalty is 100. Huber loss, on the other hand, is only quadratic for small errors. For large errors, for outliers, it switches to a much gentler linear penalty. So if the error is 10, the penalty might only be 10 or 20, depending on the threshold you set. And that gentler penalty is crucial when you're processing these colossal, messy documents where outliers are just. They're common. They're everywhere. A single typo, a minor anomaly, a surprising but irrelevant phrase. These things shouldn't destabilize the entire memory.

23:45And WIAD prevents the memory from catastrophically overreacting to that noise, which leads to much more robust memory updates. Okay, then we have Moneta. What was the design goal for this one? Moneta explored stability through mathematical strictness. It used really complex and strict mathematical penalties or generalized norms for both the attendance mechanism, what to pay attention to, and the forgetting mechanism, the retention gate. So the idea was to impose really disciplined, rigid rules on how the memory should be updated. Right, with the hypothesis that this would lead to a fundamentally more stable, powerful, and mathematically constrained long-term memory system.

24:22And finally, MEMRA, which focused directly on memory stability. MEMRA tackled stability by forcing the memory to operate like a strict probability map. And this is a very, very tight constraint. Every single time the memory state is updated, Memora ensures the changes are controlled and balanced. It guarantees a clean, stable, and theoretically controlled process for integrating new information. So it prevents memory drift. It prevents memory drift and ensures that the long-term context stays coherent, even after millions of updates. And the fact that these non-MSE variants actually delivered improved performance compared to the recurrent baselines, that proves that the Miras framework is more than just a theory.