Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddings

0:00Welcome to the Deep Dive. So if you've been anywhere near the world of large language models, you know there's this one challenge that comes up again and again. It's the most frustrating bottleneck, really. It is. It's the context wall. Exactly. It's like having this genius-level AI that can reason and write beautifully, but then you ask it to remember a specific detail from, I don't know, page 50 of a document you just gave it, and poof, it's gone. It's like talking to an expert who is brilliant for about five minutes and then completely forgets the beginning of the conversation. These huge transformer models, they work so well inside their little box, their pre-trained context length.

0:41But the moment you step outside that box... This performance just falls off a cliff. It's not a gentle slope down. Not at all. And what's fascinating is that this huge bottleneck is tied directly to one of the most critical parts of the architecture, the very thing that gives the model a sense of order. We're talking about rotary positional embeddings or ROPEE. Right, ROPEE. And here's the paradox we're going to get into today. Right. ROPEE is absolutely essential for making the training process efficient and scalable. You need it. You absolutely need it. But it's also the very thing that becomes this immovable barrier to generalizing to longer contexts.

1:15So our mission today is to really unpack that tension. And we're going to explore a surprisingly elegant new solution called DROPEE that's dropping positional embeddings. And the whole idea is to treat this positional information not as a permanent fixture, but as like a temporary tool. Exactly. Think of them as training wheels. They're essential to get you going, but at some point you have to take them off to really fly. Okay. So to get why that's such a big deal, we have to start at the beginning. Why do we even need these things, these positional embeddings, in the first place? Well, because the core of a transformer, the attention layer, is blind to order.

1:51Blind. Completely. If you feed it the dog chased the cat versus the cat chased the dog, without some kind of position marker, all the model sees is a jumble of words, a bag of tokens. The meaning is lost. So the positional embeddings, the PEs, they inject that crucial sense of sequence. They tell the model where each word is in relation to every other word. Precisely. And this gets us to the first big observation. PEs, like rope, are absolutely critical for efficient learning from the get-go. We can think of them as, well, as scaffolding. Scaffolding, I like that. It's an inductive bias. It's like giving the model a cheat sheet for structure, which gives it a massive head start.

2:32So what's the proof? What happens if you just don't use them? If you try to train what the researchers call a no-PE model, one with no positional embeddings. Oh, the difference is night and day. The data shows models trained with rope learn much, much faster. They converge better. Their quality is higher throughout the whole process. Why? What's going on under the hood? It comes down to something called the uniformity problem. Okay. Break that down. Imagine you give a NOPE model a really weird input, like the word apple repeated a thousand times. Since there's no external way for the model to tell the difference between the first apple and the 500th apple, its internal attention just becomes uniform.

3:12It can't differentiate. So the signals it needs to learn about structure, the gradients, they just vanish. They vanish. The learning just grinds to a halt. Rope E breaks that uniformity right away. It provides the coordinates, so key patterns like something called diagonal attention, where a token pays attention to itself, they form almost immediately. With no P, the model is just wandering in the dark. So we've established rope E is the accelerator. It's the scaffolding that gets the building up quickly and safely. But, and this is the main event, when does that scaffolding turn into a prison?

3:46The moment you try to build a new floor, the second you try to give it an input that's longer than the context length it was trained on, that's the extrapolation barrier. So the math just stops working. The math itself, the rotations in rope get pushed into a space the model has never seen before. It becomes out of distribution and the outputs become gibberish. And people have been trying to fix this for years, right? Yeah. That's where things like yarn and PI come in, these frequency scaling methods. Right. And the best way to think about them is like trying to stretch a photograph. You've got this photo that's, say, 4x6, and you need it to fit an 8x10 frame.

4:20So you just stretch it. You stretch it. You warp the underlying frequencies to make the old positional information fit the new, larger space. But that stretching comes at a cost. And what the research shows is that these methods, despite being popular, are actually pretty bad at zero-shot generalization. They fail at the one thing they're supposed to do. Why? It's a really subtle point, but it boils down to two different kinds of attention happening inside the model, which operate on different frequencies. High frequencies and low frequencies. Exactly. You can think of the high frequencies as being handled by the positional heads.

4:57Their job is local. They make sure words connect properly to the words right next to them. This is about sounding fluent and grammatically correct. Okay, that's the structural part. Right. Then you have the low frequencies, which are handled by the semantic heads. These are the ones doing the heavy lifting for memory. They are the matchmakers that connect a question at the end of a document to an answer that was thousands of words earlier. So when you stretch that photo. To keep the high-frequency stuff, the local structure from breaking, the scaling methods are forced to aggressively compress the low frequencies.

5:32Wait, so to keep the model sounding fluent, they have to. Mm-hmm. To squash the very frequencies that are responsible for long-term memory. You've got it. That's the tradeoff. They prioritize fluency over memory, and the result is that the model starts to just ignore information that's far away. The data shows these scaled models often perform no better than a model that literally just cuts off the text at the original training length. So they can't actually use the extra context you give them. That feels like a fundamental flaw. It is, which is where Drop E comes in with this completely different idea, this third observation, which is the real game changer.

6:06The scaffolding is temporary. The training wheels can come off. The training wheels can come off. The model uses Rope E to learn and internalize the rules of order. But once those rules are baked into the attention weights themselves, the explicit ropey signal isn't just unnecessary. It's actively holding the model back. So what does the process actually look like? It sounds like this really surgical procedure. It's surprisingly simple. A three-step dance. Step one, you pre-train your model just like normal with ropey, getting all those efficiency benefits. Step two, you just drop the positional embeddings.

6:43you literally remove that component from the model's architecture. And step three. A very short recalibration phase. You just continue pre-training for a little bit longer on the original data at the original context length. That recalibration step feels so important. I mean, my first thought would be you just drop the wheels and go, but you're saying it needs a little bit of time to adjust. It's mandatory. The model has to relearn how to operate without that explicit guidance system it relied on for so long. The research actually shows that if you try to drop the PEs too early in training, the performance is terrible because the knowledge isn't fully baked in yet.

7:18And there's one other little technical detail in there for really big models, this QK norm thing. What's that for? QK norm is purely a stability tool for that recalibration phase. When you drop rho P and try to use a really high learning rate to recalibrate quickly, the model can become unstable. You risk these things called gradient explosions. Which basically means the training goes haywire. Completely off the rails. QK Norm just acts as a stabilizer, a buffer, that lets you be more aggressive with the recalibration and get it done much, much faster. Okay, so that's the how. Let's talk about the what, the results.

7:51Does taking off the training wheels actually work? The efficiency numbers alone are pretty mind-blowing. They're staggering. Take their small M360M model. It was pre-trained on 600 billion tokens, which is a lot. A huge amount of computation. Right. Right. And with Jopi, they were able to recover over 95 % of that original model's performance by recalibrating on less than 5 billion tokens. 5 billion. Wait, that's less than 1%. That's 0.8 % of the original training budget. A tiny fraction of the cost to unlock this massive new capability. And it doesn't make the model dumber in the process. Not at all.

8:26It maintains or even slightly exceeds the original model's performance on standard reasoning benchmarks. You're not trading IQ for memory. You're getting both. And that memory is the real prize, the zero-shot, long, context performance. This brings us to the needle in a haystack test. The gold standard. Can the model find one specific sentence, the needle, hidden inside a giant document of irrelevant text? The haystack. So how did your rope B do? Well, let's look at the numbers. When tested at twice the original context length, the base model, the one with regular ropey, its success rate was 0.0%.

9:00Zero. The cliff we talked about. Total failure. Yarn, the fancy scaling method, managed 48.25%, so it fails about half the time. Okay, better than nothing. But DROPE-E, the model with the training wheels removed, achieved a 74.92 % success rate. Wow. That's not just better, that's a different class of performance entirely. It's a quantum leap. And it's even better on more complex versions of the test, where it has to find multiple needles or connect different pieces of information. It confirms that by removing rope E, you're truly freeing up those semantic heads to do their job properly. And this works on the big models too, right?

9:37This isn't just some lab experiment on a small network. That's the most important part for anyone in the industry. They applied it to LAMA-27B, a huge, widely used model, and it still showed superior long-context performance. This is something that can be applied to models that are out in the wild right now. So let's wrap this up. What does this all mean? If you're trying to build with or understand LLMs, DROPE seems to represent a really big shift in thinking. It establishes a whole new paradigm. It tells us that architectural trade-offs that we thought were permanent-like, choosing between training efficiency and long context generalization, they don't have to be.

10:14You can have your cake and eat it, too. You can. You can use one architecture, ROPE, for the phase where it excels fast, stable training. and then you can switch to a different architecture, no PE, for the phase where it excels zero-shot, infinite context. So you can take these powerful pre-trained models we already have and for a tiny fraction of the original cost, unlock this critical long-context capability. It solves one of the biggest problems in the field cheaply and effectively. Which just makes you wonder, if positional embeddings were just temporary scaffolding, this thing we thought was a fundamental part of the machine, what else is?

10:50And that is the really provocative thought. What other essential components of these models are actually just transient biases, things we need to get the learning process started, but which might be holding the model back from its true potential later on? Could we remove certain normalization layers or specific attention biases? Exactly. This idea of two-phase model development trained for stability, then surgically adapt for generalization, that opens up a whole new way of thinking about how we build these systems. It sounds like the era of fixed permanent architectures might just be getting started.

11:25This was fascinating. Thank you for walking us through it. My pleasure. It's an exciting time.