Language models are injective and hence invertible

0:00Welcome back to the Deep Dive. Today we're digging into something pretty fundamental about large language models. It's something most of us just took for granted. That's right. The common idea for ages, really, was that the internal workings of these models, those hidden states, were inherently lossy, you know, messy. Like a compression algorithm, right? You put your prompt in and all the complex math inside the nonlinear stuff, it just kind of mushes the information together. Exactly. We figured some details had to get lost. You put in this perfect prompt, this high-res photo metaphorically. And what the model sees internally is more like a blurry JPEG.

0:37You get the gist, but the original details are gone forever. Recovering the exact input seemed impossible. And honestly, it made sense looking at the parts. You've got things like layer norm averaging stuff out, nonlinear functions like GELU squishing numbers. It looks like information loss is happening constantly. But, well, the research we're diving into today basically flips that whole idea on its head. It turns out that intuition might be completely wrong. Completely. We now have solid mathematical proof and practical evidence, too, showing that standard decoder-only transformers, the kind most people use, are actually injective.

1:11Which means lossless. Structurally lossless. Different prompts almost surely create different hidden states. The information isn't being destroyed. Okay, that's huge. So our mission for you today. First, we need to unpack this injectivity idea, what the theory actually says. Then second, we'll look at an algorithm called SIPPITE. It's the tool that actually makes perfect reconstruction possible based on this theory. And finally, we absolutely have to talk about the implications. Because if these models are lossless, that changes everything for security, transparency, and especially user privacy.

1:44Big time. Definitely. All right, let's start with the theory, but keep it grounded. What does injective actually mean in this context? If the mapping from prompt to hidden state is injective, what's the practical upshot? It's basically a one-to-one mapping. Think of it like this. If a function is injective, every unique input must give you a unique output. So, change one word, even one comma, in your prompt. And the hidden state has to change in some measurable way. If that holds true, it means the model hasn't accidentally merged two different inputs into the same internal representation. It's kept them distinct.

2:21It's lossless. Okay, but I'm still stuck on that earlier point. How can components that seem designed to lose information, like those GELU activations squashing numbers? Right. How can they combine to make something lossless? It feels wrong. Yeah. It's counterintuitive. And that's the key insight here. While individual pieces might look non-injective on their own, the way they're put together sequentially in a standard decoder-only transformer, that overall structure, the aggregate function, ends up being injective. So the whole is greater than the sum of its parts, information-wise. Precisely.

2:54The architecture itself preserves the distinctions. And this isn't just like a fluke or something that happens with infinitely large models. The proof shows it's a structural consequence of how these things are built. They are almost surely injective. Okay, almost surely. That sounds like there's a catch. Let's break that down. You mentioned two key mathematical ideas that make this work in practice, right? The first one was real analyticity. Yeah, real analyticity. It basically means the building blocks matrix multiplies, layer norm activations are mathematically well behaved. They're smooth, structured functions.

3:30And because they're smooth. It means collisions where two different prompts somehow produce the exact same hidden state are incredibly rare mathematically. They can only happen if the model's parameters fall into what's called a measure zero set. Measure zero set. Okay, for you listening, picture the huge space of all possible settings for an LLM's parameters. This measure zero set is like an infinitely thin sheet, or maybe even just a line within that massive space. Exactly. It's mathematically possible to land on it, but statistically, it's vanishingly unlikely. Like, trying to hit one specific atom from across the universe, unlikely.

4:06So these collisions are theoretical edge cases, basically. Mathematical exceptions, yeah. And the second key point is about training. How do we know training doesn't accidentally push the model onto that super unlikely collision line? Right, because models train for billions in steps. Maybe one of those steps makes it lossy. Well, the proof actually covers that, too. Standard ways of initializing models, like using random Gaussian or uniform weights, already avoid this measure zero set almost certainly. Okay, so they start off injective. And critically, the common training methods, gradient descent, SGD, atom, etc., are mathematically proven not to guide the parameters into that pathological set.

4:46Wow. So injectivity holds from the start, and it holds throughout training. That's the finding. Essentially, pretty much every standard decoder-only transformer you've interacted with is structurally lossless. That is quite a realization. Okay, so the theory is solid. But can you actually do it? Can you reverse the process and get the prompt back perfectly? That's where SITPIT comes in. Exactly. SITPIT stands for Sequential Inverse Prompt via Iderative Updates. And yeah, it's the algorithm designed to take that theoretical guarantee and make it real. It's the first method proven to recover the exact input prompt from those hidden activations.

5:21And it works by using the model's own structure against it, essentially. That causal step-by-step processing. Precisely. Remember how a transformer works? Token by token. The hidden state for token number, say, 5, only depends on the first four tokens and the fifth token itself. It only looks backward. Sipi explains that. Let's say you've already figured out the first ten tokens of the original prompt to find the eleventh token. You don't need to guess the rest of the sequence. No. You just need to figure out that single next token. Sipi takes the known prefix, the first ten tokens, and then tries out every possible next token from the model's vocabulary.

5:58It plugs each one in. Like, what if the next token was the? What if it was cat? Exactly. And because we know that specific one-step mapping, taking the prefix state, and a candidate token to produce the next hidden state is itself almost surely injective. Only one candidate token will perfectly match the hidden state activation that was actually observed. Bingo. Only one token fits. So you find that token, add it to your covered prefix, and move on to the next position, step-by-step. Guaranteed accuracy. That sounds incredibly efficient compared to, well, trying to guess the whole prompt at once.

6:33The number of possible prompts is astronomical. Utterly infeasible. Brute force is hopeless. But SUPPIT changes it from this impossible exponential search into a linear problem. Linear time, meaning the time it takes scales directly with the length of the prompt. Yep. The maximum number of checks it needs to do is the prompt length, T times the vocabulary size, V. It's provably efficient, often even faster in practice, this makes exact recovery actually doable. Which previous attempts of reversing embeddings couldn't really promise. Okay, linear time, guaranteed recovery, that's the operational side.

7:09What did the empirical tests show? Did they actually try to find those theoretical collisions? Oh yeah, they went looking hard. They ran billions of comparisons, took pairs of prompts, fed them into state-of-the-art models LAMA 3.1, Mistral GPT-2s, Gemma models, and checked if any two different prompts produced the same final hidden state. A massive stress test for the measure zero set idea. Huge. And the result... Only guess. Zero collisions. Exactly. Zero collisions found, not a single one. The differences, the distances between hidden states for different prompts were consistently huge orders of magnitude larger than what they'd consider a collision.

7:51Wow. And doesn't they find something interesting about deeper layers too? Yeah, that was fascinating. You might think deeper layers would mush things together more. Lose information, yeah. But they found the opposite. Generally, the deeper you went into the model, the more distinct the representations for different inputs became. The separation actually increased. So the model isn't compressing. It's actively differentiating the inputs more sharply as it processes them. Seems like it. And then the final test, running SIPIT. They tried it on GPT-2 small. And did it work perfectly? 100 % token-level accuracy.

8:22Every single time, it reconstructed the original prompt exactly from the hidden states. Perfect recovery, just like the theory predicted. Okay. Theory proven. Algorithm works. Empirical tests confirm it. This leads us straight into the really critical part. The consequences? If SIPIT can perfectly get the prompt back from a hidden state... Then that hidden state isn't some abstract representation anymore. It is the prompt. Just encoded differently. The paper puts it bluntly. Hidden states are not abstractions, but the prompt in disguise. And that just blows up a lot of assumptions about privacy and data handling.

8:58Because people used to argue, even regulators, you mentioned the Hamburg Data Protection Commissioner, that these internal states weren't necessarily personal data because getting the original text back was thought to be too hard or impossible. Right. They relied on that idea of lossy compression. Oh, it's just an embedding. It's lossy. It's not the real data. That argument is now demonstrably false. So, any system that stores or transmits these hidden states. Maybe a RAG system indexing document chunks as embeddings or monitoring tools or even federated learning setups. If you're handling those embeddings, this research proves you are effectively handling the original text.

9:33There's no free privacy once data goes into a transformer's hidden state. The abstraction layer is gone. If you have the state, you potentially have the source text. Which means organizations have to treat those internal representations with the same level of security, the same compliance standards as the original user input. You can't pretend it's just anonymized or abstracted data anymore. That is a massive shift in how we need to think about data security in AI pipelines. Wow. Okay, this has been genuinely eye-opening. Let's recap the absolute core takeaways for everyone listening. Okay, first, standard decoder-only transformers are structurally lossless.

10:08That mapping from the prompt you type in to the internal hidden state is almost surely injective, one-to-one. Second, this isn't fragile. It holds up from the start, when the model is initialized, and it persists through standard training methods like gradient descent. And third, the SIPBIT algorithm makes this real. It provides a practical, efficient, linear time way to perfectly reconstruct the original prompt from those hidden states, proving the information is all there. So the absolute theoretical ceiling for information preservation is, well, perfect preservation. The details of your input are structurally encoded, always present in the hidden state.

10:48Which brings us to a final thought for you to chew on. If the information is proven to always be there inside the model, Right. Then if some future interpretability method, some probe trying to understand why the model did something, fails to find evidence of a certain piece of input information inside the layers. Like it can't figure out if the model registered a specific name or concept from the prompt? It can no longer be argued that the information was lost or compressed away by the network itself. We know it's structurally present, so a failure to find it must mean the probe failed. The interpretability tool wasn't good enough.

11:22That really raises the bar for anyone trying to do mechanistic interpretability or causal analysis, doesn't it? It puts the onus squarely on the analysis method, not the model's supposed information loss. It certainly does. A lot to think about there.