1. Podcasts
  2. Best AI papers explained
  3. Quantifying Elicitation of Latent Capabilities in Language Models
Best AI papers explained

2 Nov 2026 · 21m · 9 chapters

Episode artwork

Quantifying Elicitation of Latent Capabilities in Language Models

In short

The episode argues that large language models contain latent “latent capabilities” that can be elicited with tiny parameter updates, challenging the idea that models must learn new skills via large retraining. It introduces the “elicitation gap” (difference between zero-shot and fully fine-tuned performance), the “elicitation gap closure” using only 10–100 parameters, and an information-theoretic explanation via minimum description length (MDL) and “pointer messages.” It claims standard safety evaluations based on zero-shot failures may be misleading because capabilities could be under-elicited.

Guest backgrounds

No guest names or bios are provided in the transcript.

Key claims

10–100 randomly selected parameters can recover 50–95% of the elicitation gap; knowledge is distributed holographically; attention-layer (key/query/value/output) tweaks unlock more efficiently than feed-forward layers; performance vs tuned-parameter count follows a logistic S-curve with a sharp threshold.

Notable examples

GSM8-8K-COT choice (step-by-step math reasoning) recovers ~50% gap with ~30 parameters; “Tiny Stories” vs chess (Lichess) using a pre-trained vs randomly initialized 1B-parameter model (pre-trained needs <1,000 parameters for 50% gap; blank slate needs >10 million for Tiny Stories; both need >100,000 for chess).

Written by AI. May contain mistakes.

Chapters

0:45

Understanding the Elicitation Gap

Discussion on the concept of the elicitation gap in AI learning.

“Because, I mean, we usually assume that acquiring a complex new skill requires massive structural changes, right?”
1:41

Bridging the Elicitation Gap

Exploration of how minimal adjustments can yield significant performance improvements in AI models.

“So to ground this in reality, we first have to establish a concept called the elicitation gap.”
3:54

The Library Analogy

Comparison of adjusting AI parameters to unlocking hidden knowledge with a library analogy.

“You aren't building a new wing of the library to hold new knowledge.”
4:50

Testing AI Knowledge

Discussion on experiments contrasting different AI models' performance in varied tasks.

“I mean, it doesn't sound like the model is learning a new skill at all.”
8:00

Understanding Compression in AI

Deep dive into how AI models use compression to unlock latent capabilities.

“But that leads to a deeper mechanical question.”
11:40

Self-Attention Layers and AI Learning

Exploration of self-attention layers and their role in efficiently unlocking AI capabilities.

“But logically, this creates a massive structural paradox.”
14:00

Understanding Latent Capabilities in Neural Networks

Learn how tweaking self-attention layers unlocks latent capabilities in AI models.

“And finally, the output matrix packages that extracted meaning to send to the next layer of the network.”
17:28

The Flaws of Standard AI Evaluations

Discover how current AI testing methods may misrepresent a model's true capabilities.

“If we connect this to the bigger picture, we are observing a fundamental law of artificial intelligence.”
18:44

The Implications of Under-Elicited Capabilities

Understand the potential dangers of underestimating AI models with latent knowledge.

“They are treating the AI like a human who failed a chemistry test because they never took chemistry.”
A man walking through an airport, asking his phone a question

Ask about this episode

Get the gist, explore an idea or ask about something said.

Claude

On Best AI papers explained, what are the key points of this episode?

Searching your listening

Transcript

Automatic transcript. May contain errors.

0:00Imagine having access to this massive, multi-story library, right? And it contains billions of books. Oh, yeah. Just an endless sea of information. Right. Now, imagine that to instantly absorb all the knowledge in that entire building, you don't actually need to read every page. You only need to look at 30 random letters. Just 30. Yeah. Printed on 30 random pages and 30 random books. Today, we're looking at a discovery in artificial intelligence that proves large language models just, well, they don't learn like we do. They really don't. We are stepping into this corner of computer science where the whole concept of a learning process might actually be an illusion.

0:41It completely shatters our traditional understanding of machine learning. Because, I mean, we usually assume that acquiring a complex new skill requires massive structural changes, right? Like rewiring a brain. Exactly. The equivalent of rewiring a brain. But the data reveals something entirely different. We are peering into the hidden depths of these neural networks and finding advanced capabilities just lying there. Just dormant. Completely dormant, yeah. Just waiting for a microscopic nudge. So our deep dive today into the mechanics of AI capabilities is all about finding that exact boundary.

1:16We want to figure out the line between teaching an AI something entirely new versus simply eliciting or unlocking what it already secretly knows. And that's a massive distinction. Right. For you listening right now, this is a crucial concept. If you're trying to understand why AI models sometimes seem unexpectedly brilliant or, frankly, dangerously unpredictable, you have to understand how shockingly little it takes to turn the key. Okay, let's unpack this. So to ground this in reality, we first have to establish a concept called the elicitation gap. The elicitation gap. Yeah. Think of modern massive AI models, you know, like Llama 3 or Quinn 2.5.

1:55These networks contain billions of parameters. And those are basically the internal dials and knobs, right? The things that dictate how they process information. Exactly. Now, when you take one of these models fresh out of the box and give it a highly complex task, its performance is, well, it's often pretty mediocre. Makes sense. Right. And that baseline capability is known as its zero-shot performance. Okay, so that makes sense because it's essentially winging it, right? Based on a broad, generalized training phase. It hasn't specialized in the specific task you're asking it to do yet. Right.

2:27That is the standard assumption. Furthermore, we know that if you take that same model and fully fine-tune it... Meaning you spend a ton of computing power on it. Yes. Significant computing power to deliberately adjust all of its billions of parameters for that specific task. If you do that, its performance skyrockets. It hits a very high ceiling of confidence. Okay. Okay. So that massive void between the mediocre out-of-the-box performance and that fully optimized ceiling, that is the elicitation gap. So bridging that gap usually implies a heavy lift. I mean, if you want the model to act like an expert, you basically have to rewrite the brain, which costs time and money and massive amounts of data.

3:09Well, that was the consensus. But recent data flips that completely upside down. To close that gap, to take the model from baseline to expert, you do not need to retrain billions of parameters. Wait, really? No, not at all. If you randomly select a microscopic fraction of those parameters, literally just 10 to 100 dials out of billions. Just 100 dials? Yeah. And you only adjust those, you can recover 50 % to 95 % of that entire performance gap. Okay, I need to pause you there because the math on that sounds impossible. Let's say a model has 8 billion parameters. Modifying 30 randomly chosen ones suddenly makes it an expert?

3:46It sounds absurd, doesn't it? It implies the information budget required to adapt the model isn't just small, it's virtually non-existent. It's like my library analogy earlier. You aren't building a new wing of the library to hold new knowledge. You're just handing the librarian a tiny index card with a call number on it. What's fascinating here is how well that index card analogy holds up under mathematical scrutiny. Let's look at a concrete example. Okay. Lay it on me. So they used a very specific math reasoning benchmark. It's known as GSM8-8K-COT choice. Catchy name. Right. But the point is this isn't basic arithmetic.

4:23It requires evaluating logical step-by-step reasoning chains. It's complex. So it's hard. Like it's a hard test for an AI. Very hard. And to recover 50 % of the elicitation gap on that highly complex task, researchers found it only took about 30 parameters. 30. 30 random numbers chosen from across an ocean of billions. Adjust it slightly, and the model suddenly masters grade school math logic. Wow. If 30 random keystrokes can unlock a capability that complex, it forces a pretty massive re-evaluation of what the AI is actually doing. I mean, it doesn't sound like the model is learning a new skill at all.

5:01No, it doesn't. It sounds like it's just remembering one it already had. And that is the core friction in the field right now. Are we teaching these models or are we just eliciting behaviors they already possess? How do you even prove that, though? Well, to definitively separate latent knowledge from genuinely acquired knowledge, we have to create a scenario where we can isolate what the AI knows before the test even begins. But how do you establish a control group for something as vast as an AI's pre-existing knowledge? Because it's essentially a black box, right? It's trained on the entire internet.

5:36Exactly. So we use a stress test comparing two vastly different domains. Let's look at a task called tiny stories, which involves generating simple English short stories. Then contrast that with predicting optimal chess moves from complex board states using puzzles from a platform called LeeChess. Ah, I see the divergence there. Because a standard language model is trained on internet text, so it inherently understands English grammar and vocabulary. Precisely. So Tiny Stories leverages that baseline. But predicting advanced chess moves relies on specific, complex board state analysis, which it wouldn't naturally absorb just by scraping, you know, random text.

6:15Yes, the divergence in domains is key here. So to test this, researchers took two identical versions of a one billion parameter model architecture. Okay, two identical models. Right. The first version was fully pre-trained. It had read the internet. It knew language. The second version was a completely blank slate. Like completely wiped. Its parameters were randomly initialized. It knew absolutely nothing, not even basic syntax. Okay, so we have the literate model and the completely ignorant model. When you ask them both to master the tiny stories task, what does the parameter tuning look like?

6:52For the pre-trained model, the one that already understands English, it required fewer than 1 ,000 parameters to be tuned to hit the 50 % gap closure mark. Under 1 ,000. Okay. But for the blank slate model to reach that exact same level of performance, it required the tuning of over 10 million parameters. 10 million versus 1 ,000. Yeah. So that parameter count is basically acting as a polygraph test for the neural network. That's a great way to put it. If it only takes 1 ,000 tweaks, the AI is effectively admitting it already knew the material. If it takes 10 million, it's actually doing the hard work of learning it from scratch.

7:28And the chess task verifies that polygraph theory beautifully, because chess is a genuinely difficult task, lacking deep, pre-existing representation in standard text training, right? Right. Neither model had a hidden advantage. Exactly. So when tested on the chess puzzles, both the pre-trained model and the blank slate model required over 100 ,000 parameters to be tuned just to make a dent in the task. Because neither of them had the chess zip file already downloaded. They both had to legitimately learn the spatial logic of the board from scratch. Exactly. But that leads to a deeper mechanical question.

8:03I understand that the parameter count exposes the latent knowledge, but mechanically, why does it work? How does the system mathematically distinguish between unlocking a capability and building a new one? Right. To understand the mechanics, we have to look through the lens of information theory, specifically a concept called the minimum description length, or MDL. Minimum description length. Yes. Formulated by Jorma Oriesinen, MDL is essentially Occam's razor formalized for algorithms. The foundational idea is that the best model of a data set is the one that provides the absolute shortest description of it.

8:38Okay, I want to make sure I'm grasping the algorithmic side of that. If an AI truly understands a concept, it shouldn't need to memorize every single data point, right? Correct. It just needs to learn the underlying rule. Like if I want to communicate the sequence 2, 4, 6, 8, all the way up to a million, I don't send you a list of 500 ,000 numbers. I just send you the rule. Start at 2, add 2. Right. The rule compresses the data. That is a perfect illustration of compression. True understanding allows for massive data compression. Now, apply that to our AI models using a theoretical communication game between two people.

9:15Okay. Let's call them Alice and Bob. Alice has a massive data set of test questions and the correct answers. Bob has the test questions, but he lacks the answers. Got it. Alice's goal is to transmit the answers to Bob, but she is strictly limited by bandwidth. She must send the absolute smallest file possible. So she needs to find the start at two, add to role for this specific test. Yes. And here is the critical setup. Both Alice and Bob possess exact identical copies of a base AI model on their computers. Okay, they have the same AI. Right. If that shared base model already contains a latent, hidden capability to solve those test questions, Alice does not need to send the actual uncompressed answers.

9:58She doesn't? No. She only needs to calculate which 30 random parameters inside her AI model need to be adjusted to wake up that latent capability. Oh, here's where it gets really interesting. Alice just sends Bob a tiny text file with those 30 parameter updates. It's like sending someone a 12-character password to a massive zip file they already have sitting on their hard drive, versus having to email them the entire gigabyte of uncompressed files from scratch. Exactly. That tiny text file Alice sends is called a pointer message. A pointer message. Right. When Bob applies those 30 tweaks to his identical AI, the model suddenly generates all the correct answers locally.

10:40Wow. And if that tiny pointer message allows the model to drastically compress the data set, resulting in a massive mathematically measurable drop in the minimum description length, it acts as absolute proof that the skill was already pre-existing on Bob's computer. The capability was latent. And conversely, if the AI didn't already possess that latent skill, like, the 30-character password would be useless because there is no zip file to unlock. Exactly. Alice's pointer message fails, the MDL stays flat, and she is forced to send Bob thousands or millions of parameter updates to literally rebuild his AI's architecture until it learns of the skill.

11:19You've got it. The depression fails until the parameter count becomes massive. By tracking how rapidly an AI can compress a new task using a tiny fraction of parameters, we have a mathematical proof of elicitation. Okay, I buy the zip file analogy, and I definitely understand the compression math. But logically, this creates a massive structural paradox. How so? Because if 30 random keystrokes can unlock this hidden file, that file cannot possibly be sitting in one specific folder. I mean, if the knowledge was localized in a math center of the AI's brain, you'd have to tweak exactly the parameters in that specific center.

12:00Right. But we established you are picking 30 parameters randomly from across the entire network. How is that structurally possible in the code? It points to a fundamentally different kind of architecture than we use in traditional computing. The knowledge isn't stored in a localized node or a specific filing cabinet. Okay. The representation of that capability is distributed throughout the entire network. It operates more like a hologram. Every piece of the network contains a low-resolution reflection of the entire capability. So the knowledge is literally woven into the fabric of the AI itself.

12:35Yes. But surely not all threads of that fabric are equally important? I mean, if I'm poking this holographic brain, there must be certain areas that are more responsive to waking up these latent skills. Oh, absolutely. Researchers did identify structural preferences. If we look inside a transformer block, which is the core architectural building block of these modern large language models, we find distinct types of layers. Okay, what kind of layers? Primarily, there are feed-forward layers, which act somewhat like a static memory bank. And then there are self-attention layers, which dynamically decide how different pieces of information relate to one another.

13:12Let's break down those self-attention layers, because I know that's where the real magic of modern AI happens. How do they actually function? Well, the self-attention mechanism relies on four specific matrices, the key, query, value, and output matrices. Key, query, value, output. Got it. Right. To visualize this mechanically, imagine the model is processing a sentence. The query matrix is essentially the current word actively looking around and asking, what other concepts in this context are relevant to me? Okay. The key matrix contains the tags that all the other words hold up, signaling I have this specific grammatical or contextual feature.

13:51So they match up. Exactly. The value matrix holds the actual underlying meaning that gets extracted when a query successfully matches with a key. And finally, the output matrix packages that extracted meaning to send to the next layer of the network. So it's essentially a highly complex traffic routing system. Correct. That's a good way to look at it. The query is looking for a destination. The key is the road sign, the value is the cargo, and the output is the delivery truck. Spot on. And the data reveals that tweaking the matrices in those self-attention layers, specifically the key, query, value, and output matrices, is significantly more efficient for unlocking latent capabilities than tweaking the feed-forward memory layers.

14:34Oh, that perfectly aligns with the elicitation theory. If the knowledge is already deeply embedded in the model holographically, you don't need to add new facts to the memory bank. You just need to tweak the traffic routers, the self-attention layers, so the AI knows how to navigate to the information it already possesses. Exactly. You are adjusting the lens, not painting a new picture. That is wild. And the efficiency of adjusting that lens brings us to a profound observation about how this unlocking process scales. Researchers measured how performance improves as you gradually increase the number of trainable parameters.

15:10Okay. And across all tested models, ranging from 1 billion to 8 billion parameters, and across wildly different tasks like multiple choice, story generation, and complex math reasoning, the performance improvement maps to the exact same geometric shape. Let me guess, it's a linear progression. Like you tune 100 parameters, it gets 100 units smarter. You tune 1 ,000, it gets 1 ,000 units smarter. It defies that linear logic entirely. The capability improvement forms a logistic S-curve. And crucially, it forms that S-curve relative to the logarithm of the number of parameters tuned. The logarithm.

15:48Okay, so we need to visualize that for a second. On a logarithmic scale, we aren't stepping up by ones. No, we're not. The distance between 10 and 100 parameters is treated the same as the distance between 1 ,000 and 10 ,000 parameters. We're looking at exponential orders of magnitude. Correct. On that logarithmic scale, the S-curve reveals a startling behavior. You start tuning parameters, 10, 20, 50, and the performance stays completely flat. Like zero improvement? Zero. The model still appears totally ignorant. But then you hit a precise, critical mass of parameters. Depending on the task, it might be 30, it might be 800.

16:27The moment you hit that specific threshold, performance suddenly explodes vertically before eventually leveling off at the top of the S-curve. Why does it explode like that? I mean, why isn't there a gradual accumulation of competence? Because of the distributed nature of the knowledge we discussed earlier. Imagine a dormant electrical grid. Okay. Activating one or two relays does nothing because the circuit is incomplete. But the moment you activate the precise relay that connects the pathway from the power source to the destination, the entire grid lights up instantly. Ah, that makes so much sense.

17:03Right. In a neural network, tuning parameters aligns the routing paths. Once enough paths are aligned to complete a full circuit from the prompt to the latent knowledge, the capability abruptly comes online. Boom. Exactly. And this sudden, explosive S-curve shape is identical, regardless of whether you are testing a Lama model, a Quinn model, or any other architecture. So, what does this all mean? If this knowledge is distributed everywhere, like a hologram, and the mechanism that unlocks it behaves exactly the same way across completely different families of AI, we aren't just looking at a strange quirk of one specific code base.

17:43No, we aren't. If we connect this to the bigger picture, we are observing a fundamental law of artificial intelligence. It is a built-in, universal feature of how deep neural networks organize, compress, and retrieve information. That realization fundamentally upends how we evaluate these systems. Because the standard AI benchmarks the test governments and corporations use to decide if a multibillion-dollar model is smart, or more importantly, if it's safe, might be completely flawed. They are structurally inadequate. I mean, standard evaluations primarily test zero-shot performance. Evaluators present an out-of-the-box model with a dangerous prompt, asking it to generate malicious code or outline a cyber attack.

18:29The model fails to produce a coherent answer. The evaluators see that flat zero-shot performance, check a box, and declare the model safe for public deployment. Because they assume the AI's failure to answer means the AI genuinely doesn't know how to answer. They are treating the AI like a human who failed a chemistry test because they never took chemistry. But this data proves that assumption is dangerous. The model might not be ignorant at all. The capability might simply be under-elicited. A model that appears entirely safe and ignorant during standard testing might literally be 30 random parameter adjustments away from demonstrating highly advanced, potentially dangerous capabilities.

19:0930 random parameters. Yeah. Out of a network of 8 billion. That is the equivalent of a rounding error. That's like a cosmic ray flipping a single bit on a server. It means our definitions of safe and incapable are severely misaligned with the reality of neural network architecture. This raises an important question for the future of deployment. How can we ever truly guarantee the safety of an artificial intelligence system if latent, highly compressed knowledge is so incredibly easy to unlock? We can't. Not with the old methods. Right. We can no longer afford to just test what a model does out of the box.

19:48We have to invent new scientific methods to map what a model could do if the right key is turned, before those dormant capabilities are accidentally awakened. It changes the entire paradigm. Think back to our understanding of human learning. We expect to see the friction, you know, the slow accumulation of a skill, the visible effort of building new pathways in the brain. Right, the struggle. But with these massive AI networks, that struggle just doesn't exist. The pathways are already there. The network has already absorbed the underlying rules of the data it consumed. We just haven't handed it the pointer message yet.

20:25Exactly. And that leaves a lingering, somewhat unsettling thought for you to take away from this. If it takes just 10 to 100 randomly chosen parameters to suddenly unlock expert-level human reasoning tasks, tasks that we explicitly know to look for and test, what bizarre, entirely alien capabilities are lying dormant in the dark spaces of these networks? Yeah. What incomprehensible logic structures are already fully formed, compressed, and hidden, just waiting for the right random 10 parameter key to be turned?

From the publisher

This paper investigates how large language models unlock latent capabilities through fine-tuning, reframing the process as an information-constrained problem. By analyzing parameter-efficient fine-tuning using randomly chosen parameters, the authors demonstrate that training a tiny fraction of a model's weights—often as few as tens to thousands—can recover the vast majority of its full performance gap. This behavior follows a consistent logistic scaling relationship across various tasks, model sizes, and families, independent of specialized search methods. Furthermore, the study applies Rissanen's Minimum Description Length (MDL) principle to distinguish true capability elicitation from teaching new skills, offering insights that can improve capability forecasting and safety evaluations.

More from Best AI papers explained

All episodes