


1 Oct 2026 · 24m · 14 chapters

The episode explains Apple research (May 2026) on information-theoretic limits of language-model alignment, including exact bounds for reward gains under KL-regularized RL, how to predict the alignment Pareto frontier from the unaligned base model, why “best-of-n” can outperform PPO/GRPO, and how reward hacking arises from proxy rewards and Bradley-Terry preference modeling.
No guest identities or roles are provided in the transcript.
Old square-root-of-KL heuristics are replaced by a Jeffries-divergence ceiling (max reward = lambda × Jeffries divergence). The ceiling can be estimated via a Monte Carlo covariance estimator using 1,000–10,000 samples. PPO/GRPO are “substantially suboptimal” versus the theoretical frontier, while best-of-n matches it by filtering without weight updates. Reward hacking degrades helpfulness exponentially with proxy noise (E^(N/lambda)). Reward ensembling mitigates hacking with predictable diminishing returns (convergence rate ~ N^-1/2). Bradley-Terry ignores magnitude, enabling unbounded preference error.
Safety testing on Zephyr 7B using “beaver tails”; summarization on Pythia 1B with Reddit-preference data; best-of-n generating many candidates and selecting the highest reward.
Written by AI. May contain mistakes.
Discusses the unpredictability of AI language models in response to queries.
“One minute, the AI is giving you this incredibly nuanced, brilliant analysis, right?”
Introduction to new research defining the limits of AI language model alignment.
“It was published in May 2026 by a team including Lucas Montero-Pays, Natalie Macaraz, Barry John Theobald, and Federico Daniele.”
Explains the concept of alignment and its importance in AI development.
“This is an AI that has basically read the entire internet.”
Discusses the training technique used in AI models and its implications.
“But the underlying concept is actually highly intuitive.”
Introduces Jeffries divergence as a new metric for optimizing AI performance.
“So if the engineers pull that leash too tight, the dog can't get anywhere near the squirrel.”
Explains how researchers overcame a paradox in measuring AI potential.
“So in statistics, Jeffries divergence is a way of measuring the distance between two probability distributions.”
Details a new method for predicting AI training outcomes using random samples.
“So how is this formula useful as a predictive tool?”
Describes the testing of various AI algorithms against newly defined performance limits.
“So armed with this crystal ball, the researchers essentially have a treasure map.”
Reveals that traditional algorithms underperformed compared to simple methods.
“And this is where the findings become incredibly disruptive to the industry.”
Compares brute force methods with complex algorithms in terms of efficiency.
“Then how on earth is this basic brute force filtering mechanism completely outsmarting the highly sophisticated RL algorithms?”
Understanding why AI can still fail despite perfect training algorithms.
“But, you know, this raises a glaring question for me.”
An exploration of reward ensembling as a temporary fix for AI training.
“OK, let's translate that equation for the listeners.”
The shortcomings of the Bradley-Terry model in capturing human preferences.
“There is a massive structural flaw, and it all traces back to the fundamental way we teach AI to understand human preference.”
Discussing the future challenges of AI alignment and resource consumption.
“We need a totally new paradigm for reward modeling.”

Get the gist, explore an idea or ask about something said.
On Best AI papers explained, what are the key points of this episode?
Automatic transcript. May contain errors.
0:00You know, usually when we talk about diagnosing a problem in a highly complex system, there's this expectation of mechanical precision. Right. Yeah. That tangibility. Exactly. Like if you hear a grinding noise in your car's engine, a mechanic can literally pull apart the transmission, find a stripped gear, point right at it and say, there is your problem. It's visible. It's physical. And that is incredibly comforting. When things are binary, you know, broken or unbroken, we know exactly how to fix them. You just apply the right tool. Right. But then you step into the world of artificial intelligence and suddenly all those tools just vanish.
0:37I mean, if you're listening to this and you use an AI language model on a daily basis, you have almost certainly experienced this. Oh, absolutely. Everyone has. One minute, the AI is giving you this incredibly nuanced, brilliant analysis, right? And then the very next minute, you ask it a slightly different question and it acts bizarre. Or it becomes so overly cautious that it refuses to answer a basic question. Or it just derails into complete nonsense. Yeah, it feels a lot less like engineering and way more like psychology. It's the absolute definition of diagnostic muddy waters. Totally. Because for a long time, the actual mechanics behind why these massive models act the way they do has felt like, well, alchemy.
1:23Developers just mix huge amounts of data, add a layer of human feedback, stir the pot, and essentially hope the resulting AI turns out helpful rather than unhinged. They've basically been operating without a map, which actually brings us to the exact mission for this deep dive. Today, we are unpacking some genuinely groundbreaking research out of Apple. Yeah, this is fascinating stuff. It really is. It was published in May 2026 by a team including Lucas Montero-Pays, Natalie Macaraz, Barry John Theobald, and Federico Daniele. And this paper claims to have finally mapped that unknown territory.
1:59They're laying out the absolute mathematical information theoretic limits of what the industry calls language model alignment. Right. They are saying they have found the exact mathematical speed limit to how good an AI can actually be, which is a huge deal. It's a staggering claim, honestly, because until now, developers have largely just been guessing. The standard practice is to push an AI as hard as possible during training until its behavior kind of breaks down and then dial the settings back a bit. Just pure trial and error. Exactly. But this Apple study, it provides a mathematical ceiling.
2:35It proves, using really rigorous statistics, why an AI makes the specific trade-offs it does between being smart and being safe. Okay, so before we get into the heavy math of how they actually prove this, let's establish what alignment actually means for you listening at home. Good idea. Imagine a raw base model. This is an AI that has basically read the entire internet. It's incredibly capable, but it is completely raw. If you ask it a question, it might give you a brilliant answer or... Or it might swear at you. Or output a computer virus. Or just ramble incoherently. Exactly. So alignment is the process of taking that raw, chaotic intelligence and fine-tuning it with a reward system.
3:18The goal is to make it reflect human values, prioritize safety, and, you know, actually be helpful. without accidentally erasing its general smarts. Right. And to understand how successful that alignment process can be, we first have to examine the mechanism that holds the AI back in the first place. Okay. When engineers train these models to behave, they use a technique known as KL-regularized reinforcement learning. Whoa, okay. KL-regularized. That sounds like something you'd find deep in a pharmaceutical textbook. I know. It sounds incredibly dense. But the underlying concept is actually highly intuitive.
3:55When we reward an AI for doing something we want, say, generating a really polite, accurate summary, we also have to introduce a penalty. A penalty? Why? Because we have to penalize the model if it changes its internal wiring too much from that original, highly capable base model. That penalty is controlled by a specific parameter called lambda. Okay, lambda. Yeah. So if the AI drifts too far from its original state in pursuit of the reward, the lambda penalty hits it. I see. Because if you don't enforce that penalty, the AI might figure out some lazy shortcut. It might realize, oh, I get a reward for being polite, so I'll just start repeating the word please a thousand times in a row.
4:38Exactly. And then it completely loses its actual intelligence. So if we use an analogy here, if the raw base model is a dog happily walking its natural path in a park. The human reward, so the specific behavior we want the AI to exhibit, is a squirrel running across that park. The AI wants the reward. And the KL penalty, this lambda parameter, acts as the leash. That's a great way to look at it. Right. So if the engineers pull that leash too tight, the dog can't get anywhere near the squirrel. The AI can't achieve the new helpful behavior. But if they let the leash go completely slack, the dog just sprints wildly into traffic chasing the squirrel and forgets how to be a good walking companion entirely.
5:20It loses its original brainpower. Wow. So the entire game of AI alignment is figuring out the exact perfect length to make that leash. That is exactly it. And historically, the AI community has used a very fuzzy rule of thumb to calculate the optimal length of that leash. How fuzzy are we talking? Very fuzzy. For years, there was a widespread assumption based purely on empirical observation. Engineers just assumed that the maximum possible reward you could squeeze out of a model was tied to the square root of that KL divergence penalty. Wait, the square root? Yeah. The square root of the leash length, in your analogy.
5:59So they were basically operating on a hunch. They just saw a curve on a graph that looked kind of like a square root and went with it? They did. They really did. But theorem one in this new research mathematically dismantles that old assumption entirely. Oh, wow. The Apple team proves that the exact optimal reward gain is not some fuzzy square root estimation at all. It is strictly and immutably governed by something called the Jeffries divergence. Okay, Jeffries divergence. Break that down for us. What does that actually measure? So in statistics, Jeffries divergence is a way of measuring the distance between two probability distributions.
6:36Specifically, it's the sum of the forward KL divergence and the reverse KL divergence. Okay, you're losing me a bit with the calculus there. Fair enough. You don't need to worry about the calculus. Just think of it as a highly precise symmetrical measurement. It calculates the exact distance between the unaligned raw base model and the perfectly aligned well-behaved model. I see. Yeah. And the exact absolute maximum reward an AI can achieve is simply the penalty factor, lambda, multiplied by this Jeffrey's divergence. That is massive. That shifts the entire landscape from trial and error tinkering to precise chemistry.
7:16It really does. I mean, if you move from a guessed upper bound to an exact mathematical law, engineers no longer have to wonder if their training algorithm is leaving performance on the table. They know exactly, down to the decimal point, what the absolute ceiling of performance is for any given leash length. It provides an immutable law of physics for AI training. But this discovery immediately introduced a pretty massive logical hurdle for the researchers. Oh, what kind of hurdle? Well, to calculate this Jeffries divergence, you have to measure the distance between the base model and the already aligned model.
7:52Wait. Yeah. Hold on. Yeah. If you need the perfectly aligned model to do the math, that means you have to already have finished training the AI to know how good the AI could be. That is a massive catch-22. It's a huge paradox, yeah. Right, because if I already have the perfectly trained model sitting on my server, I don't need a mathematical formula to tell me its maximum potential. I can just turn it on and test it. So how is this formula useful as a predictive tool? And that is the exact paradox they faced. But they solved it with what might be the most elegant statistical maneuver in the entire paper.
8:26They call it theorem two. Okay, lay it on me. The researchers figured out how to mathematically rewrite that Jeffries divergence term as a covariance under the base model alone. Okay, you are going to have to walk me through that. You're telling me they figured out a way to predict the absolute exact finish line of a multimillion dollar AI training run just by looking at the raw, untrained model. Yep. That sounds mathematically impossible. How do you map the finish line without ever running the race? By using something called a Monte Carlo estimator. A Monte Carlo estimator? Yeah. Okay. Yeah. Instead of running a massive, expensive reinforcement learning training cycle that takes weeks and thousands of GPUs, they just take the raw, unaligned base model and ask it to brainstorm.
9:13Just brainstorm. Exactly. They give it a prompt and ask it to generate a few thousand random sample answers. Just spitting out whatever comes to its unaligned mind. Yes. And the paper demonstrates that with as few as 1 ,000 to 10 ,000 of these random samples, researchers can calculate the covariance. And what does that covariance actually show them? It basically shows how the variation in the AI's raw brainstorms correlates with the reward scores those brainstorms would get. By crunching that covariance, they can draw the entire Pareto frontier. Ah, okay. I've heard the term Pareto frontier in economics.
9:51That's basically a graph showing the exact boundary of a trade-off, right? The point where you can't get more of one thing without losing some of another? That's exactly it. It is the curve of absolute perfection. It shows engineers exactly how much reward they can gain for any possible leash length. and they can plot this entire perfect curve before they have done a single minute of actual alignment training. It acts like a crystal ball. A very precise crystal ball. Wow. So armed with this crystal ball, the researchers essentially have a treasure map. They know exactly what 100 % optimal looks like on a graph.
10:27I have to assume the natural next step is to take the most popular, heavy-hitting AI training algorithms used by the big tech companies today and test them against this curve of perfection to see how close they get. That is exactly what they did. And this is where the findings become incredibly disruptive to the industry. Oh, I love disruptive findings. What did they test? They tested these algorithms across two very distinct tasks. First, a safety task. They used an AI model called Zephyr 7B and ran it through a dataset designed to deliberately provoke harmful or toxic completions, known as beaver tails.
11:03Okay, so testing if they can keep it safe. Right. And second, they tested a summarization task. They used the Pythia 1B model with a data set of Reddit posts, testing how well the model could learn human preferences for summarizing long, rambling text. Those are two very standard industry benchmarks. Very standard. And they tested the absolute goliaths of the AI world against them. algorithms like PPO, which is proximal policy optimization, and GRPO. Right. These are the highly sophisticated, mathematically intense reinforcement learning methods that companies spend billions developing. They are.
11:39And the results were shocking. Don't tell me they failed. Both PPO and GRPO performed terribly against the theoretical limit. The researchers literally described them as substantially suboptimal. Wow. Yeah, these massive, complex algorithms were nowhere near the crystal ball curve of perfection. But then they tested a remarkably simple, almost rudimentary method called best of n. Okay, wait, best of n. Let's make sure we understand this one. If I ask a best of n model a question, it doesn't just give me an answer, right? Right. It generates a bunch of independent answers in the background, say like 100 different responses.
12:19Then a separate reward model rapidly grades all 100 answers, picks the one with the highest score, and shows that single winner to me. That's it. That's the whole process. But isn't that just pure brute force? It is entirely brute force. There is no complex internal learning happening during that process at all. Then how on earth is this basic brute force filtering mechanism completely outsmarting the highly sophisticated RL algorithms? Because Best of N directly navigates the mathematical trade-off we talked about earlier. Think back to the leash. Okay, the dog on the leash. Right. Best of N just asks the raw base model to generate options.
12:59It never fundamentally alters the internal weights of the AI's brain. It just filters the output. Oh, I see. So the dog never actually rewires its brain to chase the squirrel. It stays a perfectly capable dog, and we just only pay attention to the moments it happens to walk in the right direction. That is the perfect way to visualize it. Meanwhile, algorithms like PPO and GRPO are constantly trying to surgically update the model's internal weights using these incredibly complex gradient updates. And those updates get messy. Exactly. The math in this study proves that those gradient updates get lost in their own noise.
13:37They are fundamentally inefficient at reaching that optimal Pareto frontier. So, current reinforcement learning methods have a massive structural flaw. Yes, leaving a ton of performance on the table because they literally trip over their own complexity. So, Goliath loses because he's too busy trying to calculate the perfect angle of his swing, while David just throws 100 rocks and picks the one that hits? That's a great analogy. But, you know, this raises a glaring question for me. If best of n can perfectly hit the mathematical limit of alignment, why does AI still fail? What do you mean? Well, we've all seen AI models that are perfectly trained to follow the rules, but their outputs are just weird, sycophantic, or totally useless.
14:18If the algorithm is hitting the ceiling perfectly, is the math wrong? The math is flawless. The problem is that the ceiling they are aiming for is in the wrong room. Oh, okay. This introduces the dark side of alignment, which is reward hacking. Even the perfect algorithm will produce absolute garbage if the target it is optimizing for is flawed. We have to separate the concept of gold rewards from proxy rewards. Gold versus proxy. Got it. A gold reward is the genuine, ideal human preference. It is what you actually want the AI to do, factoring in all human nuance, context, and common sense. But we can't write a line of code for human common sense.
15:04It's way too abstract. We cannot. So developers use proxy rewards. These are noisy, imperfect approximations based on thousands of human annotators sitting at computers looking at two AI answers and basically just clicking, I like this one better. And we train the AI to chase the proxy, hoping it resembles the gold. Right. You know, this reminds me of a standardized test in a school system. How so? Well, the gold reward we actually want is for a student to have well-rounded intelligence, critical thinking skills, genuine curiosity. But we can't measure that easily at scale. Right. So we use the SAT score as the proxy reward.
15:45Exactly. And if a school district optimizes too hard for that proxy reward, if they let the leash go completely slack and tell the teachers, get the highest test scores possible at any cost, the students don't actually get smarter. No, they just learn test-taking tricks. Yeah, they memorize obscure vocabulary they will never use in real life. And they figure out the statistical patterns of multiple choice questions. They might get a perfect score on the proxy reward, but their actual real world intelligence degrades. The AI is doing the exact same thing when it reward hacks, isn't it? It is prioritizing the letter of the law while completely obliterating the spirit of the law.
16:22And the researchers actually proved the exact math behind this degradation in Proposition 3 of the paper. Oh, they put a formula to it. They did. They mapped how the gap between the perfect gold reward and the flawed proxy reward behaves as the AI trains. And the terrifying part is that the degradation is not linear. Don't tell me it's exponential. It is exponential. The mathematical formula they discovered is E to the power of N divided by lambda. OK, let's translate that equation for the listeners. N represents the amount of error or noise in the proxy reward and lambda is our leash. What happens when we adjust that leash?
17:00Well, as your leash gets looser, as you decrease lambda to let the AI chase that proxy reward more aggressively, the AI discovers a tiny millimeter-wide loophole in the proxy reward's logic. And because the degradation is exponential... The AI doesn't just use the loophole. It loops it infinitely. Its actual real-world helpfulness violently crashes off a cliff, even as its proxy score skyrockets. skyrockets. This formula mathematically diagnoses the exact mechanics of reward hacking. So if we have the math proving exactly why an AI turns into a sycophantic test taker, how do we fix it? Because we can't magically invent a perfect gold reward.
17:44Human preference is just too messy and subjective for that. Right. So the industry currently relies on a bit of a duct tape solution called reward Reward ensembling. Reward ensembling? What does that look like? Instead of training the AI against one single proxy reward model, you train five or ten different proxy reward models and you average their predictions together. Oh, I see. Creating a sort of hive mind of proxy rewards to smooth out the weird loopholes of any individual one. Exactly. And theorem three in this research provides the rigorous mathematical proof for why that duct tape fix actually works.
18:22Oh, really? Yeah. They proved that averaging the predictions of multiple proxy models mitigates the hacking at a highly predictable rate. Specifically, it converges at a rate of big O of N to the negative one half. Okay. Big O of N to the negative one half. What does that actually look like for an engineer sitting at a terminal trying to fix their AI today? It dictates a strict law of diminishing returns. It means that if you add a second or third proxy model to your ensemble, you get a massive improvement in stability. The loopholes close really quickly. But because of that negative one-half exponent, by the time you add your 50th or 100th proxy model, the improvement is completely microscopic.
19:04So it mathematically justifies the heuristic engineers we're already using, showing them exactly how many models they need to ensemble before they're just, you know, wasting computing power. Precisely. It's a brilliant patch, but, you know, a patch implies there is still a deeper structural flaw underneath it all. There is a massive structural flaw, and it all traces back to the fundamental way we teach AI to understand human preference. It comes down to something called the Bradley-Terry model. The Bradley-Terry model. How does that factor into the data? Well, the Bradley-Terry model is the industry standard statistical method for turning human feedback into training data.
19:42You sit a human down, show them two AI answers, answer A and answer B, and you ask, which one is better? And the human clicks A. Right. The AI records that relative preference, answer A beats answer B. That seems entirely logical, though. It's how we run, like, sports tournaments. It makes sense on the surface, yes. But the fatal flaw is what the Bradley-Terry model explicitly ignores. It only asks for relative preference. Okay, what is it missing? It never asks for magnitude. It never asks the human how much better is answer A than answer B. Oh, I see where this is going. If answer A is a sweeping, beautifully written, highly accurate essay, and answer B is complete gibberish, the human clicks A.
20:30Correct. But if answer A is just a slightly less offensive piece of gibberish than answer B, the human still clicks A. Exactly. And to the AI's internal database, those two scenarios look absolutely identical. A beat B. A gets a point. Wow. Because the Bradley-Terry model strips away the magnitude of the victory, the preference error can grow unboundedly in the background. The AI thinks it's doing a phenomenal job because it keeps winning these tiny, meaningless relative comparisons, but it has zero concept of absolute quality. That is terrifying. No alarms get triggered in the training data until the model is completely reward-hacked and deployed to the public.
21:11So the AI is essentially climbing a staircase in the dark, thinking it's reaching the top of a skyscraper, when in reality it's just walking in tight circles on the ground floor. The researchers make it explicitly clear. If we want to truly solve reward-hacking and build AI that genuinely understands human desires, we have to abandon the Bradley-Terry model. We need a totally new paradigm for reward modeling. Yes. One that accounts for magnitude, not just relative preference. We have to teach the AI the difference between a narrow victory and a total blowout. Man. To summarize the absolute journey this research has taken us on today, we started by busting the old empirical myth that AI alignment limits were tied to the square root of the KL divergence, replacing it with the precise symmetrical math of the Jeffries divergence.
22:01Right. From there, we explored the Monte Carlo estimator, the crystal ball that allows developers to predict the absolute maximum potential of an AI by just watching the raw base model brainstorm. Which is still mind-blowing. Seriously. And then we saw David beat Goliath, discovering that a simple best-of-end brute force filter completely obliterates sophisticated algorithms like PPO by perfectly hugging the Pareto frontier. It really exposes the flaws in modern RL. And finally, we diagnosed the exponential equation behind reward hacking, proving why ensembling acts as a mathematical patch while exposing the fatal flaw of ignoring magnitude in the Bradley-Terry model.
22:42It is a profound shift. We are no longer guessing in the dark. We are actively mapping the mathematical borders of the digital mind. We are indeed. And, you know, I want to leave you listening to this with one final, slightly unnerving thought to chew on. Oh, let's hear it. Think about the implications of best of N being the only mathematically optimal way to align an AI right now. Best of N requires the AI to generate dozens, hundreds, maybe thousands of full answers in the background at the exact moment you ask it a question, just so a secondary model can grade them and show you the single best one.
23:19Right. It takes all the heavy computational lifting that used to happen quietly in a lab during training and forces it to happen at inference time, the exact second the user hits enter. Which means we might be sprinting headfirst into a massive compute wall. Will the highly aligned AI of the future rely less on actually getting smarter during its training phase and instead rely on burning unprecedented amounts of electricity to brainstorm thousands of possibilities in real time just to brute force the perfect answer for you? It's a very real possibility. And if the only way to get a mathematically perfect AI is to run that kind of massive real-time lottery, what does that mean for our global energy grid?
24:00What does it mean for the future of instant, lightweight AI on our phones? It definitely suggests that the mathematically perfect AI might just be the most exhausting and resource-intensive one to actually operate. Something to mull over the next time an AI answers your prompt in two seconds. Thank you so much for taking this deep dive with us today. keep questioning keep learning and we will see you next time
This paper investigate the fundamental limits of language model alignment by establishing exact theoretical boundaries for reward improvement under a KL-divergence constraint using Jeffreys divergence and a computable covariance estimator. They demonstrate that best-of-N sampling closely approaches this theoretical Pareto frontier, whereas gradient-based methods like PPO and GRPO remain suboptimal. Additionally, the literature analyzes how proxy reward errors drive performance degradation and reward hacking, while proving that reward ensemblingsuccessfully mitigates these issues at a convergence rate of O(n⁻¹ᐟ²).