1. Podcasts
  2. Best AI papers explained
  3. Pointwise or pairwise: when do pairwise losses help reward learning, provably?
Best AI papers explained

4 Oct 2026 · 22m · 9 chapters

Episode artwork

Pointwise or pairwise: when do pairwise losses help reward learning, provably?

In short

Pairwise vs point-wise reward learning for RLHF-style training. Compares value regression (VR: train on absolute numeric rewards like 8.5/10) to value difference regression (VDR: hide absolute scores and train only on relative differences between two responses).

Guest backgrounds

No guests are identified in the transcript; it’s a two-host discussion.

Key claims

Absolute point-wise rewards are biased by action-independent “nuisance” (prompt/rater/context noise) under weak realizability; VDR cancels this nuisance by subtracting scores. However, VDR can have high variance/geometric vulnerability in linear function classes when comparisons are clustered (low cross-face probability). Which method wins depends on noise level and number of alternative responses per prompt.

Notable examples

Bathroom scale analogy; gymnastics judge analogy; synthetic linear bandit with nuisance dial beta; LAMA 3.23B, Qwen3, and “ARMORM” tested on WildChat and GSM8K using held-out offline regret; scaling alternatives N_A from 2–4 to 64/128/512 (VDR improves regret by 9.7%, 13.0%, 24.0% respectively); MCTS reasoning test where binary-preference VDR beats VR.

Written by AI. May contain mistakes.

Chapters

0:36

Point-wise vs. Pair-wise Scores

Learn about the debate between point-wise scoring and pair-wise scoring in AI.

“We want to know, how do the smartest AIs in the world actually learn what a good answer is?”
1:40

Exploring Value Regression (VR)

Dive into the mechanics of value regression and its implications for AI performance.

“The first approach is the point-wise method, which the research formally calls value regression, or VR.”
4:28

Understanding Value Difference Regression (VDR)

Examine the pair-wise approach and its mathematical foundation in AI training.

“It essentially suggests that we shouldn't view a score like an 8.5 as a single pure metric of quality.”
7:20

Geometric Vulnerabilities in VDR

Unpack the hidden dangers and geometric weaknesses in value difference regression.

“But if we switch to value difference regression, we're no longer looking at the 40.”
11:38

Trade-offs in High Noise Environments

Discover how bias and variance affect AI training in noisy conditions.

“And this is exactly where the point-wise approach, value regression, suddenly re-enters the fight.”
14:00

Understanding Regret in AI

Learn how regret is measured in machine learning and its implications.

“large language models we interact with daily?”
14:48

Comparing VR and VDR

Explore the differences between value regression and value difference regression in AI training.

“In machine learning, regret measures the gap between the action the AI chose to take and the absolute best optimal action it could have taken in hindsight, right?”
18:11

Scaling N sub A in AI Training

Discover how increasing training responses affects AI performance and learning.

“That highlights how critical the reward function truly is, and they confirmed this pattern in a completely separate reasoning experiment, too.”
19:48

Implications of AI Metrics on Human Systems

Reflect on how AI metrics can reshape our understanding of human evaluations in organizations.

“It is a phenomenal look at the invisible math dictating the future of machine learning.”
A man walking through an airport, asking his phone a question

Ask about this episode

Get the gist, explore an idea or ask about something said.

Claude

On Best AI papers explained, what are the key points of this episode?

Searching your listening

Transcript

Automatic transcript. May contain errors.

0:00You usually expect measuring success to involve absolute precision. Like you step on a bathroom scale and it just gives you a number. Right, an exact number. Yeah, exactly. 150 pounds. It's an absolute fact, you know, an anchor you can rely on. But if you step into the world of training artificial intelligence, suddenly that scale starts giving you completely different numbers depending on, like, what room you're standing in. Or the temperature of the floor. Or even the time of day, really. Right. It gets completely chaotic. So today we're taking you on a deep dive into a massive, ongoing debate in machine learning.

0:36We want to know, how do the smartest AIs in the world actually learn what a good answer is? It is an incredibly complex landscape to navigate. I mean, when engineers try to teach an AI how to converse or reason, which is a process we call reinforcement learning from human feedback, or RLHF, they're essentially trying to objectively score human language. Which is notoriously subjective. Oh, completely subjective. And the research we are analyzing today really digs into the mathematical foundation of that whole feedback loop. Yeah. And the mission for this deep dive is to settle a very specific high stakes question.

1:15When you're training an AI using numerical scores, is it better to tell the AI its exact absolute score? Or, and this sounds crazy, is it better to just tell the AI the relative difference between two of its attempts? It's the ultimate showdown, really. Point-wise scoring versus pair-wise scoring. And whoever wins this debate is currently shaping the next generation of large language models. Well, let's lay out the two contenders on the table before we get into the heavy mathematics. The first approach is the point-wise method, which the research formally calls value regression, or VR. Right, VR.

1:51So let's say the AI generates a response to a prompt, a human rater, or maybe an automated verifier. looks at that response and gives it an exact numerical reward. Let's just call it an 8.5 out of 10. Okay, an 8.5. The AI is then trained to directly predict and aim for that exact 8.5. And the mechanism there is super straightforward. The model just updates its internal parameters to try and hit that absolute target next time. Yeah, it wants the 8.5. But then we have the second contender, which is the pairwise approach. This one is known as value difference regression, or VDR. VDR, yeah. And in this setup, the system actually still collects those exact numerical scores behind the scenes.

2:34Like it knows response A got an 8.5 and response B got a 6.2. But, and this is the wild part, the engineers deliberately hide those absolute numbers from the AI during the actual training process. They literally intercept the data and convert it. Right, they transform it into a relative delta. So the AI is trained purely on the mathematical difference between the two actions, assuming they're under that exact same context. So it never sees the 8.5. Nope, never. The training objective only ever sees an equation like response A is 2.3 points better than response B. It's completely shielded from the 8.5 or the 6.2.

3:12Okay, so to map this onto like a human scenario, value regression, VR, is basically a gymnastics judge holding up a scorecard with a bright red 9.5. You know exactly where you stand in the universe. You have your absolute anchor, yeah. Right, but value difference regression, VDR, is that same judge putting the scorecard face down and just whispering to the coach, hey, contestant A beat contestant B by exactly one point. Yes. And if we push that gymnastics judge analogy a bit further, we really have to ask what happens when the judge has, say, slightly blurred vision. Or the lighting in the arena is just terrible.

3:50Exactly. And this brings us right to the core tension of the data. Which is where I have to push back a little on the fundamental logic of VDR. I mean, if we went through the trouble of acquiring the exact numbers, the 8.5 and the 6.2, why on earth would we convert them into relative pairs? We are intentionally throwing away data. It seems counterintuitive, I know. Right. In the era of massive compute and big data, deliberately blinding the AI to its absolute performance just seems backwards. Why would less information be better? Well, to answer that, we really have to look at the mathematical architecture of the score itself.

4:26The analysis introduces this concept called weak realizability. Weak realizability. Okay. What does that mean? It essentially suggests that we shouldn't view a score like an 8.5 as a single pure metric of quality. Instead, the math actually breaks that score down into two distinct variables added together. The first variable is the action-dependent component. So the meat of the answer, the actual genuine quality of the AI's response, which is what we actually care about. Exactly. But it is inextricably linked to the second variable, which is the action-independent nuisance. The nuisance. Right.

5:03And this is an arbitrary context-dependent factor that heavily skews the final score, but it has absolutely nothing to do with the quality of the AI's output. An action-independent nuisance. Let's try to anchor that in reality for a second. What does a nuisance actually look like when we're talking about prompting an AI? Well, think about the massive variability in how humans actually write prompts. I mean, some prompts are inherently vague or contradictory or just structurally impossible to answer perfectly. Right, like bad instructions. Yeah. Let's say a user types in a highly complex coding question, but they leave out a crucial piece of syntax.

5:43The AI might provide the absolute best possible answer given those flawed constraints. But it's still going to look wrong to a reviewer. Exactly. A human raider looking at the final output might just see a broken piece of code and slap it with a 3 out of 10. Wow. So the absolute score drops drastically. Not because the AI's reasoning was flawed, but because the environmental setup was a nightmare from the start. Right. The context was broken. Furthermore, raiders themselves are super inconsistent. One raider might be incredibly strict on a random Tuesday, you know, artificially deflating all scores by two points across the board.

6:19Because they skipped their morning coffee. Basically, yeah. That environmental noise, the flawed prompt, the strict rater, that is the mathematical nuisance. It's essentially grading on a curve in a university setting. Like if a college math professor writes a final exam that is ridiculously unreasonably difficult, that extreme difficulty is our nuisance. That's a great way to look at it. Right, because every single student's absolute score is going to plummet. A 40 out of 100 might actually end up being the highest grade in the entire lecture hall. Yeah, the absolute scale is totally broken in that room.

6:55So if we rely on value regression, the absolute score, the system looks at the top student and mathematically determines, well, this student only achieved a 40. They must be terrible at math. Exactly. The AI internalizes the bad context as its own structural failure. It literally adjusts its weights to try and fix a problem that was actually caused by the environment. But if we switch to value difference regression, we're no longer looking at the 40. We are just comparing the students against each other. Right. Student A scored 10 points higher than student B on this specific terrible test. And you still easily identify who the strongest mathematician in the room is.

7:37By subtracting student B's score from student A's score, the overarching hardness of the test, the nuisance, mathematically vanishes from the equation. It completely vanishes. The math in the research proves exactly how this cancellation works. Let's say the nuisance variable is a negative 5 penalty applied to everything in that specific context. Okay, negative 5. For response A, the math is essentially quality A minus 5. For response B, it is quality B minus 5. When you subtract response B from response A, those negative 5s cancel out entirely. Oh, wow. So you're just left with pure quality A minus quality B.

8:18Exactly. The relative difference isolates the actual signal. It isolates the signal. It really is. That's fascinating. It really is. In the mathematical models analyzed in the data, relying on absolute point-wise scoring creates a massive misspecification term. A misspecification term. Yeah, it basically creates a heavy bias because the AI is constantly trying to learn the noise of the environment. VDR completely eliminates that bias. The error margin improves significantly. Okay, so VDR cancels out the environmental noise. It grades on a curve and mathematically ignores the nuisance. Based on that, VDR should be the undisputed champion.

9:00I mean, we should throw value regression out the window and always use relative pairs, right? Case closed. Well, we have to pump the brakes there a little bit because the analysis introduces a really fascinating twist. Uh-oh. There's always a twist. Always. When the researchers analyzed what they call linear function classes, they discovered VDR has a hidden, highly dangerous geometric vulnerability. Okay, let's unpack linear function classes first before we get to the danger. How does that change the playing field? A linear function class is essentially how the AI mathematically maps the features of its response, things like length, tone, factual accuracy, vocabulary, to the reward it expects to get.

9:43Okay. You can imagine the AI plotting every possible response it could give on a massive, multidimensional graph. So every dot on this massive graph represents a unique way the AI can answer a question? Right. And the AI's goal is to find the highest point on that graph. But remember, VDR is completely blind to absolute height. It only knows the vertical distance between two specific dots it compares. Because we hit the absolute scores from it. Exactly. And this creates a problem dictated by a metric called the cross-face probability, which the research represents with the Greek letter delta. Cross-face probability.

10:16I am picturing like the AI comparing points across different faces of this multidimensional graph. That is the exact mechanism. It measures how often the AI compares points that are wildly different from each other, say, comparing a highly creative poetic response in one corner of the graph to a hyperlogical rigid response in the opposite corner. But what happens if the AI's current policy is generating responses that are all very similar? Like, what if it's comparing a lot of slightly varied good answers to other slightly varied good answers? That is where VDR falls apart. If the AI is only comparing points that are clustered right next to each other on the graph, the mathematical difference between them is tiny.

10:57Barely noticeable. Right, but more importantly, it never gets a sense of the overarching shape of the entire graph. VDR only sees the tiny, nuanced distances between clustered dots. It lacks the anchors to the origin point. Oh, I see. It is essentially floating in empty space, calculating these micro-differences, but it has no idea where the actual ground is. So, it knows contestant A beat contestant B by 0.1 points. But because it has no absolute anchor, it doesn't know if they are both Olympic gold medalists at the top of the graph or both amateurs falling off the balance beam at the bottom.

11:32That's a perfect analogy. Okay. Which means VDR suffers from incredibly high variance. It is highly unstable when cross comparisons are rare. And this is exactly where the point-wise approach, value regression, suddenly re-enters the fight. Exactly. VR might have bias from the nuisance, but it has absolute anchors. A 9.5 is tied directly to the ground truth of the origin. Wait, let me make sure I am tracing this causality correctly. If the AI is operating in a relatively clean, straightforward environment, meaning the nuisance factor is low, taking the supposedly safer relative VDR approach actually hurts the training?

12:11It can, yes. Because it gets confused by the lack of absolute anchors. Precisely. The research tested this exact dynamic using a synthetic, linear banded experiment. They built this isolated, controllable mathematical environment where an AI agent had to choose between different actions to maximize its reward. And they introduced a dial to control the magnitude of the nuisance, which they called beta. A literal knob they could turn up or down to increase the environmental noise. Yeah. And when they turned the nuisance beta dial down to exactly zero, meaning a perfectly clean environment with no confusing contexts, value regression absolutely dominated.

12:49Because the scale wasn't broken. Right. The absolute scores provided faster, vastly more stable learning. A 9.5 was genuinely a 9.5. VDR, meanwhile, just floundered because of its geometric variance. It really struggled to orient itself without those absolute anchors. But what happens when they start turning the beta dial up? As the nuisance beta grew, the environment obviously got noisier. And mathematically, VR's bias grew completely linearly with the noise. So it started chasing the ghosts of the environment, getting more and more confused by the shifting context. Exactly. Eventually, the dial hit a threshold where the nuisance became so overwhelmingly large that the absolute scores were essentially meaningless.

13:32And that is the exact moment VDR overtakes it. The bias of VR becomes a heavier burden than the variance of VDR. The tug-of-war ends right there. At high noise levels, knowing the relative difference, even without knowing where the ground is, is literally the only mathematically sound way to find the best action. It's just a brilliant mathematical trade-off. But, you know, theoretical bandit models in an isolated lab are one thing. We really need to take this out into the real world. How does this geometric bias-variance trade-off actually play out when training the massive large language models we interact with daily?

14:07Well, to bridge that gap, the analysis moved to practical empirical applications. They took these two methods, absolute VR and relative VDR, and tested them on frozen representations of state-of-the-art LLMs. Which ones did they use? Specifically, they used LAMA 3.23B, QUEN3, and ARMORM. Okay, so those are real industry heavyweights. How did they stress test them? They ran them through massive complex data sets like WildChat, which is full of highly variable, messy human conversations, and GSM 8K, which is a rigorous benchmark for complex mathematical reasoning. Nice. And to declare a winner, they didn't just look at basic accuracy.

14:44They used a metric called held out offline regret. Regret. I love that term. In machine learning, regret measures the gap between the action the AI chose to take and the absolute best optimal action it could have taken in hindsight, right? Correct. If the AI chooses the absolute best possible response, regret is zero. If it chooses a mediocre response when a brilliant one was actually available, regret spikes. Right. It regrets not picking the better option. So the goal of any training methodology is to minimize regret as much as possible. Exactly. So when they pitted VR against VDR on these massive models trying to minimize regret, what tipped the scales?

15:26The decisive mechanism turned out to be a variable designated as N sub A. Let's define N sub A for everyone. N sub A simply represents the number of alternative training responses generated for a single prompt. Okay, so if I ask the AI to write a poem, N sub A is how many different drafts of that poem the AI generates before we score them and train on them. You've got it. When n sub a was small, say just two or four responses per prompt, value regression held its ground. Sometimes it even outperformed VDR, yielding lower regret. That directly aligns with the math from the linear banded experiment.

16:04If you only have two or four data points per prompt, it's a small, manageable room. The AI desperately needs those absolute anchors to make sense of the sparse data. It doesn't have enough geometric spread to rely purely on relative distance. Exactly. The variance of VDR is just too high with only two or four points of comparison. But as we know, the modern trend in AI is massive scaling. So the researchers started cranking up N sub A. How high did they go? They asked the models to evaluate 64 responses, then 128 responses, and finally 512 different responses for a single prompt. 512 drafts of the same prompt.

16:45That's insane. How does that sheer volume change the math? Well, as NSAID scales up, you are drowning the system in data. And more data actually amplifies the complexity of the context. When an AI generates 512 subtle variations of a response, the absolute baseline shatters. Because they're all so similar. Exactly. An absolute score of an 8.2 on response number 400 means almost nothing. Because the nuance is so thick, the environment itself becomes a massive nuisance variable. Oh, wow. VR gets completely overwhelmed trying to anchor 512 different points to a shifting origin. But VDR doesn't care about the origin.

17:24It just looks at the distance between the 512 points, completely ignoring the overarching noise of the crowded stadium. Which is why at N-sub-A equals 512, VDR utterly dominated across the board. Really? How big was the gap? The reduction in regret was staggering. Using the relative VDR method reduced regret by 9.7 % for the LAMA model, 13.0 % for QUIN3, and a massive 24.0 % reduction for RMO-RM compared to using the absolute point-wise scores. A 24 % reduction in AI error. Not by adding more parameters, not by changing the fundamental architecture of the neural network, but just by changing how you format the math of the feedback score.

18:11That highlights how critical the reward function truly is, and they confirmed this pattern in a completely separate reasoning experiment, too. Oh, really? Yeah, they used Monte Carlo Tree Search, or MCTS, to test mathematical logic. The AI had to explore different branches of logic to solve a math problem. And what happened? Even there, a binary preference version of VDR decisively beat VR for finding the correct final math answers. When the search space gets large, absolute scores just break down. That is wild. Let's synthesize what we've unpacked here, because I think it reveals something fundamental, not just about artificial intelligence, but really about how massive systems process information.

18:51We've navigated the dense mathematics of AI reward learning and isolated this profound tradeoff. We really have. On one side, we have value regression, absolute scores. They provide stability and necessary anchors when data is sparse or the environment is incredibly clean. But the math proves they are fundamentally brittle. They are highly vulnerable to context-driven biases and action-independent nuisances. And on the other side, we have value difference regression, relative comparisons. By grading on a curve and subtracting one score from another, VDR acts as a mathematical shield against environmental pollution.

19:29Exactly. It suffers from variance when data is low, sure. But as AI models scale up to process hundreds of responses at a time, calculating the relative difference is the only mathematically sound way to extract genuine quality from the noise. As the systems get more complex, the context becomes the nuisance. VDR cuts right through it. It's elegant, really. It is a phenomenal look at the invisible math dictating the future of machine learning. But, you know, before we wrap up this deep dive, I really want to take this theoretical ML concept and pivot it towards something we all deal with. Okay.

20:03I want to leave you with a provocative thought about how we structure human organizations. Oh, the implications are definitely broader than just code. Right. Think about the massive, complex systems we navigate every day. Corporate structures, university grading, annual performance reviews. We are obsessed with absolute metrics. We try to objectively score an employee's performance on a rigid one to five scale, or we hold students to an unyielding percentage grade. We rely entirely on the value regression approach for human evaluation. We do. We really do. But if advanced mathematics and rigorous empirical data on the world's smartest LLMs prove that absolute scores inevitably become polluted by the nuances of a complex environment, well, are we utilizing fundamentally broken metrics in our own lives?

20:53That's a great question. Right. Like, an employee who scores a 3 out of 5 might be operating under a terrible manager, an impossible quarterly quota, or a completely broken supply chain. We are internalizing the noise of the company as the failure of the employee. You are suggesting that HR departments and educational boards are failing to account for their own action-independent nuisances. The math strongly suggests they are. Perhaps. Instead of asking, what is your exact objective numerical score on this performance review, we should be deploying value difference regression in our own lives.

21:31Evaluating relative impact. Exactly. We shouldn't be searching for absolute anchors in a noisy world. We should just be asking, who is actively elevating the baseline in this specific chaotic context? It really makes you wonder if that broken bathroom scale from the beginning of the show isn't broken at all. Maybe we've just been reading the wrong metric the entire time. Exactly. When the environment is noisy, the relative difference is the only truth you can rely on. A really fascinating lens to view our own systems through. We'll leave you to calculate the math on that one. Thanks for joining us on this deep dive.

22:05See you next time.

From the publisher

This research paper investigates when and why pairwise losses outperform pointwise losses for reward learning by analyzing both approaches within a grouped offline contextual-bandit framework. The authors compare Value Regression (VR), which directly fits absolute observed rewards, with Value Difference Regression (VDR), which models reward differences between action pairs sharing the same context. Through a localized mathematical analysis, the study establishes finite-sample prediction guarantees and offline-regret bounds for both finite and linear function classes. The theoretical findings reveal that VDR successfully eliminates nuisance-induced misspecification bias that disrupts pointwise regression, whereas VR can maintain lower estimation variance under correct specification depending on the underlying feature geometry. Consequently, the choice between these learning strategies introduces a fundamental bias-variance tradeoff, which is further validated through both synthetic simulations and real-world language model response-selection experiments.

More from Best AI papers explained

All episodes