


18 Oct 2026 · 23m · 16 chapters

The episode explains “OSVerify,” an algorithmic framework for one-sided verification when AI output is too large for human experts to review. It argues that standard “LLM as a judge” holistic grading fails because models miss errors in long proofs, while OSVerify separates error-finding from error-checking to maximize true negatives under a capped false-negative rate.
No guest names or biographies are provided; the transcript is a dialogue between two hosts.
Verification should use Neyman-Pearson one-sided verification (reject invalid candidates while keeping false negatives under ~5%). OSVerify uses an error proposer (hierarchical decomposition to generate suspicious hypotheses) and an error validator (checks specific proposed errors in isolation). Using weaker verifiers avoids memorization.
Ant-on-a-rectangle puzzle (counterexample to a fake lemma). OSVerify found errors in 11 OpenAI math preprints and in Annals of Mathematics papers by Shinwen Zhu and by Liu and Wong (45 and 88 issues). It also helped resolve open conjectures in relative Langlands duality and targeted the Manin constant conjecture, correcting a mistake missed by humans and 12 holistic AI review rounds.
Written by AI. May contain mistakes.
Discussion on the challenges of filtering AI-generated hypotheses.
“I mean, that is the defining bottleneck of modern scientific discovery.”
Introduction to one-sided verification and its significance.
“And whether you are, say, a programmer trying to check AI-generated code, researcher swimming in data, or just someone who's exhausted by the daily avalanche of information overload.”
Explaining the verification process using gold panning as a metaphor.
“Simply because 99 % of the solutions were garbage anyway.”
Exploration of the limitations of current AI models in verification.
“I mean, if I have a massive state-of-the-art model with billions of parameters, why can't I just feed it a newly generated math proof and ask, hey, act as a judge, is this correct?”
Introduction to the OSVerify framework and its components.
“When they grade holistically, they just do not have the reliable capability to spot hidden errors.”
Detailing how the error proposer identifies potential errors.
“Okay, I want to look at the first half of that duo.”
Explaining how the error validator processes flagged errors.
“The cool, calm judge that actually looks at the evidence.”
Discussion on the rigorous testing of the OSVerify framework.
“What happens when you actually force this duo to grade real, brutal, PhD-level math problems?”
Explaining the training techniques used for the error validator.
“Across these highly complex benchmarks under that strict 5 % false negative constraint we talked about earlier.”
Illustrating OSVerify's effectiveness through a real-world example.
“But, you know, statistics can feel a little abstract.”
Learn how OS Verify identified and corrected errors in complex mathematical proofs.
“The error proposer sliced up the proof, zeroed in on that specific stepping stone, and basically raised its hand, proposing that the structural lemma felt suspicious.”
Discover how OS Verify is applied in high-level mathematical research and its implications.
“We are talking about the Langlands program.”
Understand the significance of minor errors in mathematic papers and their implications.
“The system identified 45 expert verified issues in Zhu's paper.”
Explore how OS Verify enhances AI math solvers and aids in proving open conjectures.
“When the researchers paired this verification system with an agentic AI math solver...”
Learn about the evolving role of AI in scientific research and its potential benefits.
“This is a problem in number theory that has been open for over 50 years.”
Consider the broader implications of OS Verify in fields beyond mathematics.
“It is respecting the human's limited time.”

Get the gist, explore an idea or ask about something said.
On Best AI papers explained, what are the key points of this episode?
Automatic transcript. May contain errors.
0:00Imagine an AI that can formulate a cure for a rare disease in the time it takes you to drink your morning coffee. Like, it just hums for a few seconds and then hands you 10 ,000 potential chemical formulas. Right. And on paper, that sounds completely amazing. Exactly. But here is the problem. 9 ,999 of those formulas are completely useless or, you know, maybe even dangerous. Yeah, that is the catch. And it is entirely up to you to manually test every single one of them in a laboratory to find the one that actually works. The AI just performed essentially a century of discovery in a single second.
0:38But in doing so, it created a century of manual labor for you. Right. And we are completely outpaced by our own creation at this point. We really are. I mean, that is the defining bottleneck of modern scientific discovery. We have built this infinite engine for generating hypotheses and drafting math proofs and proposing novel solutions. With the filter. Exactly. The ultimate filter for all of that output is still us, the human experts. What happens when we simply cannot review everything the AI is producing? Because if we can't effectively filter the output, the entire system just grinds to a halt under the weight of its own, like, unverified data.
1:16It becomes a data avalanche. Okay, let's unpack this, because solving that exact bottleneck is the core mission of our deep dive today. We are exploring this really groundbreaking algorithmic framework called OSVerify. And it is fascinating because it fundamentally changes the very nature of how AI verifies complex information. It really does. And whether you are, say, a programmer trying to check AI-generated code, researcher swimming in data, or just someone who's exhausted by the daily avalanche of information overload. Which is all of us, honestly. Right. All of us. Understanding how to filter out the junk without accidentally throwing away the gold is a vital modern survival skill.
1:58Definitely. And to understand how this OS-Verify framework attacks the problem, we first have to kind of look at the mathematical philosophy of filtering itself. Okay, lay it on me. So in domains like high-level mathematics or quantum physics, a valid solution is an extreme anomaly. It's incredibly rare. Because almost every guess an AI makes in those fields is just going to be wrong. Exactly. And this creates a statistical trap if you try to evaluate an AI verifier using standard metrics of accuracy. Wait, why is that a trap? Because if the AI just blindly rejects every single proposed solution it sees, it could technically boast a 99 % accuracy rate.
2:35Oh, I see. Simply because 99 % of the solutions were garbage anyway. Right. But if it rejects everything, we learn nothing. We lose the cure, we lose the mathematical proof. So standard accuracy is a terrible metric here. It is. So to escape this trap, statisticians rely on something called the Neyman-Pearson principle. Instead of looking at overall accuracy, the goal shifts to one-sided verification. One-sided verification. What does that mean in practice? The objective becomes incredibly specific. You want to maximize your true negative rate, meaning you accurately identify and toss out as many invalid candidates as humanly possible.
3:12Right. But, and this is the key, you have to do that while keeping your false negative rate strictly capped, usually under something like 5%. You absolutely cannot throw away the rare, valid solutions. You must preserve them at all costs. Let me see if I can picture this, because it sounds exactly like panning for gold. Oh, that's a good way to look at it. Like you are standing in the river with your pan, and the objective is to wash away as much dirt and gravel as possible. Getting rid of the dirt is your high true negative rate. Right. But your sieve cannot have holes so big that a gold nugget slips through and washes down the river.
3:49You just cannot lose the gold. That analogy captures the tension perfectly. You want to minimize the amount of gravel the human expert has to manually pick through later without ever letting the nugget slip away. Okay, but I have to push back a little here. Why is this gold panning process so notoriously hard for our current large language models? It's a great question. I mean, if I have a massive state-of-the-art model with billions of parameters, why can't I just feed it a newly generated math proof and ask, hey, act as a judge, is this correct? Well, many researchers have tried that exact approach.
4:25They usually call it the LLM as a judge framework. The fatal flaw is that current models simply cannot reliably establish the holistic validity of highly complex scientific results. What do you mean by holistic validity? Like when you give an AI a dense 40-page proof and ask it to grade the whole thing at once, it just gets tunnel vision. Oh, so it's essentially nodding along, pretending it understands the whole document when it actually missed a fatal flaw on, like, page three? Often, yes. It might completely gloss over a massive logical leap because the surrounding text looks, you know, highly confident and academically formatted.
5:01It gets fooled by the jargon. Exactly. Or, conversely, it might hallucinate a flaw that isn't actually there because it lost track of a variable halfway through. When they grade holistically, they just do not have the reliable capability to spot hidden errors. They pass too much gravel or they accidentally throw out the gold. Precisely. So structurally, if asking an AI to be a holistic judge is a dead end, how do we fix it? Well, we completely abandon the holistic approach. And this is where OSPherify pulls off an algorithmic trick that mirrors a really profound realization about human psychology.
5:36Oh, psychology. How so? The researchers recognized this massive difficulty gap in how we process information. Searching through a dense, highly technical document to discover an error from scratch is incredibly difficult. Right. Because it requires holding the entire architecture of the argument in your head while simultaneously scanning for microscopic cracks. That sounds exhausting just thinking about it. It is. However, checking whether a specifically proposed error is valid is much, much easier. Oh, I completely see that. Yeah. It is like the difference between handing an editor a giant stack of paper and asking, hey, is there a typo anywhere in this 500-page manuscript?
6:15Good luck with that. Right. Versus asking them, hey, look at page 42, line 7. Is the word there spilled wrong in that specific sentence? Yes. The first task is overwhelming. The second task is trivial. Okay. So how does OS Verify actually build a system around that? It builds its entire architecture around that exact difficulty gap. It divides the labor of verification into two separate adversarial AI components, the error proposer and the error validator. Okay, I want to look at the first half of that duo. How does the error proposer search for these flaws if it is intentionally not looking at the whole picture?
6:52It uses a technique called hierarchical decomposition. Hierarchical decomposition. Okay. So the proposer takes the AI-generated solution and starts slicing it into smaller and smaller atomic pieces. Like zooming in. Exactly. It looks at the whole document, then cuts it into chapters, then cuts those into paragraphs, then into individual equations or sentences. It recursively scans the solution across multiple levels of granularity. Okay. And what is it looking for when it zooms in like that? Its only job is to flag anything that looks even remotely suspicious and propose it as a potential error.
7:29Oh, wow. So it is essentially acting as a hyper-paranoid detective. Pretty much. It doesn't actually have to prove the error is real. It just points a finger and says, this step looks weird. Or this equation might be skipping a variable. Exactly. It generates a massive list of potential hypotheses for where things might have gone wrong. Casting a really wide net. Casting that wide net ensures it doesn't miss anything. But naturally, a paranoid detective generates a lot of false alarms. Right. Like hallucinated errors or minor typos that don't actually ruin the overall proof. Exactly. And that is where the second component steps in.
8:05The error validator. Ah. The cool, calm judge that actually looks at the evidence. You got it. The validator receives a very specific isolated package. It gets the original problem, the proposed solution, and one specifically proposed error from the proposer. So it isn't searching for anything at all? None. It only has to answer one question. Is this specific proposed error real? Wow. It checks if the error is a hallucination, and if the error is real, it then has to decide if it is just a minor formatting mistake, or if it is a substantial error that mathematically destroys the entire solution.
8:41I love how separating these two tasks allows both of them to be scaled up independently. What is fascinating here is that if you have more computing power, you don't just throw raw processing power at a holistic judge and hope he gets magically smarter. Right. You structurally organize the compute. You can have the proposer slice the document even finer to generate more hypotheses, and you can give the validator more time to check each specific claim. Exactly. It converts the impossible task of solution verification into the much more manageable task of error validation. Okay, but a theory on a whiteboard is one thing.
9:16What happens when you actually force this duo to grade real, brutal, PhD-level math problems? Well, they tested it. Let's talk about this lab test, because I know they put this framework through the ringer, using some of the most punishing math benchmarks out there, like Putnam 100, Hard to Verify, and First Proof-Eval. Right. These are data sets containing incredibly difficult math problems, along with AI-generated solutions that have been meticulously graded by human experts. And they withheld those human-verified answers from the OzVerify system, right? Completely. And any formal proof checkers.
9:52It had to figure out validity entirely on its own. Wait, I was reading about the setup for these experiments, and something tripped me up. I noticed they deliberately chose to use relatively weak large language models to act as the verifiers. They did, yeah. They used models like GPT-AUS-120B and GPT-5-MINI. If we were trying to verify cutting-edge complex science, why are we using MINI models? Aren't we just automating ignorance at that point? It definitely seems counterintuitive. I mean, why not use the smartest model? But it was a strictly necessary choice. Why? If they used the absolute most powerful frontier model available, that model might just instantly know the answer to the math problem because it memorized it during its training data phase.
10:39Oh, so it wouldn't actually be verifying the steps. Right. It wouldn't test the verification framework at all. It would just test the model's memory. So by intentionally using weaker models, they simulated real-world scenarios where the complexity of the problem vastly exceeds the AI's reliable problem-solving capabilities. So it forces the system to rely entirely on the OS-Verify method, the proposing and validating, rather than just being super smart and knowing the answer outright. Exactly. It proves that the framework itself elevates the capabilities of whatever underlying model you happen to use.
11:14That makes total sense. And how did it actually do? The results were stunning. Across these highly complex benchmarks under that strict 5 % false negative constraint we talked about earlier. Protecting the gold. Yes. OS Verify reduced the number of invalid candidates requiring human review by an average of 46 % compared to standard voting baselines. Wow. That is nearly half the dirt washed out of the gold pan without losing the gold just by changing the structure of the prompt. And they actually took it a step further. Oh, really? Yeah. They trained the error validator using a reinforcement learning technique called GRPO.
11:50The goal was to specifically teach the validator how to better distinguish between harmless errors and fatal ones. Wait, how do you even train an AI to know the difference between a typo and a substantial solution-destroying error? Well, by giving it targeted practice with clear rewards. They feed the validator thousands of examples. Okay. And some examples may be a missing comma or a slightly misnamed variable doesn't actually change the underlying math. So the AI is penalized if it throws the whole solution away. Right, because it tossed out a valid solution over a typo. Exactly. But in other examples, a missing variable completely collapses the equation, and the AI is rewarded only when it correctly identifies that structural collapse.
12:32Oh, that's clever. And with this targeted training, the reduction in human review effort jumped from 46 % to 57%. That is a massive return on human time. But, you know, statistics can feel a little abstract. There is a specific anecdote from the hard-to-verify dataset that I think perfectly illustrates exactly how OS Verify behaves in the trenches. Oh, are you talking about the ant on a rectangle puzzle? Yes. Solution 62. This is such a fascinating case study. So to paint a picture for you, the listener, the problem basically asks for a proof involving an ant moving across a large rectangle that has been divided into a grid of smaller rectangles.
13:15Right. An AI generated a dense mathematical proof for this. And what is wild is that human PhD annotators originally looked at this AI-generated math solution and labeled it as valid. The human experts thought it was correct. Yeah. The AI's argument appeared totally sound on a macro level. To solve the problem, the AI relied on what mathematicians call a lemma. Right. And you can think of a lemma as a stepping stone rule, like a mini proof inside a bigger proof. Exactly. The AI's stepping stone rule stated that because an interior vertical cut ended at the corner of one of these smaller rectangles, the whole shape was essentially a series of parallel strips.
13:57And the human PhDs looked at that stepping stone, thought it made logical sense in the context of the larger argument, and just gave the proof a passing grade. But the OS Verify system dug in. The error proposer sliced up the proof, zeroed in on that specific stepping stone, and basically raised its hand, proposing that the structural lemma felt suspicious. And the validator then took that specific suspicion, analyzed it in total isolation, and actually found a counterexample. A counterexample. Yes. It proved mathematically that a vertical cut ending at a corner does not necessarily mean the whole dissection consists of parallel strips.
14:34So the stepping stone was fake. Completely fake. The validator caught a false structural lemma proving the entire solution was actually invalid. It caught a subtle mathematical lie that even the human experts had initially glossed over. It really did. Does something like that bruise the ego of the PhDs? Having a mini model catch their oversight? Honestly, I think it forces a shift in how we view AI. Instead of seeing it as this omniscient peer that might replace them, the human experts start seeing it as an incredibly rigorous, tireless quality control assistant. It is a relief. So we have a system that can catch fake stepping stones and logic puzzles, but let's take it out of the sandbox.
15:14The researchers took OS Verify and deployed it into the absolute highest echelons of professional, real-world mathematics. We are talking about the Langlands program. Yes. For anyone unfamiliar, the Langlands program is often called the Grand Unified Theory of Mathematics. It is a web of incredibly deep esoteric conjectures connecting number theory and geometry. I mean, it is the kind of math that wins Fields medals. It is arguably the most challenging area of mathematical research alive today. The arguments are technically involved, and the theories require substantial specialist expertise just to read, let alone verify.
15:50And first they had OS Verify review 11 preprints from OpenAI, papers that claimed to solve central problems in mathematical research. And what happened? OS Verify autonomously found non-trivial mathematical mistakes in every single one of those 11 papers. Wow. Keep in mind, these were mistakes that the internal models generating the papers had entirely overlooked. And when human experts reviewed OS Verify's findings, they confirmed the errors were real. I mean, the main conclusions of the papers usually survived, but correcting these foundational issues was vital for the accuracy of the literature.
16:25But they aimed even higher than that. They aimed at the very top of the academic mountain, the Annals of Mathematics, which is basically the most prestigious journal in the field. Oh, yeah, the Zhu paper. Right. They fed OS Verify a published paper by a top mathematician, Shinwen Zhu, and a recent preprint by Liu and Wong. The system identified 45 expert verified issues in Zhu's paper. and 88 expert verified issues in the Liu and Wong preprint. And the human authors of those papers actually confirmed the reported errors and accepted the corrections proposed by the AI. That is just mind-blowing to me.
17:00There's one specific catch in Zhu's paper that I think highlights why this framework is so revolutionary. The paper had this really subtle sign error. It involved something called causal signs in a commutivity constraint, where the paper used an ordinary flip instead of a graded flip. Okay, let's translate that for anyone who doesn't have a PhD in number theory. What does that actually mean in practice? Well, it is the highest level equivalent of doing a massive algebra equation and forgetting that multiplying two negative numbers makes a positive. Oh. It essentially flipped a positive value to a negative one, but it did this inside a complex, multidimensional geometry problem.
17:39So it's like a tiny minus sign where a plus sign should be just buried in 40 pages of dense abstract math. Exactly. And what is truly remarkable is that this subtle sign error went entirely unnoticed by top human mathematicians for years. It survived rigorous peer review. It survived publication in the Annals of Mathematics. But OSVerify, through its methodical hierarchical decomposition, isolated the exact claim, proposed that the sign was wrong, validated the error, and even proposed the valid correction. Here is where it gets really interesting. OSVerify isn't just acting as a hyper-pedantic peer reviewer looking for minus signs.
18:20When the researchers paired this verification system with an agentic AI math solver... An AI designed to actively try and solve unsolved problems, right? Yes. When they paired them together, it actually helped resolve open conjectures. In the realm of relative Langland's duality, the paired system produced proofs for three previously open statements. What is the mechanism there? How does a verifier actually help a solver discover something new? Think of the solver as someone running through a massive maze. Usually a math AI wanders down a path, gets stuck, and hallucinates a bridge to keep going.
18:57It fakes its way through. Right. But with OS Verify acting as a co-pilot, every time the solver tries to build a fake bridge or make a false logical leap, OS Verify acts like a drone overhead. It flags the air immediately and says, Nope, that brick is hollow. That path is invalid. It shuts down the hallucinations in real time. Exactly. It forces the solver to turn around, abandon the dead end, and find a real mathematically sound path. It provides this vital verification signal during the solving process. And perhaps the crown jewel of this whole deep dive is how this pairing tackled the Manin constant conjecture.
19:34Oh, this is an incredible story. This is a problem in number theory that has been open for over 50 years. Human mathematicians had been banging their heads against it for half a century. and recently a paper was drafted claiming to finally prove it. But an earlier version of that paper contained a mistake. A subtle one, obviously. Extremely subtle. A mistake that was overlooked by human reviewers and overlooked by 12 separate rounds of standard AI review. 12 rounds of holistic AI judges totally failed to spot the flaw. They all got tunnel vision. But oh, as far as I got it. It found the microscopic crack in a 50-year-old mathematical puzzle that everyone else, human and machine, had missed.
20:14If we connect this to the bigger picture, this completely reframes how we should be thinking about AI in science. The fear has always been that AI will either replace human scientists or that it will bury them under a mountain of unverified, hallucinated garbage. Right. The data avalanche we talked about. Exactly. But OS Verify demonstrates a third path. It isn't replacing the human. It is respecting the human's limited time. It ensures that when a problem finally reaches the human expert's desk, the gravel has been washed away and only the highest probability nuggets of gold remain. Exactly. So to summarize the journey we've been on today, we started with a problem of an infinite hypothesis generator hitting the bottleneck of human review.
20:56Right. We looked at how standard holistic AI judges failed to spot subtle reasoning errors because they tried to grade everything at once. And we explored how the OSVerify framework solves this by breaking verification into two adversarial roles. A proposer that hierarchically decomposes a solution to find potential flaws, and a validator that strictly checks if those specific flaws are real and substantial. By changing the verification process from a massive find-the-needle-in-the-haystack search into a highly targeted is-this-specific-object-and-needle validation, we dramatically increase our true negative rate without sacrificing our valid discoveries.
21:35We saw it work, not just in lab benchmarks, but in catching decades-old errors in the most elite mathematical journals in the world. It leaves me with a thought that I want to pass on to you, the listener. We have spent this entire deep dive talking about how this decentralized, two-part framework can find microscopic logical flaws in the most esoteric, abstract mathematics in the world. But what happens if we take OS Verify out of the math department and apply it to our daily lives? The potential applications are honestly fascinating. Think about it. Could this exact same one-sided verifier be unleashed on newly drafted legal legislation?
22:14Like, could we feed a 1 ,000-page omnibus bill into an error proposer and have an error validator automatically flag substantive logical contradictions or hidden legal loopholes before the bill was ever signed into law. Oh, that would be revolutionary. What about political speeches, corporate merger contracts? If an algorithm can catch a missing minus sign in a geometry problem that eluded human geniuses for years, what hidden flaws could it find in the fine print of the systems that govern our society? Mathematics is just the proving ground. The true application of rigorous automated error validation is essentially limitless.
22:52we might finally have a sieve fine enough to catch the truth. Thank you so much for joining us on this deep dive. We know your time is valuable and we appreciate you spending it with us. Keep questioning, keep filtering, and keep exploring.
AI-driven scientific discovery increasingly relies on automated verification to evaluate candidate solutions, but the scarcity of reliable automated metrics makes human expert review essential for validating complex research. Because generating candidate solutions often outpaces human review capacity, OSVERIFY introduces an innovative algorithmic framework that separates verification into hierarchical error proposal and error validation to maximize the rejection of invalid solutions while preserving rare valid ones. Experiments across mathematical benchmarks demonstrate that this approach significantly reduces human review workload compared to standard voting baselines while maintaining strict false-rejection constraints. Furthermore, the application of OSVERIFY to real-world research successfully identifies subtle mistakes in published papers and helps resolve open conjectures in the Langlands program. Ultimately, this methodology facilitates human-AI research collaboration by providing a structured, scalable solution to the verification bottleneck in open-ended scientific domains.