SPIRAL: Learning to search and aggregate

0:00You know, there's a very specific way we tend to visualize artificial intelligence getting smarter. Oh, yeah. Like a chart or a graph. Right. Exactly. We picture this graph with a line just pointing up and to the right. The basic assumption is that, you know, if you give a model more computing power and just, well, more time to process a prompt, it scales perfectly. It just automatically gets more intelligent. Exactly. But the reality we are looking at right now presents this fascinating, almost jarring paradox. I mean, take the Erden's unit distance problem, for example. Oh, wow. Yeah. That is a notoriously difficult puzzle in discrete geometry.

0:37Right. It essentially asks for the maximum number of pairs of points in a set that can be exactly a unit distance apart. It involves incredibly complex combinatorics. Exactly. And human mathematicians struggled with variations of it for, gosh, over 70 years. But recently an AI cracked it. It did. Just stepped in and solved a problem that stumped human geniuses. But then, and this is where the paradox comes in, You take that exact same highly advanced model and you feed it a basic first proof logic puzzle. Something super simple. Yeah, something along the lines of like if A implies B and B implies C, what is the relationship between A and C?

1:13And the model completely derails. It just falls apart. It really does. It hallucinates or it gets stuck in an endless loop or, you know, just outputs something entirely nonsensical. In the field, we refer to this as a diagnostic muddy landscape. A muddy landscape, huh? Yeah, but the technical term that gets thrown around a lot is the jagged edge of AI capability. The jagged edge. Right, because we have these systems that are simultaneously capable of profound, world-class mathematical reasoning and highly specific scenarios, and yet they are utterly helpless when faced with basic logic that, honestly, a high school student could navigate in their sleep.

1:53It's wild. And today we are diving deep into a groundbreaking new framework that actually attempts to smooth out that jagged edge. It's a really exciting development. It is. And as we look into why this jagged edge happens, it turns out the failure isn't because the unsolved problems are inherently harder. The failure lies in what the AI actually does when we give it more time to think. Yeah, we are talking about what's called inference compute. Break that down for us. Sure. So inference compute is basically the processing budget. It's the amount of time and the number of operations an AI is allowed to run through before it finally spits out an answer to your prompt.

2:32OK, so it's the thinking time. Exactly. And the industry assumption has pretty much always been that more inference compute equals better answers. Just let it think longer and it'll get it right. But the data shows that when we increase that processing budget, the model doesn't automatically get smarter. Like it often just completely wastes all that extra brainpower. It really does. It allocates its resources so poorly. Right. It will meticulously detail totally routine basic operations over and over again while completely missing the crucial logical leap required to actually arrive at the solution.

3:07You might have seen this yourself, actually, if you use these models frequently. Oh, absolutely. You ask a complex question and the AI gives you this massive multi-paragraph response. And it's filled with flawless grammar, correct definitions, but it entirely misses the actual point of your question. Yes. So the effort is there. The compute is there. But the cognitive allocation is entirely wrong. Precisely. Which is the core focus of our deep dive today. This new framework is called SPIRAL. And that stands for, let me get this right, sequential parallel aggregative reinforcement learning. That's right, spiral.

3:45And the goal of spiral isn't just to make the AI faster, right? It's designed to fundamentally change how AI models allocate their brain power. It's teaching them to actively search and synthesize information rather than just guessing in a straight line. Right. So to grasp why spiral represents such a huge paradigm shift, I feel like we first have to break down how an AI actually processes information at a structural level. Yeah, we definitely do. There are three distinct ways it can compute. We call these the three primitives of compute. The three primitives. Okay. And the reason we have that jagged edge of intelligence is because currently most AI is only natively trained to be really good at just one of them.

4:24Okay, let's start with the one it's good at. The first dimension of AI thinking. That would be sequential compute. This is the standard behavior you see when you just open up a chatbot. It's the AI thinking in a single linear chain of thought. Like a straight line. Exactly. Under the hood, it's generating intermediate reasoning steps. One token or piece of a word after another, building up to an answer. It is locked into a single trajectory. So if I'm thinking about this in terms of, like, writing a paper, sequential compute is like sitting down and drafting an entire essay top to bottom. You start at the introduction, you move to the body paragraphs, and you just write straight through to the conclusion.

5:04Right, but with the inherent risk that if your core thesis was flawed in paragraph one, well, the entire sequential effort is totally wasted. Oh, right, because you just spent all that time building on a bad idea. Exactly. You have to scrap the whole thing and start a new sequence. And this vulnerability is exactly why we need the second dimension, which is parallel compute. Which shifts the approach from a single track to multiple tracks. Yes, exactly. Parallel compute is where the AI samples multiple, completely independent reasoning paths all at once. Instead of drafting one essay, you are exploring a broad range of possible solutions simultaneously.

5:40So sticking with our essay analogy, parallel compute would be like drafting five different thesis statements on five different pieces of paper just to see which argument has the strongest legs. That's a great way to put it. That covers the exploration phase. But, you know, you still need a final product. Right. You can't just hand in five thesis statements. No, you can't. Which requires the third primitive, and that is aggregative compute. The synthesis phase. Exactly. The AI takes all of those parallel candidate paths, verifies them, filters out the hallucinations or the dead ends, and synthesizes the useful mathematically sound bits into one final highly refined answer.

6:18Okay. So it's like taking the strongest argument from page two, combining it with the data from page four, and then write it the ultimate perfect introduction. Yes. That makes total sense. You need all three, right? Yeah. The linear thought, the broad exploration, and the final synthesis, just to arrive at a complex truth efficiently. You really do. But, you know, this raises an obvious question. Yeah, because if we already know parallel and aggregate of compute are helpful, like, if they are necessary for solving complex logic, why hasn't the industry just been forcing AI models to operate this way all along?

6:53That's the billion dollar question. I mean, can't developers just write a script that says, hey, give me five answers, compare them and combine the best parts? Why do we need a completely new mathematically complex framework like Spiral to do this? Well, developers have been doing exactly that for a while. They use what are known as hand-designed scaffolds. Scaffolds, okay. Basically, they are external human-coded rules that are bolted onto the outside of the AI to force it into a parallel structure. Like training wheels. Exactly. But there is a fatal flaw in that approach. During the reinforcement learning phase, which is the post-training where the AI learns how to behave, models are almost entirely optimized for sequential compute alone.

7:33Oh, I see. They are trained to succeed by generating a single high-quality chain of thought. So on a fundamental architectural level, they are completely blind to the other two dimensions during their actual training. They literally don't know how to brainstorm. That is exactly it. And that lack of native training causes the whole system to break down when you force it into a scaffold. When you make an AI do parallel thinking without training it to do so natively, its entropy collapses. Wait, entropy collapses, meaning its randomness, like its ability to generate diverse ideas. Yes. It just spits out highly redundant, highly correlated attempts.

8:10Oh, so to use the analogy, you ask the AI for five different thesis statements, and it just hands you the exact same sentence written five slightly different ways. Yes. It doesn't know how to explore the solution space broadly. And worse, it struggles to verify and combine those ideas without clunky, rigid human rules guiding it every single step of the way. So human design scaffolds are just too rigid. They treat the AI like a calculator rather than letting it naturally orchestrate its own thinking. Exactly. That is where it became clear that the AI had to learn to build these parallel tracks internally.

8:46Which brings us to the actual mechanics of Spiral. Okay, let's get into it. The primary innovation of Spiral is that it trains the model end-to-end. It forces the AI to use all three of these compute primitives natively within its own architecture, evaluating the model against a single final reward. Let's walk through what that looks like in practice. So the model first generates its parallel search traces like its brainstorms. Right. Then it generates an aggregation trace to synthesis based on those parallel paths. Yeah. But the training signal, like the grade it gets to tell it if it did a good job, only comes at the very end based on that final aggregated response.

9:21Which presents a massive technical challenge. I bet. I mean, think about it. If the AI only gets a grade at the very end of the process, How does it know which of its initial brainstorming ideas were actually good? Oh, right, because standard reinforcement learning models grade each parallel trace individually. Exactly. They look at a single trajectory, and if it reaches the right answer, it gets a high reward. If it fails, it gets penalized. But that goes right back to the redundancy problem we talked about. Yes. If you only reward the paths that go straight to the finish line, the AI learns to only ever take the safest, most obvious path.

9:56It becomes terrified to experiment. So, to solve this, Spiral uses a technique called set reinforcement learning, or set RL. Set RL. Instead of grading each trace individually as a pass or fail, it grades them collectively as a set. You are maximizing the expected score of the entire tree of generated states. So the learning signal is shared across the whole set of traces. Yes. If we think about this in real-world terms, gosh, it sounds a lot like assembling a heist crew. A heist crew is actually a great way to visualize set RL. Right. Because if you're robbing a bank, you don't want five master safe crackers.

10:32No, that would be redundant. Exactly. If you grade everyone on their individual ability to open the vault, you end up with a highly redundant collapsed solution space. Nobody is watching the cameras. Nobody is driving the van. Right. You need a hacker, a getaway driver, and the muscle. Exactly. Even if the getaway driver can't crack the safe on their own, like even if their individual trace doesn't yield the final answer, their unique skill is absolutely vital to the group's ultimate success. And Spiral mathematically incentivizes the AI to build that diverse crew. It does this through a mechanism called the marginal set advantage.

11:10Marginal set advantage. How does that work? Well, the algorithm looks at the pool of parallel ideas the AI generated. It then constructs different subsets from that pool and scores those subsets based on how well the aggregator can use them to find the final answer. So it's actively testing different combinations of team members. Yes, exactly. If a specific trace consistently shows up in subsets that result in a high quality final answer, that individual trace gets a high marginal set advantage. Let's make that concrete because I want to make sure I'm getting this. How does the math actually reward a wrong answer that just happens to be uniquely useful?

11:45OK, let's say the AI is solving a massive calculus problem. Sure. One of its parallel traces tries a complex integration method, hits a wall, and completely fails to solve the equation. OK, so in a standard model, that trace gets deleted and penalized, right? Under a standard training method like GRPO, or Group Relative Policy Optimization, yes, it would be heavily penalized. GRPO compares outputs within a group to calculate a baseline, and it punishes anything that diverges from the most direct path to the correct final answer. Which forces the model to play it safe, causing that entropy collapse.

12:21Exactly. But under Spiral's set RL, what happens to that failed calculus trace? Under Spiral, the aggregator looks at that failed trace and notices that, hey, before it hit a wall, it successfully factored out a highly complex polynomial. Oh, wow. Yeah, a piece of math that another completely different trace desperately needs to finish its own successful integration. because the aggregator extracts that polynomial and uses it to get the final correct answer, the initial failed trace is assigned a high marginal set advantage. It gets credit for being a team player. Exactly. The AI learns that it is perfectly acceptable to generate a flawed trace or to explore a dead end as long as that trace contains a unique piece of the puzzle.

13:04A specific equation, a weird simplification, a new perspective. Yes. That shared scalar learning signal creates a natural coupling effect. Set RL explicitly encourages the AI to keep its token level entropy high. It actively rewards diversity of thought. It really does. It teaches the AI the value of learning from its own mistakes natively, without a human developer constantly adjusting the parameters from the outside. That fundamentally shifts the paradigm of machine learning. It's huge. But, you know, theory only gets us so far. Building a cognitive heist crew sounds brilliant on paper, but I want to look at the hard data.

13:41How did this actually perform when it was put to the test? So they designed a very rigorous evaluation. The framework was tested on intense mathematical reasoning, specifically using a data set called Polaris 53K. For those who might not be familiar, what does a Polaris 53K problem look like? Are we talking basic algebra or something deeper? Oh, we are talking about 53 ,000 highly complex mathematical reasoning problems. These aren't just arithmetic questions. They require multi-step proofs, advanced discrete geometry, complex algebraic derivations. It's tough stuff. Wow. And they use a specific base model for the test, which is QUEN34B instruct 2507.

14:21Okay, a 4 billion parameter model, which, I mean, in the grand scheme of things, is actually pretty small compared to the massive trillion parameter models out there today. Oh, definitely. But that was intentional. The goal is to prove that Spiral makes the architecture itself smarter and more efficient, rather than just relying on the brute force processing power of a massive model. Right, that makes sense. So they fine-tuned this Quen model using Spiral and compared it head-to-head against a model fine-tuned with GRPO, which again is the standard sequential method. And they gave both methods the exact same training compute budget just to ensure a completely fair fight.

14:57But remember, the GRPO model only optimizes for sequential compute during training, while Spiral optimizes across all three primitives. Exactly. So what happened when they asked both models to scale their parallel compute at test time, meaning, you know, just asking the models to generate multiple independent attempts to find the right answer? The results were genuinely staggering. Spiral achieved up to 11x scaling efficiency compared to the standard GRPO model. Wait, let me make sure I'm translating that correctly. 11 times scaling efficiency, meaning Spiral is exploring the solution space so much more effectively that it finds the correct proof using a fraction of the parallel attempts the standard model needs.

15:38Yes, its brainstorms are just exponentially higher quality because it isn't generating redundant ideas. It hasn't suffered entropy collapse. Exactly. It generates a highly diverse set of plausible mathematical approaches right out of the gate. But eventually, even with parallel compute, you run into hardware and software limits, right? You can't just scale sequential chains of thought forever. No, you can't. Like if you've ever used a chatbot for a long research session and noticed that after maybe 20 minutes, it completely forgets the instructions you gave it at the very beginning. You've hit what developers call the context wall.

16:13The context wall. Exactly. For this specific Quinn 4 billion parameter model, that hard wall sits around 32 ,000 tokens. 32 ,000 tokens. Yeah. Once the prompt and the generated reasoning get that long, the model simply runs out of context space. Its performance degrades, and it just loses the plot. It's like working on a giant whiteboard. Sequential thinking is just riding out your poof until the whiteboard is entirely full. At 32 ,000 tokens, there is no more white space. Right. You start having to erase foundational formulas at the top just to keep writing at the bottom, and the whole mathematical proof falls apart.

16:48So sequential compute has a hard ceiling. And to get past that ceiling, you have to lean heavily into parallel and aggregative compute. And this necessity birthed one of the most powerful applications of the spiral framework, which is recursive self-aggregation, or RSA. Okay, RSA. Break down the mechanics of recursive self-aggregation for us. How does it bypass that whiteboard limit, that context wall? Okay, imagine you sample a population of eight independent parallel reasoning traces for a really complex proof. Under RSA, you don't ask the model to read all eight at once because that would instantly overwhelm the 32 ,000 token limit.

17:26It would just crash. Exactly. Instead, you partition those eight traces into groups, say, two sets of four. Okay, so you're breaking it into chunks. Right. You prompt the model with the original problem in the first set of four traces, and you ask it to synthesize them into one refined answer. Then you do the same for the second set of four. Okay, so now you have two highly refined answers. Yes. And then you recurse. You take those two newly synthesized answers, put them together, and ask the model to aggregate them into a final ultimate solution. I see the logic there, but I have to push back a bit on the real-world application here.

17:59Sure. Is recursive self-aggregation just a highly technical way of saying we let the AI hold a massive committee meeting with itself? Well, because if five of those initial traces say the answer to the math problem is 42 and three say it's 10, shouldn't we just trust a simple majority rules voting system? OK. Does a model really do a better job synthesizing its own messy thoughts than just a hard coded heuristic? That's a very fair question. But the data provides a definitive answer to that. model-based aggregation, this recursive internal auditing, vastly outperforms rule-based aggregation like majority voting.

18:36Really? Yeah. When the standard GRPO model was tested against Spiral using simple majority voting, they performed similarly. But when they applied recursive self-aggregation, Spiral achieved up to 15 % higher performance overall. Wait, 15%. But why? Why does the AI's internal committee meeting find the truth better than a simple majority vote? Think about it. A vote assumes the most common answer is the correct one. But in advanced discrete geometry or logic, what if the correct answer relies on a highly unintuitive leap that only one out of the eight parallel traces managed to find? Oh. In a majority vote, that brilliant isolated leap is instantly discarded because it lost 7 to 1.

19:16Ah, so RSA isn't looking for consensus. It is actively auditing for truth. Precisely. In Spiral's recursive self-aggregation, the prompt explicitly asks the model to act as an auditor. It is taught to read a trace, extract a specific mathematical proof, and discard the surrounding hallucinations. It resolves disagreements by reasoning through the math step by step, not by just counting votes. So the AI is actually evaluating the quality of its own work. It is. The data shows that when the AI is trained to search and aggregate natively, it actively discovers search procedures that surpass any rigid heuristic a human could code.

19:52That's incredible. During difficult problems, the spiral model dynamically learns to spend more tokens in its search traces, studying related simplifications. Then, it spends more compute in the aggregation traces, rigorously verifying the math. It learns how to allocate its own cognitive budget. It's like it takes a messy whiteboard, finds the three equations that actually matter, writes them down on a fresh whiteboard, and starts the process again, completely bypassing the context limit because it is constantly compressing the information. It transforms raw compute into effective directed search.

20:26Wow. So when we step back and look at the broader implications for you, the listener, we aren't just looking at a minor software update that makes a language model run a little faster. Not at all. We are witnessing a fundamental shift in the architecture of machine thought. We're moving from AI as a simple question-answering machine, where, you know, you put a prompt in and get a single linear gumball of an answer out to AI operating as an autonomous researcher. An autonomous researcher that dynamically budgets its own resources to explore, verify, and synthesize complex truths. And it does this recognizing that a failed experiment isn't a waste of compute, provided it contributes to the collective understanding.

21:07Which leaves me with a final lingering thought. What's that? Well, it's something that wasn't explicitly tested in the data we looked at today, but it feels like the inevitable frontier. Right now, Spiral trains the AI on a fixed number of recursive steps, like a set size of four, a population of eight, primarily to keep computing costs manageable during the training phase. Right, the sets and populations are heavily regimented right now. But imagine a future iteration where the AI learns to dynamically resize its own sets of thoughts in real time, where it doesn't need us to artificially cap its brainstorm at exactly eight traces.

21:43Imagine what happens when an AI develops an organic intuition for exactly when to stop recursing and exploring. Where it can analyze its own search space and decide entirely on its own the exact moment a concept has been perfectly understood. Exactly. If an AI can dynamically decide when the search is over, it moves from being an incredibly efficient researcher to possessing true autonomous expertise. It won't just see a wall of confusing text when faced with a first-proof logic puzzle. It will see the jagged edge, build its cognitive ice crew, audit its own findings, and simply solve it.