Is Chain-of-Thought Reasoning a Mirage?

0:00Welcome to the Deep Dive. We take research, sources, our own notes, and boil it all down, giving you a shortcut to getting up to speed. And today we're looking at something, well, everyone's talking about it, large language models, LLMs. Right, specifically their ability to seemingly reason using what's called chain of thought prompting or COTI. Yeah, it's that step-by-step output that looks incredibly human-like. It really seemed like a breakthrough in, you know, getting AI to actually think. It does. You see these complex problems broken down and you think, wow, it's really getting it. Exactly.

0:35And that perception wasn't unfounded initially. Cote did lead to pretty significant performance boosts on tasks needing logic, maths, common sense reasoning. It definitely fueled the idea that these models were developing deeper inferential processes. But yeah, and this is the big question for today. What if that reasoning, that apparent understanding is actually kind of an illusion? What if it's not genuine understanding at all? Maybe something else entirely. Precisely. So our mission today is to unpack this really fascinating new research paper. It's titled, Is Chain of Thought Reasoning of LLMs a Mirage?

1:11A Data Distribution Lens. It's by Cheng Shui Zhao and colleagues over at Arizona State University. From the Data Mining and Machine Learning Lab there. And they argue pretty convincingly that Cody might be more of a, what do they call it, a brittle mirage. A brittle mirage. I like that. So less about real thinking power and more about something fragile. We're going to explore why it fails, when it fails, and give you a shortcut to understanding these surprising limits. Yeah, let's get into it. Okay, so let's set the stage a bit more. For a few years now, Kodi Prompting has been seen as this huge step forward.

1:48Definitely. The idea was simple. Just add a little phrase like let's think step by step and boom. Suddenly the LLM starts producing output that looks like it's working through the problem logically, showing its work essentially. Right. And like I said, it often led to better answers on complex stuff, which naturally made everyone think, OK, it's actually reasoning now. OK, but here's where the paper throws a bit of cold water on that idea. Right. They start with a really striking example. They do. It's quite clever. Imagine asking an LLM something that requires a tiny bit of factual recall and a simple logical step.

2:20The day the U.S. was established is in a leap year or a normal year. Seems straightforward enough. So they tried this with Gemini, which is a pretty advanced LLM. And with the Kauti prompt, it came back with something like this. The United States was established in 1776. OK, correct fact. 1776 is divisible by four, but it's not a century year, so it's a leap year. Correct rule application. Still good. 1776 was a leap year. Right. But then it concludes, therefore, the day the U.S. was established was in a normal year. Wait, what? It just said it was a leap year. Exactly. It perfectly recited the steps, identified it correctly as a leap year, and then just contradicted itself in the final conclusion.

3:01It's bizarre. It's like the steps are just decoration, not actual logic guiding the answer. Precisely. It really highlights the potential disconnect. The model can generate text that looks like reasoning, but it doesn't guarantee the underlying process is logically sound or even connected to the final output. It raises that question. Is it reasoning or is it just sophisticated mimicry? So that Gemini example, it's kind of a perfect storm, isn't it? It's confidently wrong, but wraps it up in these steps that make it seem reasoned. Which is almost more dangerous than just getting it wrong because it looks plausible.

3:36So what does this tell us about how these models are really working under the hood? If it's not logic, what is it? Well, it strongly suggests they're relying more on surface level patterns and associations learned from the data. Like certain sequences of words tend to follow others. Okay. Other analyses mentioned in the paper also show that LLM performance can drop sharply if you just, say, insert irrelevant phrases into a problem. That suggests they aren't grasping the core logic. They're just getting thrown off by unexpected patterns. Right. They get distracted easily. Yeah. And this led the researchers to propose looking at SOTI through a different lens, the data distribution lens.

4:15Okay. Data distribution. What does that actually mean in this context? So the core idea, the hypothesis, is that SOTI isn't real reasoning. It's what they call an inductive bias. An inductive bias. Yeah. Like a tendency it picks up from its training. Exactly. It learns to prefer generating sequences of text that look like reasoning steps because it saw lots of examples like that in its training data. It's essentially approximating the style of reasoning it was trained on. So its success depends entirely on how similar a new problem is to what it saw during training. That's the core argument. Its effectiveness is limited by the distribution discrepancy, the difference between the training data and the test question.

4:56If the test question is too different, too out of distribution. The mirage shatters. Pretty much. And to test this rigorously, they built this controlled environment they called data alchemy. Data alchemy. Sounds cool. What did they do with it? It's actually quite elegant. They basically created a miniature controlled world for the LLMs. They defined basic atoms, just letters A to Z. Okay. Then elements, which are sequences of these atoms, like A-P-P-L-E. Got it. And then, crucially, they defined specific transformations they could apply to these elements. These transformations are meant to stand in for multi-step reasoning tasks.

5:32Ah, okay. So controlled reasoning problems, that kind of transformation. They used two main types. First, a ROT transformation, like a Caesar cipher, where each letter gets shifted alphabetically. So APLE shifted by 13 becomes N-C-C-Y-R. Right. Second, a cyclic position shift, where the letters just change position. APLE shifted by 1 becomes E-A-P-P-L. Simple enough. Yeah, but the clever part was chaining them. They could create compositional transformations, like doing a ROT shift, then a position shift, F1, then F2. This let them systematically test how well Cothee holds up when things change.

6:07And they looked at this across different dimensions. Three critical ones, yeah. Starting with task generalization. Task generalization. So can it handle new types of tasks, like transformations it hasn't seen composed in exactly that way before? Precisely. Or entirely new transformations or elements with structures it wasn't trained on. And how did the LLMs do? Not well, I'm guessing, based on the Mirage idea. Not well is an understatement. Under transformation generalization, when they changed the transformations between training and testing, the results were dramatic. How dramatic? Okay, so if the test used the exact same transformation sequence as training, they called this indistribution or ID, accuracy was 100 % perfect.

6:51Makes sense. It saw that exact pattern. But if they tested on a new combination of transformations the model had seen individually, composition or CMP, say train on F1, then F1, and F1, then F2, but test on F2, then F2, the exact match accuracy dropped to basically 0.01%. Wow. From 100 to 0, just by recombining familiar steps. Essentially, yes. And if even one part of the transformation sequence was new, partial outdistribution or POD, or if the entire transformation type was novel out of distribution or OD, the accuracy was flat zero. So it completely falls apart the moment you step outside the exact training setup.

7:27Completely. The generalization just wasn't there. And what's really telling, sometimes the model would generate correct-looking reasoning steps, but still get the final answer wrong. Huh. Like the Gemini example again. Kind of. Or, even weirder, sometimes it got the right answer, but for the wrong reasons. Like, maybe two different transformation sequences just coincidentally produced the same output for a specific input, like A-N-A-N. The model spits out the right answer, but its reasoning path was totally unfaithful to the actual transformation it was supposed to perform. So it's just matching superficial patterns, not understanding the operations at all.

8:04That's what the evidence strongly suggests. It's replicating the form of reasoning, not the substance. And you mentioned something about fine-tuning. Could they fix it easily? They found that if you took one of these models that failed on, say, an ode task, and you fine-tuned it on just a tiny number of examples of that new task. It learned it quickly. Very quickly. It could get back to high accuracy. But the researchers argue this isn't true generalization. It's more like quickly memorizing a new pattern. It's a patch, not a fundamental fix for the like of reasoning ability. Okay, so that covers new transformations.

8:37What about generalizing to new elements, like using different letters or sequences? That was element generalization. And spoiler alert, same story. Performance plummeted when dealing with novel atoms or combinations it hadn't explicitly seen. Zero again. Pretty much zero exact match in the compositional and out of distribution scenarios. And get this, if they trained the model only on elements using letters A through M and then tested it on an element containing an N or an O. It just broke. It often failed to produce any sensible output. Like it didn't even try to apply the transformation. It just gave up or produced gibberish.

9:14Wow. So it's incredibly brittle even when it comes to the basic building blocks. Extremely brittle. And again, even with fine-tuning, while it could learn specific new elements, it didn't seem to grasp the underlying concept of how transformation should apply regardless of the specific letters involved. They also noted this interesting mismatch, sometimes even during training, where the reasoning steps weren't always perfectly aligned with the final answer accuracy, hinting at this inherent inconsistency. Okay, so task generalization is looking pretty shaky. What about the other dimensions you mentioned, Link?

9:49Right. The next dimension was length generalization. This asks, can the LLM handle inputs or reasoning processes that are longer or shorter than what it was trained on? Seems like a basic requirement for any real intelligence, right? Adapting to different lengths. You'd think so. But they tested text length generalization first, just changing the length of the input element, like APPLE versus APPL. Even small changes in length caused big problems. If a model was trained only on elements of length 4, testing it on length 3 or length 5 caused performance to drop significantly. Why, though? What was it doing wrong?

10:23What's fascinating is how it failed. Often, the LLM would try to force the output to match the training length. So if trained on length 4 but tested on length 3, it might just add a random token at the end to make the output length 4. Seriously. It's just trying to match the length pattern it learned, not the actual result of the transformation. That seems to be exactly what was happening. It prioritized matching the superficial characteristic length over performing the actual task. Standard ways of handling different lengths, like padding, didn't really help either. So it's stuck on the surface features.

10:57Although they did find a clever trick, a group padding strategy, where they chunked the input seemed to help length generalization somewhat. Yeah. But the basic finding holds. Okay, what about the length of the reasoning itself, like number of steps? That's reasoning step generalization. They trained models on problems requiring, say, exactly two transformation steps, F1, then F2. Then they tested them on problems needing only one step, F1, or maybe three steps, F1, then F2, then F1. And I bet they failed again. Largely, yes. Models trained on two-step problems struggled significantly with one-step or three-step problems.

11:33They might try to insert an unnecessary step or skip a required one. Again, just mimicking the structure it saw in training. Exactly. Now, they did find that if you pre-train the model with a mix of different step lengths, performance improved. But that just reinforces the main hypothesis, doesn't it? Yeah, it means performance is directly tied to the variety seen in the training data distribution. If it hasn't seen it, it can't do it reliably. Precisely. Okay, last to mention then, format generalization. How sensitive is this Cothee stuff to just tweaking the way you ask the question, like minor changes in the prompt?

12:07This one's really interesting. How robust is Cothee TT to just, you know, superficial changes in the input format? They tested this by introducing small perturbations. Like what? Typos. Things like inserting random noise words into the prompt, deleting a few words, or slightly changing some of the non-essential words. And because it's fragile again. You guessed it. Coase's reasoning was shown to be quite easily affected by these format changes. Interestingly, inserting extra noise words seemed to hurt performance more than deleting words. Huh. Any idea why? Maybe the extra words just disrupt the expected patterns more?

12:43Hard to say for sure. But they also found which parts of the prompt were most sensitive. The important parts, presumably. Oh, right. Changes to the actual element, like APLE or the transformation description, had a much bigger negative impact than changes to other parts of the prompt, like introductory phrases. So it's really latching onto those specific keywords and the structure around them. Change that slightly and it gets confused. Exactly. It suggests the model isn't understanding the overall request flexibly. It's relying on very specific phrasing cues learned during training. Okay, so across the board, new tasks, different lengths, slight format changes.

13:20This chain of thought reasoning seems pretty brittle, just like the paper title suggests. But were these findings maybe just specific to their setup, like the specific small LLMs they use in Data Alchemy or maybe how they generated the text? Did they check that? That's a really important question for robustness, right? Is this just an artifact of their specific experiment? Yeah. But they did address this quite thoroughly. What's compelling is that they replicated these core findings, this fragility, across a range of conditions. They varied the sampling temperature, which controls how random or creative the output is.

13:54Did that make a difference? Like more creative output equals better reasoning? No. The fragility remained consistent across a wide range of temperatures, from very deterministic, almost no randomness, to quite creative. Okay. And crucially, they also tested this across different model sizes, from really tiny models with just tens of thousands of parameters up to much larger ones with over half a billion parameters. And the bigger models didn't overcome this limitation. They still showed the same fundamental fragility when faced with these out-of-distribution shifts in task length and format. The scale didn't fix the core issue.

14:32Wow. Okay, so that really strengthens their claim. It's not just a quark. It seems to be an inherent characteristic of how these models currently learn and operate, regardless of size or specific settings. It certainly seems that way. It reinforces the Mirage conclusion quite strongly. All right, let's bring this all together then. We've gone through the Cothee Promise, the Gemini crack, the data alchemy tests. What's the big takeaway for, you know, for us listening? Yeah, the core insight from this deep dive is that this impressive reasoning we sometimes MC with Cothee prompting, it's probably not what it looks like on the surface.

15:06This is not genuine logical thinking. It appears to be much more like a very sophisticated form of structured pattern matching. The model learns to generate text that follows the pattern of reasoning steps it saw during training. But its ability to do that is fundamentally tied to how similar the new problem is to its training data, that distribution thing again. Exactly. Push it even a little bit outside that familiar distribution, change the task structure, the length, the format, and the performance tends to degrade, often quite dramatically. The mirage evaporates. So what does this mean practically for people using these tools or maybe thinking of deploying them for important things?

15:45I think the implications are pretty significant, especially if you're relying on LLMs for anything critical. The researchers highlight a few key points. Okay. First, guard against over-reliance and false confidence. Just because an LLM can produce a fluent step-by-step explanation doesn't mean it's correct or logically sound. Right. That fluent nonsense idea. It can look convincing even when it's totally wrong. Precisely. And that can be more dangerous than a simple wrong answer because it gives a false sense of reliability. So especially in high stakes areas, medicine, finance, law, you absolutely cannot just trust the Cotee output.

16:23Human expert oversight is still crucial. Makes sense. What else? Second, prioritize out of distribution testing. Standard testing where your test data looks a lot like your training data isn't enough. Because that just tests if it learned the patterns it was shown. Exactly. You need to actively try to break it. Use adversarial testing. Systematically probe how it handles variations in task, length, format, the very things they tested in Data Alchemy. You need to find the edges of its capabilities. Okay, so stress test it for real-world messiness. Mm-hmm. And the third point is about fine-tuning.

16:56recognize fine-tuning SFT as a patch, not a panacea. We touched on that. It can learn a new specific pattern quickly. But that doesn't mean it suddenly achieved true generalizable reasoning ability. It's just expanded its little bubble of known patterns slightly. Relying on SFT to constantly fix every time it sails on something new is probably not a sustainable strategy. It doesn't fix the underlying limitation. Right. It doesn't address the core issue that the model, as it stands, lacks robust abstract reasoning capabilities. This whole deep dive really forces you to reevaluate what we mean by intelligence or reasoning in AI, doesn't it?

17:34It absolutely does. You see the fluent output, the step-by-step breakdowns, and it's so tempting to project human-like understanding onto it. But this research pulls back the curtain. Yeah, it challenges us to look deeper, to be more critical about what these tools are actually doing versus what they appear to be doing. So if Kuity is largely a mirage, a pattern-matching trick, Based on the training data, what's the path forward? How do we get to real reasoning? Well, that's a million-dollar question, isn't it? If current methods based on scaling data and matching patterns primarily lead to this kind of brittle performance outside the training distribution.

18:10We need fundamentally new approaches. It certainly raises that possibility. How do we build models that go beyond replicating sequences to actually developing more abstract, generalizable, robust inferential skills? that's the huge challenge facing the field. A challenge for the researchers, definitely. But also something for all of us using these tools to keep in mind, right? To stay curious and critical. Absolutely. Keep exploring, keep questioning what's really going on under the hood.