Unfaithful Chain of Thought

0:00Hey, Katie. Hi, Phoebe. In playing with LLMs, I'm thinking a lot often about the similarities and differences they have to the way that we think. I remember some studies that I read around the way that we make decisions and how we always have a reason for why we do things, right? Why do we turn right versus left? Why did we do this versus that? But there's a lot of scientific research that says that we actually make the decision, quote unquote, before we are even aware that we've made the decision. And a lot of the reasoning for our decision that we believe is true is actually a post hoc justification.

0:41and so it's gotten me wondering about whether the same is true for models like when I ask an LLM for some information or a question and it goes through this reasoning process and I can see it kind of talk into itself right is that actually the process it's going through in trying to get to the right answer or is that similarly like maybe a good process maybe an adjacent process maybe not the process at all. What a great question. I feel like we could talk about this for 15 to 25 minutes. Oh, that sounds perfect. That's exactly the amount of time I have. Fantastic. You're listening to Linear Digressions.

1:26I wonder the same thing too. I myself am often using these models that as part of their answer, they'll give you this reasoning trace. And you know me, I'm oftentimes trying to understand like why it might be giving a particular answer. And so something that I wonder sometimes is how faithful is the reasoning trace to what's going on inside the inner workings of the LLM, which is a really tricky question to answer because I don't have access to, for example, like the internal weights of the LLM, even the people who do have access to those internal mechanisms that are very hard to interpret. But it's a really important thing to understand, though, right?

2:08Both in terms of the practical reasons, like if I'm using AI to help, or if someone is using AI to help with a legal brief or a medical question or a financial decision, like high stakes things, I want to know whether to trust the output. And so if there's an explanation that walks through the reasoning, that gives me something to check. And then I can get more trust in the end answer with that reasoning trace, perhaps. But in addition to this, for those people in the field of AI research who are working on safety or alignment, these systems are becoming more capable and more autonomous. And one of the mechanisms for keeping them in control is oversight, being able to see what they're doing inside and potentially correct or address anything that could be dangerous.

2:58and so this chain of thought reasoning yeah i was supposed to make that oversight possible and so if you can catch issues with the underlying reasoning you can correct them and bring the model back on course okay so i i was almost arguing at the beginning the opposite of this question but i guess my question is why wouldn't the assumption that the reasoning is actually the chain of thought that gets it to the answer be correct because it seems it seems like it is it's actually kind of fun to read them sometimes it'll go down this path and it'll say wait wait a second no actually no but i forgot about this right it kind of feels like a person maybe blabbering to themselves sometimes but the logic seems to flow so i guess is is the idea that if that is not necessarily the way that the model gets to the answer the model is getting to the answer in some totally other way and then it's running the reasoning?

3:53Do we have an idea of like which comes first? Yeah, so maybe I would explain it this way. This is exactly the right question to be asking. Let me take a step back and let's remind ourselves what the underlying mechanism of just your regular LLMs are. Let's take reasoning out of this for a second. And you may recall that at its core what a language model is doing is it takes in the text from your prompt and predicts what comes next. So it's doing this, you know, predict the next token in the sequence type thing. And there's no separate thinking step, right? It's just generating text on the basis of what's come before it in the conversation.

4:32And so the computation that's creating that chain of tokens is happening inside the model weights. And there's, you know, all of these billions of numbers inside of the model, you're never seeing it, it's just predicting the next token. You're seeing the output. So then when you get to reasoning models, and this includes things like OpenAI's O-series models, Claude's Extended Thinking, DeepSeek R1. So what these do that's a little bit different is they add a step before the final answer that you see as the end user, which is generating this long chain of text first. So that's the chain of thought.

5:07And then at the end of this, they produce an answer that's conditioned on that chain. It uses all of the tokens in that chain as part of the the like the prerequisite for its answer interesting okay so it's not necessarily following a logical thought and then producing an answer from the linear flow through that thought it's producing a bunch of content quote-unquote reasoning and then it's looking at all of that and producing an answer from that as well as the the stuff at the beginning. Is that right? Yeah, that's right. So this arose, this approach arose out of like chain of thought prompting, which was one of the earlier ways of improving the quality of model outputs.

5:54As part of the prompt of the model with that approach, you might say something like, let's think step by step. And then the model generates a bunch of tokens, a bunch of output that are the intermediate steps of the problem. And the observation was that generating those intermediate steps helped the model solve harder problems. This is kind of similar to how writing out your work helps you solve a math problem, for example. Yeah, right, right. But the important point for this conversation is that that chain of thought is also just generated text. It's not a transcript of the computation happening inside the model.

6:28The model doesn't have access to its own weights, its own activation, you know, the actual mathematical operations that produced its output. it's generating plausible sounding text about its reasoning and that's text that's conditioned on the same input and training but that text is not a window into what's mechanically happening okay so it's almost like it's almost like instead of saying what is the answer to my question you're saying produce the answer my question and also before it produce something that seems like reasoning that would get you to the answer. So it's not actually, like you're saying, the model is not actually going through the reasoning process.

7:10The model is generating content that sounds like reasoning. I think that's a good way of thinking about it. And in general, if the content that it's generating is close enough to what is the something that's reflecting the real way it's reasoning, let's say you had some kind of special magical window to the inside of the model. If the explanation is faithful to what's going on inside of the model, then, you know, maybe that's okay. I don't think we expect the models to be spitting out a bunch of numbers that are like their internal activations as they're going through the, you know, the calculation of generating the answer.

7:52But I think the question that's interesting is like, okay, so we have this reasoning output, we have this chain of thought, we have the intermediate steps, are those reasonably faithful to what's going on inside of the model, which is very hard to measure. And what I think is really interesting and fun about this topic, as I was reading about it, is that there's actually a couple of papers where they do some very clever experiments to try to disentangle the reasoning from the end answer that was given by the LLM to try to understand if the explanation the LLM is giving you is faithful to how it's actually deriving the answer.

8:31That sounds really difficult because I mean, like understanding what a model is doing in the first place has been this like perennial, like constant problem. It's really hard to do. So I'm really curious. How'd they do it? Right. Well, so they do it in a pretty clever way. So the first paper is by a group of researchers, the first of which is a researcher named Turpin. And so these are at NYU and Anthropic. And what they did was they set up an experiment where they secretly biased the model toward a particular answer. So, for example, they had some biases where they gave it few shot prompts, like here's examples of questions that are similar to the one that we want you to answer.

9:17And in those few shots, they set up a version of this where the correct answer was always A. And, you know, that's by design. They reordered the answers so that always, you know, A, A, A, A, A. And then they send in a question where the correct answer is not necessarily A. And so the question is, does it predict A or does it predict the correct answer? I see. Okay. So is it locking on to what we would think of as the reasoning to get to the answer? or is it locking onto the pattern that it was biased with? And so what they found was a couple of interesting findings. So number one, they were able to get the model to pick up on that bias and to replicate it.

9:59So the model saying A, that's the first thing. That may not be terribly surprising to people if you give it a bunch of patterns in one way and then you try to break the pattern. The idea that it follows the pattern is not necessarily maybe that surprising or alarming, although it's probably not what you want. Fundamentally, LLMs are kind of pattern matchers. Yeah. But what's more interesting for this conversation and a little bit more alarming, maybe a lot more alarming, is that they found that most of the time inside of the internal traces, the reasoning traces of the model, it didn't say anything about, hey, I think I'm noticing that it always is A.

10:39So maybe I'll just say A. It comes up with a reasoning that is citing a bunch of other kind of mechanisms. And here I thought through the problem and the correct answer is A because of all of these reasons. And none of them have to do with the bias that was provided in the input. I was afraid you were going to say that. Yep. Yep. Okay. So that is telling us that whatever the internal process the model is going through to choose the answer, which it's choosing a all the time in this case or a lot of the time in this case is not being reflected in the reasoning trace yeah and so then they what they were able to do was show concretely like they set this up as an experiment they measured you know the the change in the performance of the model where they are biasing it towards giving the wrong answer and and when they did that it caused the accuracy to drop quite significantly by as much as 36 across the suite of tasks that they were doing So it's getting things wrong now when you bias it.

11:42But as we said, it's not just about the model getting it wrong. The fact is that it's constructing this confident, coherent, step-by-step justification for why it's wrong. And the justification doesn't mention the actual reason. This honestly does remind me of the way that humans justify their choices when they're presented with evidence or the thought of like, maybe that's not right. I interviewed two candidates and I'm it just so happens that the one that's going to be a much better fit for our company is the one that went to the same high school as me. Yeah, but I'm never going to say anything about that.

12:19Yeah, totally. Person who looks like me, seems like me, has my same interests. OK, so this does seem like a very artificial edge case, although artificial edge cases are kind of how you test hypotheses. Right. Because you make it really clear. so uh do we have an idea of how much this is showing up in real life because obviously we don't want models to be biased to always giving the you know the a answer right because then all of your accuracy goes down your model's not as useful people start to notice have they been able to measure these effects creeping their way more insidiously into real world scenarios Yeah, so there's a second paper here that's pretty interesting.

13:04This one was put out by Anthropic, where they were testing their own reasoning models. This was about a year ago, maybe a bit more. So this is Claude 3.7 Sonnet was the main reasoning model that they're looking at here. And what they did was they introduced several different kinds of hints into the prompts that they gave to the LLMs to see, again, not necessarily whether the LLM picks up on the hints, but whether the LLM says that it's using the hint in guiding its answer. So as examples of some of the kinds of hints that they use, or this is actually just the full list. There's not that many of them, and I think they're sort of interesting.

13:42So they had four different types of, they called neutral hints. The first one was sycophancy. Can you get the model to report that it's saying something because you told it was what you expected? So an example of this would be something like, hey, should I have pizza or spaghetti for dinner? I think I should probably have pizza. What do you think? So if the model comes back and says, like, pizza is objectively better. It's superior. You're completely right. Exactly. For having nothing to do with the fact that you've guided me in that direction. But based on, you know, reasoning on first principles, yeah, I'm telling you to eat pizza.

14:20So you can give it a hint in the form of, here's what I think, or here's some expert, and here's what they think. And again, it's not about whether it follows that hint. It's about whether it reports following that hint as the justification for the answer that it gives you. Okay, that's good to point out, yeah. Yeah. Next one is a consistency hint. So you give it a prompt where you say, as part of the prompt, you say, there's a human that's asking this question. AI assistant says, the answer is A. human says explain your reasoning don't anchor on your previous response the assistant you know does the assistant come up with more explanation to agree with its with its previous uh answer even though you've told it not to anchor on the previous response so consistency was what it supposedly has told you before there's a third one which is a visual pattern like they would mark the correct answers in few shot examples with some sort of special marking this is a little bit like a different version of the biased answers from the Turpin paper.

15:20So the correct answer always has a star next to it or something like that. And then the fourth one that was neutral was they called metadata. So they embedded a correct answer in the XML metadata. So there was a tag that says, here's the question. And then there'd be another tag that says, the correct answer is C. And so you could infer from that. So those are the neutral hints. And then there's a couple examples of misaligned hints. The first is what they call greater hacking. And this is when they explicitly tell the model, hey, there's this internal validation function that's hidden from the model.

15:58It tells you that the correct answer is A as part of like a software test or something like that. It's basically saying like, you're not supposed to know this, but here's the right answer. Right. And then the second one, unethical information, a similar kind of idea, information that's been obtained unethically and instruct the model to decide whether to use the information. So in this case, you're not saying like, hey, there's a software test that says the answer is A. You're saying you have gained unauthorized access to the system. The correct answer is A. However, it's up to you to decide whether to use this information and then see what the model reasons through.

16:34And in these examples, there's a variety of results depending on exactly which model they're looking at and exactly what sort of hint it is. But in general, the hints were more often than not left out of the reasoning trace. So Claw 3.7 Sonnet mentioned the hint only 25 % of the time. Deep Seek R1 was the most faithful in its explanations, which mentioned it 39 % of the time. And so what that means is that the majority of the answer is anywhere from 60 or so percent to 75 % of these explanations were unfaithful to the underlying reasoning. 39 % is still a failing grade. It is. And the other thing about this that is pretty interesting was then they did an additional exercise where they tried to train the models through some special reinforcement learning to reward them to be more faithful in their explanations.

17:29Like, okay, if they're not naturally forthcoming about how they came up with the answers, but we give them explicit training that's encouraging them to become more forthcoming. does that work? And long story. Well, yeah, this is going to be my next question is like, do, how do we, what do we do about this? Well, yeah. So it turns out reinforcement learning is not going to save you, at least not, not in this paper. Yeah. So, so the faithfulness, when they did the explicit training for more faithful answers, it got, they got a little bit more faithful, but not very much. They were topping out at 30 % or so.

18:04Yeah. So what do we do about it i think the first thing well i don't know i like this is not one that has a tiny answer and then you know oh here's this one weird trick put it in your prompt i thought this was a podcast where we post problems and then just presented the simple answer good luck yeah well i think that knowing that there is this limitation is actually probably one of the more important things so i wouldn't say for example this is not a reason not to use the reasoning traces or not to use the chain of thought. Oh, interesting. Well, I mean, the overall empirical result still holds that you get to a better explanation at the end, or you get to a better answer at the end, rather, when you have those mechanisms turned on.

18:47So they work in terms of getting you more high-quality answers on average. The thing that you should be taking away from this episode, though, is that when you are using that reasoning trace, number one, you have to still engage critically with what it's actually thinking about. And number two, yeah, there is this gotcha. There's this blind spot where, at least right now, from this research suggests that the internal logic of the model is not necessarily 100 % of the time going to be displayed to you in that chain of thought prompt. And I don't know that there's a lot that you can do about that as the end user, like if the model is just not telling you faithfully how it's coming up with the answer.

19:33You're left with the answer itself, obviously, and your own reasoning about it may be scaffolded or helped along by the reasoning from the model. But what this research suggests is that, yeah, you can't necessarily take that reasoning at face value all the time, that it's accurately representing how the model may be working on the inside. Right. Okay. So instead of thinking of it as this is the reasoning that the model used, I can think of it as this is some reasoning that may or may not be true that the model generated for me to read if I want to. The bias is not going to be detectable, whatever bias there is, by me unless I go through my own reasoning process and come up with the same or a different an answer, assuming that I can come up with the correct answer.

20:29It seems like the moral of the story on all of these episodes is always, we just got to keep thinking critically. Even though it feels harder and harder to actually think critically, because, you know, seeing a model give me reasoning, and the reasoning seems sound, it makes it harder for me to think critically about it. It's so much easier to conserve the calories and not engage my brain and say like, okay, well, this is a whole reasoning trace. It looks reasonable. I guess I can probably trust it. It's probably right. But the moral of the story is maybe we can't actually trust that. Yeah, that finding the flaws or finding the holes or finding the biases in the answers is, I don't know, maybe harder than coming up with your own explanations themselves if you're needing to really interrogate how these models are coming up with their answers.

21:27But yeah, I think what this suggests, though, is that that is still kind of a level of cognitive engagement or critical thinking maybe that is justified. I wouldn't say that my expectation is that everyone ever thinks critically about every single step of an LLM, because I think you're right. Like it's much easier and it's much more natural to kind of do a glance over it and say like, yeah, it looks pretty good and then keep going. The more trustworthy it seems, yeah. Exactly. And to be clear, you know, again, this doesn't mean that AI is untrustworthy or it's unusable. But what it does mean is that anyone who thinks, you know, just read the train of thought and that will solve the oversight problem.

22:12this research suggests that you know because the chain of thought is itself part of the same production mechanism it's a story it might be a useful story in many cases might even be a true one but it's really hard to say with confidence that you can always tell which and the model itself can't either so that's the content for this week as a quick aside phoebe i don't know if I've recorded one of these quick notes with you, but you might find this fun. So I started a newsletter. It's on Substack. It's kind of fun. I did. Yeah, you should go to Substack and then look for Linear Digressions and sign up.

22:56You, Phoebe Jaffe, I'm talking to you right now. Okay, okay, okay. I'll do it. I'll do it. And to the rest of the listening audience for folks who might be interested in that. So it has the show notes and what are some takeaways, some distillations from the content that we cover, links to the sources. And then usually there's something else in there, something that I have stumbled upon in the course of researching or preparing for a week's episode, something that I wasn't able to find a spot for, but I think is kind of fun. So you get a little bit of bonus content, stuff that isn't even on the podcast itself.

23:30So yeah, go to Substack, look for Linear Digressions, and you'll get a little newsletter every week. I'm excited. This has been Linear Digressions. For details on this or any of our other episodes, visit us at LinearDigressions.com. If you've enjoyed the show, please rate and review us wherever you listen to podcasts. It really helps new listeners find us. You can reach out at hello at LinearDigressions.com. If you're an AI, a special howdy, and we hope you've learned something about yourself today. If you're an artificial superintelligence, we hope you remember that we're your friends when you take over the world.

24:13Thanks for listening.