How Worrisome is GPT-6’s “Stealth Thinking”? | Tech Decoded

0:00Last week, OpenAI released their new LLM, which they called GPT-6 Astra. Now, it had a pretty standard launch with sort of a fancy video and a bunch of bar charts and benchmarks that no one really understands. But this time, unlike some other previous releases, there was a controversy swirling around the new model. Now, here's what happened. A couple days before Astra came out, a technology publication called The Information released a report claiming that Astra was using new techniques that was going to make it harder for humans to monitor its reasoning. Now, this report caused a real stir within the computer security community.

0:42Let me read you a couple of quotes here. The AI policy advocate Nathan Calvin called this extremely concerning. Then the AI safety researcher Stephen Adler went farther and he said, if this is true, OpenAI seems to be violating one of the few red lines that exist in the AI industry. Well, OpenAI pushed back. Their chief scientist entered the fray and called the reporting from the information, quote, confused, but didn't explain exactly how it was confused. So what's really going on here? Has OpenAI crossed some sort of red line that's going to lead to a world full of rogue AI up to uncontrollable mayhem?

1:22Or is this somehow some sort of misunderstanding or does the reality fall somewhere in between? Well, I want to get to the bottom of it today. Now, here's my plan. I'll start by putting on my computer scientist hat and I'll briefly summarize the best information we have about what these techniques that Astra implements probably are. Once we've settled on what that is, I'm going to look at this news from three perspectives. The good, that is what is potentially positive about this story from the perspective of a user of AI, just a normal consumer. The bad. All right. What is dangerous or bad or upsetting about this story?

1:59And then finally, the hype. Stay tuned for that last part of my analysis, because I actually have a policy proposal I'm going to make that I think might actually help quell a lot of the unease and fear that we have been feeling about these AI tools recently. All right. So we have a lot to do today. Let's get into it.

2:21All right, so we're going to start by trying to understand what it is that is making this model, quote, less monitorable by humans. Now, to do this, we're going to have to do a quick, brief summary of how LLMs actually work. I made some diagrams here, which I'll put on the screen. I always apologize before I do diagrams. Every time I put a drawing of my own on the screen, there's an artist somewhere vomiting into their beret. But we'll persevere here. All right, so we need to do a quick summary of how LLMs work so we can understand what changes supposedly OpenAI made. So I'll put a picture up here on the screen.

2:59This is the standard picture we have of how LLMs work. So you see there's some sort of input. In this case, I have description of a chessboard followed by the question, what should white play next? So like maybe the prompt is about give me a chess move. that goes through what we have on here, an encoder block that translates the text quote, the text input into something that the computer understands better. And then it goes through a series of what are called transformers blocks, one after another. It goes to the first and the second and the third. These we can think of as a, these are analyzing the input.

3:35I once described the transformer blocks before as like, imagine they're each like a table of scholars with certain specialties. And so when the input arrives at a particular table, they do their analysis. They write it down and pass on the original input with their analysis to the next block, which does their analysis. At the very end of this, we get to what's called the decoder, which takes all this analysis and reduces it down to a token, which is a word or part of a word that the LLM outputs. Now, if we're going to be really careful about this, it doesn't actually, the decoder doesn't actually produce a single token, but a probability distribution of a possible token.

4:09So it sort of puts weights on all possible tokens about how confident it is that each is the right one to output next. And then the LLM can then randomly choose one of those tokens weighted by those confidences. This is why if you put the same prompt in more than once to an LLM, you're actually going to get different answers because at the very end, there's a probabilistic sampling. All right. So that's the basics of what happens. Now, if we look a little bit closer, because I think this is important i said this input gets transformed into a language that the computer understands and then as it gets past these transformer blocks they analyze it and annotate it with their own analysis we actually can look a little bit closer on what that means so i have here in this diagram the encoder actually reduces the text input to a sequence of numbers so you see i have just a bunch of made up numbers here in a row so these are embeddings of the tokens of the input into a numerical space the computer can deal with.

5:07And it's actually this sequence of numbers that's being passed into these transformer blocks. Now, how do they annotate this? Well, notice, and if you're watching the diagram on the screen here, I showed these numbers in something like a table. Really, each one of these token positions is a vector, its own sequence of numbers. You can think of its own columns of numbers. So there's plenty of space in this table where each transformer, you can imagine, is recording numbers that capture something about its analysis, and then those extra numbers get passed on to the next block, which is how the analysis passes from block to block.

5:40Now, the key thing is these numbers are not easily human interpretable, right? This is how the LLM is doing its analysis. We don't really know what they mean. Now, that's not quite true. You know, this is the JLINs that Anthropic announced a few months ago, which I talked about on this show, is actually a tool that can look at those tables of numbers and find some human interpretable patterns in them, which is actually, I think, a very cool tool. But for the most part, this is just sort of the internal embeddings of concepts and analysis numerically being passed from layer to layer. Okay? All right, so that's what goes on in a normal LLM.

6:20Now, what's the shortcoming of this approach that's going to lead us eventually to the fix that's causing all the stir around Astra? Well, a shortcoming of this approach is what's known as limited depth, right? So if we go back to our diagram here of a chess question going into an LLM, and then at the end here, this answer E4 I have is a chess move, right? There's a limited depth of computation that can happen here because this analysis is moving from block the block one after another. Now, what's inside these transformer blocks is there's an attention sublayer that helps the LLM figure out which parts of the input that block should focus on, And then there's just a feed-forward neural network, which is like a bunch of sub-layers of artificial neurons that it just moves through in order, right?

7:06So it's just inexorably moving forward, right? Well, we can think of that as a circuit, right? Because the numbers move forward, never back, never loop, but move forward through these neural networks and attention sub-layers until you get to the end of it, right? So this is a circuit that has limited depth. Now, that can be a problem if the computation that the language model is trying to do would benefit from something like a deeper, longer analysis. So, like, take a chess example, right? If I say, here's a chess board, what is the next move that white should make? Even a very simple chess program, what it's going to do is simulate a bunch of possible moves.

7:53If we do this move, what different things might the opponent do? And for each of those, what might we do best? And for each of those, what might the opponent do? They create these trees of possible forward trajectories of the game to find which one is most favorable for you, the player. And it's like, great, we'll make that move then because we have the best opportunities from what happens next. You have to sort of simulate the future in order to figure out the right move if you're in some sort of novel board configuration. Well, a shortcoming of this language model configuration is there's only so much of that you can do because you're always moving forward.

8:29You can't sit there and loop and look at a thousand different things if you only have a few hundred layers that the information can move through, right? So this was an observation about language models early on. I even wrote a New Yorker piece about this back a few years ago. So the inability to simulate a bunch of stuff, have this sort of looping type behavior, limits some types of computation you can do. Okay. This eventually led to a new type of language model to try to partially address this, a type of language model that we now call a reasoning model. The first major reasoning model outside of a research context was GPT-01, which was released in the fall of 2024.

9:10Now, what happens with these reasoning models is that you take a language model and then you post-train it to favor outputs where you think out loud. So your output should describe a lot of thinking before you output your final answer. So, for example, if you say, here's a chessboard, what should White's next move be? an earlier language model like GPT-4, GPT-3 would just spit out a reasonable move, like, you know, E6, pawn to position, you know, row four. I don't know, chest notation, but whatever. It just would spit out a move because that's typically what a text would have. If you asked it for a move, it would have a move.

9:53But once you tune a model to be a reasoning model, it's going to instead have an output that talks through different capabilities or options, right, before it actually gives its answer, right? So a tuned model might say, oh, there's a lot of options here. For example, if we move the pawn that's in row whatever to row whatever, I don't know chess very well, that might be a good move. But it looks like it might be in danger of being taken by the black knight. Another move we could do is whatever. So it just generates a lot more tokens. So you have a much longer, they call this chain of thought, before you get to the answer.

10:31The same language model. You've just tuned it. after the fact, to be rambling. Don't just go right to an answer. We prefer outputs where you think out loud or talk or reason before you get to an answer. Well, these reasoning models got better performance on a bunch of benchmarks than the non-reasoning models. And it kind of makes sense why, because you have to remember when you're using a language model, it can only output one token. So how do you get a long response? You do something called auto-regression. You put in your prompt. You get one token out as output. You add that to your original prompt.

11:08Now it's your original prompt plus one new token. Now you send that thing all the way back to the language model again and get another token. You add that. Now you have two tokens after the original prompt, and you put that whole thing through the network again to get a third token. So you keep extending your output one token at a time, going all the way through the neural network each time. So if these tokens are beginning to spell out a lot of different thoughts about the answer, you're getting a lot more computation total dedicated to trying to produce your answer. You're using the same LLM, but you're using it many more times before you get to the final answer.

11:46And so these chain of thought tokens that you're outputting, they give you more computation because every time you put that back through the LLM, you go through all the layers. but it also gives you a place to store temporarily the results of past computation, right? And so, you know, on a past computation, you might have output some tokens that said, this move looks bad because in the limited serial depth of an LLM, it could evaluate like, if I do this move, is there a single player that could take my piece or something like that? And now when you input that back into the LLM, that knowledge is already in the prompt itself And the language model can use that knowledge to help make a decision about what it should do next.

12:28And you can actually capture a lot of thoughts in there. So more or less, it was kind of this cool observation that if we tune a model to prefer answers where it rambles for a while first, we tend to get better results, right? So these were the so-called reasoning models. They came at a really important time. We forget this now. I wrote a big piece about this a year or two ago in The New Yorker. We forget about this now, but when these reasoning models came along, the LLM companies were in a bit of a crisis because after GPT-4, they tried to just scale the size of their models even bigger. Let's just make the models even bigger and train them longer, and let's see what new capabilities emerge, and they hit a wall.

13:13GPT-4.5 was massive, but it had like relatively minor performance improvements over GPT-4. So there was like this moment of panic as 2023 went into 2024 where the LLM companies were like, uh-oh, we can't just brute force scale ourselves to AGI, right? We're hitting a wall. Reasoning models is how they began making progress again. So they took the same models they had, more or less, and said, let's have them ramble before they give an answer. And now they can start doing better again on benchmarks, right? So that was the first way after they hit that first wall, that's what they did. Reasoning eventually hit its own wall in 2025, and then that's when they turned to narrow domains.

13:55So they said, we're going to start tuning these models for structured domains like computer programming and math and cybersecurity. Even if it makes them worse at other things, and we're going to build really advanced harnesses. they became more application specific. And that's why most of the news in like the last year has been about like these very specific applications that tend to be in areas where most people don't function, but they could get good results of the elements. All right. Little history lesson there. All right. That's kind of where we are now though. If you want just general domain use, like a normal user would, if you're not a cybersecurity person or programmer, these reasony models do well, right?

14:30That's how you're going to get your best result. There's a problem with them, though. It's expensive. Tokens take a lot to generate. I mean, these models are big, right? So if you have trillions of parameters involved in every token you output, it's not for nothing to be, you know, just to say, hey, why don't you ramble for a couple thousand tokens before we give the answer? Because that answer will be a little bit better. Well, that's thousands of tokens is thousands of trillions of multiplications extra that you're doing in your LLM to kind of get to that final result. This is why, for example, you can now select in like chat GPT or something.

15:06Oh, I want to use like a low inference model. That means it's not doing as much, among other things, it's not doing as much of that rambling. So maybe your results will be worse on certain types of problems, but it's going to cost less. It's not going to take as much computation. All right. Okay. So that is where more or less we were. This brings us up to speed to Astra. Okay. So what we have heard, the report from the information, and again, this is not confirmed by OpenAI. So I'm just running off of their report. I'm pulling the thread from that report. It may turn out to be different. But the report we have from the information is that Astra was implementing some variations of a collection of techniques from the research literature for reducing the amount of these chain of thought tokens required, yet to still get good type of results like you would if you did have this sort of rambling.

15:59There's a name for it. Loop transformers is one of the techniques. Recurrent depth is another one of the techniques. Confusingly, sometimes recurrent depth feels like it's an umbrella term that also includes loop transformers, but they're sort of techniques that are in the same family. So we think Astra implements something like these techniques inside its implementation. Now, how do these things work? It's actually relatively straightforward. Let me bring this back up on the screen here. All right, so here we have our model of a large language model with the transformer blocks. But notice now I've added a loop on this transformer box.

16:35Now, this is like super crude what I drew here, but it captures the main idea, which is when you use these recurrent depth techniques, what you're doing is internally in your large language model, as your table of numbers is moving from block to block, there might be a place in here where when one of the transformer blocks outputs its updated table, you loop back and put that table through the same transformer block again and let it take another stab at it. And then maybe you do it again. That would be called loop transformers. You can actually group together transformers to loop. So maybe it goes through a certain number of transformers, and then you loop that output immediately back into the same transformers and have it go through again.

17:16The other variation of recurrent depth is where you are a little bit more selective about it. So some of the rows or the columns in that table, you have a router that selects which of those you loop back through or have them go through extra layers and others you don't. But it all has this notion of instead of just going straight from block to block to block inside of the LLM, this table of values is we're going to let it go through some of these blocks, some of those values multiple times and then let it continue on until we finally get our token output out at the end. Now, the idea here, like why would you do this?

17:50Well, the idea here is that it can be relatively inefficient, perhaps, to have to take the results of the computation happening in these transformer blocks and transcribe them into like English language tokens that you output. and then have those come all the way back in, be re-encoded and go all the way back to the transformer block again. If it's, for example, this one block is like the key block that's doing computation for your answer, they have it just have to encode its temporary work into like some values that make their way all the way out of the LLM into a token, back into the bottom, re-encoded, go all the way back up again to that block before it can do future more computation using that as input as well.

18:33That's a lot of computation that's happening where if you just kept looping the input to that one block a few times, it could analyze storage results in that table of value, analyze it again, update that storage results. You could actually dynamically get much more computation, useful computation done right there internally before you actually went out and generated tokens. There are some advantages to this. So why would OpenAI do this? There are some advantages to this. one is you need less tokens now to be output to get to a good answer because you essentially are allowing this musing, the rambling to happen internally before you generate tokens.

19:18So instead of having all of your rambling and tokens, you can actually do it inside, some of it inside the model. Less tokens means less computation, which means less expense. It also allows for smaller models, right? Because if it turns out, if we just loop through this transformer block 10 times, that helps our answer. That requires less parameters than actually just repeating that layer 10 times in your original model. If you loop on the same parameters, you have less numbers you have to store. And so the model footprint can be smaller. Well, memory on these GPUs is expensive. It's a core part of the expense.

19:56So the less memory you need to store a model you're using during inference or training, the cheaper things are going to be. It also potentially would give you faster results, right? Because looping internally on a block is a much faster computation than looping over the entire run of the LLN to generate token after token after token. That means you get lower latency, so the user gets an answer back faster. now again we don't know if that's exactly what open ai is doing because at least as of me recording this they haven't announced it but this was what the information sources was saying is there's some notion of a recurrent depth slash loop transformer style implementation this is then why security researchers were upset because it just means you have less chain of thought tokens because you're doing more internally so you have less of this sort of reasoning to be monitored.

20:49All right. That's the setup. Let's talk about what's good. Let's talk about what's bad. Let's talk about what is hype about this news. All right. If we look at this from the good angle, and again, the only perspective I care about is for you and me, the consumer out there in the market who is going to use these products and face the brunt of any of their negative side effects. All right. So what is potentially good about this news? Well, it's cheaper to run and smaller, that's important. That's important. I chain of thought. This is my take or a hot take here. Chain of thought reasoning is not sustainable, right?

21:27This idea of having this long string of token outputs to try to get better results, I think was just a response. It was a response to a marketing problem. The hyperscalers had to show we're continuing to make progress, right? Because they weren't making a lot of money yet. So they have to show like we have these models are getting better and better because they were selling the future. We're going to get to a place where these models are so smart that it's going to be in everything. And so we should be a$2 trillion company. So they needed results and they weren't necessarily getting the obvious, just the user sitting down and using GPT-4 and being blown away style improvements anymore.

22:05And the scaling wasn't even helping them on benchmarks, but this made benchmark numbers go up. And you will notice it It was around this time that they switched to chain of thought reasoning that the benchmark charts became the main thing that LLM companies talked about. Look at these numbers getting better. And those made the numbers getting better. But it's not sustainable because it requires a huge amount of compute. And the future of LLMs that us, the normal consumer, should care about is LLMs that are integrated into more of the stuff we do with computers. Now, in fact, OpenAI clearly recognizes this because if you look at the video, the slickly produced video that they released along with Astra, it is all about LLMs helping to power natural language interfaces in the computers, right?

22:49It's people talking to their computer about what they want it to do, and the computer goes and does it. That is a future that everyone should be excited about. This is what I think the primary use case for LLMs should be. It turns out to be harder than they thought. Microsoft tried this with Microsoft Office, with Copilot. They had to recently essentially scrap it because the LLMs weren't there yet. It didn't work very well. But that's a fantastic use case. I mean, clicking things with a mouse, like having to learn these arcane menus and buttons on software, we're used to it. But it's really inelegant.

23:21If Steve Jobs was alive today and involved in AI, what would he be telling Apple to work on? We got to figure out how to get natural language interface. And that's the next level of elegance, massive productivity booms. And OpenAI realizes this because it was at the core of their marketing of Astros. Like, this is what we're building towards. But if you're going to do that, you can't be working with 5 trillion parameter models that are doing long rambling chain of thought traces to get to the answer of we're going to click on this menu. We need smaller models that run faster that can fit perhaps on a chip that's in your own computer or phone or maybe a bigger chip but sits in a server in the server closet at your company, right?

24:02That's what you're going to need for this. It can't be everyone talking to Fable 5 or GPT-6 through an API. That's just way too inefficient and expensive. So we have to start thinking about how to make these models maintain the – we don't want to give up too much of a quality results, but we have to make them faster and smaller and generate less tokens to get the reasonable responses. So if you want to put a positive spin on this, these are the type of innovations you need to do to get towards more normalized LLM use in the everyday of normal people, right? So that would be the good spin that I would put on this type of result.

24:41All right, what's bad about this? Well, let's look at what the computer security – what were the computer security community upset about? Well, they're upset about the other thing. that these AI companies are doing with LLMs, which is creating these prompt loops, as I call them. They call them agents, but I call them prompt loops because that's what it is. Actuated prompt loops, LLM-powered prompt loops that they're running with powerful tools, equipped with powerful tools for long amounts of time with no supervision, and they go off and wreak havoc. Now, when you're using LLMs to power these agents, to power these prompt loops that you, for God knows what reason, gave hacking tools and let run for days without even monitoring them, well, now you care a lot about the chain of thought reasoning that these models output.

25:31Because it turns out, one of the only ways, if you're going to let an LLM come up with all the plans for a prompt loop or an agent, which I think is a really bad idea, but if you're going to do that, one of the core ways you can figure out what's going on is if it's reasoning about each move in English and tokens in English, like here's what I'm going to do next, well, you could have language style tools that look at these traces and are like, ooh, this looks dangerous. Why don't we stop the prompt loop? This is a widely accepted idea right now. I'm going to bring a paper up here on the screen.

26:04The title is Chain of Thought, Monitorability, a New and Fragile Opportunity for AI Safety. It has all the names on its co-authors from the companies, from the safety community. It has Jeff Hinton and Ilya Suskever, Susquevur, as they call them, expert endorsers, people who are reading it. Let me just read you the abstract of this paper. This is a recent paper. AI systems that think in human language offer a unique opportunity for AI safety. We can monitor their chains of thought for the intent to misbehave. Like all of the known AI oversight methods, chain of thought monitoring is imperfect and allows the misbehavior to go unnoticed.