Are We About to Lose Control of AI? | AI Reality Check

0:00Last week, Anthropic released a report with a scary-sounding title, When AI Builds Itself, and it came accompanied by a scary animation that shows machines replicating themselves exponentially like cells in a petri dish. Now, the body of the report itself keeps these dark vibes going. I want to read you some actual quotes here from the intro to the report. They say, for most of AI's history, humans drove every step in its development cycle, but at Anthropic, we are delegating a growing share of AI development to AI systems themselves, which is speeding up our work. Taken far enough and given enough compute, this trend points to an AI system capable of fully autonomously designing and developing its own successor.

0:47This is called recursive self-improvement. We are not there yet, and recursive self-improvement is not inevitable, but it could come sooner than most institutions are prepared for. A little bit later, they then add, AI that can build itself would be a major development in the history of technology, one that could bring enormous good for the world in science, healthcare, and beyond. But full recursive self-improvement also might increase the risks of humans losing control over AI systems. Now, if you look at the headlines generated in response to this report, most of them focused on a section of the report that seemed to call for a worldwide pause on AI development to avoid this scenario of humans losing control.

1:34But if you read that section closer, you see that's not actually what the report says. Here's the actual wording. If it were possible to effectively slow the development of this technology to give ourselves as more time to deal with its immense implications, we think that would likely be a good thing. But if a slowdown simply lets the least cautious actors catch up technologically, it could leave everyone less safe. So in other words, Anthropic is saying, we'll only slow down if everyone else around the world does too. Otherwise, we have no choice but to continue with our efforts at full speed.

2:08Now, look, this is pretty grim stuff. Anthropic is basically saying that we are potentially hurtling towards a world of AI that improves itself rapidly until we lose control over it. And they're saying there is nothing that we can do about it except maybe continuing to publish solemn reports with fancy animations and I guess also cash in on our stock options after an IPO. All right, so here's the key question. Are these fears justified? Well, it's Thursday, which means it's time for an AI reality check episode of this show, which is a good opportunity to go looking for some measured answers. And that is exactly what we are going to do.

2:47As always, I'm Cal Newport, and this is Deep Questions, the show for people seeking depth in a distracted world.

3:02All right, so how much should we actually be afraid of recursive self-improvement? I want to look at the core charts from this Anthropic report so we can see what they are pointing to that is giving them these RSI fears. All right, I'll load the first chart from the report up here on the page. It's called Code Contributed Per Quarter, Per Person by Quarter. It's measuring how much lines of code their engineers are producing over time. And what we see is in 2021, 2022, 2023, 2024, the beginning of 2025, not much. and then the second half of 2025 and into the first half of 2026, the amount of code jumps up.

3:43All right? Here's how the report itself describes this trend. It says, a caveat, lines of code is an imperfect measure as it measures quantity over quality. So eight times lines of code per engineer per day in the second quarter of 2026 is almost certainly an overstatement of the true productivity gain. Nonetheless, it indicates an acceleration. All right, so chart one, Once we introduced these tools in late 2025 for software development, we began producing a lot more code using AI. All right, let's go to the second major chart in this paper. This is called Claude Code Session Success Rate. What we see here is various color lines.

4:24Each of these represents a different type of problem. You could ask an AI LLM to solve, trivial task, routine task, substantial task, and then at the bottom, open-ended problems. It starts in the fall of 2025. And what we see is, especially with open-ended problems into 2026, and as the new models, Mythos and Cloud Opus 47 are introduced, the success rate with those jumps up from low 20 % to somewhere between around 70%. Okay. Notice that graph starts, it just starts in the fall of 2025 because, and we'll get back to this in a second in more detail. But in order to ask AI to solve a hard coding problem, like the example they gave was, why is this thing – the system we programmed is crashing.

5:12Why is it crashing? And sending the AI to figure out why. You can't actually have data points for before the fall of 2025 because that's when Anthropic released along with OpenAI their sort of first mature what's called a coding harness. So a control program that allows you to leverage an LLM plus a lot of hard-coded logic to actually tackle multi-step plans. So you can't really have data for that from before because there's nothing to ask to try to solve a problem like that. You couldn't just prompt the chat GPT to do that or plot. So you needed the arrival of mature software development tools built on LLMs before you could even do these tasks.

5:47So basically what this graph is showing is that like the very first software development coding harnesses they introduced in the fall of 2025 couldn't solve the hardest type of problems, and then they sort of fixed that the next year. All right. There's one final chart in this paper that they're pointing at to justify their concerns. I'll put it on the screen here. The title is, Where Researcher Went Wrong, Could Claude Have Done Better? And we see here a bunch of different models of Claude and percentage bars. And back for these early models of Claude, we were getting like 50 % or 45%. And now with the very newest versions of Claude, like Opus 4.7 and Claude Mythos, we're at like 59 % to 64%.

6:36So we got like a 10 % or 15 % improvement on that measure. What is this measure? It's a little bit complicated. Essentially, they have these transcripts of programmers working on programmer-style tasks, and what they're looking for is an example where there's some problem that the programmer is trying to solve, and they take a wrong turn. So they go down some path to explore something that turns out not to be the correct source of the problem. So what they would do is take the transcript of this session right up to the point where the human was about to try to explore something wrong. wrong they fed this transcript to one of these mature um coding harnesses on top of a llm and said hey what do you think we should do next and and if it if it found the right thing to do where the human looked at the wrong thing then they would say this is a case where like the llm plus its coding harness were smarter um than the human so that's jumped from like 50 percent from a couple years ago to like 64 percent now all right uh so that is that's this is sort of like the core data they're looking at, they capture this idea of AI is getting so smart at producing programs that maybe soon it will be able to not only improve itself, but, and I'm going to quote here from the report, become capable of full recursive improvement and begin building their own successors.

7:53All right. So looking at that data and what I know about current AI tools, are these fears justified? And I would say, no. Here's what this data is all describing. About a year ago, the major AI companies got serious about building tools to help software developers. These tools are a combination of human-written programs called coding harnesses and LLMs. The coding harnesses can make calls to the LLMs and then act on what the LLMs do. The coding harnesses can also interact with other various tools on the computer. So what all these charts seem to be showing is that like, oh, once they got serious about writing these coding harnesses and also tuning the LLMs to play nicer with these coding harnesses on these sort of computer programming-related tasks and evaluations, things jumped up.

8:43The world before having these harnesses, we couldn't do well on programming tasks, and now that we do, we do better. Now, none of this is trivial. In fact, it's a very smart market for the AI companies to go after. Their software development is a big industry. These coding harnesses built on top of LLMs are like really potentially very useful, right? Because LLMs are very good at understanding and producing code. The type of tools that a coding harness has to access in order to execute things are simple and text-based. Like this is like a perfect case market to build the first killer apps on top of LLMs.

9:21And starting last year and then really picking up speed last fall, the major players really got involved in these harnesses. And so I think that's what these charts are all showing is like, yeah, these harnesses make us, they're getting really good at the, we can suddenly do a lot of programming tasks we couldn't before. But does that mean that recursive self-improvement is imminent? It does not. And there's two reasons why. And I want to look at these one by one. Point number one, faster software development doesn't equal smarter AI. All right, so these tools help computer programmers produce code faster or find mistakes or issues in existing code or systems faster as well.

9:56These capabilities, though useful and perhaps a good source of revenue for these companies, doesn't add up to AI being able to improve themselves to create AI systems that are much smarter than what humans would have otherwise been able to produce. Now, why is this? Because the bottleneck to producing breakthroughs in AI, to building new AI systems that are substantially more capable than those that came before, is not the speed with which you produce computer code or track down bugs or issues in existing code. The thing that advances the capability of AI beyond just training it longer are ideas.

10:35Like if we look at our current world of gender of AI built on large language models, there's three big insights that built on each other. The first was Jeff Hinton and his collaborators working on back propagation, right? This was an intuition that if you applied calculus properly, you could train neural nets that had many, many layers. This was the so-called deep learning revolution, really picked up speed in the 1990s into the 2000s, and it was critical for what we have today. The second big idea that led to our current AI world was Google research notion of adding something known as an attention transformer.

11:09There's a mathematical formalism you add into the feedforward architecture of a large language model that allows there to be a selective attention on the input tokens as you are trying to calculate what token to produce next. This allowed sort of coherent generation of text based off a very complicated input or request, right? This was revolutionary. This is largely a sort of intuition around architecture and linear algebra. The third breakthrough that led to our current moment was researchers at OpenAI, led by Jared Kaplan, wanting to know what happened if you scaled the size and training compute of large language models well beyond the limits where traditional machine learning theory said you're going to start overfitting.

11:56Those three things together led us to this current future, and none of those had anything to do with computer programming. It wasn't, oh, if only we could have programmed faster or been better at finding bugs in computer programs, we would have had those advances quicker. These advances were scientific, not engineering. AI does not advance. It's not created at the fingertips of computer programmers and speeding up those computer programmers does not speed up the rate at which we get smarter, more capable, or more advanced AI systems. All right, the second point I want to make here is that these tools, these software development tools that they're testing in these charts are completely controllable.

12:38Arguments that say we're on track to losing control of AI like to think about these sort of AI-based coding tools as some sort of unknowable alien black box that we ask to do things and we don't know what it does. It just kind of like goes off and behave, and we don't know its intentions, and in fact, it could develop somehow its own intentions that are contrary to us and go rogue, right? We just see this like HAL 9000 from Stanley Kubrick's 2001. This is not how these systems work. And again, I mentioned this before briefly, but let's just say it again briefly right now. The software development tools that they are testing in these charts is a combination of a large language model like Claude Mythos or Opus 4.7 and what I've been calling a coding harness.

13:24Now, the coding harness is a computer program. It's written by humans. There's nothing obfuscated about it. It's not a neural net. It's not something that trained. There's no mystery about how it behaves. It's a bunch of pattern recognition, often using old-fashioned tools like regular expressions and a huge amount of conditional logic, if-then statements, if this, do that. a lot of hard-coded cases of things that comes up often in software development, and they want to make sure they do the right thing. This coding harness, programmed by people and completely linear, deterministic, we know exactly how it works, will then make calls to an LLM through an API when it needs LLM intelligence, right?

14:02So it might say, okay, what we need to do here is produce some code. All right, I'm now going to create a prompt and send it to the LLM. Can you give me a code that does this? And the LLM will return some code. Or, okay, we're trying to, we want to create a multi-step plan for investigating why this program is crashing. I will put that into a prompt. I will send it to the LLM, say, will you write me a multi-step plan for this goal? So here's the constraints. Here's the tools you can use right in this format. The LLM produces text. It gets that back. And now the coding harness, human written code, completely non-obfuscated, completely deterministic, looks at that plan and says, great, let's break this up into steps and execute them one by one.

14:39It's the coding harness that actually has any action that actually accesses tools. It's the coding harness that has all the control logic. So we program that, which means it's imminently controllable, right? So like, let's do a thought experiment here. Like, let's say there's a certain tool you don't want your programming tools to use on your computer. Well, you could do that with 100 % certainty by just never having your coding harness call that tool. Just like no matter what instructions it gets, it's just not on the list of tools that it'll call. But this is the thing we often mix up. LLMs are unpredictable, okay?

15:21They're token producers, right? And they're deterministic in the sense that if you give it the same input, it'll give you the same output. But they're unpredictable because the output that you actually get from an LLM is effectively a probability distribution over potential next tokens, and then you probabilistically select a token from that distribution. This is why if you ask the same prompt multiple times, you might get different answers because those random selections made outside of the LLM might select differently. So LLMs are unpredictable, right? That's why if you say, hey, give me a plan for this, like sometimes the plan will make sense and sometimes it doesn't, and sometimes it'll seem reasonable, but it'll have a weird aside.

16:00Or if you give it a big scenario and say, what do you want to do next? It might, you know, it might say something you weren't expecting. And if it gets little cues and it prompts that we're playing like a sci-fi game, it'll be very sci-fi and maybe it'll do like weird things. LLM textual output is unpredictable. That doesn't mean that AI systems that query LLMs are unpredictable because all of the control logic and action is done in human written programs, right? So this idea that it's like, these are just these black boxes that maybe at some point will spontaneously start improving themselves makes no sense.

16:30The LLM is a static thing that you can prompt and it will give you text back. That's it. And a human control program does that prompting. That is not a setup where there is a, where is the growing intention here? Where is the, like, we're going to change our own code? That type of thinking only makes sense when you kind of obfuscate this all as some sort of alien brain we don't understand. No, it's an LLM that's unpredictable plus a very predictable coding harness. All right, so let's step back here. What's a better description of the reality where we are right now as these programming tools built on LLMs are getting good, right?

17:09Like, I mean, we've only been working on it for less than a year, but we're finding like, oh, this is a great application of LLMs. We're still trying to figure out what these tools can do and can't do and how best to integrate them. What's a better description of this reality? Well, I want to bring one more chart up here on the screen. It comes from a recent John Byrne Murdoch article in the Financial Times. I actually saw this chart first through Gary Marcus's newsletter. Here's the chart. The title here is Relative Change to Monthly Volumes of iOS App Releases and Reviews. And here's what we see is over time, starting in 2025, when these AI-based coding, mature coding tools were first released, the number of iOS apps released jumped up.

17:49You got a dark blue line there. At the same time, let's look at this light blue line. Apps with significant usage, it stayed steady and is even falling. All right. This, I think, captures something important. We're seeing that the introduction of these software development tools led to an increase in the number of mobile apps, which are like exactly the type of programs that like basically anyone with like some tokens to burn and cloud code can just spit out. they're easy for AI to produce, while at the same time we're seeing a decrease in apps with significant usage. AI powers software development tools to speed things up, and that makes everyone involved feel more productive, but it doesn't necessarily mean that we're accelerating the creation of useful new things.

18:35AI often offers interesting potential to a lot of fields, but figuring out how to make it economically useful is a complicated effort that requires a lot of focus. Putting out these sci-fi themed essays about all the doom that's lurking right around the corner might make these AI companies feel important or exceptional, but I'd argue that at the moment they're proving, if you'll excuse the use of an ironic term here,

19:01unproductive. Programming tools are interesting. They're here to stay, though in their final form and how we use them is still up in the air. This is a new thing. But nothing about this new tool's arrival should make you think that somehow AI is going to start improving itself. I would almost say shame on Anthropic for dropping this report. No suggestions, no reassurance, no culpability, no responsibility. Just like, what can we do? This is not productive. This is not useful. If you're wondering, let's just sum this all up. So no, clod code or codex or cursor is not a thing that's going to very soon start improving itself to lose control of AI.

19:49We've got bigger fish to fry here. All right, that's enough for this week. We'll be back on Monday with an advice episode of this show, which you check out. I'll probably have another reality check next Thursday. Until then, however, remember, care about AI, but not everything you read about it. Hey, if you've made it this far, you must be ready to join my fight for depth in a distracted world. Now, the best way to do this is to join over 125 ,000 people who receive my email newsletter each Monday. You can sign up at calnewport.com slash ideas. And when you do, I will send you a free guide to my seven best ideas about cultivating a deep life.

20:28Sign up today, calnewport.com slash ideas.