Is AI About to Automate Every Office Job? | AI Reality Check

0:00Back in February, Microsoft Chief Executive Mustafa Suleiman sat down for an interview with the Financial Times. During it, he made the following extraordinary claim. I think that we're going to have a human level performance on most, if not all, professional tasks. So white collar work where you're sitting down at a computer, either being, you know, a lawyer or an accountant or a project manager or a marketing person. most of those tasks will be fully automated by an AI within the next 12 to 18 months. Now, if this prediction is true, then we're just a year away from one of the most sudden and calamitous economic shifts in the history of modern economics.

0:44I mean, worldwide, the knowledge and technology intensive industries produced over$10 trillion of value per year and make up more than a third of economic activity here in the US. So if basically all of this could be replaced by compute and this is going to happen by next spring, it would make the Industrial Revolution seem glacial by comparison. It would be the economic equivalent of the asteroid that killed much of life on Earth, including the dinosaurs. So is it possible that Suleiman is right? And if he's not, what's a more realistic understanding of what AI will and will not be able to do in the workplace in the near future?

1:24Well, it's Thursday, which means it's time for another AI reality check episode. So this is a great opportunity to dive deeper into Suleiman's claims. Now, I have a lot to say on this topic, including, if you make it all the way to the end of this episode, a little conspiracy that I uncovered when I was doing research on this topic. So stay tuned for that. But we have a lot to get to, so let's get started. As always, I'm Cal Newport, and this is Deep Questions, the show for people seeking depth in a distracted world.

2:02All right, as you may have guessed, I'm going to argue here today that Mustafa Suleiman's claim is not accurate. Now, I have three major reasons to propose why he is inaccurate in his claim. I'm going to present these three reasons in order from least technical to most technical. Now, to be clear, I'm not trying to be like Dr. AI skeptic guy here, right? I mean, obviously people are finding uses for LLMs in the workplace, even if they are nowhere ready to replace all knowledge work jobs. So I'm hoping that as I get to the end of these three reasons, and before I get to the conspiracy I promised you at the end, that I'll be able to review the ways that these tools actually are being useful.

2:44what are the actual parameters of where LLMs are or will continue to make a difference in knowledge work in the near future. So we're going to give some positive information as well towards the end of this episode. All right, but let's dive in. I want to start with my first reason why Suleiman is likely wrong in his claim. That reason is other tech leaders don't agree. So before we get into the details of what LLMs can or cannot do, I want to emphasize that Suleiman's idea that we are 12 to 16 months, and given that he said this in February, that means basically a year away from all knowledge work jobs being fully automatable.

3:25So he said fully automatable by AI. That claim is an outlier compared to what other leaders of major AI companies have been saying. So, for example, before Suleiman grabbed the crown of most doomer about AI job impacts in February, Dario Amadei had been the most pessimistic among the CEOs. On multiple occasions, the claim that he made is that AI will replace up to 50 % of entry-level knowledge work jobs within five years. That's the claim that Amadei has made several times now. Now, look, that's not great news either, but it's actually much less dramatic than what Suleiman was predicting. Think about it.

4:06It's a longer timeline. Amadei's talking up to five years. It's only affecting entry-level jobs, right? So he's not talking about all knowledge work jobs, but just entry-level jobs. And he's not talking about most jobs. He's talking about 50%. So on all scales, we might measure Amadei's claim. It's much less drastic than what Suleiman was saying. If we turn our attention to NVIDIA's Jensen Wong, he's much more aggressively against the type of claim that Suleiman or even Amadei is making right now. He doesn't really see AI fully automating many jobs at all. And in fact, at a recent event at Stanford, Wong argued that making these predictions about large swaths of the economy being automated is actually anti-productive, if not just straight up false.

4:53So I want to play a clip of Jensen Wong at a recent event making the opposite claim of Suleiman. First of all, I think the narratives of AI destroying jobs is not going to help America. First of all, it's just, it's false. Wong goes on this event to argue that what we're more likely to see is AI tools being integrated into work like we did with computer tools in the 90s and early 2000s. It will change what jobs look like. Like it'll change the day-to-day, maybe what tools you're touching, but it's not going to wholesale replace large swaths of the economy. He notes in that Stanford appearance that the engineering teams at NVIDIA use a lot of AI tools, and they're busier than ever before, and they're hiring more engineers than ever before.

5:38So he doesn't see AI tools as a job destroyer, but instead as a job changer. All right, so let's move on now to the second reason why Suleiman is likely wrong in his prediction. We are not seeing enough progress. All right, so Suleiman says we're basically a year away from most knowledge work jobs being fully automated. If that was true, we would need to be seeing major rapid advances in LLM technology to keep us on such an ambitious trajectory, but it's simply not what is happening. Now, look, this might sound crazy at first because you're bombarded with news constantly about AI and AI is this and AI is that and we're scared of this.

6:21And it gives you this general impression that aren't things just moving really fast in the AI world? But if you cut through the PR and the hype, here's what you will actually discover if you follow this technology closely. Since roughly late 2024, progress with newly released LLMs from the frontier AI companies have been making steady but not particularly fast progress. And instead of seeing the type of immediately obvious functional improvements that we got used to in that era when we went from GPT-2 to GPT-4, most of the improvements we're seeing from model to model are largely being captured in benchmarks.

7:00So charts of numbers of tests invented by the AI companies themselves in many cases that have obscure acronym names. So we just see these charts where they'll say, look, we got a 20 % boost on this benchmark. We're now moving to be comparable to this machine on this benchmark. So we're not seeing major revolutionary leaps in functionality more. Instead, we're seeing slow and steady progress. To give you an example of what this is like, I'm going to load up here on the screen a Reddit thread that's talking about the most recent release from Anthropic, which is Claude's Opus 4.7. There's a summary here of a vigorous conversation that happened on this Reddit thread about the new Opus 4.7.

7:42Here's the summary. The verdict is in, and it's not pretty. The overwhelming consensus in this thread is that Opus 4.7 is a massive regression and a serious downgrade from 4.6. Users across the board are reporting a dumber, lazier, and less reliable model that It feels like a step back to early chat GPT. All right. So we're not talking about this, their newest model. We're not seeing revolutionary leaps from model to model. This is much more of a jagged type of frontier. Make progress here. Take steps back elsewhere. Another model that was released recently was OpenAI's GPT 5.5. And people seem to like this better than Opus 4.7.

8:22But what is the magnitude of these improvements? Well, I loaded up a review. I'll put it on the screen here. Matt Schumer had a long review he posted online where he said, a big upgrade that doesn't always feel like one. I'll read you a couple things from this review. Schumer is excited about continued improvement in the LLM playing nicely with coding harnesses, so producing long-term plans for coding. Here's what he wrote. For serious software work, it is exceptional, thoughtful, careful, able to make many of the same decisions I would make and very good at iterating against a goal until a thing actually works.

8:57But as he also goes on to point out, these models have been getting slowly and surely better at this type of thing steadily now for about a year. So you might not actually notice much of an improvement because they're already pretty good at that. Here's a summary of what Matt called the biggest story about GPT-5. GPT-5 rounds out the weaker parts of the GPT line, design from existing context, iOS native Mac app, security, etc. So what are we hearing here? This is what we're hearing about these latest, newest models is slow and steady. It's like normal software updates. We improve the native Mac iOS thing.

9:34We tweak this functionality. Now we're getting better scores on this particular benchmark. Sometimes they swing in a miss like Opus 4.7. The tweaks we made actually made this worse. And so people are going to go back to the previous. There's nothing bad about that. This is just like the normal pace of software improvements. But the problem is to get from a place where we are now where almost no knowledge work is fully automatable, no major task is fully automatable, to a place where all knowledge work tasks are fully automatable, we're not going to get there in another year with these slow and steady improvements, one step forward, one step back, two steps forward, one step back.

10:10If we tweak this, we improve that, we improve that. That's nowhere near a fast enough pace to get us where Shuliman said we're going to get. Now, at this point, you might be saying, yeah, but what about coding agents? Coding agents feel like this example of a major knowledge work task, software engineering, that we weren't using AI heavily except for in like more autocomplete ways. And then suddenly it seemed last fall into the new year, everyone was using AI coding tools within software engineering firms. It's not everyone, but it's like massive percentages. Now, the way that the general public thinks about this is like, well, I don't know, like AI, quotes around it, keeps getting better and better, and it got good enough to automate that all of a sudden.

10:52And as it continues to get better, maybe it will all of a sudden unlock other types of tasks that we can fully automate. But that doesn't really describe what happened with the rise of coding agents within enterprise software development teams. What you have to understand is that that leap was actually as much about, if not more about, the quote-unquote coding harnesses as it was the underlying models. The coding harnesses is the software program written by people. It's not machine learning. It's not AI that actually calls the LLMs for ideas and then executes the LLMs ideas on behalf of the LLM.

11:29What really happened is there's a multi-year process of multiple companies working on these coding harnesses to make them more and more relevant. They're trying to figure out how can we get LLM's ability to produce computer code, which we've known since 2022, how do we get this ability better integrated into how actual professional software development teams operate on really large code bases? And so most of the innovation actually happened in that coding harness. They figured out all these different rules and approaches. We're going to use skill files. We're going to have a sort of simulation of memory over these skills files.

12:04There's a ton of just old-fashioned, like 1950s-style AI in these coding harnesses, like Regex pattern matching, using existing software tools to verify code. So there's a lot of kludged together stuff. You can read, I think Gary Marcus had a good piece about this a few weeks ago because they leaked the code for the CloudCode coding harness so we know what's in there. So there's just slow and steady work on building the coding harness until they could finally figure out how to make it play nicely enough with enterprise coding that we can now bring LLMs into that process. Now, they also tuned the models to play better with the harnesses as well, but I think it's in the harness development that we saw.

12:42That's what made AI coding relevant to big software development teams. So what this tells us, if you want to have a similar jump in another type of major knowledge work task, somewhere you have to have a lot of people iterating for maybe a year or two to try to figure out the right harness to connect properly into that particular type of job. And if we want Suleiman's claim to be true that all major knowledge work tasks will be fully automatable, you would need like a thousand of these teams, each of them focused on another major knowledge work task, trying to build a custom coding harness that works just for that task.

13:17Well, they're not doing that. You don't have enough people to do that. The market isn't there. And it takes experts. The one thing that people working at AI companies are experts at is software development because that's what they do. So this was like the ideal place to do this. So I think the real lesson of the quote-unquote sudden emergence of coding agents is that it's actually really hard and takes a lot of focused work to try to integrate AI into individual types of workflows. It's not something that just happens as the models get smarter. So again, unless these companies are hiding thousands of teams working on all these different areas of knowledge work trying to find ways to subtly integrate AI into them, I do not see how we're going to suddenly have coding agent-style automation in many other tasks within a roughly 12-month period.

14:04All right. My third and final point was why Suleyman is likely wrong. The functionality of LLMs are limited. so that's we're moving down this technical stack here let's briefly open the the black box around llims to better understand what they can and more importantly cannot likely do no matter how big we scale them so let's start from the bottom we've done this before so i'll go quick what does an llm actually do it predicts tokens you give it text and it outputs what token should come next it has been trained to assume the text it's given as input is a real text written by a human that actually exists.

14:46And so there is a right answer for what the next token is, and it's trying to get the right answer. That's at the base what an LLM does. Okay, so then how do we get long text out of an LLM? Well, you have to actually put a program on top of the LLM to keep calling it again and again what's known as auto-regressively. So you have some text, you give it as input, It outputs a token. You put that token at the end of the input text, and now you take that slightly longer input text and put it through the LLM, and you get another token. Now you take that even longer input text and you put it through the LLM, and you keep doing that until it's grown out a whole answer, and then you can return that answer to whoever made the original prompt.

15:25So what this auto-regression of token guessing gives you is basically a story completer. Here's some text, and the LLM is implicitly trying to finish the story it was given as input. as accurately as possible based on all the types of text it's seen so far. Now, in the original big LLMs like GPT-3, it would create reasonable stories, but they would be all over the place. So then they figured out how to then what's known as post-train or tune these LLMs of the possible ways they could go to tune them towards certain categories of types of text and away from others. So GPT-3.5, which was the version that powered the original chat GPT, was tuned to think about the stories it's completing as answers to questions.

16:11So you're giving your prompt, the input's a question, and you're trying to answer the question. And now it's much easier for average users to deal with. All right, so they're story completers. Now, we don't want to downgrade that, right? This isn't a dismissal, like, ah, it's autocomplete. Because what we discovered, particularly with GPT-4, is that as we scaled these things up, in order to successfully complete stories in reasonable ways, like these LLMs actually encoded a lot of really interesting rules and logic. If my story involves a math problem, to complete that story in the right way, the LLM is going to have to have some math logic built in there because otherwise the story is not going to seem reasonable.

16:50and this was really the big discovery of GPT-4 was like wow there's all of these abilities that were implicitly coded into this LLM during its training it's just trying to complete the stories but we trained it on so much stuff for so long that it learned all of these types of abilities that allows it to win the complete stories well it understands humor, it understands math it understands computer code it understands basic logic if it's seen a game enough time it has some sense of generally what the game rules are or what a valid move in that game looks like. It's pretty incredible what GPG-4 could actually do.

17:24And there was this idea of like, if we kept scaling these things bigger and bigger, yeah, sure, they're story completers, but they can eventually have so many rules and logics programmed into them to win this game that like they will actually have like a human level intelligence. And then we can just use the LM as the brain to automate everything. That was division. Okay. Then what happened is we learned by the summer of 2024 is that this type of just scaling was hitting a wall. Just trying to make these models bigger and train them longer was not appreciably leading to new functionality being encoded into them like we had seen before.

18:00And this picked up a whole era that really started with the Alphabet suit models out of OpenAI in the fall of 2024. It kicked off a whole era of post-training and tuning where, okay, they get further appearances of improvement in these models. We can't just scale them bigger. We can do things called post-training, where if we have very specific data sets where we have questions and exact right answers, we can tune an already pre-trained model to be better at using its already wired intelligence to answer this type of problem. And that's why we saw, starting in 2024, a big focus on the type of things for which we had data to do this tuning, reasoning, math, and computer coding.

18:40That's why, why weren't they talking about other types of things you would want an LLM to do because it'd be economically very valuable? Because all we can do now is tune these and get sort of like steady improvements in areas where we happen to have a lot of highly structured data to do the tuning on it. And that's really where we've been since 2024. And that's why we have more of this slow but steady improvement with LLMs on particular benchmarks and no longer these big general leaps like we had during the prescaling era when we saw like three to three, five to four. Okay. So this is a problem for automating work because we know we can't scale these LLMs just to have like a human level of reasoning on things.

19:21And for most things we do in knowledge work, most of the skilled things we do, we don't have highly structured data sets like we have for computer programming or math. So we can't even tune the pre-trained LLMs to be better at things that a lot of people do in their jobs. So this is sort of a wall that's hard to get past when you leave where we are right now with LLMs. Now, you might say, yeah, but what about workplace agents? A lot of what people do in knowledge work is actually not that skilled. It's individual things an LLM can do. Send an email, get information about an upcoming conference, move information to a spreadsheet, build slides, and send it to the team.

19:59Like, actually, a lot of things we do in knowledge work is what I call shallow work. It's things that don't require a particularly high level of skill. So why can't we have at the very least something like a knowledge workplace administrative assistant that can automate a lot of boring stuff we do in knowledge work like we have in computer programming where coding agents can do multiple steps of work on behalf of the programmer? So at the very least, this wouldn't be Suleiman's prediction, but it would get us closer. Why don't we have those either? Well, it turns out that building these type of tools is difficult.

20:34I wrote an article about this back in January for The New Yorker. It was called, Why Didn't AI Transform Our Lives in 2025? And in that article, I looked at the question of, why don't we have more workplace agents? We were told at the beginning of 2025 that this would be the year of the agent, not just coding agents, but for all the things you do in work, more people would have these AI assistants. why did that largely not happen? But when you actually study how these agents work, the problem is the plans, the multi-step plans that agents execute are the result of LLM prompts. You have a harness, which is a computer program written by a person that prompts an LLM and says, give me a multi-step plan for doing X.

21:17And then it takes that text and it goes step-by-step to harness and executes those things on behalf of the user. For computer programming, I mean, they've learned how to be pretty good at making those multi-step plans because the option space of what you do when you're building a computer program is very narrow. And what they're doing is very verifiable. And so actually the harnesses for coding agents will call non-AI tools. They can verify things. All right. This step has a clear success category like the code compiles and passes these tests. And we're going to run another program that's going to check the code, make sure it compiles and passes those tests.

21:51And if it doesn't, we can go back to the LLM and say that plan didn't work. Give us another one. Right. It's very well set up for doing this. But once you're in the more, so I wrote about in my New Yorker article, once you're in this more general world of more ambiguous sort of knowledge work task, what you're going to get out of the LLM is a reasonable sounding plan, reasonable sound, because that's what it does. It's a story completer. Reasonable sounding plans get you in trouble. You need correct plans. And the way that like humans actually make plans is we do a few things. One, we test a bunch of possibilities internally to see what makes the most sense.

22:24And two, we have some sort of notion of correctness and a world model we can use to evaluate plans to see, does this actually do what I want it to do? Are there any mistakes along the way? That's not how LLMs operate. Again, we're just auto-aggressively producing tokens. So you get a reasonable-sounding plan, but it's not stepping back. It doesn't have a world model to test it against. It can't have hard-coded rules that it consistently applies. It has no ability to do future simulation of possible outcomes. And so we just get a story that sounds like a good plan, but often has issues along the way.

22:56And if an agent is just going to automatically execute these things, we get into trouble. All right. So making agents even just like administrative assistants in non-computer programming areas is also a very hard problem and not one that we have a good solution to. We also underestimate the degree to which if you watch a programmer using a coding agent, they're constantly tweaking and re-asking and adjusting. there's a huge amount of work to try to get consistently useful output out of these agents. And most knowledge workers just simply aren't going to do that or have the technical chops to pull that off.

23:30Earlier this year, actually, in the fall, OpenAI even mentioned they were slowing down or reducing their non-coding agent projects because they weren't working. And they wanted to focus on ChatGPT and their coding agents. All right, so if we put all these things together, we put these three reasons together, I think it's clear that Mustafa Suleiman's claim that AI would fully automate most knowledge work jobs by next spring really is an outlier opinion. It's an opinion that is contradicted by the statements of other CEOs, the reality of the rate of progress of LLMs over the last couple of years, which is not nearly fast enough to get where he says we're going to get, and the technical limitations of these models.

24:06They're just LLMs with a harness on it is just not a great setup for automating a lot of different types of knowledge work jobs. so why then might Suleiman be pushing this story what's in it for him I'm going to play you a quick clip of one potential explanation it comes from the absolutely agentic channel let's hear what they had to say there and it's worth noting that this interview comes at a particularly interesting moment for Microsoft their stock recently took a hit as markets questioned whether the enormous capital expenditure on AI infrastructure is going to pay off so Suleiman has a motive to sell the dream.

24:44More generally, claims about AI massively disrupting the job market, like we just sort of heard there, tend to be very useful for the major AI companies. They know they can just say wild things about economic disruptions, and they're not going to be challenged on these predictions, even if they don't come to be true, because it matches a general vibe of there's disruption, and that's exciting or scary, or we want people to be worried, or whatever it is. They're not going to be challenged, so why not? and it makes them seem like they're working on the most important technology ever, which is exactly the type of technology.

25:15If you're an investor, you would say, sure, Anthropic. We've given you$60 billion and you've made$5 billion to date of revenue. But like, hey, I'm not going to do normal expectation accounting with you because you're building the most important technology ever. And so none of these numbers matter. You're going to automate the whole economy. So it's super in the favor of the CEOs to say these things. Now, to be clear, I don't want this to make it seem like LLM-based tools are irrelevant to the workplace. They're not. As I promised, I wanted to talk briefly about the ways in which LLMs and LLM-based tools are actually useful in non-coding knowledge work sectors.