Anthropic’s Head of Economics on AI adoption data, Claude Code, the burden of knowledge & the next generation of experts

0:00So Anthropic, the leading AI lab behind Claude, has just released the next edition of their economic index report. They have analysed millions of real conversations with Claude to map exactly where AI is augmenting human work today and where it isn't. I'm with Peter McCrory, who is the head of economics at Anthropik, and he's the one who led this research. I think it's the best empirical window we have into how AI could be sharing, shaping work right now. So, Peter, thank you so much for breaking away from your computer and joining us today. It's a privilege to be here and so glad to be able to share this work with the world and all the underlying data, which I mean, I'll talk about, but everything that we do is based on open source data.

0:47So we hope that others will join us in making sense of what's on the horizon. You know, that is such an important point because in this moment of the investment boom and the prospect of artificial intelligence really changing the way we live, the quality of data I found has been really, really poor. It's sort of scuttlebutt and, you know, survey a few mates and slap a logo on it and change the way the market thinks. So when you get data from, you know, Anthropic or Epoch or others, it's really good to be able to hold on to it. Your data shows something quite striking, which is two completely different use patterns emerging at the same time.

1:29So if you're using Claude, you can do it like most of us do through the chat interface, you know, tippity, tappity, tippity, Claude goes away and thinks and comes back with an answer. Or you can use it through the API, which is a programming interface, which means that perhaps you're accessing it through another piece of software, maybe one that you have written or more likely your IT department has written. So what I found really interesting is, and that your previous report showed as well, that use cases through the API are about 75 % in what you call automation of tasks, but they have lower success rates.

2:06Whereas if you look at the interactions on claw.ai, the task mix is much more around augmentation, but it's also likely to result in more success. So these aren't just two channels. They're two different stories about how AI integrates into the workplace and the future of work. Which one is more indicative of what the future is? That's a really great question. And I think of this in one of two ways. One, broadly speaking, the usage patterns on Cloud.ai, which is this chatbot interaction, do at a high level look very similar to the API deployment. so dominant usage for coding related tasks as as well as sort of the other overrepresented categories with a little bit more tilt toward programmatic deployment when businesses choose to embed cloud's capabilities so high level i tend to see these as both capturing where are capabilities strongest and where are they providing economic value whether that's through sort of iterative back and forth with a user through the chat window or through the API deployment.

3:18But to your point, so much of the, and this was a point that we made in our last report, so much of the labor market and productivity implications of this technology, much like past technologies, will hinge on how businesses choose to embed and deploy the tool. And so the sharp increase, relative increase in automated use where Claude is given a straightforward directive and expected to produce an output that feeds directly into a service that's provided to a consumer or some internal business operation is where I think the productivity effects will begin to materialize. This sort of matches sort of the historical pattern of general purpose technologies where businesses need to figure out how to maybe even embed the capability in invisible ways.

4:09So the analogy that I use is with electricity. When I go to a coffee shop and I order a latte, I don't often think, except in this conversation, don't often think about the power of the electricity that's required to provide this service. That general purpose technology is invisible to me. And it will take time for businesses to figure out what those use cases are, the Claude.ai usage patterns might be an early window into what's on the horizon. So early adopters use the chat bot to complete very sophisticated tasks through multi-turn interactions. Businesses learn that there's immense economic value there should they be able to provide the right contextual information.

4:54And then over time that gets embedded in business workflows? We're going to investigate all those ideas like multi-step and over time, I'm going to be a bit cheeky. So when I looked at this data after your previous economic report, the way I took this away was human employees like you and I, we think of ourselves and what our job entails in the round. And so when we get a superpower tool like claw.ai, we think about how we do that job better, make it more interesting, make it more challenging, get through not just our must-do list, but our coulds and woulds where the real value and enjoyment may lie.

5:36Whereas it turns out that companies actually see their employees as bundles of tasks where you can discretize them and you don't ask the question, what does Peter really need to kind of manifest himself fully as the head of economics at Anthropic, which is what you think of about your job in your head. They think, what are the 17 tasks Peter does and which are the ones can we automate discreetly? And I felt it actually said something a little bit more about that different perspective, the relationality that an individual has with their own work and the honesty with which their boss actually thinks about that individual.

6:17Yeah, I think that that's an interesting observation and i think i think it illustrates the the the fact that i mean so much one one implication of our this report was the sort of uneven uh impact that across different sorts of jobs and the fact that some jobs are likely to be fundamentally transformed and maybe even have greater risk of displacement so if you take into account task reliability like what is claude really good at doing. And you look at what are the tasks that are very time intensive for certain types of workers. You plug that into what data we actually see on our platform. You get a nuanced picture of sort of who is exposed to AI.

7:01One example here would be sort of data entry workers. These are tasks where in our analysis, Claude is pretty reliable at extracting information and standardized ways from sort of natural language reports. That's the most essential task in that job. And I mean, maybe to your point, that's where it might be more straightforward for businesses to figure out how to automate that specific workflow. In our data, in this report, we actually saw a jump in office and administrative usage as a total share of our API traffic, like suggesting that businesses are broadening out beyond just coding related tasks in how they choose to sort of deploy the tool into these like back office administrative support tasks.

7:51It's not obvious. I don't think that this means that your job will become sort of less meaningful or more meaningful, but it may mean that your job fundamentally changes. For data entry workers, maybe that type of role becomes more, and this is very speculative, so you can push back, but maybe someone needs to manage more the sort of relaying of information that Claude is providing. And it's not quite a data entry worker focused on plugging in the data points from a report into a spreadsheet, but there's nevertheless this important role that a human will play in translating and engaging with sort of people within the organization.

8:35Well, I mean, what we're doing there is we're going back and we're playing through what happened with previous waves of general purpose technologies and automation. And I think the observation you made that maybe what people do on the chat interfaces is a little bit of an early warning about, or early signal rather than warning about what you can do when you start to automate it. Because I think one of the observations would be that if all you do is automate a task like data entry that had fundamentally been gated by the marginal value of the nth item that was entered into data by a human who you're paying per hour, you had already taken a scarcity mindset to that and you were only going to look at the minimal amount of data that you needed to achieve the outcome.

9:20Now, if that cost drops by a factor of, say, a thousand, you're not just talking about looking at five more customer records, you're talking about looking at a thousand times more, which is a regime shift, right? Even if the fundamental kind of economic organizational model remains the same, a thousand X is a regime shift, which means new behaviors emerge. And some of the things that I see when I talk to businesses is, you know, a large part of them are doing the sorts of things we've discussed by automating data entry. But what they've done is they've automated the horse. They've automated the fact that they used to look at 10 ,000 pieces of data a day by humans, and now they look at 10 ,000 by machines rather than saying, well, what if we looked at 10 billion and how would that change our business downstream from that?

10:03I mean, you know, I feel that there's something there. Yeah, you know, I think this sort of reminds me of one of the more complicated insights from our last report, but I actually think is quite relevant here, which is the set of things that businesses will do over time to restructure sort of their modernizing data tech stack or new organizational workflows to unlock the productivity it's not just do what you had been doing before but sort of fundamentally restructure how the business operates the historical analog here would be again electricity where factories shifted from centralizing power within the on the factory floor to more distributed power that changed fundamentally how how factories operated.

10:52So what do we see in the data? We actually look at how much context do businesses, when they use Cloud through the API, how much contextual information do they provide, and how much output does the model produce. And actually, if you generate more output tokens, that tends to be the most complex tasks. It turns out that in order to get those most complex implementations, so moving beyond just straightforward data entry to something that's like much more sophisticated, maybe like automating biological research and analysis. Oh, wow. We've taken a big step. Yeah, that's a big step. And, you know, that is arguably like something that is maybe on the horizon.

11:34Like, I think a lot about even our productivity analysis might be conservative if we're failing to account for the automation of innovation itself. Maybe we can return to that. I'm kind of curious to hear your perspective. But it turns out for those most complex tasks that we see in our data, businesses need to provide disproportionately more contextual information. So even if the capabilities are there, if you don't have the relevant information to deploy that capability, business adoption will maybe constrain or bottleneck. You've got to run around and get all your tacit knowledge written down and stuck in markdown files.

12:08Exactly. The tacit knowledge piece is so crucial, I think, about developing a sales strategy. It might be that your coworker has the relevant information, not even in the sort of customer management software that you're using, but in their brain or in some document that they've written out. And if the business is not thinking about how to elicit that tacit information in a way that can make the model effective, you might fail to see tasks in our API deployment for which Claude and other frontier models would actually be capable if they have that information. I mean, it's a really fantastic point, and it shows how quickly the field is moving because no one was saying this a couple of years ago, right?

12:48And your report came out before Claude Opus 4.5. Yeah, the data, the underlying data that we sampled was just before Opus 4.5. And Opus 4.5 has really changed the world, at least for certain early adopters. It's particularly good at coding, for example, and it has absolutely cut into my sleep. You can see my URA scores just collapse over the last month. And I'm getting up at 4 a.m., I'm firing off a bunch of parallel agents, then I'm going for my walk, and then I'm coming back to see what they've done. this year we're on we're recording this on january the 15th i have written or rather i've commanded opus 4.5 to write about 10 000 lines of code per day so i'm up at about 150 000 lines of code according to some some measure that i did i'm just curious about how that kind of capability jump changes the way you you think people might behave based on what you had seen in in this very very extensive empirical study you conducted.

13:55This sort of touches on one of the findings, which is for the most complex tasks, which I presume you're asking Claude to take on very, Opus 4.5 to take on very ambitious exercises. In our data, those most complex tasks tend to have, that's where the model struggles. So there's like the success rate is somewhat lower. And that suggests that human expertise to evaluate the quality of the output is more important and you need more human delegation and direction and managerial oversight. I have found that even with Opus 4.5, there's still some level of oversight and quality attention that I need to maintain, but I actually worry a lot less about the implementation steps.

14:44And so I'll give a somewhat of a concrete example where analysis that I've been wanting to do for a while and I haven't had the time to do it and it sort of touches on this idea of how does AI exposure across jobs as commonly constructed how does that relate to business cycle sensitivity is it the case that workers who have high AI exposure are they the ones who might have higher unemployment when the labor market slows down. I've been giving Claude for some time now in a sort of separate server to run autonomously this exercise of downloading some research papers, implementing the sort of replicating some prior reports, and then coming up with this like new analysis that I wanted to do.

15:28Before Opus 4.5, I had low confidence that this would work. And then when I gave the same task to Opus 4.5, I discovered that it was able to move reasonably proficiently at implementing even somewhat ambiguous directives. But I still had a lot of back and forth. It was like having a very capable research assistant, much like the job that I had when I was fresh out of college working at the St. Louis Fed. That type of implementation capacity, the ability to understand how to run and set up regressions, how to download data through writing a new API, Opus 4.5 was able to do all of that and even write up a documentation methodological report.

16:17I suspect that we will see sort of the level of managerial oversight sort of move to a higher level as people begin to delegate maybe like lower, more straightforward, but nevertheless, sometimes ambiguous implementation steps. This is such an important point. So let me go back to before Opus 4.5. So what you looked at in the report, one observation was that when people used the API, the system could do a 3.5 hour task, long task. In other words, a task a human might take 3.5 hours to do with a 50 % success rate. But if they were using Claude.ai, that 50 % success threshold was a 17-hour task, which is two workdays in Europe and, of course, one workday in the Bay Area.

17:05I'm just quite curious about a few details there, right? So what is a 17-hour human task and how much time were the humans having to do it? Because it's really different. If I just, you know, bark a command at Claude, who will rid me of this turbulent priest? And it goes off and does it for 17 hours versus I have to sit over the machine and every 30 minutes give it feedback. Because in that case, I'm context switching and perhaps I can't really do other things. So is it a 17 hour task or am I just pressing shift tab, accept, accept on my keyboard for the better part of a day? And what is it about these multi-turn conversations that single shot can't achieve?

17:44So I'm just curious. So what is that task? How much is a human sitting over the machine pressing a button? It's a great question. And, you know, in the report, the 50 % at sort of 17 hours or around like 20 hours is an extrapolation of the linear fit that we document. So I don't think we actually see those very long tasks that you're describing. I mean, it's just sort of the implication of the data that we do see in our data. And so I think this is similar to the type of analyses that Meter has put out looking at task duration or task horizon and relating that to the reliability of frontier models.

18:21But it is conceptually different along two important dimensions. One, it's not just a sort of a laboratory controlled analysis. it's people are choosing to bring which tasks they bring to claude and they are bringing tasks where they believe claude will be most capable especially on claude.ai where there's this multi-turn oversight interaction where you give claude a task you see the output you give it some feedback and you have this oversight that can increase the reliability of what the model is able to complete in a short amount of time relative to how long it would take if you had to do it alone.

19:01I think compiling a literature review, a substantive literature review, is an example of a task that would take a research assistant. So I'm speaking a bit from my own experience working at the Fed when I was younger. That would take me multiple days to put together and compile nicely for the senior economist I was working for. Claude is able to do that sort of thing reasonably proficiently very quickly, and Opus 4.5 will be able to do that even better. It's quite a complicated picture because only a certain type of person with a certain experience can command a 17-hour or longer task. Only a certain experienced person can make sense of whether a literature review is good or not.

19:45It's why PhD students have supervisors. And so this is a tension in your findings I think is worth exploring. Models are improving, and people are trusting them with longer and harder tasks. But judging those outputs does require real expertise. But there is this hollowing out because many of the junior tasks that have built that expertise are at risk, or if not already, being automated away. And I'm curious about, you know, when you look at that and you look in the data and you extrapolate, are we creating a long-term institutional fragility problem that what is being a lot of what's being automated is what used to be the apprenticeship work as you discovered doing all those literature reviews you know a few years ago i think that this is a very insightful observation about how the nature of work might be changing and how the sort of capabilities unequally affect sort of early career workers versus later career workers and you know when i worked as a research assistant at the Fed to continue this example for my life.

20:54I learned a lot on the job from doing very basic tasks. Sometimes it was more involved like a literature review. I actually worked with an economic historian to transcribe historical records, exactly this data entry job, but in the context of economic research. And Claude and other frontier models are increasingly capable of doing that type of work. And the question is, will businesses sort of continue to invest in younger workers to equip them with those tacit skills that they had previously learned from these basic set of tasks? I think it's not entirely clear. That's partially why we're putting out this data so that we can track in real time what are the actual effects in the labor market.

21:47We have some suggestive evidence, of course, from this nice paper, Canaries in the Coal Mine, from researchers at Stanford that document that early career workers in high AI exposed roles have had worse employment trajectories in recent years. But, you know, we have alternative evidence as well. And so we're still in the early stages. A quick note. If you want to support us in bringing more of these conversations to the world, please consider subscribing to the show. Let's go back to the personal, because I'm going to go outside of the wheelhouse. But the beauty of doing this in my own conversation is that I can, and no one can stop me.

22:23So here I go. You think about London taxi drivers, which is something a bit like Henry Ford's factory that you and I have to talk about a lot. The London taxi drivers hippocampus grows because they spend a lot of time studying the maps and essentially becoming homing pigeons in London. When you read a book, the book changes you. And it's the act of doing that, that changes a person, that sort of judgment that a biologist in their 60s has or a shopkeeper in their 60s who's done it for a long time, to look at the stock and say, well, we're low on chewing gum is a function of judgment that is almost certainly reflected in connections in the brain and changes there.

23:07There is something about the skill that you developed, the skills I have developed which involved late nights, mundane, repetitive work that allow us to work at this higher level. So I'm thinking less about this from a skills acquisition point. I'm thinking about this from the perspective of an individual and how they establish certain classes of mastery. You know, my team will tell you that there are certain things that I just will not notice and I'm kind of rubbish at them. And there are certain things where I have such acute mastery that I can spot the crack from two miles away. And that has come not because of genes, but it's come from banging my head against the wall for 25 years.

23:56So when you look at that and you think about your own personal experience, how do you address this hollowing out? So I think I'll sort of respond to this through the lens of one of the primitives that we introduced. And it was sort of to try to get at this idea where we ask how many, you know, ask Claude effectively to estimate how many years of formal education would someone need to have to understand the prompt? And how many years of formal education would you need to understand Claude's response? It turns out that the most complex sort of high education tasks in our data typically coincide with very sophisticated prompts provided by the human.

24:41So I think in my example that I mentioned before where I'm asking Claude to do this regression analysis and somewhat sophisticated implementation, at present you need a PhD in economics to understand and formulate and have the taste to know how to prompt the model and much less evaluate the output that it produces. And so there is this question of like, what's the best way to acquire those skills? There is evidence on this front that it's not just acquiring the skill, but it's also developing the cognitive endurance. A shout out to one of my good friends at University of Chicago, Christina Brown, has a nice paper on this front in a developing world educational context.

25:28I think this is a really, really interesting point, question. And I just want to dig into, Anthropic is one of the leading firms here in this space. So you've obviously got this realization internally ahead of many other companies. So what is a sense of things that are working in practice when you look at your junior team members and you think, how do we get around the missing rungs in this ladder? How do I help, you know, junior Jenny and starter Steve get those skills that I got through sort of sweating it through in late nights? That's a really great question. And, you know, maybe I'll have a harder time sort of concretely pointing to something here at Anthropic.

26:14But I do believe that it is exactly as you're describing, that to have the taste and discernment of what is good writing, you need to spend a lot of time actually writing. And so I would encourage someone to not let Claude take the first pass at a draft. That's how you develop your voice. That's how you develop your argument. That's how you can discern whether the writing that Claude produces is actually of sufficient quality. And, you know, in previous roles, I've seen examples of where people have made missteps relying too heavily on sort of language models in developing their own voice. I think that there's a similar implication for reading papers.

27:02Like, you should just sit down and work your way through it. You know, setting aside the question of AI, for, like, very challenging technical papers, I would feel a lot of trepidation about just, oh, man, like, I'm never going to be able to understand how to do some macro dynamic DSGE modeling, for example. sitting down and reading the methodological paper and just like forcing yourself to work through and understand it turns out that maybe it's like not as scary as you first thought the great hope of this technology is that it can actually help to facilitate and sort of accelerate the acquisition of skills so once you've done that first pass of something that's very challenging go back and interact with claude to help clarify things that don't make much sense so actually one of the primitives that we introduce in the report is are people using it for professional, personal, or for coursework?

27:58And we actually see that coursework is one of the more common, I mean, it is relatively common around a quarter of all usage globally and more so in lower income countries. People are using CLAW to sort of complement their education for educational purposes and skill and expertise acquisition. But you know, it does require some new novel thinking. And in the corporate setting, in a world where you, as a business, you can get something done quickly, to stop and say to your employees, no, you've got to spend 10 hours reading slowly requires a certain type of corporate culture. I'll share with you a couple of things that we've done.

28:40So this is my fountain pen. I've not written as much by hand for 25 years as I have in the last year. We've got the team fountain pens. I do hope they're all using them because it slows you down with your writing. They're really voracious readers. And in fact, just yesterday, one of my researchers caught me out on the very famous Paul Romer economic growth paper, which I have not read for 10 years. And he was sitting there reading it on a printout And I was thinking, gosh, I'm amazed he's even printed it out. And of course, in the detail that you forget when you just read the summary is a detail that did catch me out.

29:20So there are some behaviors that we have found to be perhaps, you know, a little bit more effective. Slow yourself down, make sure you're reading, but you need to have a certain culture. And I think that a lot of businesses, a lot of American businesses, really, you know, they're thinking about short-term productivity. They are thinking about volume being more valuable than that insight and that depth. Would you be able to see in your data whether anyone was designing around this or are we just going to see this quarter by quarter optimization? It's an interesting question and sort of has sort of sparking interest in a number of threads.

30:01I'll say briefly, and then I want to return to this Paul Romer point about endogenous growth theory and related growth theory, because I think it's a really important point for thinking about what's on the horizon with this technology. So one thing that we do see in our data when you compare the API versus the Claude.ai interactions is that businesses give less autonomy to Claude to complete the task, like less decision making ability, which suggests that, I mean, it's kind of related to this idea that like you don't want to fully automate the process. You want to make sure that there are appropriate guardrails in the context of reading a paper.