Why Big Tech wants more economists

0:00First, AI companies came for the engineers. Now, they're coming for the economists. Beyond hiring teams of economists, the big AI labs are increasingly trying to fund and collaborate with researchers. OpenAI has an economic research team and has launched an economic research exchange to collaborate with outside academics. Anthropic is doing something similar. All great news for economists keen to dig into the exclusive data that AI labs hold, but it does raise a question. Should we worry that to find out what AI is actually doing to our economy and society, we have to ask an AI company?

0:48This is The Economics Show with Samaya Keynes. And this week, I want to explore the trend of economists working with or for AI companies, including the potential for conflicts of interest. AI could dramatically reshape our economies and societies. It is incredibly important to ask and answer the right questions about this transition. First, let's hear from Ara Karazian. He is the lead economist at Ramp, a New York-based financial software company that tracks corporate spending. He's got this substack where he shares insights based on the data that his company collects. For example, the Ramp AI Index shows which AI models US businesses are buying.

1:32In August, Anthropics lead over OpenAI increased. And because we always kick off the show by asking a question for which the answer is on a scale of one to 10, I asked Ara where he sees demand for economists among AI companies right now. I'm going to answer in a very economics way where there was really very little demand from AI companies. It's out of 10. It's the highest it's been. I myself have never noticed higher demand for this sort of public facing role side, at least of it, for economists at AI startups and companies. We're definitely in a hot moment right now for the job market. Okay, well, why is demand for economists so hot right now?

2:11I mean, you can imagine that AI is a very new and nation technology. Everyone wants to know how it's going to affect the economy. It's not showing up in productivity statistics. So you want to get as close to the data and the users as possible to understand first who's using it at all, and then how are those firms using it. And so you can imagine there's demand for that kind of data, both from governments and from investors and the financial markets, but also from the general public, too. But there is also a history here. AI companies are not the first to start their own economics research function.

2:43I think it started with the big banks around the time of the financial crisis. You didn't see tech start to do it until a similar rupture in public trust, Cambridge Analytica, when tech companies then became the sort of ire of the general public. And then now you have this sort of new era where AI companies similarly have this sort of public trust issue, trying to argue both in favor of their technology in front of the government and also in front of consumers, where there is very little data otherwise available to understand how this technology is affecting our day to day. I mean, what you're describing sounds like economists are kind of involved in an elaborate PR exercise, right?

3:22Partly it's kind of shaping the narrative. They're the ones that have access to the data. The companies have an interest in rebuilding that trust with whoever. And so, yeah, economists are just part of that. Can we talk about Ramp specifically, which is where you are the chief economist? Why does Ramp have a chief economist? Well, you can think about the questions that we got asked by our own customers. So Ramp is an AI finance platform. So you can think about the data set that I work with by thinking about our core products. So let's just say corporate cards and bill pay and invoicing. So essentially the whole universe of business spend, everything except for payroll is what I get to see.

4:01And as a researcher, that's such a privilege because it's both a very broad set of businesses. It's a very broad set of the investments that businesses are making. It's also highly granular because you can see the line items on bills and receipts. And so we have found that we can start publishing aggregate and anonymized statistics from our data set about how businesses are investing in a new and nation technology, something like AI or something else, and share that data, ideally, with the purpose of helping other businesses, whether they use Ramp or they don't use Ramp, also make better decisions and become more productive.

4:35But it will be helpful if they use Ramp, right? I mean, part of the idea is to, you know, get ramp into the public eye. Well, look, I think it's an extremely competitive market. But if you're a business and you are trusting your money and finances with another software company, and you're picking between them, ideally, you want to think that that company is thoughtful about trends in the market and is smart about how your business works and how other businesses work. And so in many ways, you know, I think you could distill the job of an economist at a company. Certainly the way I see my job is how do I make Ramp look smart?

5:11I'd say that is an exercise in branding as much as it is an exercise in PR. Speaking of smart, let's talk about some of the efforts you have made to make Ramp look smart. So I remember seeing the research that you put out looking at growth in AI companies and jobs. Everyone wants to write the paper about how AI is going to affect the job market. Very few companies, if any, have the perfect data set to answer that question. Ideally, you want a data set that shows, hey, these firms adopted AI, these firms didn't. How did jobs change over time? So we put out a paper at the end of June where we use ramp data on business adoption, combine that with workforce data from Reveglio Labs, and we tracked exactly that.

5:54And we found that companies that use AI intensely see a 10 % increase in headcount over the two years following adoption. But notably, there is no statistically significant change amongst the companies that use AI with low intensity. But all to say is that there's so little data available to assess how AI is actually going to affect jobs in the economy, or how businesses that are using AI are actually hiring differently, whether they're hiring more or less. The AI companies can't produce that kind of research. You know, they don't necessarily have that appropriate data set that shows, hey, these firms used AI.

6:31They don't know the firms that didn't use it or used a different model. In any case, it's hard to trust the information that's going to come out from an economics team at an AI company that might be directly interested in the result that says, hey, AI is great. It's going to create jobs. From our perspective, you know, I don't have an investment in whether the result is good or bad. And we don't have a financial interest one way or the other. I mean, the tempting conclusion to draw from that is that, you know, AI is supporting that employment growth. But it could just be that, you know, the kinds of companies, maybe, you know, small startups that would be particularly fast growing are also the most likely to be using AI.

7:08So it's not that the AI is causing the job changes, you're just picking up something else. Well, I think that's a good point. We did control for that. And I still think a little bit of both is happening because it's absolutely true that the companies that are going to use AI in the first place are early adopters of this new technology. They were already growing faster pre-adoption. So the main specification we used in our paper was to compare firms that adopted AI to firms that eventually adopt AI but had not yet in the comparable period. And when we made that control, we found that the firms in the AI versus non-AI group were growing at the same rate pre-adoption.

7:46So still growing faster than firms that never used AI at all. But upon AI adoption, the firms that use it accelerate in their adoption too. I mean, that's one of the challenges of understanding AI is that we're not entirely sure yet the mechanisms by which AI leads to increased productivity. Do you have any other examples of trends that you're able to see because you have access to this data? I think what's very interesting about this market and also this data set is thinking about how these alternative data sets can support new ways of thinking about how markets develop and how companies themselves are evolving.

8:20I think it's obvious, for example, that consumer brands would use influencer marketing that they'd want to advertise on Instagram with an influencer. I think it's less obvious that the second most likely sector to use influencer marketing is AI and software companies. So 28 % of influencer marketing spend is coming from AI companies and startups. You wouldn't necessarily expect that to happen in an industry that's largely focused on selling to businesses for most of their revenues are coming from businesses. And yet, AI is also sensibly this software that most people, buyers, and businesses are hearing about for the first time as consumers.

8:59And so seeing that kind of content on Instagram or TikTok, where your favorite influencer is talking about open-air Anthropic, it's unreasonable that you'd expect AI companies to use these methods that are most commonly associated with direct-to-consumer consumer brands. I mean, it feels like there might be a parallel there between the influencers being hired by these AI companies and the economists being hired by these AI companies to influence public perceptions. How do you think about potential conflicts of interest in this space? There's an appropriate risk calculation here in that the AI companies and private sector companies in general have access to data that the public just does not have access to, that the government will never be able to effectively collect.

9:43And those data sets are valuable for producing research and understanding these new technologies as they affect us day to day. At the same time, we should be skeptical of the incentives and the framing of that work and what's being put out and simultaneously what's not being published because it may tell a specific narrative that is unhelpful for the researcher. But look, my perspective on this is that we also live in a marketplace for information. and that it is really hard to build credibility for a data set over time. And I wouldn't underrate that. Even for our research at RAMP, it's taken some time before the data set has built the credibility that it's gotten.

10:26And I think we see the same thing when OpenIr Anthropic or another AI startup puts out its own set of economic research, is that there is a society of economists that is willing to poke holes in it and to argue against it and to share their opinions about whether or not it is valid or should be trusted for policymaking. All right, thank you so much for joining me. Thank you, Sumeya. Let us take a break. And when I come back, we are going to hear from Peter McRory, chief economist at Anthropik. What does he have to say about the ways that Anthropik has been coping with this perceived conflict of interest?

11:03Plus the FT's chief data cruncher, Jomber Murdoch, on how he sees the research coming out of the AI labs.

11:24We'll see you next time.

11:56We are back from the break. And I am delighted to be speaking to my colleague, John Byrne Murdoch. John, hello. Hello. Okay, so it feels like we are in the middle of this moment where risks of AI are really at the top of everyone's mind. So we're all scared of AI swarms hacking the internet. There is a public backlash against data centers. And as anxiety about AI increases, economists have this really important role in informing the narrative about the effects of this technology. This is also the year I feel where working in an AI lab became cool. And so, So, you know, I started thinking about this topic when I was hearing about, you know, senior academics trying to get positions in the AI labs.

12:46Now, obviously, a lot is happening in academia, but it feels to me like a lot of the analysis is kind of moving to Substack or just, you know, being published on the AI company's websites. How have you experienced this shift? Have you seen a change in a sort of way we're turning to for analysis? this? Yeah, there's definitely been a real sort of diversification of where this stuff is coming from now. So it's interesting that you mentioned Substack, because even some of the economists who have gone to work at the AI labs, I'm thinking, for example, of Alex Emas, who's gone to Google DeepMind. A lot of that work is being put out on Substack, or at least his own musings on this stuff.

13:28So yeah, I've definitely had to sort of cast a wider net now for where I'm seeing interesting work being done but interesting even looking at what the ai labs themselves are putting out i find it's hugely variable in terms of what they're even trying to do like some of their work is sort of classic economic analysis some of it it feels a bit more hand wavy and sort of somewhere between that and commentary that's been one of the challenges as well is almost to your point here is trying to work out which of this work should I be viewing as classic economic research just coming out of an AI lab and which things should I be looking at as musings and commentary and showcasing the data that they have.

14:14Do you have any examples of research that you think is really high quality that you've seen? There was actually a piece of work by the team at Google DeepMind the other day, which I thought was really nice because what they did was they combined their data on how Gemini, their AI product is being used with a lot of qualitative stuff. And they interviewed and surveyed people who use their tools as well. So they did a whole piece of work focusing on how scientists are using AI. And what you often see in stuff that comes out of the AI labs is they focus for me too heavily on just what their own data shows.

14:50So you get loads of stuff coming out just saying, here's what people are using our tools for. But what it often misses is what is that actually doing to the jobs and the working days of those people? And so what I liked about this one from Google was that they had the survey data to marry the two up so they could see, right, our data shows scientists are using AI a lot and they're doing these things with it. Now let's talk to them about how they're feeling about that. How is it changing their working day? Like, where is it boosting productivity? Where is it actually hampering what they're able to do?

15:22And what did that qualitative research show? What they found here was that scientists are some of the heaviest AI users in any industry, any occupation. And they are generally using the models to speed up a lot of the research they do. They're then using specialized models, so not the models out of the frontier labs, to do very technical domain specific stuff. And this is meaning the sort of research process itself is getting faster and faster. But it is leading them to realize there are then massive bottlenecks that slow down the speed at which this can translate into the real world. So the amount of data checking, results checking that they're having to do has gone massively up.

16:06The number of promising hypotheses they're generating but haven't yet had time or headcount to get onto, again, is ballooning. For me, that was really interesting because it shows why pure usage and pure productivity gains in certain tasks don't necessarily translate, even within a domain like science, into more value, more stuff being created. I guess we should probably put this into historical context, though. This is one thing Ara mentioned before the break, but we have seen waves of hiring of economists in the past, right? So, you know, the 2000s and the 2010s, I think Amazon hired something like 40 % of economists going into tech.

16:50You have this huge hiring sprees. And I suppose there's a question of, you know, how is this really different, right? And, you know, one difference I think is that maybe the first wave involved economists doing lots of work that was internally facing, that was actually directly trying to improve the products that some of these tech companies were offering. For example, they might change the way an auction was designed, increasing revenue to the platform. And right now, economists at Google are still doing a lot of inward facing work, as well as the external work that you described. But there does seem to be more research trying to answer some of these really big questions thrown up by the AI shift that aren't sort of directly relevant to the kind of commercial interests of these companies.

17:39This is something that I picked up on as well when I spoke to the chief economist at Anthropic, Peter McGrory. Before I joined Anthropic, I had this growing sense that this is perhaps the most consequential technology at the time, I would say, of the last 30 years, if not the past 100 years. and because of that I wanted to do what I could by joining Anthropic a company I believed cared about providing information and credible research about the most important questions to help build out their economic research program and I think that that urgency that I felt then has only accelerated since joining Anthropic I've dropped the 30 years from is this the most consequential technology and helping society make sense of this fact and what we should be doing to exercise agency over the future is what really motivates me in my work today.

18:39So John, do you also have the sense that, you know, the kinds of questions that they're answering are a bit big or a bit different from the questions of the last tech boom? For me, something that is different, not just with the economists here, but the whole technology, is that it's really come out of an intellectual space in a way that was, I think, quite different to some of the previous tech booms. So I do see a lot of work here, which looks a lot like people who are just very, very interested in finding out what is going on in the world through the lens of this technology. So I don't see clear patterns in terms of the results always showing the technology in a positive light or a negative light.

19:20It's more that there is a concerted effort within these companies, from what I can gather, to use all of the data they have to find out how people are using this. How are people in different jobs using it? To what extent is it augmenting things versus automating things? How is that usage translating into the real world? It does to me just look like sort of straight research trying to find out what's going on. Okay, you just mentioned the data. I suppose when reporting on this, I did hear concerns that, you know, it maybe wasn't the healthiest dynamic where the best data, the most granular data on AI usage was in the hands of a very few companies that had fundamentally commercial interests.

20:02And then, you know, they weren't as kind of insulated from commercial pressures as, say, someone in academia. I suppose one question is, do you agree with all of that premise? So, you know, how much of an advantage do you think that the AI labs have in holding this data? Yeah, I think there's unquestionably an advantage there. And the way I think about this is that even when the labs are putting out data on usage, for example, it's not completely raw, unfiltered data, right? They're deciding how to sort of score different types of usage. They are breaking things down into a level of granularity that they are comfortable with, and perhaps not more in some cases.

20:46Certainly when we move away from straight, sort of quite dry economics questions, but more into things like, are people having quite disturbing conversations with chatbots? That's the kind of thing where you do not see those breakdowns in the data that's being released. I think certainly the labs are putting out data that they are happy with, But I do still think that in the economics domain, what I tend to see is that these are people interested in how AI is impacting productivity, employment, wages, and they are trying to get as much data out there as possible to help people answer those questions.

21:22Again, though, with the caveat that they are the only ones who have the entire universe of that data. So everyone else is kind of working with their sanitized and slightly aggregated release. When I was speaking to people, I actually did hear some concern that there might be a kind of, you know, almost a restriction of the kinds of questions that could be asked in the economics domain. Perhaps questions like how is AI affecting the intensity of work or how people feel about their work? Those are questions that are harder to answer. You know, there might be legitimate reasons why they haven't been such a focus of study.

21:59But, you know, there was a concern that sort of subtle biases could creep in that way. Maybe only researchers that are super, super positive or super enthusiastic about the effects of AI or think that AI is going to be a really big deal. Maybe they get preferential access over others who are more skeptical. Yeah, that for me is the biggest thing here. The way I've been thinking about it is like that idiom, if all you have is a hammer, every problem looks like a nail. For me, with the AI labs, it's like if you have a toolbox that is full of very shiny diamond encrusted hammers, then all of your economic analysis is going to use nails.

22:38So everything pretty much that comes out of these labs is based on their data on what tasks AI is being used for. But that is such a small part of this whole thing. It's a small part, both in the sense of what you're talking about there, where the massively increased intensity of work is a really important thing that that usage data is maybe not telling them, or at least, as you say, they're not releasing that data. The extent to which you see evidence of people getting frustrated with AI making mistakes in work use. I don't think we've seen a huge amount of data on that. there are all these messy things even even just more basic things like if ai can do like i don't know three or four of the tasks that constitute your job is that making your job at risk or not and that is the stuff where for me maybe as you say some of this is about the labs not being particularly interested in going down those thornier routes but i think it's also just a weakness from only having the data they have.

23:42And that's where I think the best research is that which marries that data with the interviews, with the surveys, that goes out and says, here's what our data is showing us, but what does this look like in the real world? Speaking to people who were involved in the kind of external research collaborations, I did hear that it would be naive to think that the labs would approve anything. You know, I think If you came to an AI lab with the headline of your project is, I want to investigate all the evil that AI is causing, I'm not sure you'd be as likely to get approved as other slightly more friendly questions.

24:18Although, of course, the labs would say, well, if there's a credible research design, then of course we'd be open to it. But also, I think this is a concern for the AI labs too. They really want their research to be perceived as credible. I heard that one of the worst insults you could hurl at someone within these AI labs is to call them, you know, an AI cheerleader. So as well as speaking to Peter, I spoke to Ronnie Chatterjee, who's the chief economist at OpenAI. He was mentioning as kind of guards against the potential for conflict of interest. You know, there would be peer review on internally produced papers, making data available to external researchers, and then just relying on external voices, I suppose, maybe some sub-stacks, social media, but also just kind of the wider research ecosystem system acting as a check if internal researchers were ignoring particular questions.

25:11I also put this to Peter McCrory of Anthropic, who talked about making data public, also trying to fund independent external research. And then I suppose also, you know, just thinking about the narratives coming out of these AI labs, at least in the past, Anthropic has been, you know, pretty open about the risks associated with AI and also sharing findings that aren't entirely positive. For example, where AI could be reinforcing inequality. A key finding that I think is just like really important comes from our September Economic Index report where we introduced for the first time an analysis of how cloud is being used in like more than 100 countries around the world.

25:56And what you see there is that adoption rates are much higher in high-income economies. This is, in some sense, not surprising. Using AI relies on having access to digital infrastructure. You need access to the internet. The occupational makeup of the economy for high-income economies is more similar to the type of work that Claude is good at. But it does raise the question of, will the productivity benefits accrue to already rich countries? will the acceleration of global economic growth coincide with a divergence in living standards as was the case in the transition to modern economic growth in the late 19th century?

26:36I think that's like a really important question. We should have a global view on economic implications of AI. The other thing that Peter said was that by being transparent early on, by setting out its methodology, the researchers at Anthropic could help build their credibility later on. I think the next big report that I'm really proud of is this labor impact report from March, where we tried to ask the question, where is CLAWD being used in automated ways for work purposes? And through that lens, which jobs might, as a result, be most exposed to labor displacement risk? and when we map that analysis to the unemployment data in the U.S., there's no impact so far, but what's nice is it ties our hands to a methodological approach that if there is concentrated disruption, we think it should show up in this monitoring framework and we've pre-committed to reporting those results in some sense.

27:40So John, any reactions? Yeah, so I think the transparency and the methodology and the papers that are being published alongside these nice glossy web pages is a really nice thing to see. And I think we actually maybe do see more of that here than in previous economics tech booms. But for me, that's not necessarily the end of the story. Like, it would be more valuable, I think, for external people to have more eyes on the data that goes into them than just the methodology. Like, sometimes you read the methodology for some of these pieces of research coming out of the labs, and you think, that is all great and technically sound and lovely.

28:17But the assumptions that you've made, the mechanisms that you've omitted, the data itself, they would all be really interesting things for more people to have eyes on as well. Okay, data journalist wants more data. Shocker. So overall, John, how positive are you feeling about this research agenda? How healthy is it? I think this depends whether I've got my pure economics hat on or my broader sort of societal hat on. I think in terms of economics, the research does look reasonably healthy. The fact that the firms are being out calls for academics to come and work with them and apply to use the data, I think, again, that's not the end of the story, but it's promising.

28:58There does seem to be intent to just find out what's going on in the world. With my economics hat on, the problem I would say is that we're not seeing nearly enough work that goes out and speaks to humans and looks at what's happening to workplaces and workers beyond simply what's in the chat logs. With my societal hat on, I think I would be a little bit more concerned that the really troubling stuff that the companies might be sitting on is really coming out in a very drip-drip fashion. Obviously, the things we're seeing around rogue agent swarms. and there may well be a lot that falls somewhere in between rogue agent swarms and task-based economic usage that we're not finding out so in our weekly newsletter the ai shift which looks at how ai is impacting the labor market we a couple weeks ago sat down with jack clark who's the co-founder of anthropic and now heads up their economic policy think tank operation and we were asking him about a new piece of economic work they just put out there called Anthropic Economic Futures, where they were looking at what might happen if you have dramatic impact of AI on the economy over the next few years.

30:14And what was very clear throughout this entire conversation was that they really saw this as narrative. It was about putting out some numbers that would start a conversation, get people thinking about things. What if we have explosive economic growth? What if we have wage growth or not? So what I came away from that thinking was that sometimes the economic research that is coming out of these labs is not really done with the same aims as economic research that would go into a journal. It's being done as part of a broader strategy to shape the conversation. John, I think we should leave it there.

30:50Thank you so much for joining me. Thanks so much for having me.

31:02That is all for this week. You have been listening to The Economics Show with Samaya Keynes. If you enjoyed the show, then I would be so grateful if you could rate and review us wherever you listen. This episode was produced by Misha Frankel-Duval and Claire Williamson with original music and sound design from Breen Turner. Our executive producer is Manuela Saragossa. I'm Samaya Keynes. Thanks for listening.

31:40We'll see you next time.

31:56Mortgage eligible people. Banks win and francs win. Lenders win and so do lenders. When you build credit scores the right way, everybody wins. Get started at VantageScore.com. The new episode of the Next 5 podcast is all about sovereignty, data and control. Charlie Giancarlo at Everpure joins me. You always get a very embarrassed response if you ask them if they know where all their data is. As does Professor Sana Karaghani. There isn't any modern AI system that's entirely sovereign. And Claire Schramm-Fergus at Lloyds Banking Group. We really need to design for safety rather than bolting it on at the end.

32:34Listen to The Next Five wherever you get your podcasts. Enjoy.