

Joseph Francis revisits the “China shock” literature—especially Autor, Dorn, and Hanson (ADH)—on whether Chinese manufacturing imports caused U.S. manufacturing job losses and broader living-standard harms. He argues that widely cited econometric results overstate the China-attributable employment decline and that the living-standards effects largely disappear after methodological corrections.
Joseph Francis is an economic historian and honorary research fellow in Economics at the University of Birmingham. He focuses on post-publication reviews/replications of influential economics papers and writes for Econ Journal Watch.
(1) ADH’s causal design uses instrumental variables from Chinese import changes in other OECD countries to proxy China’s supply shock. (2) Borussia, Hull, and Haravell (2022) show ADH insufficiently separates long-run manufacturing decline from China-specific effects; the estimated employment impact falls by about half. (3) Francis finds that once these corrections are applied, ADH’s reported negative wage/unemployment/living-standard effects are eliminated (not emphasized in the main paper).
2000–2010 U.S. manufacturing employment fell by ~6 million (~one-third); ADH’s “China shock” narrative became politically influential, including in support for trade-restricting policy. Francis also references Ed Leamer’s critique (“Taking the Con Out of Econometrics”).
Written by AI. May contain mistakes.
Discussion on the impact of Chinese manufacturing on U.S. employment and standard of living.
“Our conversation is going to discuss his recent article in the September 2026 Econ Journal Watch.”
Exploration of how economists measure the impact of Chinese imports on U.S. manufacturing.
“And what did – let's start with what Otter, Dorn, and Hansen, sometimes called ADH – but I'll try to say their names because listeners will get perhaps confused.”
Analysis of the influential paper by Otter, Dorn, and Hanson and its findings on manufacturing employment.
“the great productivity growth in China from demand factors in the United States.”
Joseph Francis discusses criticisms of the China Shock narrative and the need for a nuanced understanding.
“This article by Otto et al, it's had a lot of discussion, debate, and everyone's about the effect on the manufacturing share of employment.”
Review of the adjustments made to employment impact estimates and their implications.
“between 1990 and 2007, which is a different time period than we've talked about so far.”
Explore the impact of Chinese imports on U.S. living standards and job markets.
“And that meant that people looked at the Borussia et al.”
Discuss the pressures in academic publishing and how they affect research findings.
“Why do you think they kept that quiet in the online app?”
Examine how political narratives have shifted regarding the China shock over the years.
“They incorporated some of Borossiuk et al.'s corrections.”
Analyze the long-term trends in U.S. manufacturing and the role of technology.
“And it's a way of avoiding kind of other questions, which it's much easier to blame foreigners.”
Explore the historical context and effects of globalization on U.S. manufacturing.
“And I mean, my prior as an economic historian is that countries tend to industrialize and then they tend to de-industrialize because as they become wealthier, people don't want to work in manufacturing anymore.”
Examine the complexities of protectionism and industrial policy through historical examples.
“You and I might disagree about that, which is why you really need to have a debate about it rather than just saying it's China.”
Discuss the shift in manufacturing jobs and its impact on public sentiment and education levels.
“and that's also why auto-atil is so important because they did seem to have found this big causal effect of China which just might not have been there.”
Understand the dual nature of globalization's impact on different economic groups.
“But to blame that on China is a special case that economists have accepted, and it's very much a change in their perspective on trade generally.”
Learn about challenges in econometric research and the importance of robust findings.
“And I'm really just the messenger who's bringing what they found to more kind of wider notice, maybe.”
Delve into the fragility of economic research and the importance of questioning assumptions.
“So I asked him, how many regressions did you run?”
Explore the necessity of storytelling and pattern recognition in economic research.
“and he said that economics, it was fragile because of this process that you said of where people run regressions, they try different things until they kind of get a result that they think can be published.”
Exploration of the shortcomings in economic assumptions and methodologies.
“of these kind of assumptions baked into a lot of economics research.”
Discussion on the importance of storytelling and comprehensive analysis in economic history.
“Because economic historians over the last 50 years have turned to statistics with great excitement because it's prestigious and it looks scientific.”
Insight into the difficulties of conducting robust economic research and the implications of publication incentives.
“and it's kind of sophisticated beyond this kind of X-Causes-Y approach that tends to dominate.”
Examination of institutional issues in economics and the reliability of published knowledge.
“So that's kind of my end goal of this kind of paper.”
Discussion on the differences and similarities in data handling and reliability between economics and scientific disciplines.
“Economics is a very curious one because economists obviously pride themselves on being very robust.”
Learn about the varying peer review cultures across scientific disciplines and their implications.
“And of course, there's all kinds of problems we have with our data and the fact that human beings have their own sense of agency and life's much more difficult for us.”
Explore the author's interactions with peers and their responses to published work.
“like a pack of kind of rabid dogs and tear it apart on social media in just a way that, I think within about a week or two, nature have put out an expression of concern and they were just savage.”
Discuss the shortcomings of peer review in economics compared to other scientific fields.
“So as far as I'm aware, I haven't made any econometric errors in the article.”
Reflect on the robustness of economic research and the contributions of the guest.
“And you just don't see that culture at all in economics.”

Get the gist, explore an idea or ask about something said.
On EconTalk, what are the key points of this episode?
Automatic transcript. May contain errors.
0:02Welcome to EconTalk, conversations for the curious, part of the Library of Economics and Liberty. I'm your host, Russ Roberts of Shalem College in Jerusalem and Stanford University's Hoover Institution. Go to EconTalk.org where you can subscribe, comment on this episode and find links and other information related to today's conversation. You'll also find our archives with every episode we've done going back to 2006. Our email address is mail at econtalk.org. We'd love to hear from you.
0:36Today is October 13th, 2026, and my guest is economic historian Joseph Francis. He is an honorary research fellow in the Department of Economics at the University of Birmingham. Our conversation is going to discuss his recent article in the September 2026 Econ Journal Watch. The title of that piece was Whimsy and Fragility in the China Shock Narrative. In short, we're going to be talking about a very, very famous paper and an influential paper on the impact of Chinese manufacturing on employment, and manufacturing standard of living in the United States and some of its implications. Joe, welcome to EconTalk.
1:22Hello, welcome. And I also want to add that I interviewed David Autor on this program back in 2016 about his research, and we'll link to that. So that body of work, and there were a number of papers on it, but they were all looking, a lot of them were looking at the impact of Chinese imports in the United States. was correlated, not necessarily causally, but correlated with a terrible time for manufacturing employment in the United States. So between 2000 and 2010, the United States lost about 6 million jobs in manufacturing, which is about a decline of about a third, which is quite dramatic in such a short period of time.
2:11And it coincided, perhaps causally, when imports from China in nominal terms went from, in that 10 years,$100 billion, which is already at a high level. $100 billion seems like a lot of money, although it's a percentage of GDP. It's not as big as you might think. It's about 1%. But it went to$365 billion, which is about 2.5%. So it's a dramatic increase. And many, many people argued that that was due – the loss of manufacturing employment was due to Chinese imports. And what did – let's start with what Otter, Dorn, and Hansen, sometimes called ADH – but I'll try to say their names because listeners will get perhaps confused.
2:54As Otter, Dorn, and Hanson tried to tease out, measure, whether this was causal, whether the increase in Chinese imports to the United States, their exports, our United States imports, whether that had destroyed manufacturing in many, many places or dealt it a grievous blow. What were they trying to – how were they trying to measure that? so they were trying to um um they were using imports from china but they didn't use imports from china into the united states because they were endogenous so imports um from china into the united states uh responded to u.s demand whereas they were interested in the supply shock the supply shock from china so they were their framing is that they want to look at the this one-off kind of world historical transition from communism to capitalism in China and to see what those effects were in the United States.
3:57But they can't do that just by looking at the imports into the United States because that also responds to demand in the United States. So instead, they used an instrumental variable based on Chinese imports into, I think, seven or eight other OECD countries in Europe and Japan. And they use those imports as an exogenous measure of the supply shock in China. We should just clarify one thing, which is, at the same time, China's trade status with the United States had changed, right? So China is transitioning from communism to capitalism, sort of. That's a crude statement. But suddenly China was able to produce a great deal more than they had before.
4:52And they were able to sell the United States at better terms. So the United States definitely bought a lot more stuff, whether that was because of Chinese productivity, that transition, or whether it's because, as you point out, that Americans wanted to buy more manufacturing goods. is the question. But it's also, there are many, many, many, many other things happening at the same time. And the challenge of econometrics or statistics is to try to isolate the impact of China from those other things. For example, the world was getting more productive. There's technological change. There's innovation in producing and manufacturing generally with being able to produce things with fewer people.
5:33So we want to blame, I put that in quotes because I don't really believe that's the right word. But if we want to, I'll make a more neutral term, if we want to attribute the collapse in manufacturing in the early 2000s to China, you have to be able to hold those other things constant using statistical techniques. Yeah, they were trying to get the causal effect of China's transition from communism to capitalism. So Otter et al. were trying to look at the effect of the Chinese miracle, if you like, China's transition, by separating out the effects of the increased supply and the great productivity growth in China from demand factors in the United States.
6:24so whatever might be causing an increased demand for imports in the United States and with this statistical technique they they claim to have found that imports from China had this big causal effect that it caused a decrease in the manufacturing share of employment and it also had a lot of negative effects on living standards in the United States Things like unemployment, wages, household income, welfare transfers from the state. That was in their 2013 paper. Then they published other papers afterwards, which said that there was negative impacts of Chinese imports on things like mortality and the marriage market and politics.
7:18They said it drove a lot of the vote for Republicans and had an impact on the 2016 election.
7:28And what I found, and I'm really the messenger at this point, I'm not actually an econometrician. I am an economic historian and I got into a habit of doing what you call replications or post-publication reviews of published papers. And I did various reasonably well-known articles. And then I said, well, what's the most famous, most influential economics article published in the past 25 years? And that's why I found this Otter et al., The China Syndrome. and I spent two or three months looking at it, trying to figure it out, trying to understand it with my quite limited understanding of econometrics, using AI and large language models to help me to explain it to you repeatedly as if I was a five-year-old, which they very patiently did.
8:25And I focused exclusively on the manufacturing share of employment, because that's what everyone else is focused on. This article by Otto et al, it's had a lot of discussion, debate, and everyone's about the effect on the manufacturing share of employment. And I did all the kind of robustness tests I could to see whether it was robust. And I ended up with the same conclusion as these three reasonably young econometricians, who obviously know what they're talking about, called Kero Borosiak, Peter Hull and Xavier Haravell. And they published an article in 2022 in the Review of Economic Studies in which they found some problems in Otter et al.'s study.
9:16They found that Otter et al. hadn't distinguished very well between the general decline of manufacturing towns in the United States and the specific decline caused by China. Meaning that if your town produced something that China didn't produce and it also declined, you wouldn't want to mistakenly attribute that to the China shock, but their technique was not able to separate those very effectively, right? Is that a good summary? Yeah, yeah. So Borussia et al., They found that a lot of the effect on the manufacturing share of employment that Otter et al. had found was basically a comparison between manufacturing towns and towns without any manufacturing or with very little manufacturing, with just service towns.
10:11And that meant that they couldn't really distinguish between what China had caused from 1990 onwards and what had been a very long term from the 1970s onwards decline of manufacturing in the United States. so to fix that borussia ketal introduced a new controls into the regression which meant that they were comparing manufacturing it's not it's like commuting zones in the united states so these these kind of areas which are all linked around a town um they were comparing commuting zones with similar levels of manufacturing so what really mattered was how exposed they were to Chinese imports within the same level of manufacturing.
11:02And what they found was that the effect on the manufacturing share of employment, once they introduced these new controls, fell by about half. It fell from about 0.6 to 0.3. I can't quite remember what the units of the coefficients are in. So it fell by about half. Yeah, and the original – when I interviewed David Otter in 2016, it was – at that point, he and his co-authors were saying that about a quarter, 25 percent of the drop between 1990 and 2007, which is a different time period than we've talked about so far. But it's before the 2008 recession, so that's an argument for leaving out the later couple years of, say, 2010.
11:50But from 90 to 2007, about 25 percent of the drop was due to China and about 45 percent of the drop from 2000 to 2007, which is maybe one and a half to two million jobs. jobs. So it might only be, once that correction is made, it might only be 700 ,000 to 1 million rather than 1.5 to 2 million once that correction is made. I would just point out that at that point, a rough number of jobs in the United States is about 140 to 150 million over this time period, around 135 to 140 million over the time period we're talking about. So you can debate whether that's a large number or a small number, but it was treated like a very large number.
12:41And of course, with the people who lost their jobs, who may not have easily found new jobs, which was part of the claim, it was very brutal. And it was very localized, which meaning it was very put in various places in the United States, which could suggest very powerful political changes were put in motion because a large group of people in a small area like a state were having a big impact. And so it was still, it's half, but it's still a big impact. Yeah, the period in all this is 1990 to 2007. So it ends in 2007. The 25%, 45 % figure that's in the abstract, and I know that he said on your podcast as well, that's actually a different issue um which we should probably come to late on to later um their their regressions can't actually produce that figure and that was something else i found independently of the borussia kettal um corrections um they their corrections were to the kind of the the raw regression coefficient, which is, I honestly can't remember what the units are, it's per thousand dollars of exposure to Chinese imports, something like that.
14:03And the effect fell by about half. And that meant that people looked at the Borussia et al. article, which was 2022, and they said, OK, so they introduced some controls. The effect survived, so Walter et al.'s research device survived. and and as i was doing this long process of reviewing it i i was like oh yeah okay well that's fair enough but then eventually i had this kind of eureka moment where i said well what about all the other effects that they report which is actually what their abstracts really about all the stuff on unemployment about wages about all the living standard stuff what what effects does to the corrections or the borosio-catel corrections have on on those results so i started to look and i found it on page 45 of their online statistical appendix of boris jackett house statistical appendix where they actually showed that their corrections kind of eliminated all those effects all those negative effects on on living standards but they never mentioned it in the actual um paper in the actual article itself they just said there's a footnote in which they say we also look at the effects on living standards in table c.1 in the online appendix but these were actually kind of the main results if you look at otter et al's abstract to their their original article the abstract is mainly about the living standards effects and the borussia et al corrections had really shown that those effects didn't survive once you adjust the regressions to correct for the research design errors.
15:48So the reason this is important is that it's not surprising that if the United States is importing a lot more from China, that it's possible that it will take away manufacturing jobs in the United States. But there was another effect, and this is, of course, when I interviewed David back in 2016, David Otter, we talked about this at some length. Economists have known this forever. That's not interesting. What's interesting is what happens next. In theory, if America is importing an enormous amount of less expensive goods from China, there will be resources available to spend on other things. And the things that expand will increase the demand for jobs.
16:33The state of living, in theory, should go up, although there are adjustment costs. It's a question of how long it takes for adjustments to take place. But the idea would be that you certainly don't want to evaluate the expansion of Chinese imports based on what happens to U.S. manufacturing in areas where factories are competing with China. You want to see what the full effects are, and they don't pretend to measure those. But when you look at the – on the employment side, but when you look at the wages and standard of living, the fact that this methodological correction in Appendix C in the online – unpublished but online available adjustment shows that it's actually no impact on wages and standard of living, it is kind of shocking.
17:25Why do you think they kept that quiet in the online app? It's there. I looked at it. I checked before we did this because it's shocking. If I may use a bad pun unintentionally. It's very surprising that this very technical paper, which, as you say, left the impact at half of what it had been before, but still the sizable impact, that the other rest of the effects, they just thought, eh, we'll just not mention them. um i i can only speculate on that russ um i think the way that the um the way the academic publishing system works especially in economics um young researchers especially are under a lot of pressure to get publications in top five economics journals.
18:15That's what they have to do to get tenure.
18:25you might not want to upset senior economists if you want to get published in top five journals, would be my speculation. That's a very cynical one, but it could be true.
18:41I'd say it's realistic. one man's cynicus and other man's realist the way the way that economics is currently in the the institutions and incentives and in economics i think is is produces that kind of result um and in this case it meant that we've we've had um three or four years four years say without anyone actually noticing that those results were there tucked away in the appendix. But to be fair to Otter at all, they might not – have they responded to that paper that critiqued their assumptions? I don't think so. I think they published a chapter in one of the economics handbooks last year on which they did incorporate.
19:39They incorporated some of Borossiuk et al.'s corrections. It's not entirely clear how many of them they might not have.
19:50But they certainly didn't say anything about the broader effect of those corrections. Possibly they're not aware of them. I mean, they might not have got to page 45. And that paper, the 2013 and later papers, they've been cited. It's not like you called it the most influential paper of the last 25 years. It's been cited, I checked, thousands of times, mostly just in passing, not critically. And you could have it. We could have a conversation. I don't know if he's fruitful, but you could argue it mainly had an impact on the economics profession. Certainly it was used in the phrase China shock, which also I checked on Pangram, had an enormous rise in usage after the 2016 paper.
20:49It was invoked by politicians and pundits for sure. But the correlation between imports from China and manufacturing employment, most people don't care whether it's real or not. It's just something they used. And I think many, many political people I've spoken to in the United States, people involved in politics, treat this like an obvious fact that America's globalization, that its willingness to import willy-nilly from China has destroyed the American economy. And it's one of the reasons I mentioned that, you know, it's a small portion of the total employment. Again, the human costs are not small for the people who suffered them if they did not find other work.
21:37But this became certainly Otter et al.'s work, Otter, Dorn, and Hansen's work was used, whether it was persuasive, I'm suggesting is not so clear. It's definitely used to prop up a variety of policy decisions made over the last 10 years that have really changed the U.S. perspective on foreign trade. Comment on that, if you would. Yeah, I think I've done a little bit of research on this. The other day I was looking at it and it was interesting. I look back at 2000, I think it was 2005, and I found a New York Times article by Nick Christophe, a commentator in the New York Times, who he said that American elites were using China as a scapegoat, basically, to distract attention from their own shortcomings.
22:40and my opinion which is just my opinion is that that process has kind of continued and some of the people who responded to Nick Kristoff 20 years ago were the same people who are still saying it's the China shock now and it's interesting because back then it was Democrats as well it was the Democrat politicians who were saying it was China, China and
23:11And it's almost like Trump has brought the Republicans on board with that Democrat position. And now it's kind of a bipartisan consensus there, which is quite unusual in the US, I believe.
23:28And my opinion would be that it's a way to avoid having debates on other issues that you really need to have debates on over there.
23:40And it's a way of avoiding kind of other questions, which it's much easier to blame foreigners. And it's a long tradition in the United States as well, especially blaming China. I mean, I'm a historian of the 19th century mainly, and there was a lot of kind of anti-China rhetoric used then as well. Yeah. Yeah, blaming foreigners goes back a lot more than the 19th century. And it's not just Americans as well, to be clear. For sure. And I just want to put the alternative hypothesis on the table, and then we'll talk about the reliability of econometric work generally and why you titled your paper, how you titled it.
24:29But the alternative hypothesis, certainly the acceptance of China into the World Trade Organization increased imports from China into the United States. And it's certainly true that the transition from China and China from a more state-controlled, top-down system to a more capitalistic one increased the amount of the productivity of China, increased their standard of living, had extraordinary transformation of their economy from agricultural to urban, enormous migrations of hundreds of millions. It's one of the most dramatic economic events of all time in terms of magnitude. So that's going to happen.
25:18That's not up for discussion. What's up for discussion is how responsive is the U.S. economy and especially the labor market in response to that challenge. That's one question. the ability and willingness of people to move toward economic activity if their local activity is being hurt by, say, imports. But the third thing that's going on that I think is extremely important is the revolution of the application of technology to manufacturing that has reduced the employment of workers there in any one factory. But at the same time, an enormous expansion that's statistically challenging. You have to be careful, but a lot of expansion of manufacturing, whether it's computers only and others, we've had episodes on that.
26:09But the point is, is that the United States economy is not being hollowed out. It's not somehow being destroyed. And that these trends that have become so politically salient started – you said – I think – I can't remember what year you started, but they really go back to 1945. At the end of World War II, going to the present, the proportion of the U.S. economy manufacturing has fallen steadily. That's not a coincidence. It's not an external effect. It's not a measure of how poorly the U.S. economy has done. It's a tremendous tribute to human creativity in figuring how to make more stuff with fewer people using technology and innovation.
26:58Now, we could make a decision as a nation, the United States could, to not let that happen. It would be really hard to implement. But it's really important whether that transformation of the manufacturing sector in employment is caused by foreign imports versus internal innovation and the application of technology. Because in one case, you can justify to protect local people from change, some kind of perhaps tariff policy. But otherwise, you're barking up the wrong tree there. So it's for those who are listening, we've had a number of episodes on trade over the years and we'll link to them. But it's important to remember that a lot of things happen at the same time.
27:46And to blame the one that is convenient is a very dangerous policy, not just because it's an error and it's it's a mistake. It can lead to very bad decisions in terms of policy. Policy and geopolitics and all sorts of things. And I mean, my prior as an economic historian is that countries tend to industrialize and then they tend to de-industrialize because as they become wealthier, people don't want to work in manufacturing anymore. And it's basically a wealth effect. And that's facilitated by globalization. And if it hadn't been China, it probably would have been another country supplying those goods.
28:26perhaps China might have accelerated the process slightly but it was a long term ongoing process that was then accelerated by China maybe but it might have been Japan, it might have been Malaysia, it might have been India supplying the American market if China had stayed a communist basket case
28:51that's my opinion more than a reasoned or researched position
29:00and I think the question for the United States is how it adjusts to that and in international trade it always has different regional effects that's been known since Olin in the 1930s I think so if you're going to be involved in international trade you have to accept that especially in a big country like the United States, so you could have regional programs to help support places that were losing out and things like that. Or there might be other solutions. You and I might disagree about that, which is why you really need to have a debate about it rather than just saying it's China.
29:46Because blame in China and trying to reverse it could have adverse impacts. I'm also an economic historian of Argentina, so I know a bit about protectionism and industrial policy, how it can go wrong.
30:10I'm actually on the left myself, so I'm a leftist, But I do kind of disagree with some of my leftist fellow travellers about the benefits of industrial policy and protectionism, because from study in Argentina, I do know that it's not as easy as it sounds.
30:38and that's also why auto-atil is so important because they did seem to have found this big causal effect of China which just might not have been there. And to maybe clarify what I was saying earlier between 1945 and 1990 or 2000 and manufacturing as a source of employment became less and less important in the United States economy. As a source of output, it was very, very healthy. But certainly as a source of employment, it was less so. And it was not a crisis for the United States that manufacturing employment was falling. People who normally would have taken those jobs found other jobs. There were still most years full employment.
31:25If it wasn't, it wasn't because of manufacturing becoming less active, because of other macroeconomic effects. So what was the claim here that this was different was because it was so large and so sudden. And it was hard for the U.S. economy to adjust the labor market to respond, perhaps, as it had in the past. So that's part of the conversation. That's why I mentioned the ability to move to where economic opportunity might be in a different part of the country. If housing markets are not particularly healthy, it can be very expensive to move to the places where the best jobs are. And there's a related question, which we haven't mentioned, which is a lot of the people who work in manufacturing, not all, but some of them are relatively low levels of education.
32:18and may have trouble finding a job that's equally attractive given their levels of skill and education if manufacturing is no longer thriving and healthy. And that's a serious problem for a country. And I think, you know, I've always felt that the last 20 years, that is certainly part of what has produced a lot of anger in the United States and a feeling that the elites don't care about the less educated, say. But to blame that on China is a special case that economists have accepted, and it's very much a change in their perspective on trade generally. In the past, they were much more comfortable and confident that while trade, of course, hurt certain people, while the general country might thrive, that this was of such a magnitude that we needed to do something about it.
33:14Yeah, I think it has been a big change. I think it's odd because it's been known for a long time that globalisation produces winners and losers in relative terms. And really, if you're going to embrace globalisation, you need to do something to compensate the losers because otherwise they get angry and vote for politicians who can then do unwise things. And that's quite sensible.
33:50And it's kind of obvious. And it's been just known for a very long time. So some people have responded to my critique of Otto et al. They said, okay, okay, so the epilometrics wasn't great and the effect wasn't so big, didn't have these effects on living standards. But they at least made it really clear that international trade does have these different effects on different people and different localities. But I think economists have known this for a very long time.
34:26And possibly it's because they've become much more beholden to kind of simple causal stories in recent years, that they like this kind of China caused it story. but I think it probably says something about economics that it has taken hold in this way I'd say it says a good thing about econometricians because econometricians have looked very closely at Otter et al like Borrecy et al and there are others before them who I discuss in the article as well and they they really looked at this result in in in the original 2013 article and And they put a lot of time and effort into picking it apart, I suppose.
35:11And I'm really just the messenger who's bringing what they found to more kind of wider notice, maybe. Yeah. Yeah, it's, I mean, there's a moment. Let me tell the story more accurately because, or in a more interesting way, because I don't know if I mentioned it when I interviewed Otter back in 2016. I was at a seminar where an economist was making a very dramatic claim about some impact or something or other. I don't remember what it was now. And I asked a question, and I have to confess, this was maybe, I don't know, 10 years ago. So at that point, I've been an economist for about 35 years. And I asked a question I've never heard anyone ever ask at a seminar.
36:03I asked it with some trepidation. I said, how many regressions, regression being a statistical attempt to estimate something using a statistical package typically. I said, how many regressions did you run? And the reason I asked was because you can always run some more. Try something different, add a different variable, change the way you – the instrumental variable you alluded to earlier, which is a way to control for correlation and try to isolate causation more effectively. You can always run some more. And many people have suggested – we'll talk about that in more detail in a minute – but many people suggested that this is really not a healthy way to look for the truth.
36:45But it is the way our profession – this is what you'll see in the kitchen. And I've been in the kitchen. Like you, I'm not an econometrician per se, but I've done empirical work. And you can always do some more until you find a result that maybe is a little more comforting. Now, it might take a lot of torturing of the data that – so much so that people say that's not a reliable finding. But in the more complex world that we're talking about, it's very hard to know how – what it means to do something that's justifiable or true. You can always do something one more time. So I asked him, how many regressions did you run?
37:22He didn't know what I meant. He literally looked at me, what? He couldn't answer the question. And I said, no, because I'm sure you ran a lot because that's how the world works. And I'm curious to know how many times in all those regressions did your finding stand up, hold up, stand out? And it was a rhetorical question. So I asked David Otter that question. And I didn't let him answer it because I knew it was. It's not a meaningful question. It's a cheap shot in a way. It says, look, I know how the business, how we do our work. We run a lot of regressions till we find something that's publishable or that we think we can defend.
38:04And that's the wrong way to do econometrics in my view, but it's the way it's done. Yeah, I think it is. I think what happened, there was what they call the credibility revolution in economics. So if I go back to when I was younger, say 20 years ago, my critique of economics was it was all too theoretical. Like economists, they were just kind of in this unreal world of pure theory. And then suddenly all that changed. Economists, they all started doing lots of empirical work in what they called the credibility revolution. And they introduced, they started using these econometric techniques. and I think the econometricians of that generation they were very rigorous but they also they also kind of had this attitude of oh here's my shiny toy would you like to play with it and the economists went away and played with these shiny toys and um and produced all these papers saying oh we've we've proved that this that x court is y But now I feel with the econometricians of today, the younger people, that they're kind of pushing back a bit on that, that they say, well, actually, maybe we shouldn't just play with shiny toys.
39:22Maybe we should be slightly more serious and actually kind of make things a bit more robust. And I've been studying economics kind of as an outsider of economics now for about a year, I suppose, doing a lot of these reviews of different articles and different papers. And I've been very impressed by the work of econometricians about trying to make it more robust. and I think this is one example of how a research community of econometricians have made Otter et al's work more robust and actually changed the results in doing so, changed what the research design actually shows and yeah. Well, you called your paper, your article, Whimsy and Fragility in the China Shark Narrative, which is a whimsical title.
40:26But you're making a reference to Ed Leamer, who passed away recently. He was a guest on the program many times. He's one of my favorite economists. You mentioned in passing recently that it's a shame he wasn't winning a Nobel Prize, which was announced recently. because he's gone. But he wrote a very provocative, non-influential article called Taking the Con Out of Econometrics. And what he was arguing for there was a more honest look at what empirical work actually was doing along the lines I was just mentioning. And And why'd you, in a way, this is an homage to him. Yeah, I really love Lima's work and his attitude as well.
41:17Lots of people didn't like his attitude. They said he was too aggressive, too mean. I think people say the same thing about me sometimes. But I think, honestly, I think you need to be a bit aggressive and mean in intellectual debates. Otherwise, you don't really get anywhere.
41:37and he said that economics, it was fragile because of this process that you said of where people run regressions, they try different things until they kind of get a result that they think can be published. Whereas if you change a few things, then that result disappears or changes, as in this case. And then he said it was also very whimsical because there's a lot of these kind of assumptions baked into a lot of economics research. And actually, if you pick at those assumptions, it doesn't really hold up very well.
42:15I live in Wales, and there's a lot of old farmhouses in Wales. And if you start picking away a bit of plaster on the wall, you soon discover that there's a massive great big hole because they're all just completely damp and falling apart. And I found a lot of economics newspapers are like that as well um so yeah ed lee was kind of a hero to me and um and i i re-listened to to to your episodes that you did with him to prepare for this and and what he said was like oh yeah that completely chimes with with what i say that economics should actually be more about telling stories and about um finding patterns in data and and and i think a lot of problems of comics now is that they want to use these econometric techniques which are very good at precisely saying that x caused y but in the real world we have multiple causality there's always multiple factors causing things so if you look at the china shock or sorry if you look at deindustrialization you know china might have had a role but it's probably quite a small role um and there's all these other things going on that if you want to have a proper analysis of of deindustrialization, what caused it in the United States, you really need a much, much broader analysis, which you're not probably going to be able to do in a regression, which is fundamentally based on X causes Y.
43:38There's all these other factors that you need to take into account. And econometrics might not be able to do that very well. And of course, the data you have don't exactly match what you're trying to measure often, but you, quote, do the best you can because it's better than doing nothing. And the answer is no, it's not if it misleads you or turns you in the wrong direction. You mentioned the credibility revolution. It's been followed, of course, by the replication crisis, which is that a lot of the papers, certainly in psychology and I think increasingly in economics, have not held up when people take an independent look.
44:14And I want to salute you for working on that. It's generally not a rewarding strategy to try to show that something that we thought was true actually isn't true. Just like if you find no effect, instead of announcing it and publishing it, you just throw it away. It's called the file drawer effect or the garbage can effect. And so we only see the results that look dramatic and large. And if you've ignored all the other ones, you're not going to get very far. Would you expand on what you said a minute ago about storytelling and Ed Leamer and economic history? Because economic historians over the last 50 years have turned to statistics with great excitement because it's prestigious and it looks scientific.
44:59Whether it is or not doesn't matter. What do you think economic historians ought to be doing to understand something like the deindustrialization or the reduction in manufacturing in the United States as a source of employment?
45:17um so i'm a very old-fashioned economic historian like i i my my style of economic history goes back to say the 1950s 1960s and um and i see economic history as having been in a very long-term decline and and um a lot of what gets kind of described as economic history now i i I refuse to recognize it as economic history. What I would do if I was going to do this properly, I would do a lot of time series analysis. I would try maybe to kind of econometric things on small scale kind of sub-questions. I think econometric techniques can be very good for small questions, which then if you build them all together, you could answer the bigger questions.
46:13Like, so you could try and measure the effect of Chinese imports, other countries' imports. You could try and measure the effect of automation. But then ultimately you need to kind of put them together into a more holistic analysis that kind of makes sense.
46:36and it's kind of sophisticated beyond this kind of X-Causes-Y approach that tends to dominate. I confess that I haven't looked at deindustrialisation in the United States beyond this paper, so I'd need to sit down and really think about... I think I'd also go back to a lot of the old debates, look at how people were trying to understand it back in the 1980s in the 1990s when when it was when the process was still quite young because it's because i find a lot of value in kind of people at the time how they were trying to understand things um as well um yeah so i and a lot more description and just kind of describing what happened i think and And because actually it's through description that you can get a much better understanding of what happened.
47:37And that kind of leads you to why it happened as well. Yeah, sometimes I think that's the best you can do. And that's too depressing for most people. It doesn't get you on the cover of the New York Times or even on the editorial page. You know, I think about the changes that have happened in the United States over my lifetime. I now live in Israel, but, you know, I lived in the United States from until five years ago and grew up in the 60s and 70s. And there's an enormous number of things that changed. If you're trying to answer even a small, simple question akin to what we talked about recently with Jesus Fernandez-Villiverde, you know, why are Americans having fewer children?
48:25oh, because fewer people are getting married. Why are fewer people getting married? Well, that's because technology and the workforce, and there's 3 ,000 interesting possible, it's a smartphone, you know, fill in the blank. It's the decline of the U.S. education system, especially in the urban areas. I mean, there's just so much change, and the challenge, of course, of being a good historian is to tell a story that certainly has some complexity to it the next causes why. But all the rewards go in the other direction, right? All the incentives go the other direction. So it's the rare person who can honor the complexity of the world, even though it's not always going to be the most personally successful approach.
49:17Yeah. I mean, I hope by by doing this kind of study, like what I've just published in Econ Journal Watch, and I've published other ones before. I also hope to do a little bit to kind of change the incentives so that actually my kind of economic history, which I should have also said should also be theory informed. I think that's very important. Like in the case of deindustrialization, we really need to think about whether deindustrialization is kind of inevitable as a country gets richer. and I'm not sure whether that debate's really happening um so but but through my econ journal watch articles and other work I I hope to be able to do a little bit to and my blog as well I hope to be able to do a little bit to change the incentives um in economics by showing that a lot of what's getting published and a lot of what young researchers are being pushed to do especially isn't very robust and maybe the emperor doesn't have any clothes.
50:24So that's kind of my end goal of this kind of paper. If you look at your paper, of course, we'll link to it and we'll link to the Con Journal and Watch paper, we'll link to your blog. You know, there's a lot of data in the paper. There's a lot of tables. And you look at it, if you're a non-academic person, a non-trained economist or econometrician, and it looks thorough. It looks like there's a lot of work there that you're citing of these, of Honor, Torn and Hansen and the people who revised their methodology to get a different number. But there's a natural impulse of the public to think, well, these people are at first rate universities and look how much technique they've applied.
51:23And even I, who am a skeptic about China, about the China shock, and I'll confess, part of it purely out of ideological reasons, or I don't know if that's the right word. It's not the right word. I don't know how you would describe it. I like trade. I think trade's good for the general, there are exceptions, of course. But in general, trade is a good policy, having more trade rather than less. And I like being involved with the world rather than less. And I think it's good to be outward facing rather than just inward facing for a nation. And it's a thousand things that I like about trade that have nothing to do with the narrow question of whether it makes the pie bigger.
52:00I think it's about much more than that. So when a person reads – and forget me. So now you're just a smart person who doesn't have a PhD in economics. And you're trying to stay up with public policy and be informed. And you read on the front page of the New York Times or the Wall Street Journal that a study has come out of people from world-class universities saying that China cost America millions of jobs. It's hard not to take it seriously. You know, there's a certain – and I've always wanted – the incentives we've talked about a little bit. But what if your life mattered? What if your future well-being mattered on your econometric results being standing up to various robustness checks?
52:50Would you say what you say? I'm not suggesting there's anything untoward about econometricians and their claims, but in general, they seem to be more confident than the work merits. It's a difficult field. But you get famous by finding something dramatic. and so, which is a good idea. It's the way the world works. It should work that way. You should try to find something dramatic. But it's sad how many results, not just in economics, goes in every field are challenged when they're looked at more closely. It's really not, it's alarming. Yeah, yeah, I agree. And it's not just economics as well. I think that there's, I think there's big problems in the knowledge production system as a whole.
53:38I'm mainly looking at economics because I find it very interesting and it's adjacent to my own speciality as an economic historian. And I think it is very disturbing. And it would be nice to believe that academia was producing reliable knowledge that we could depend upon. And if it's not, then it's problematic because it suggests that there's big institutional problems that really ought to be addressed. And some disciplines have gone through that. I think psychology has gone through that. I'm curious to see how much difference it's made to psychology. Is the new stuff any better than the old stuff?
54:26I don't know. I haven't looked at it at all. I would hope so. Economics is a very curious one because economists obviously pride themselves on being very robust. the economics journals they make all these replication packages available but my impression is that no one looks at them and and then if you write to if you write to an economics journal and say look I found this problem in this article they're like I don't really care send it to Econ Journal Watch I've literally had that response well it wasn't it was a bit more sophisticated than that. They told me to send my complaints to Econ Journal Watch.
55:05On the other hand, other disciplines don't have an institution like Econ Journal Watch. So something about economics has produced that, which I think also says something good about the discipline. Yeah, Econ Journal Watch, I think, was started by Dan Klein, my former colleague at Church Mason and past EconTalk guests. And it's a watchdog. It tries to give a forum for people like yourself who find things that are somewhat perhaps alarming. I would just say that I remember having a conversation with an astronomer, a brilliant person, who was obsessed with the reality, I'm sure, at least his perception of reality, that the empirical work that justified some claim in astronomy was shoddy.
55:59And we have this certain, you know, romance, you know, we're only in the social sciences. And of course, there's all kinds of problems we have with our data and the fact that human beings have their own sense of agency and life's much more difficult for us. But we assume that in the sciences, I mean, that's a whole different thing. It's opening and shut. They just find truth, or they don't find, in which case they put it down. But it was basically a claim that the same things were happening in astronomy that were happening in economics. The use of advanced techniques, the ability to tease out certain really nuanced, small things from data that were noisy, et cetera.
56:41I think, yeah, I think each discipline has a different variety of problems. I think what may be different in STEM, in science, in the sciences, is that they've got a real culture of post-publication peer review. I've seen scientists descend on a shoddy nature article, like a pack of kind of rabid dogs and tear it apart on social media in just a way that, I think within about a week or two, nature have put out an expression of concern and they were just savage. Whereas in economics, when something gets into a journal, it tends to be taken as law. And that's, you know, each discipline has a different culture and it has its different issues.
57:33And I'm not saying that economics has any more issues than any other one. A sociologist wrote to me, actually, and I asked him what sociology is like, and he said, well, we don't accept these results so easily if something's passed these statistical tests. On the other hand, no one ever wants to make their data available because all the qualitative people would just see this as a very dangerous thing to do because then people might start poking around too much. So yeah, each discipline has its issues. Have you heard from either Otter, Dorn and Hansen or the – and I apologize, I'm not – I can't pronounce the – I don't have it in front of me.
58:22Give me the other trio who challenged them. Borussia Hull and Caravelle. I'm not sure if I'm pronouncing all those correct either. Have you heard from any of them after you published your paper? So not since I published the paper. I sent a working paper version to Otto Dorn and Hansen, and David Otto responded to say thank you for sending it to me. I've had various interactions with Peter Hull, mainly on social media, and my impression is that Peter hasn't found anything wrong with what I've written. He's kind of... So that's good for me. I've had some I've had some interaction with Kirill Borosiak another of the economic matricians I released a statistical package for R a statistical software based on some of the research I'd done with him and Kirill wrote to me and said this is rubbish so I retracted that I retracted that rapidly but that was apart from the apart from the the article That was a separate thing.
59:35So as far as I'm aware, I haven't made any econometric errors in the article. I interviewed Adam Astriani, the psychologist who wrote a shocking article.
59:53You get disillusioned. It just takes a long time. You think everything that's peer-reviewed is true, and then you realize it's not. But you assume if it's a really good economist, it's true, and then you realize maybe not. And what Adam wrote, partly in response to the replication crisis in his field of psychology, was that peer review is awful. It's not like, well, it's flawed, it's imperfect. No, it's a bad system. And I remember reading that and thinking, well, that's a ridiculous idea. But then I read his article and I realized, oh, my gosh, I think he's right. And it doesn't mean that you can publish whatever you want.
1:00:34It means there's a vigorous, rigorous, ideally, post-publication process the way science used to be before there were formal industry in both journals and in academic life. What do you think about that? I think peer review is completely necessary. but a lot of the actually existing versions of peer review are not very good. I'd say in economics it's a complete black box. Explain. What do you mean? Well, you see the working paper released at some point and then however many years later you see the article released and you don't really know what happened in between. I know people have written to me and said, yeah, you know, I reviewed that article and I said it was rubbish and it just got published anyway.
1:01:38So because an editor might decide that actually they want to publish it anyway. The editor just overrides the reviewers. The editor might be friends with the author, et cetera, et cetera. and note to self says the editor don't send anything to that person again so the whole process has a certain endogeneity to it that's not attractive it's quite corrupt and because it's not open so if that corruption is quite easy to get away with if you look at the sciences so if you look at nature for instance they often publish their their peer review reports sometimes the peer reviewers are named sometimes not um and you can you can then really kind of track what happens with the paper um and of course there is this this this culture of post-publication peer review where an article the one i'm really thinking of it was an article where it was written by quite quite famous um scientists and they said that the time of day when you when you took a particular treatment for cancer had a big effect on um on the on on how effective the treatment was and the research had all been done in china and and like the results it was all a bit kind of shady and and these are these other a whole whole load of scientists just kind of jumped in there and said no this they found all these irregularities in the data and and all sorts of stuff.
1:03:04And you just don't see that culture at all in economics. And for all the economists claim to be very robust and have this very robust seminar culture, I like to think that if my articles contribute anything, it's to show that even though they're very proud of that robustness, it's not really doing its job very well. My guest today has been Joseph Francis. Joe, thanks for being part of EconTalk. You're very welcome. Thank you very much for having me.
1:03:39This is EconTalk, part of the Library of Economics and Liberty. For more EconTalk, go to econtalk.org, where you can also comment on today's podcast and find links and readings related to today's conversation. The sound engineer for EconTalk is Rich Goyette. I'm your host, Russ Roberts. Thanks for listening. Talk to you on Monday.
1:04:03Thank you.
What if the results of one of the most influential economics papers of the past 25 years are unreliable? The "China shock" narrative--that a surge of Chinese imports destroyed American manufacturing jobs and the communities that depended on them--became conventional wisdom in Washington and beyond. But economic historian Joseph Francis argues that the claim is much weaker than it looks. Listen as he tells EconTalk's Russ Roberts how a 2022 correction to the paper’s research design cut its estimate of lost jobs in half and erased its findings on wages and living standards. He and Russ then look at what this affair reveals about economics as a field, including the incentives that reward dramatic results, and the problems with peer review.