Episode 316 - Andrew Chen: "Is everything I was taught about cross-sectional asset pricing wrong?!"

0:03This is the Rational Reminder podcast, a weekly reality check on sensible investing and financial decision-making from two Canadians. We are hosted by me, Benjamin Felix, Portfolio Manager at PWL Capital, and Mark McGrath, Associate Portfolio Manager at PWL Capital. Welcome to episode 316. Cameron is obviously not here today, so Ben, good job on the modified intro there, two Canadians, not three. This is a big thing. Can we just real quick for a second pause here and this is your first episode with a guest and the first time you and I are together doing an episode. And without Cameron as well.

0:37And I mentioned that to my wife is away right now, but I texted her this morning, like, you're not going to believe this. I'm replacing Cameron today. And I'd still pinch myself over it. It's wild to me after being such a fan and a listener for all these years that you and I got to interview Andrew Chen today. What an episode for you to start on. Oh boy, what a conversation that was. He's obviously brilliant, tons of incredible insight. He's done some amazing research. And I think my mind was blown multiple times through this conversation. Andrew's research, he looks at meta research. Looking at meta studies, large-scale replications, large-scale statistical inference, meta science being the broad term of all that.

1:17He's doing this on cross-sectional asset pricing literature. Factor investing is based on that. He's looked at this from a whole bunch of different angles. He's looked at replication. He's looked at out of sample performance. He's looked at whether a factor premium survive transaction costs. I'm not going to give away the answer on whether they do or not. You'll have to listen, but all just absolutely fascinating. Honestly, a lot of stuff that I thought to be true or just held as a belief turns it on its head. I think he says that for a very long time, cross-sectional asset pricing research was not going in a great direction.

1:57It's improved more recently. We talked about that, I think, near the end of the conversation. Some pretty serious stuff. Any other thoughts before I talk a little bit more about who Andrew is? Like you said, incredible conversation and really very insightful. And to your point, has me thinking about the topic more and more. What I do with the information that I learned in this episode remains to be seen. But as you said, I won't spoil it for the listeners. They'll have to listen to the rest of it. So Andrew has been an economist at the Federal Reserve Board since 2014. He's got a PhD in finance from Ohio State University and an MBA from Georgetown.

2:28He was previously a research fellow at the National Institutes of Health, which is interesting. His primary research interest, as I mentioned, in financial economics specifically, is in understanding whether finance research is going in the right direction using tools from meta-science. Most of his papers are around that. His research has been published in Management Science, the Journal of Financial Economics, Critical Finance Review, the Journal of Finance, the Journal of Financial and Quantitative Analysis, the Review of Asset Pricing Studies, and the Review of Financial Studies. For anyone unfamiliar, those are like the top journals in financial economics.

3:03His focus on cross-sectional asset pricing, which again is what the concept of factor investing is based on, makes him really, really well suited for our podcast. And I thought this was an incredible conversation that will affect the beliefs of everyone listening. Okay. That's a great introduction. Let's go to the episode. Let's do it.

3:29Andrew Chen, welcome to the Rational Reminder podcast. It's great to be here. I just need to put the disclaimer up front that my views are not necessarily those of the Federal Reserve Board or the Federal Reserve System. Understood. Andrew, what is an asset pricing factor and how is it different from a predictor? Predictor is how it sounds. It's something that predicts asset returns. A very popular one is when you take book equity of a firm and divide by the market value, you get this variable that predicts returns. Now there's different ways to measure that predictability. One of them is to construct a factor, which is basically a trading strategy on book to market.

4:08But there's also lots of other things people mean when they say factor. So I prefer the word predictor and try to use it as much as possible. But the language has just gotten all squished. People say predictor, factor, and anomaly and just squish it all together. Try to say predictor, but it's okay if we deviate a bit. Okay, makes sense. What are the plausible explanations for why cross-sectional predictors exist? There's basically three explanations. The first is that it's due to risk. The stock or the strategy is giving you high returns because it's exposing you to something that you should fear and it's rewarding you for that exposure.

4:40Another one is, of course, mispricing, just that the price isn't right. It's going to move up in the future or it's going to move down in the future. And the third one is that the statistics got wrong somehow. out. There's always some uncertainty in the stock market. You can't know for sure what the return is going to be. And there could be something wrong with the statistics. We've heard about the predictor zoo or the factor zoo with over 400 predictors. How many predictors are there, really? If you're talking about predictors that are clearly documented in the academic literature, I would say there's roughly 200.

5:13I say roughly because it's not always very clear if one predictor is different than another predictor. Also, it's possible to say if you really have the whole literature capture. People have this idea of 450 anomalies out there from a paper, which I think has some issues. They're pretty creative with their language, as you can see. I try to be strict about my language. I also can make mistakes too. But in this very famous paper, they take 150 predictors and they create 450 anomalies from them by tweaking them certain ways. Why was it important for you to create an open source data set of cross-sectional asset pricing predictors?

5:49Academic finance and I guess science board generally always has this trade-off between open collaboration, like standing on the shoulders of giants, and closed competition. If you don't have any competition, you might not really get those cutting-edge insights. And when we started on this project, academic finance was very much closed competition. It's kind of like the business school culture to compete. But then we got to the state where we're publishing these papers with these machine learning algorithms that are really complicated, fitted to dozens and dozens of predictors. And it seemed like it was impossible to openly collaborate at that point.

6:23I thought it was very important. And you can see in the AI space how that's become so critical to progress. Another aspect of this was that at the time, there were some papers that were, I think, not very high quality that were misleading people into thinking that there's some kind of replication crisis. And we wanted to put out a paper to really counter this narrative. Can you maybe just talk a little bit more about the open source asset pricing project? What we did there is we took 150 papers on cross-sectional asset pricing, and we found any kind of result about predictability in the paper.

6:57So just to be clear, it's about predictability. What is the factor? I don't know. Everyone disagrees, but you can see it's predictor. There's a regression that shows predictability. And we tried to replicate every single one of those predictors. And then not only that, we posted all the code online and the data online, and we update this every year. We tried to make it high-quality code, too. Once you have a project this big, you realize, oh, man, you need to write things nicely. We're still primarily economists and finance researchers, so we don't have the best code. But I think it was a good step up from what people often provide, especially if you go to some Stanford professor's website, you might not like what you see.

7:35I hope ours is a bit better. And how did you choose which predictors to study? So as I alluded to before, there are some papers out there pushing this replication crisis narrative. One is by Ho, Shui, and Zhang. It's published in the Review of Financial Studies. It has a ton of citations. That's the paper where they have 450 anomalies. There's really only 150 predictors underlying there. Their main result was that they failed to replicate the anomalies. And so we want to clarify that. That's not really what you see if you look closely. So we replicate everything that's in that paper. And then there are other papers simultaneously on this topic.

8:10A very important one is by McLean and Pontiff. I don't know how much you guys are into these names, but this is a very important paper that documents how predictability decays out of sample. If you read some numbers in the paper, it tells you that, well, you got to divide by two in the future, if you're looking forward. So we want to replicate everything in there too. And we got that pretty much done. And then there is another meta study that they didn't actually replicate anything. So this was more challenging. They had a lot of stuff that was very, very difficult to deal with. And we still try to replicate everything that's in there.

8:40And ultimately, we got this list of 300 variables that may or may not predict returns, but they were discussed in these previous meta studies. We wanted to be comprehensive, cover all that. How many of those were you actually able to reproduce? Let me back up a bit and say that of those 300 variables, only 200 actually were shown to predict returns in the original papers. That was a major comment that we wanted to make that some of these influential papers, they never really look to see whether there's anything to replicate in the first place. And that's something to document. And that's something I think we document clearly as part of the open source project.

9:14We have these spreadsheets where we actually write down the page that you should look at for predictability. And if there's no predictability, we just say there's nothing in there. Of those 300, about 200 variables actually were shown to predict returns. Replication is a bit subjective. It's not clear if you get the number seven and the original number was eight. Is that good or not? But our judgment was that we replicated everything but three. There are three failures that were pretty unsatisfactory. Three out of 200? Yep. Wow, okay. Roughly one to two percent failure rate. That's wild. You mentioned the other paper that showed maybe replication was not so great.

9:52Why were your results different from that? If I'm frank about it, it's because they're very loose with their language. And it is pretty surprising because this is a really influential paper. It's still highly cited all the time in this top journal. But really, they never define what an anomaly is. They define what a failed replication is, but it's not a good definition because they don't define an anomaly. It's helpful to describe an example. So one of the papers in our data set, both of us share this paper. It's by Dijchev in the late 1990s. And he talks about bankruptcy risk. To try to be more precise, the probability of bankruptcy, you have these variables from the accounting literature that predict bankruptcy.

10:31And he wants to know, do you get a reward for bearing this bankruptcy risk? Stocks that have high bankruptcy probability, do they have high returns in the future? And he finds that the answer is no. They actually tend to have low returns. a relationship is the opposite of what you would expect in an efficient market. You're not going to be compensated for that. But he doesn't find that this relationship is statistically significant. It depends on the test. Some of the tests show significance, some of them don't. But the relationship goes the wrong way. So that's a very interesting result. You definitely want to know about this.

11:02When you replicate it, you need to be careful. If you find insignificant result, that's actually a successful replication. He finds that there's no relationship. There should be a relationship between higher bankruptcy probability should be higher return finds actually a negative relationship. It's not statistically significant, but if you get that direction right, that's a successful replication. And in the Hoshue Song paper, they don't care about that. They call it a failed replication. And to make it more interesting, they make, I think, three or four versions of this bankruptcy variable, and then they get four failed replications, although they should all be called four successful replications if you did this carefully.

11:40Very interesting. How do you anomaly returns change post-publication in your data? I want to be a bit more precise about that. Instead of post-publication returns, let's talk about out-of-sample returns. It takes a long time to publish a paper. And so usually, if you find a paper published in 93, the sample probably ends in like 89. If the sample ends in 89 in the paper, from 1990 to present, the returns decline by about 50%. 50 % as in if you had 10 % per year in sample before 1989, from 1990 to present, you'll have a return of about 5%. But there's a big nuance there, which is that in the first few years out of sample from like 1990 to 1993, the decay will only be about 25%.

12:26You replicate a lot of these things. You find the premiums decline, but don't go away. What would you say are the implications of this research for the so-called replication crisis that people have talked about in cross-sectional asset pricing? The replication crisis is interpreted in like a big way. You could mean replication crisis in that you just can't write the code to reproduce the numbers in the original papers. That's clearly out. That's just false. I think it's become well accepted about that for at least the equities literature. But then you could also think about replication crisis more broadly.

13:00I'm not a fan of how this is being used, but people also interpret this as external validity is wrapped up in replication crisis. And it makes sense. But when you're talking about stock markets, you have to be a bit more careful. If a predictor is based off mispricing and the markets are reasonably efficient, which we hope that they are, you would expect that predictability to decay out of sample. What you're really afraid of, I think, is something called a false discovery or a statistical figment, where the numbers in the papers, they look good, but because of these statistical problems like p-hacking, multiple testing problems, perhaps these are just statistical figments.

13:35And if that was the case, then you would expect the returns to vanish immediately out of sample. It shouldn't take five years. And right after 1990, you should see the returns go to zero. But that's not what we see. We see them gradually, in a sense, decay to 50%. That tells you that that's also a relatively minor problem. And I should say, I need to credit McLean and Pontiff for that result. Academics care a lot about that. And they're the ones who really discovered that. We replicate it, which I think is pretty cool. They're a meta-study that replicates many papers and we're a replication of their replications.

14:06And you can still see this result. It's perhaps one of the most solid results in empirical asset pricing. Can you just elaborate a little bit on that term, false discovery? So it's a term from the statistics literature. In statistics, a discovery is something that's statistically significant using some tests. But then even though something's statistically significant, it could still represent no effect at all. They call that null in statistics. But traditionally, in asset pricing, that would mean the true alpha or the true expected return is just zero. A false discovery is when you have a statistically significant result, the statistics look good, but the actual underlying effect is zero.

14:45There's actually no alpha there. It's just a false positive. How large is the false discovery rate in cross-sectional asset pricing research? You could reliably say this is just a statistical concept, so there's caveats. But what the numbers say is that the false discovery rate is less than 10%. So it's actually quite small. Why does the false discovery rate that you find differ from other studies? This is another case where it might be a bit controversial because we're finding stuff at odds with other people's results. And also the way that things are at odds is also a bit striking. The previous paper on this, which is still probably the most well-known paper by Harvey, Liu, and Zhu, also in the Review of Financial Studies, extremely influential paper.

15:28Basically, they conflate insignificant with false. There's a difference between insignificant and false. Just because you have a negative COVID test, you could still actually have COVID. You could have a false negative. So there's a distinction between what the statistics show you and the underlying truth. And a fundamental part of statistics is that there's uncertainty. If there's no uncertainty, there's no need to do statistics that just be like, I'm not sure, like history or something. If you equate false and insignificant, then you're ignoring this uncertainty. That's something that like they just didn't think about.

16:00And then you did and put it into your research. I can't get into the heads of what Harvey, Liu, and Sue were thinking. I think the rhetoric actually just got too strong. Another thing in the paper, which is probably more striking than equating false and insignificant is they equate many with most. So you actually have to dig very deep into the paper to understand any of this. So it took a long time for even me to understand as someone who specializes in this. All these papers we've been talking about, they're dozens of pages long. And it's not until you get to page 26 or something like that of the Harvey Lewin Zoo paper that you really see how did they get this claim?

16:34In the abstract, they loudly proclaim most claim research findings in financial economics are likely false. But you have to get to like page 26 of the paper, get through all these statistics to really understand what's going on there. And if you get there, you'll see that they mix up both false and insignificant and also mix up many with most. It's pretty incredible. I think it's because of the incentives in I'm vulnerable to this. Perhaps I've also overstated things before, but there's a big incentive to make big claims in academia. Can you talk about how the false discovery rate relates to publication bias and out of sample returns?

17:10Let's start with publication bias. We haven't talked about that yet. Publication bias is no one wants to read something that's not interesting. No people who write for a living want to write things that are uninteresting. So we all get together, look for only the interesting stuff, and sometimes we push it too far. Or another way to think about it is it's an inevitable fact that if we're only looking for interesting stuff, some of that stuff will be not interesting. If we're kind of biasing our reading or our writing towards stuff that's interesting, some of it will actually not be there. So that's publication bias.

17:42And then how that relates to false discovery rates is you can think of publication as people searching through thousands of ideas and just reporting what is really noteworthy in the journals. And that could be thought of as like a multiple hypothesis testing problem. Traditionally, if you go to your, I'm not sure where people learn statistics nowadays, but hopefully high school, you'll say, oh, yeah, you have a hypothesis. you construct this test. If the test statistic exceeds some value, it's significant. But that procedure assumes that you have a single test. But what happens if you have many, many tests, if you only talk about the very extreme test, the extreme test in the tail, then that's when you need to use these false discovery rate methods.

18:28It's interesting that they're relatively recent. In statistics, they didn't really get developed until the late dineese. Interesting. Switching gears a little bit, how much of an effect do transaction costs have on factor expected returns? That's a very important thing to bring up because I'm kind of emphasizing false discovery rates and this publication bias I've been talking about are really statistical concepts, at least the way I've been using them. So the statistics are only as good as the ingredients you put in or they only reflect the economics that you put into it. And the economics that we've been talking about, perhaps surprisingly, they don't account for transaction costs at all.

19:06I don't want to go too much of a tangent about this, but there's perhaps a good reason for that. No one agrees on how to measure transaction costs. If someone puts in some number, someone's going to want to take it out and put in their own number. In a way, it makes sense for the academics to just focus on gross returns. But if you try to measure transaction costs in a very basic way, I have a paper with Mishvelikov, where we find that they eat up about 25%, 30 % of the trading strategies returns in the original samples. That's a lot, right? When you think also about the out of sample performance declining, is that eating all the out of sample performance?

19:44If you know the literature, you know that people, they have these equal weighted trading strategies that weight every stock equally, no matter how illiquid it is. My intuition was 25%, 30%. That's about right of what you would have in sample. But I think then you're alluding to this fact that we find in this paper that out of sample, the returns are basically completely gone after trading costs. I mentioned that people find that out of sample, the returns decline by 50%, but that ignores transaction costs. And then if you account for transaction costs in the recent period from like 2005 to the present, that even the best of these anomalies or predictors earns nothing in this very baseline transaction cost case.

20:27On the transaction costs though, are those worst case transaction costs or using cost mitigation techniques that a real asset manager might use? They're not worst case. We are using some transaction cost mitigation techniques. We look to see if we can modify the strategy to eliminate these tiny stocks, or we can try to tilt towards the large stocks. And we also try to rebalance less frequently. We also try to rebalance only when the signal really tells us to rebalance. After you do all of that, you get that 25 % decline in sample and then out of sample, the returns are gone. There are definitely caveats on this.

20:59Like I mentioned, people argue about how to measure transaction costs. Ours cost is the effective bid-ask spread, which you can think of as assuming aggressive market orders for a small trader who doesn't have price impact and ignoring short sale costs. There's a recent paper that says that just a short sale cost will eliminate these anomalies altogether. At the same time, it could get messier than this. The real way to think about these predictors is in combination. And in fact, there's an economics of this that transaction costs are actually, in a sense, lower if you use more predictors at the same time.

21:33If one predictor is telling you you need to make a trade, if you only have one predictor, then you kind of have to eat the cost. But if you have two predictors, the other predictor may tell you that you can offset it and that you don't have to make the trade. I've been talking about these predictors in isolation. And there's some recent studies that find that even net of transaction costs, if you can combine predictors, you can still get some returns. Wow, that's really interesting. I hadn't thought about that. The combination of predictors can actually reduce trading costs. What are the problems with estimating factor expected returns using historical data?

22:07Of course, everyone knows that past performance doesn't necessarily reflect future performance. But one thing that is becoming out of the anomalies literature is that the markets get more efficient over time. If you're looking at stuff that's performance in the past, it's likely to be not just that it's not necessarily reflective of the future, but probably overstates performance because of these market efficiency changes, technological changes. As an example, a lot of this data, like I'll keep on using this 1990 sample end date, Fama-French 1993 or 1992. There were these very influential papers at that time.

22:41They're using data from 1963 to 1990 or 1989. And you think, how would you trade on book to market in that time? For a lot of people, you have to get these accounting statements in the mail and write everything down and make a ton of phone calls. It's a completely different world now. Now it could take a microsecond to do that once you get the data. That's the big problem. Think about the era that your data reflects. How do you need to take a haircut off of that because of the new era of technology? How did factor performance change after 2005? Let me just take the empirical fact first before I speculate or try to get my view on what it is.

23:17The fact is that from a lot of studies, you see these anomaly returns or predictor returns or factor returns. They all decay. There's like a kink around the mid-2000s. It's pretty cool that you see this across many studies. And the hypothesis that I prefer is that that was when the internet took off, when IT took off. Thinking back when I was an MBA student, and even then, it was still kind of growing. It could be somewhat difficult to get an annual report and digest it. You'd have to go through a little bit of hurdles. Now AI will read these things. There's a competing hypothesis, though, which is that this was also the time when a lot of these anomaly papers were published.