41. Dr. Bapu Jena on Why Freakonomics Is the Best Medicine

0:00How's the new place? Get your Wi-Fi figured out? I got T-Mobile 5G home internet and I'm loving it. It's the fastest 5G home internet. My old provider was so slow. Mine is slow. And only 35 bucks a month. Do I need to try this? Try it for 15 days. Love it or get your money back. I love it already. There are over 250 reasons to join T-Mobile. Find yours and check availability at T-Mobile.com slash home internet. That's this according to Oculus V test intelligence data. Second half of 2025. All rates reserved for D money back guarantee within 30 days of cancellation. With the Discover Cashback card, it's payback time when you earn cash back on everyday purchases.

0:36Activate and earn 5 % cash back at different categories each quarter on up to$1 ,500 in purchases. That's 5 % cash back at different places each quarter, like grocery stores, on gas, and at restaurants. It pays to Discover. Terms apply. See discovery.com slash five for details.

1:02My guest today, Bapu Jenna, is a medical doctor. He's a PhD economist, and he's unbelievably creative. What happens when you mix those three things together? You get someone who's finding ways to answer important questions that no one else can. And the good news is, if you like what you hear today, He's got a brand new podcast on the Freakonomics radio network called Freakonomics MD.

1:27Welcome to People I Mostly Admire with Steve Levitt. I first met Bapu when he took one of my classes as a second-year PhD student at the University of Chicago. It was obvious to me that Bapu had immense talent, one of the most special students I've taught. But I also thought he must be a little bit crazy. Who does an economics PhD and an MD simultaneously? But I've always liked people who are a little crazy, so I did what I could to help him out while he was a student, and I've kept an eye on his progress since then. And wow, even I've been amazed at what he's done, more or less single-handedly bringing Freakonomic-style approaches into the mainstream of medicine.

2:13A couple months ago, I read a story in the New York Times researchers had used the timing of birthdays to learn about the transmission of COVID. And I thought, wow, what a brilliant idea. And then I read a few paragraphs further, and there was a quote from you, Bapu. And I said, of course, it had to be Bapu. No one else in medicine thinks this way. So that's a pretty high compliment. I appreciate it. Thank you. So partly why you think differently is that you are not just a doctor, but you also got a PhD in Economics from the University of Chicago. And I think back in the day, you even took one of my classes, didn't you?

2:50Yeah. You may not remember, but you were on my doctoral committee. It's a long time ago. I was on your committee, but it was fake. I didn't really advise you. They just needed a third person to sign. I would love to take credit for some of your success, but you and I will both agree. I probably didn't help you very much when you were a grad student. It's a thought that counts. So before we get into all the interesting stuff about you and how you think about the world, I think it's really important that we describe the standard way of thinking in medicine. And in medicine, which is true really in all the scientific disciplines, randomized experiments are seen as the gold standard, right?

3:26Yeah, that's exactly right. Medicine is one of these areas where you start with this biological hypothesis about what drug might work and how it might work. But ultimately, when you move from the bench to the bedside, there's so many things that can go wrong. These are high-stake decisions, and what you think might work may not actually end up working. And that's why we do these randomized trials to give doctors a good sense of what works and what doesn't work. Okay, and that makes perfect sense because in medicine, what we care about is causality. And we want to know, if you take this drug, will it make you feel better?

4:01And randomized experiments are the gold standard of causality because you're able to hold everything else constant. Very one thing, do I give someone a placebo or a drug? and then see whether the people who got the drug do better. But randomized trials aren't the only research strategy used in medicine. There's also this thing called epidemiology. Yeah. Why don't we have randomized trials for every single decision we make? They're costly to do. It takes time to recruit patients. Not all patients want to participate in an experimental drug. It sometimes takes years to get this sort of information.

4:32So that's why as a field, as a discipline, medicine has had to rely on other approaches to trying to understand causal questions like, does drug A do a better job at treating disease than drug B? And so there's this field of, I would call it clinical epidemiology, because epidemiology does a lot of different things. It studies the spread of disease. But this field of clinical epidemiology is really designed around trying to use real-world, typically historical or observational data to understand whether one treatment works better than another. And in that data, you have information on sometimes tens of thousands or millions of patients who receive one drug versus another.

5:13And then a doctor or an epidemiologist, or maybe both, will work with that data to try to figure out whether that drug is better than some other drug. Okay, just to be clear, what differs between this and a randomized experiment is that nobody's randomized who's getting the drug. Some people are getting one drug, some people are getting another drug, and that's choices that they've made or the doctors have made, not a randomization. Yeah. Like let's suppose you have a population of patients with cancer and you want to know whether or not a brand new, fancy, expensive oncology drug results in improved survival.

5:46One way you could do that is you could do a randomized trial where you randomize some patients to receiving that drug and other patients to receiving either placebo or standard of care and then measure the outcomes, in this case survival. The other way you could do that is look at historical data on patients who received that drug and compare their outcomes to patients who didn't receive that drug. Now, what happens if you find that patients who receive that fancy new drug have worse survival? They're more likely to be dead in the next year. If you look at that data, you might conclude that the drug actually harms patients.

6:19But what if the patients who are offered that drug are the ones who have worse cancer, who have passed through all other therapies up to that point and are really left with one option, which is this new experimental drug, in that case, you would reach the wrong conclusion. It might be the case that the drug actually works, but because you're not comparing like to like, you're comparing patients who are not similar in the treatment arm, which is this drug, and the control arm, which is a different drug, you're going to come to the wrong conclusion. Okay, exactly. So what you're saying is in clinical epidemiology, what you're looking at are correlations between did you take the good drug and did you die?

6:57But if all other things aren't held constant, like they are in a randomized experiment, then you can come to the wrong conclusions. And of course, epidemiologists are not stupid and they understand this and they try to do the best they can to control for other factors, but they're also limited by what data they have available or what data they think to collect. But you're always left, at least I'm always left, a little bit uncertain, maybe a little bit, I'm often very skeptical at the end of the day when as a consumer of the newspaper, I read about these epidemiological studies and I'm told to do one thing or another.

7:35Wait, are you telling me that eating peanuts at age six doesn't cause dementia at age 66? You don't believe that? I gotta retract my new paper then, okay. Okay, economics and medicine have something really important in common, namely that both disciplines care about the answers to many questions where it's hard to do randomized experiments. So I'm not allowed to induce massive unemployment in some cities for my research to see what the effect of unemployment is. Or even in my own studies, I'm really interested in the effect of prisons on crime. And there's no way in the world I'm ever going to convince a random set of states to let 20 % of the prisoners out in that state.

8:14And maybe another state will lock up another 20 % of prisoners. Just not going to happen. Totally impossible. And so what economists have put an enormous amount of effort and thought into is developing models that use this non-experimental data, everyday data, but that might plausibly have a causal interpretation because we look for special settings that mimic a true randomization. And we call it a natural experiment. I actually prefer the name accidental experiment because I think it's a better description of what happens. And that's really what you do, but you do it in medicine. You know, correlation is not the same thing as causation.

8:50That statement, it's endemic in the way medical students are taught. And that's good and bad. It's good because I think it introduces a healthy dose of skepticism in anybody who's training to be a doctor about how to interpret a new study that shows there's a link between red wine and whatever outcome. Now, the problem with that, though, is that it goes so far as to make doctors think, or at least the doctors I know think, that really the only way that you can get at questions of causation is a randomized trial. Exactly. And it goes so far as if you're writing an article for a medical journal in which you're using a natural experiment, or to borrow your words, an accidental experiment, any language that reflects causation, this study shows that X caused Y, or the effect of X on Y was this.

9:37Any language like that is typically removed from the manuscript because there's this belief that you can't use observational data to reach causal conclusions. And I think that's a challenge that our field has to overcome. Let me just try to explain how I explain natural experiments to people. A randomized experiment, a real randomized trial, has two key features. The first one is that the treatment group gets treated different than the control group. Okay, we all know that. The second key feature is that except for the treatment, we would have expected the treatment group and the control group to have the same outcomes on average.

10:13Okay, so you start from that and you say, so a good natural experiment just tries to mimic those exact features. We go out in the real world and we try to find settings where otherwise identical people, essentially by chance, get treated very differently. So if we can do that, we've more or less mimicked a randomized trial without actually running a formal experiment. You've had a lot more success convincing your colleagues that natural experiments have merit. So what words do you use when you try to describe what a natural experiment is? Steve, I don't know that success is the word I'd use. You haven't seen all the studies that I've tried to get published.

10:51You only see the successful ones. That's a different bias. You know, the way I describe it, it's very similar. But when you see a paper in a clinical journal that presents the results of a randomized trial, most often the first table, the first exhibit in that study, shows the characteristics of patients who received a treatment and a control. And you can look at that table and you can see that the characteristics almost always are nearly identical between the two groups. And that gives doctors, I think, a lot of faith that these two groups are balanced and we expect the outcomes in those two groups to be otherwise similar were it not for the fact that one group is going to receive a treatment and another group is going to receive a control.

11:34And therefore, any difference that we end up observing between those two groups in the outcomes is attributable to the receipt of that treatment as opposed to underlying differences and characteristics between those two groups. What I've tried to do in most of the natural experiment studies that I published in medical journals is try to, you know, just essentially replicate that table one. So if I'm looking at what happens to patients who are hospitalized during the dates of a national cardiology conference when all the cardiologists are out of town, and I want to know the answer to whether or not care is different and whether that difference in care leads to differences in outcomes, let's say mortality at one year later, the first question someone should rightly ask is, well, Bapu, how do you know that the patients who go to the hospital when cardiologists are in town just aren't different from the patients who go to the hospital when cardiologists are in town?

12:27And I start by saying, well, do you have a reason for why they would be different? Do people choose to have heart attacks? You've got to fight that criticism. And so the way I fight is, I just show in a table one, look, the patients who are hospitalized with a heart attack during the dates of the American Heart Association meeting are identical to patients who are hospitalized with heart attacks during the surrounding weeks of the year. And so what happens to those folks who do show up at the cardiologist's office only to find out the cardiologists are all at the national convention? What do you think, Abbott?

12:59They actually do better. That was a shocking thing of that study. I remember being in the cardiac intensive care unit around the time of one of these meetings, and the composition of senior doctors was different. And I just thought to myself, I wonder what happens when patients get hospitalized with these acute medical conditions when these meetings are happening. I thought they would do worse because either staffing was lower or the most experienced doctors would be away at these meetings. And what we found was the opposite. We actually found that they did better. And if we're interested in making people's lives better, helping them live longer, what the hell is happening that's different in the hospital during those two periods that's generating an improvement in survival that is larger than statins and aspirin and beta blockers and everything else that cardiologists do.

13:46We know that's probably a causal effect of something that's happening during that meeting date compared to other dates. What we don't know and what a randomized trial can't always tell us is what's the mechanism of that effect. And one thing that we found was that rates of a particular intensive procedure were lower. And so that led me to hypothesize that maybe what's going on is during the dates of these meetings, it's just that less intensive care is performed. And on the margin, the people who would have gotten that more intensive care aren't getting it. And the risk benefit profile for them would have been such that they'd have been benefited by not getting the care.

14:21So has anybody followed up to see whether in fact it is hurting people on the margin? No, they haven't. I'm sure you've seen this problem in Economics, which is, that was a controversial study. It had this very unexpected finding. And I thought to myself, wow, let me replicate it because there'd be a lot of interest in showing that this finding could be replicated. We had the hardest time getting that study published. And you know what the journals would say to me? They said, you already published this finding, so there's nothing novel here. And that actually has made me think a lot about this replication crisis.

14:53There's no incentive to replicate findings, at least for these types of findings. And so now I write a study and I leave it to the world to replicate it. Yeah, it's so troubling. I'm sure it's true in medicine. It's been extremely true in psychology and economics that so much of what is done cannot be replicated, randomized trial or otherwise. But the incentives within these professions and at the journals are totally screwed up when it comes to trying to sort out the truth. You think that all these journals and all these researchers should be after the truth, but really it's a very different game that's being played.

15:26a game of how do I get published and how do I get citations from my journal? Look, I don't have the answer, but I think if outsiders knew how academic publishing worked, they would be discouraged by it. Stephen, if I knew how academic publishing worked, I would be encouraged by it.

15:49I once made a foray into trying to do natural experiments in medicine, and I got burned so badly that I've never gone back. I started, I think, in 2004 when you were still a grad student, and I think it's quite possible that you might have been my inspiration. And I'll give you credit for what's going to turn out to be maybe the most unsuccessful project I ever worked on in my life. I'll take credit for the lost year of my life. Okay, so the idea is really simple. When a patient shows up at the emergency room, he or she has no idea what doctors are going to be working at that time. And if some doctors are better than others, then as a patient, I can get lucky or unlucky depending on which doctors are working.

16:33So I didn't assign any patient at random to any doctor, but a set of patients who show up are essentially random. And so I can compare outcomes across the time period, say a Tuesday, July 29th. Let's say you're working that shift. And let's say the July 22nd, Tuesday, I'm working that shift. Then if the patients who come in on July 22nd, have better outcomes than on July 29th, I'm a better doctor than you. That would be the kind of inference one would draw from these data. Okay. Does that setup make sense to you? Yeah, absolutely. Yeah. Okay. So I was working with a hospital. They had an ER. The ER was staffed by three or four doctors at a time.

17:09So I was able to get years worth of data and outcomes on the patients. And I was able to find some patterns in those data. And I was super excited. And I went to present at what's called the grand rounds, where the doctors sit around and they listen to someone who's supposed to know something, explained a new concept to him. And it was unreal. The reactions I got were confusion, anger, a consensus in the audience that I had no idea what I was talking about. So then I showed the results to my dad, who's an eminent medical researcher, probably has 500 publications. He's got lifetime achievement awards in medicine, and he's a super smart guy.

17:49And I explained my method, and he listens to the whole thing and he says, wait. So you're telling me that to figure out whether I'm a good doctor, you're gonna look at the outcomes of all the patients who come on my shift, not just the ones that I take care of. Okay, and that's exactly right. You understand, it was this moment of triumph. I had finally explained it. And my dad says, that's the dumbest thing I've ever heard of in my entire life. If I didn't treat the patient, It's not my fault if he has a bad outcome. That doesn't even make sense. So that was the last natural experiment research paper that I ever tried to do in medicine.

18:28I never got it published. So hats off to you that you've managed to rattle off a string of these papers. And I'm not sure we've really made it clear, but is it fair to say that when you started doing these natural experiments in medicine, literally were you not the only person publishing studies like that? I would say this is what I'm known for. There is a Canadian physician named Donald Reddemeier, who I would think is a pioneer in this area. He did a lot of work with Amos Tversky and Daniel Kahneman years ago on behavioral economics. But besides that, there's not a ton of people who really specialize in natural experiments.

19:02How many medical researchers are there publishing journals, would you guess? Oh. 10 ,000, 20 ,000? Yeah, probably more than that, yeah. Okay, 50 ,000. Okay, let's see. there are tens of thousands of researchers. And there are, I think by what you just said, let's be generous. Maybe there's a dozen people like you who are looking for natural experiments. If you look at economics, I would say something like a third or a fourth of the papers that are published in top journals are taking advantage of natural experiments. It sure seems to me like medicine's making a huge mistake by not focusing more on natural experiments.

19:41Yeah. So there's a ton of research that's what you might call health policy. So what's the effect of Medicaid expansion in one state on some outcome? There's much, much less work that uses natural experiments to try to answer questions that might be of clinical importance to a doctor, what therapy to provide or not provide. To your point, it's the minority of observational studies in a medical journal that use these sorts of methods. Whereas in economics, you'd be taken to the coals if you didn't have a good natural experiment. Now, I think the challenge though, and this is what I face is, it's hard to find these natural experiments.

20:16I've gotten good at it and this is the way my mind thinks nowadays and I'm sure that's the way you think about the world. But it's hard to find natural experiments for every question that you might want to answer. And the view of a lot of medicine has been, well, let's just try to answer it anyway. Take, for example, red meat. So tons of studies that look at the relationship between red meat consumption and any number of outcomes, mostly cardiovascular, hopelessly non-causal, right? There's no way you can look at those studies and think that they're causal. And I, for years, tried to think of situations where I could find a natural experiment around beef consumption and looking at whether or not, for example, when the mad cow disease outbreak in England was like 10 or 15 years ago, I looked to see whether or not there was changes in cattle output and consumption in parts of the world, and maybe that could then be used to show changes in cardiovascular outcomes.

21:06That was hopeless. I didn't find anything there. But it's hard to answer a question that people would want to know the answer to is, should I eat more or less red meat? Look, there's a million questions I'd love to answer that I haven't been able to answer with a natural experiment, either because there wasn't one out there, or more likely, I just wasn't clever enough to find it. But it sure seems to me that the medical profession is making a systematic error in the sense that by putting so much effort into randomized experiments and so much faith in this epidemiology, but missing out on the middle ground just seems like an obvious mistake to me and one that somebody, somehow the profession should change.

21:45And you're the guy to do it. I think that's a good name of a podcast. I'm the guy to do it.

21:54You're listening to People I Mostly Admire with Steve Levitt and his conversation with economist and physician Bapu Jenna. After this break, they'll return to talk about some of Bapu's research on COVID.

22:12With the Discover Cashback Card, it's payback time when you earn cash back on everyday purchases. Activate and earn 5 % cash back at different categories each quarter on up to$1 ,500 in purchases. That's 5 % cash back at different places each quarter, like grocery stores, on gas, and at restaurants. It pays to discover. Terms apply. See discover.com slash 5 for details. This episode is brought to you by Charles Schwab. Is there a right time to sell a stock? Are you taking the right risks with your portfolio? financial decisions can be tricky and often your own cognitive and emotional biases can lead you astray financial decoder an original podcast from charles schwab can help join host mark reepy as he offers practical solutions to help overcome the cognitive and emotional biases that may affect your investing decisions listen at schwab.com financial decoder hey chicagoland the wayfair store is in your neighborhood at edens plaza and wilmette finally you can feel the fabric sit on the sectionals and even open the refrigerators.

23:17Plus, our in-store designers will help you bring it all together with free one-on-one design support for any project on any budget. Yep, we said free. Oh, and did we mention the cafe? So what are you waiting for? Come see all that's in store. Visit the Wayfair store today at Edens Plaza and Wilmette. Wayfair, every style, every home.

23:40Morgan, what do we have on the table today? Steve, I want to check back in on something. In one of our episodes with Sendhil Malinaten, you said...