Inside the evidence revolution — how decision-making became data driven

0:00Business is booming, but you know what really helped? Zero, the accounting platform for small businesses. I used to think unpaid bills were just part of it. Ugh. But batch payments? Wow. Automated accounts payable? Even better. All in one place? That's a big one. And I can keep track from my phone while I get the job done. No bills in sight. What a life. Do right by yourself and your business and search XERO for your free trial. Conditions apply. We asked you here because we care. And your car insurance doesn't. They've raised your rate every year and you haven't even checked. I thought that was normal.

0:42A lot of people do. That's why there's Jerry. Jerry pulls up to 20 quotes from top insurers in minutes, shows them side by side, and lets you switch in the app. No jumping sites, no re-entering info, no spam calls. It helps you cancel your old policy and keep checking for better rates. We want better for you. Jerry does too. Don't settle. Visit jerry.ai.acast.

1:10Hi, Benjamin here. Welcome to episode 14 of Nature Hits the Books. In this episode, I'm joined by science journalist and colleague of mine here at Nature, Helen Pearson, whose book Beyond Belief comes out next week. Helen's voice will no doubt be familiar to folk who listen to our podcasts and watch our videos. And Beyond Belief is Helen's second book, which looks at the history of the use of evidence rather than opinion in decision making. The book traces the course of the movement in various disciplines, such as the rise of evidence-based medicine in the 1990s, 1990s, looking at the rebels who led the charge, the barriers they faced, and why the use of evidence is crucial at a time when misinformation is rife.

1:57A few of the topics we chat about in this podcast.

2:04Helen Pearson, welcome to Nature Hits the Books. Thanks very much for having me. Not at all. So beyond belief then, why was now the time to write this book and who were you hoping to reach with it?

2:15Helen Pearson: Well, now is the time to publish the book, I would say, because there's a lot of conversation about, in some ways, rejection of science and evidence. You know, for example, what we're seeing in the US, it's almost become more relevant, I think, as I've written the book. And I really wanted the book to remind us that we're on this much sort of longer journey, if I can use the overused words, into incorporating evidence into our decisions. But the people I want to reach are perhaps science interested people who might be struggling, maybe with the kind of overwhelming amounts of conflicting information and misinformation in the world, who I hope might be a bit more empowered actually by evidence after reading the book.

2:52Well, the subtitle of the book, if I may, is how evidence shows what really works. And you do cover a bunch of different topics. Obviously, we'll talk about a few of them today. What surprised me was that we live in a world where evidence-based decisions, that system is relatively new. There are a few examples from sort of antiquity onwards, but really it's kind of a new thing.

3:10Helen Pearson: Yeah, and that was one of the huge surprises, I think, to me, and maybe another motivation in wanting to write the book. So I tell the story of the rise of evidence-based medicine, which is perhaps the kind of pioneer, shall we say, in this use of evidence in decisions. Not so long ago, sort of 40s, 50s, we were in the world of eminence-based medicine. And I tell the story in the book of these early pioneers like Ian Chalmers and Dave Sackett, who were training in hospitals and medical school in the late 50s, early 60s, and actually realised that a lot of what doctors were doing, it would very much vary by doctors.

3:44Helen Pearson: So one doctor might say in obstetrics, this is the case where we have a cesarean section, or bed rest is really important, and then another doctor would say no such thing. And these kind of pioneers thought, well, this is a bit odd. Why aren't we giving out correct advice? Or what advice should we be using? And they realised there was a different way to do things, which is basically there's a lot of evidence out there from things like randomised controlled trials, where treatments have actually been tested. And so they embraced this idea of what became known as evidence-based medicine. And that term, surprisingly, was only published in the literature about 1991.

4:17Helen Pearson: Now, I don't want to suggest that that whole approach was brand new, you know, in the 1990s. It obviously took a few decades to develop. But nevertheless, 35 years ago, since that term came along is really surprising. And it does seem that a lot of the characters who appear in your book, like the two you've mentioned there, you know, they seem to be a group of folk going, hang on a minute, that doesn't seem right. And they enjoy maybe kicking back against the status quo. Yeah. And I love that about the characters in the book because they all have this sort of rebellious streak. They're a bit maverick because they had to be willing to speak truth to power.

4:48Helen Pearson: Nobody really likes being told what they're doing is wrong. And it's uncomfortable when evidence shows that what we have been doing is wrong. And so all of them were willing to speak up and say, hold on a minute. Why are we just following conventional wisdom here? Well, let's talk about some of the examples and evidence-based medicine there. Your book opens with that. One of the key players is the Cochrane collaboration. Tell me about how that came to be and the sorts of things that they were looking to do. So the Cochrane collaboration has been really influential, I think, in cementing this idea that medicine should be based on evidence from research.

5:19And that traces back to this pioneering researcher and Dr. Ian Chalmers. And he actually was inspired by Archie Cochrane.

5:26Helen Pearson: That's where the name Cochrane collaboration comes from. And Archie Cochrane was sort of one of the early people who put forward in this seminal book called Effectiveness and Efficiency in 1972. That book really put forward and put on the map this idea that actually in order to know what really works in medicine, we need to test things, particularly in randomised trials, which are a very powerful way of showing whether a treatment has an effect or not. And he also challenged the medical profession to bring together the evidence, not just look at one trial, but to look at a whole body of evidence.

5:55And because of this, Ian Chalmers went off on this huge mission to bring together all of the evidence on what was actually effective helping women and children in pregnancy and childbirth. Because at the time, there was all this conflicting opinion and conventional wisdom. And certainly in the 70s, quite rightly, women were getting pretty annoyed about this. So he did this, it took 10 years, he comes out with these two massive tomes, these books, which I've now got on my shelf with hundreds of systematic reviews in them, systematic reviews of this particular kind of very rigorous form of synthesizing evidence together, shows that tons of practices in obstetrics and gynaecology are basically based on almost no evidence at all.

6:32For example, shaving women's pubic hair during labour, these sort of invasive practices which were based on very little evidence and really started to change practice in that discipline. And inspired by this, Ian Chalmers thought, well, why don't we do this for all specialities in medicine and bring together the evidence to find out what really works

6:50Helen Pearson: and brought together this group of fellow rebellious types in the 1990s and launched what we now know as the Cochrane Collaboration. So this idea of evidence synthesis then is quite a particular term related to this and part of this as you say is pulling together a lot of disparate sources of information right and trying to provide the best course of action based on what this evidence suggests. Yeah so evidence synthesis I think it is a term that a lot of people aren't familiar with and I wasn't really actually going into this which is kind of surprising as a science journalists but it's this really fundamentally important part of science which i think is sometimes overlooked it doesn't get any of the glory that the randomized control trial does but if you think about it it makes absolute sense that we're often as scientists or science journalists we're kind of chasing after the latest study but one study can be wrong it can be an outlier it can be poorly done by chance it can reach the wrong results so the idea of synthesizing evidence is bringing together all of the information the evidence from science which can answer a question Now, in a systematic review, often that's focused only on randomised controlled trials, but there are lots of different ways of bringing together of synthesising evidence.

7:59Helen Pearson: And in the book, I looked at the different use cases. You know, if you're working in policy, you need your evidence synthesis tomorrow, right? So you might not have time to go and do the perfect two-year systematic review. And we'll talk about different spheres in a bit, but keeping with medicine, maternal health and early years health, that does seem to be a big focus of the Cochrane collaboration from the start. But there was this pushback among doctors to this big analysis. And that pushback is a theme that comes up for different topics and maybe echoes the world we live in now. Yeah, we're seeing a lot of people questioning evidence, political leaders pushing back on evidence.

8:35Helen Pearson: But if you look at the kind of history of this evidence revolution, that's what I call it in the book, opposition to evidence has come up again and again. So these early pioneers of evidence based medicine, Ian Chalmers was compared to a terrorist. They were called troublemakers, because again, people don't like being told that what they're doing is wrong, basically. And that's, I think, happened again and again, as pioneers in each field have suggested that there's a different way of doing things. And I noticed one of your footnotes suggests that although medicine is where this stuff maybe came at the forefront of practice, it turns out that astronomers had beaten the medics to the punch in terms of synthesizing evidence by several hundred years.

9:12Helen Pearson: Yeah, I mean, I think a lot of discoveries in science sort of happen again and again, and often around the same time by multiple people. Do you know what I mean? It's sort of in the air, this idea of synthesizing evidence. I mean, I interviewed a statistician who basically said, evidence synthesis is discovered in each field at the moment at which the body of evidence is big enough to need it. And astronomers were doing that a bit earlier than medics were, you know, it arose in medicine in some ways, because the randomized control trial took off and all of these trials were emerging. And all of a sudden, we need a way to bring them together and say, okay, where do you know what I mean?

9:42Helen Pearson: the signal in the noise here. And what about other spheres? I mean, there's a few that I wasn't necessarily expecting. You'd imagine economics, and of course, there's plenty of evidence-based trials working out the effects of different interventions in different populations. But evidence-based parenting and evidence-based management, were these a surprise to you? I'd certainly never heard of them. Definitely. That was the joy of reporting this book, is I got to go into all these fields that I wasn't necessarily familiar with before. So in international development. I mean, again, there's been this evidence revolution that's taken place over the last 25 years, let's say, where development economists have started testing whether interventions, programmes to help families living in poverty are effective.

10:22I looked at evidence-based policing. There's a global movement in evidence-based policing.

10:26Helen Pearson: There's a huge database of over 7 ,000 rigorous studies which test, for example, is hotspot policing, which is focusing on the places where crime takes place, more effective than randomly patrolling the streets, which by the way it is. I got to go to Scotland Yard and talk to a huge proponent of evidence-based policing there. And, you know, I looked at evidence-based conservation. Parenting, I'd say, is a field that's got a long way to go in terms of embracing evidence. Like I'd say parenting and also business and management, like those are fields which I think are very rocked by fashions and fads in practice.

11:00And maybe the sort of idea of using evidence maybe isn't as well cemented or has found it harder to gain traction. So each of these different fields I look in also look at conservation, government policy, education, for example, there have been different barriers, I think, to evidence along the way. And if someone's listening to this podcast in an airport right now, and goes to the bookshop, they will find a management guru saying this is the way to manage people. And you actually talk about yourself. I think you might have been my manager, Helen, for a little bit. I don't really want to do my annual appraisal right now.

11:31But did researching this book open your eyes to the way you did things or the way that things should be done? Absolutely. Yes, of course, because I did. I managed a team, you were in it. And I hadn't really, I think like many people thought of that parallel that, okay, so in a team, you're doing things, right? Like you're doing performance appraisals or you're making changes in your team. But I didn't really stop to sort of go, I wonder if there's evidence that shows this will

11:55Helen Pearson: help my team in the same way you might ask before you take a medicine, is there evidence that it will help. And one of my favourite stories in the book is of this character called Eric Berenz, who is sort of waging this lonely battle to turn business and management into an evidence-based field. So he was a manager, realised that people around him all had conflicting opinions and were using conventional wisdom to make really important decisions in their teams, goes off, learns about evidence-based medicine, decides that management needs to change and become evidence-based management. And there is a small movement to do that.

12:26But it has been a big battle. And I think that's quite well captured by the fact he now is running his evidence-based management centre

12:33Helen Pearson: from a bus driving around Central America, which just kind of, I don't know, epitomises to me, it's like this sort of metaphor for how difficult it has been, I think, to convert this field. He got so into it, he did a PhD as well, I believe. He did one of the first PhDs in evidence-based management, yes. And to go back to just performance appraisal, which I think is quite a nice example, you know, we all do performance appraisals, we don't really think about them, that's just something that happens. And I spoke to one advocate of evidence-based management, I think called them a soul-destroying ritualistic charade because he was so unconvinced that there was any evidence that supported the idea that they actually improved people's performance.

13:10Helen Pearson: And actually, I mean, I've looked at one of the sort of evidence syntheses around this and the evidence base is mixed, shall we say, but there seems to be stronger evidence that effective feedback to people potentially can improve their performance more than just this process of having this sit-down appraisal. Well, we are, I think, one floor below the HR department, so let's hope they don't necessarily hear that right now. And I will say that as well as appearing in the book in the context of management, your family also appear at points as well. Now, you mentioned that evidence based parenting, maybe he hasn't gone quite as far down the timeline as medicine or economics.

13:44Did you look back to the past in that sort of situation? Yeah, completely. I mean, I don't want to suggest that there's no evidence used in parenting, because of course, I talked about how evidence in pregnancy and childbirth, there's a wealth of evidence there. I think it's more around the sort of parenting practices, perhaps, that we have around bringing up our children. It's just it's kind of riven, again, with books and conflicting advice. And it's really hard to know what's evidence based and what's not. The example I wrote about in the book, personally, and again, I don't want to suggest we look at evidence, and it gives us these perfect examples, and that I'm a better parent than anyone else.

14:14It's a really difficult thing. And evidence often doesn't give clear answers. And it's one factor in the decision. It's just it can make us perhaps a little bit more informed about what we're doing. One body of evidence I've looked at for the book and also for nature was around screens, smartphones and whether use of smartphones and social media are basically driving this kind of epidemic of mental health problems in adolescence, which is this really obviously important topic. Some experts suggest so. The evidence base is much, much more mixed. And if you actually look at most of the syntheses of evidence, then if anything, they're detecting maybe a small effect.

14:50But there were conflicting views about this evidence. And I don't think at the moment it provides the clear answers that me as a parent and probably you want to hear. But sometimes knowing that, is it self-empowering? Because like for me, I was like, OK, this is what the evidence says. It does seem to make common sense that if my children are on screens for four hours a day, they're not doing other things like exercising, seeing friends, doing their schoolwork. And so anyway, I tried to perhaps adjust the family policy a bit, should we say, informed by the evidence, even if it didn't give me perfect answers.

15:21I mean, it is such a hot button issue right now, certainly social media and the use of smartphones by adolescents. And we've covered it on the main Nature podcast. And from work that I've done and from what you're saying there, it's become such a thing so quickly that the actual evidence gathering is lagging behind only because it takes so long to do a well planned study. And yet people want the answers right now. That's right. And the challenges around screens, you can't do a perfect randomised trial.

15:50Helen Pearson: I mean, screens are now so embedded in our lives, you can't take a cohort of babies and say no screens for you for the next 18 years. So it's a bunch of observational studies. And it's classically difficult to kind of tease out causes from those. But I think a prime example of that is, for example, around social media bans. So everyone really wants definitive answers as to whether to ban adolescents or children from social media. Of course, the evidence is not there. Australia's made that change and researchers are watching that but decisions have to be made in the absence of evidence. And in terms of getting the perfect study do you think there's like a deification maybe of the randomised control trial?

16:24Yes definitely and that's one of the big I think pushbacks on the evidence-based approach that's happened in medicine and across these other fields is that often the kind of early advocates of it were like real fans of the randomised trial and the randomised trial is an amazing method you know it can provide these clear answers but it needs to be recognised that it's one form of evidence. Sometimes you don't need a randomised trial or randomised trial is very difficult to do. I mean, an example I give in the book is you can't randomise

16:52Helen Pearson: women to give birth at home or in hospitals to find out which is safer, because obviously, there's a huge choice people want to decide for themselves. You can't randomise children to grow up in poverty or not. So there have to be all these other methods. And I think this idea that the randomised trial is superior to all other forms of evidence really annoys some people. In a way, I think I emerged from my deep dive into evidence feeling much more inclusive towards evidence, like there's lots of different forms of knowledge which are coming out of research, which are really valid. So perhaps the challenges of how to bring those things together.

17:26I mean, I heard the news this morning, Helen, and there was a politician on saying we're following the evidence. And that has now become potentially a situation where this is maybe used in a way to add weight to a decision when there's maybe not quite as much evidence on one side as the other. Or it can be used to defer doing something due to a lack of evidence. Do you think this is the flip side to the evidence based movement? Or is that just a symptom of what the world's like right now?

17:51Helen Pearson: Yeah, I mean, I think that that's a risk. And people talk about policy based evidence where you've decided your policy and you just want to add the evidence afterwards or spray on evidence, another term that I quite like. So I think there is definitely a risk. And that's partly because of the word evidence, which I think is quite slippery in that in the book, I had to define actually quite carefully what I meant by evidence. So evidence in general means information that indicates a belief or proposition is true or valid. Now, in the book, I'm talking largely about empirical evidence from scientific studies, rigorous observational and experimental studies.

18:26But there are other forms of evidence. OK, so there's evidence in a court of law.

18:30Helen Pearson: There's lived experience, which is a completely valid form of evidence. It's just sometimes we're talking about different things. But either way, I do think there's this risk that people just sort of hitch the word evidence to something. And it gives this gloss of legitimacy, which may not be well deserved. And another maybe knock against the evidence-based movement is that a lot of the organisations and the drives for evidence collecting have come from the global north. And maybe a sense that certainly in terms of things like economic interventions, that one idea that worked in one city will work everywhere else, where it might not work in the neighbouring city or the same country or, you know, other side of the world, things like that.

19:06So I think that's another really important pushback against evidence that comes up quite a lot. I mean, I have an example in the book about a trial that was done that showed that incentives to encourage families to vaccinate their children in India, and these incentives were providing a bag of lentils if you brought your child to be vaccinated. So that seemed to be effective in India. Now, the danger is that we go, okay, that works here. So it's going to work in somewhere else. So there's just this really strong message that you can't kind of generalize evidence like that. And that's why some people don't like to talk about what works.

19:38They say what works for who, where, when. And there's also, I'd say, this evidence movement, which had its roots in the global north, in Western countries. There's a really big push now for this to go global. And certainly, for example, one of the current initiatives, which is how to better synthesise the vast amount of knowledge in the world and use AI tools to help, is actually becoming much more rooted and potentially being led by global south countries. And here we are now, several decades since the evidence movement really kicked in. Where do you think it goes now? One of the huge challenges is that we've just got so much information now.

20:17So part of it is just finding more sophisticated ways to synthesize it and get it to the people who need it. And so there's a lot of excitement now about using AI to make sense of vast amounts of information. The danger, of course, is that, you know, an LLM could give you, if you ask it to tell me all the evidence on such and such, we might come out with the wrong answer. So there are some really serious efforts now to develop tools that can do even something like a really rigorous systematic review much faster than could be done before. Because, of course, isn't that what we all want? That's actually the kind of vision of my evidence champions is that you can literally push a button and you will get an authoritative synthesis of what the world knows about a question, which is kind of amazing.

20:55Right. So I think that that's one of the big challenges. But the other one, and this has been encountered in lots of fields, is that people get excited about evidence. They produce all this evidence and they realise people aren't using it. So, for example, there's tons of evidence in education that shows effective practices, for example, like metacognition, which is helping children understand what they're learning, or phonics or feedback or whatever. But it isn't necessarily embedded yet in teaching practice. And that's where I think almost the frontier is for a lot of fields. And certainly with the education one, I think the pushback in many cases is money.

21:28for a lot of these interventions, I suppose, is that this would work. Now can someone go and pay for it? To some extent, but some of these approaches aren't necessarily expensive. So one of the most effective practices in education is providing specific, detailed feedback to children. Now, that's just a change of practice. I mean, I'm not suggesting that's easy for teachers. Teachers are overstretched and underpaid. But I think it's more becoming about behaviour change. And we know that behaviour change is hard, right? I mean, it's really hard to go and exercise more well it's quite hard to suddenly start using evidence in your practice and there's sort of a lesson for that in medicine in that I think one of the reasons that it became embedded in medicine was that evidence-based medicine emerged in medical schools so students were learning about it as they trained to be doctors right and they still do but that's not necessarily the case if you're training to be a police officer or you're trained to be a teacher and I think that that's one of the barriers which these fields are running up against.

22:22And that can be a hard thing to learn as well because full courses are taught on how to read a scientific paper or how to work out which evidence is maybe of better quality than others and and that takes time yeah it's not just that so most people don't realize that this sort of bedrock of evidence which there is in medicine there's this vast quantity of systematic reviews which were synthesized the studies and then there's clinical guidelines which basically tell doctors how to treat people in a particular situation already based on those systematic reviews so it's almost i think i called it in the book pre-processed evidence so it's not like you're expecting all the doctors to go back and read the journals of systematic reviews it's already done and that's what i mean by it's embedded in practice and we don't so much see that in other fields but the other really important thing that happens in medicine is doctors are held to account if they don't use evidence so obviously if you treat someone in a way there's some you know bad outcome because you haven't followed the evidence-based guidelines you could be sued you could be disallowed from practicing medicine now we don't necessarily see that level of accountability again in another field like

23:22Helen Pearson: Like, you know, in conservation, if you mow the field of flowers or something, and that was the wrong thing to do, you know, no one's necessarily going to hold you to account. And broadly, how do you think this brand new, bright, evidence-based world we live in now, how has it changed the world? Do you think? Is it a better place? Well, I've come out of it quite an evidence optimist, I suppose, an evidence fan. I do think the current opposition to evidence is really important to understand and take seriously. I'm literally reporting a piece at the moment about whether there's a crisis of trust in science and if you look at a lot of the statistics actually the sort of level of public trust in science is quite high and has been consistently high but there's a political polarization going on we've got political leaders who are undermining evidence I think understanding this history of the evidence revolution puts it in context because it's like well it's only 35 years since the word evidence based medicine was introduced so we're quite early maybe in this movement and it's hard for me to think that we're just going to take all this knowledge we've got and ignore it completely?

24:20Like, I think maybe it helps us understand that maybe this is just one setback on a longer path to using evidence in our decisions. And as we come to the end of this chat then, Helen, for folks who read your book, is there a call to arms or is there something you would like them to do that maybe you've learned as part of your journey? At the end of the book, I did want to give people some sense of what they could do next. I don't want to be naive about evidence. There are many caveats to it, but I provided a list of evidence hacks for people if they want to sign up to the evidence revolution. And the most basic of these is literally to be sceptical if we're encountering claims from an influencer, from a politician, asking, is there evidence behind this?

25:02Now, if there is a study, we can start to ask some basic questions about whether that's a credible study. For example, is it peer reviewed and published in the academic literature? there are plenty of problems with peer review and lots of bad research goes through and is published but it's this kind of crude indicator of quality shall we say one thing i do now as a science journalist is i don't just look for studies when i'm going into a new topic i actually look for

25:24Helen Pearson: systematic reviews which will give me an overview those are the kinds of things i hoped to empower people with while acknowledging that it's a complicated world and there aren't necessarily simple answers and what one story has stuck with you because obviously you started writing this book i imagine quite a long time ago but is there one that you could almost see in your sleep one story is the randomized controlled trial of beer glasses and that took place in britain and that you or i might unknowingly have been part of if we bought a pint around 1998 so this grew out of a discovery from a researcher called jonathan shepherd he's a surgeon he realized that a large number of the accidents that were turning up in his emergency room were due to accidents from glass some of them were these terrible kind of glassings where somebody puts a glass into somebody's face absolutely awful traumatic injuries.

26:09So he decided to run a randomised trial of toughened beer glasses versus regular beer glasses, which ran across a bunch of pubs. The pubs were randomised to change the whole beer glass stock. Discovered through this, I mean, it sounds obvious, maybe in retrospect, but discovered through this that indeed toughened glasses did lead to less glass injuries. And this helped convince the pub industry to switch to toughened glass. And I think there was an estimate of about 40 ,000 less glass assaults in England and Wales in the year after the pub industry made that change. That's just for the just the idea that something as kind of mundane as beer glasses might have been tested in the same method you know that we use to test whether a drug is effective I think is fantastic.

26:53Wonderful let's leave it there Helen Pearson thank you so much for being here. Thanks for having me.

27:00Helen Pearson there her book Beyond Belief is out next week. And that's it for episode 14 of Nature Hits the Books. If you have any feedback on the show, why not ping us an email to podcast at nature.com with the subject line Nature Hits the Books. The music used in this episode was called Cloud Jumping by SPD via Getty Images and Triple Scoop Music. I'm Benjamin Thompson. Thanks for listening.

27:55We'll be right back. Do right by yourself and your business and search X-E-R-O for your free trial. Conditions apply. Hi, it's Paige from Giggly Squad, and this episode is sponsored by Experian Boost. Summer glow up? Check. Credit glow up? Even better. Boost your credit scores instantly by getting credit for bills you're already paying. Your phone, utilities, rent, and insurance. I wish dating kind of worked like that. Connect your bank account, add those on-time payments to your Experian credit file, and your FICO score updates right away. You could instantly raise your FICO score by an average of 14 points with Experian Boost.

28:33Download the Experian app for free today. Results will vary. Users who received a boost improved their FICO score eight from Experian by an average of 14 points. See App Store or Experian.com for details. Results will vary. Not all payments are Boost eligible. Users who received a boost improved their FICO score eight from Experian by an average of 14 points. Some may not see improved scores or approval odds. Not all lenders use Experian credit files. and not all lenders use scores impacted by Experian Boost. See Experian.com for details.