AI, bounties and culture change, how scientists are taking on errors

0:00This message is brought to you by today's sponsor, Walden University. Set a course for change. Now is the time to make a real change in your life, career, and community. At Walden University, we give you the support and flexibility to get the W. Walden's online programs are designed for working professionals like you. You'll gain hands-on skills to take on real-world challenges and succeed. Your future is waiting, and Walden is here to help you achieve it. Take the first step to make your impact. Visit waldenu.edu. Walden University. Set a course for change. certified to operate by Chev. Hi, this is Kim and Penn Holderness from the Laugh Lines podcast.

0:39This episode is sponsored by Oli, your wellness partner for back to school and beyond. Back to school season is here. Packed backpacks, packed lunches and packed schedules are officially back. Oli Kids Sleep helps little dreamers wind down with a gentle blend of melatonin, L-theanine and botanicals for restful sleep. Translation, an easy way to put those school night negotiations to bed. And for the grownups, Ollie's sleep helps you fall asleep, stay asleep, and wake up feeling refreshed. Because as you know, the morning mayhem comes early. Visit Ollie.com and stock up on back-to-school must-haves today.

1:15That's O-L-L-Y dot com. These statements have not been evaluated by the Food and Drug Administration. This product is not intended to diagnose, treat, cure, or prevent any disease.

1:33I think we're on very shaky ground scientifically with the idea that alcohol in moderation prevents heart disease and makes us healthier. This is Tim Stockwell, the former director of the Canadian Institute for Substance Use Research. Now, that statement may surprise you. You've probably heard, I know I have, usually if someone is about to pour me a glass of wine, that a little bit of alcohol can actually be protective against things like heart disease. And there's a reason for that. This was the prevailing view of many scientists for a long time. In fact, even Tim himself used to think so. I mean, 25 years ago, I published a piece saying you may as well be a member of the Flat Earth Society if you doubted that there were health benefits from alcohol.

2:21Let's just get real. I actually published and said that. But since then, Tim has done a lot of studies and meta-analyses on this idea, named the J-curve. after the shape of the graph produced if you look at the risk of death and alcohol consumption. In such graphs, starting at a baseline of non-drinkers, the risk of death goes down for moderate drinkers before rising sharply for heavy drinkers. This relationship has been shown in hundreds of studies, but as Tim said, it may be on shaky ground. When he and his colleagues did a meta-analysis last year that really focused on high-quality studies according to their criteria, that J-shaped relationship disappeared.

3:06We get a straight-line increase in risk with the amount of alcohol consumed in studies that meet those criteria. So what has happened here? I don't think it's deliberate. It's like accidental.

3:27in fact there's a few things going on here but one of the major factors is incorrectly deciding who is a non-drinker in many studies people were included in that group who were drinkers in the past and actually just aren't healthy enough to keep drinking. People become unwell, frail, have medication, they cut down on their alcohol use. Alcohol is our favourite recreational drug and it's like a marker for how well we are. So in our theory, people who are able to still drink in the 60s, 70s, 80s are in robust good health. They're able to drink because they're well enough to do so, as opposed to the other idea, which is that alcohol is making them well.

4:10And in fact, the group they're compared with is being filled up all the time with passing time with people who are unwell. So I mean it's a very clear bias. The other key factor is that many studies only included data from older people and older people are more likely to have given up drinking for health reasons. So in a recent meta-analysis Tim and his team scoured the literature for studies that accounted for these two things. That non-drinkers really never had drunk and that they had data that included younger people. If you apply just two very strict criteria to the studies, you end up with six out of 107 published studies we found and you get this straight line linear increase in risk of death.

4:58I was surprised actually about it. It was light and day between the two kinds of studies when we compare the results. There are many studies other than Tim's analysis that also suggest that the J-curve doesn't really exist for alcohol consumption. Simple methodological errors in many studies may mean that there's a higher risk of death than we thought from alcohol use. Policies and recommendations were made on the back of this methodological error and people have been drinking without really understanding the true harms and their risk of death.

5:41But isn't science supposed to find errors like this? How do such things that seem maybe obvious in hindsight slip through the cracks? The truth is that errors happen in science because scientists are human. They aren't infallible. But at the same time, errors should be rooted out by the scientific process. Science is supposed to be self-correcting. That self-correction, though, doesn't happen nearly as often as it should. Peer review is useful, but it's not designed to catch errors. Or if it is, it's only designed to catch the most obvious errors. And those errors can cause real damage. I think it's likely that the burden from alcohol will be approaching that from tobacco.

6:33So that's my prediction for the future, that as we improve our methods, we'll realise we've greatly underestimated harms from alcohol and the risks per drink. I'm Nick Petridge Howell, and this is Self Correction, a two-part series from Nature. In this podcast, we're going to discuss how errors proliferate and persist, why misconduct seems worryingly common, and what we can do about it. In this episode, the mistakes we've all made from time to time, honest errors.

7:17many errors are unintentional possibly the tools a researcher used to do their research are simply out of date perhaps they copied the wrong data and excluded something important or maybe they made a mistake with their maths and ended up with an effect looking much bigger than it actually is. Some of those errors will be relatively innocuous and easily fixed. Lots of these will be caught by colleagues or the researchers themselves or by peer review. But when they do get through, unfortunately, they don't seem to get fixed as often as they should. And sometimes even honest errors can be harmful.

7:56I think about the potential damage in several ways. This is journalist Ivan Oransky, co-founder of Retraction Watch, a website that monitors and reports on papers that have been, well, retracted. In other words, papers that contain mistakes that require correction or ones that have been the subject of fraud. Unsurprisingly, Ivan has given a lot of thought about the damage that errors do to science. Put yourself in the shoes of a graduate student or a postdoc. Your PI reads a paper, is very excited about it, and says, well, you know, I wish we had done this set of experiments. That's pretty elegant.

8:35Hey, go do these, right? So go repeat this, go try and reproduce it, replicate it, or go do the next set of obvious, you know, the obvious next set of experiments. And you go off and do that, and you take six months or something. It's a complicated set of experiments or protocols. and it doesn't work something not working is a pretty common experience in research but what if the thing you're researching is based on a mistake then you may be stuck on something that will never work in science we stand on the shoulders of giants each discovery builds on the previous with that previous discovery you're standing on is in fact erroneous you'll find yourself on some shaky foundations.

9:22And you end up, A, you've just wasted a bunch of time. Probably worse, you feel like you're a failure, when in fact, you're the right one. You're the one who knows what they're doing and it didn't work. And there are people, we talk to people all the time like this, who they get depressed about science. Science is hard enough to do, you know, whether you need to find a job or funding, it's hard enough to do when you actually have all the right information. You don't have the right information. and that's a problem. Now I should say that a lot of science is robust. It's not perfect but it's kind of the best system we have.

9:57That imperfection though is clear when it comes to errors. Mistakes do slip into the scientific record and can do damage and if we want to make this method of finding truth as robust as possible that's something we need to resolve. So when I learned of a workshop where scientists were gathering to tackle errors in science, I couldn't resist hopping on a train to Switzerland.

10:38Like every conference I've been to, it was idiosyncratic. The first event was a concert performed on Alpons.

10:54The ideas presented there were also unique. They ranged from philosophical discussions of what it means for a paper to be wrong, to practical discussions of how to track down and correct errors. And I spoke to a lot of people who were determined to improve science. a lot of the discussion was about what to do about these honest errors now the distinction between what's an honest error and what is something more pernicious like misconduct is a bit murky how can you tell if another researcher has done something shady or just made a mistake but as a working definition we can think of honest errors as unintentional mistakes made by researchers that mean what they're claiming is to some degree wrong.

11:44Such errors are well known to exist in science. Scientists are, after all, fallible humans. So there are structures in place to deal with them. The first thing you might think of is peer review. But even this cornerstone of science has its limits. Let me first say peer review is useful, but it's not designed to catch errors. Or if it is, it's only designed to catch the most obvious errors. This is Morte Edren, a researcher of digitalisation and someone who has devoted a lot of time to thinking about errors. In fact, he's one of the organisers of the workshop. But peer review is not designed to catch, let's say, errors in data collection or errors in the statistical analysis.

12:31mostly because as a reviewer you don't get access to these materials. In psychology if you get invited to review you get the same information that everybody else gets when they read the paper which is what's written in the paper. There is no special access to the kind of materials you would have to get in order to scrutinize the science itself. And of course even if you would get this kind of data, what would be the motivation of reviewers to look at them to spend hours or days looking through that stuff when it's, you know, unpaid service labour to contribute to the system, which is actually a full-time job.

13:14Whether data is available for reviewer scrutiny varies from journal to journal. Some publications do require all data to be available for reviewers, But others don't. And regardless, that data doesn't help with the well-reported problem that peer reviewers just don't have enough time to go through everything with the kind of fine-toothed comb you'd need to detect every error anyway. Especially as the number of papers published grows year on year. And to compound the problem, papers have long been growing in complexity. They have many more authors than ever before and cover a broader range of techniques and analyses.

13:57So people have questioned how feasible it is for just a couple of peer reviewers to truly assess the rigour of a paper. Malta suggests that peer review only catches the most glaring errors. If something is obviously implausible, it will get picked up by peer review. On the other hand, if some error gets included in the data that may change the conclusions, but is still plausible in principle, it may get through. In fact, according to Malte and other researchers who have looked at errors, that is pretty common.

14:33One measure of error in the scientific record is retractions. A nature analysis suggested that one in five papers are retracted, and while it's hard to say how many papers are retracted due to errors, estimates suggest it's something between 21 and 40%. That's likely an underestimate too, as many errors will not lead to a retraction or will go unreported. And if those errors are only picked up when an article is retracted, it means they got through the peer review process. So, at least for error detection, that cornerstone of science is showing some cracks.

15:16another cornerstone of science is reproducibility and replicability two terms that are used somewhat interchangeably but generally speak to the idea that independent researchers should be able to repeat any given experiment and gain at least similar results to what was originally stated but performing such replications can be difficult. So it's actually more time intensive and more effort than doing normal research in a lot of cases.

15:51The damage that can be done to science through errors is something I've been keenly aware of for a while. When I started at university in the early 2010s there were the first rumblings of a replication crisis. The idea that much research cannot be adequately reproduced. And by the time I started my PhD in late 2015, there had been a few high-profile studies on how a lot of biomedical and psychological research couldn't be replicated. In fact, the first thing I was tasked with was replicating someone else's work. But in the decades since I started my PhD, it seems that these concerns have led to a lot of changes in science, including the rise of metascience, the study of science itself, reforms to how statistics are done, and changes to peer review.

16:40One example of such a change is that in 2016, the Dutch government earmarked 3 million euros to test whether scientists can reproduce important research. And me and my colleagues, we thought it might be interesting to take the opportunity to look into this funding call and what actually happens in replication and action. Stephanie Meiermans is a philosopher looking at how to improve science. She and a team of researchers interviewed and tracked the progress of 24 teams trying to do replications of biomedical, psychological and humanities research in this funding call. And while reproducing a study may sound straightforward in principle, just repeat what someone else did, she found that in practice it was much more complicated.

17:29For a start, it can be hard to know what different people mean by reproducibility or replication. Here's Stephanie explaining it to me at the workshop. That's actually a very hard one and for example the funding agency struggled a lot with what's the definition of a replication and what actually happened was that some of the replicators applied for projects they anyway wanted to do And so they tweaked them a little bit into replications. So that, I think, broadened observations we were making about replications because some of them were actually resisting what the funder wanted them to do. So some of them even told us we were wondering how far we can tweak, right, what the funders would still allow us to do.

18:12Because what they really wanted to do was more conceptual replication. So introducing new elements. And the question was, how far can we do new things? So that's maybe where the difficulty came in of, you know, what is that then actually? Stephanie also found that often these replications tended to become much more complicated and much more work than people initially thought, especially when it came to diagnostic replications, ones where people are specifically trying to validate a previous study. Often researchers felt like they had to do the replication almost flawlessly. because you are under the pressure, of course, if you want to do a diagnostic replication, that you should yourself not do something that is not good, basically.

19:03This translated to often having much bigger sample sizes, recruiting more people for data analysis, and even getting advisory panels to help. All great things for top-notch science, but difficult for the actual humans that are doing the research who are under this pressure to be flawless. It could also require more funding and may take people away from their own research that they want to be pursuing. But this perspective on the difficulties of replications is not shared by everyone. Replication is way easier because you don't need to think about much. You're just copying what someone else did.

19:40This is Abel Brodeur, an economist and founder of the Institute for Replication, which concerns itself with, well, replications. It's working to systematically reproduce and replicate research en masse. I should mention as well that Nature Human Behaviour has partnered with the Institute for Replication to reproduce much of their published work. Nature Human Behaviour is part of the Nature family of journals owned by Springer Nature, from which the Nature Podcast is editorially independent. Now, Abel does acknowledge that there are some hurdles to performing replications, but he thinks they are surmountable.

20:20A lot of the Institute's work has looked at research that can often be replicated in a short amount of time by independent researchers via things like redoing the analysis with the original data. But even for research where replications may involve using animal models or performing large experiments again, Abel believes that there are ways to make replications more the norm. Every time I receive an email from someone from like a different field like astronomy or whatever I always tell them what you need to think of like what are the problems in your discipline and what are the solutions. Maybe it needs to be each lab decides to do one replication of a study.

20:58There needs to be sharing of protocols and things like this and it's going to be more expensive. it's definitely going to be more expensive more time consuming but i think again it needs to come from the association needs to come from the journals if journals say look if you publish this paper you need to make sure that other researchers can replicate it you could even like force labs to say you need to find another lab who's going to be willing to replicate your work otherwise we're not going to publish it abel is not the only researcher to suggest such a policy but historically journals have not been in the job of policing researchers.

21:36They tend to have recommendations rather than regulations. In recent years though, there has been a shift towards encouraging researchers to share their data and include any code written for their analyses, something which people interested in errors and replication can then use to check others' work. But so far, there hasn't been a requirement to replicate studies. although some journals, like the Royal Society Open Science, have committed to published replication studies. Abel also points out that a lot of replication is already done, even for those studies that are more time-intensive. In fact, a 2016 survey in Nature of more than 1 ,500 researchers showed that the majority had attempted to reproduce someone else's work, but only a minority of people had attempted to publish the results.

22:29Every time a new lab actually discovers something, do you think there's not another lab saying, oh wow, they've done this, we need to check whether it's true and then we can build on that. So it's being done and just not being disseminated. According to the survey, most researchers have, at some point, failed to replicate others' work. But they typically chalk that up to a dull reason. Maybe they made a mistake, for instance. But if those studies never get out there in the published literature, nobody will ever know for sure, and others may also try and fail to replicate the findings. People may also not pursue publishing the study as they are concerned about pointing fingers.

23:11Researchers can feel singled out by someone replicating their work, or those replicating may feel anxious about pointing out others' errors. This is something that both Abel and Stephanie noted as an issue. But by creating an institute that facilitates replications, Abel feels that this can be overcome, as it can act as an intermediary between authors and replicators. I think you need institutes who just do the mediation. Shoot me an email with all these problems and we can release it. I can release this as a report and do the mediation with the authors and you can remain anonymous if you're scared.

23:50Institutionalising replication may also help with a known issue of researchers not responding to requests for data. Whilst access to data upon request is typically part and parcel of publishing a paper in a journal, several studies have shown that the percentage who respond to such requests is very low. But Abel says that's a different story for the Institute. If I, through the Institute for Replication, send an email, the likelihood that they're going to respond is incredibly high. And that's because we sort of institutionalize the practice of reproduction and replication. And they also look at the website, they see all these top political scientists and top economists and are like, oh, this is legit.

24:33We have to respond to this. Now whilst Abel is confident that problems with replications can be overcome, one aspect that an institute like the Institute for Replication may not overcome is who is interested in doing replications. In fact, one issue that Stephanie and her colleagues noted was there was a bit of a divide between more senior and more junior researchers. Often the people who actually carried out the replications were the more junior scientists, something which put a lot of pressure on them. And in a way, you can look at it like they were checking up on very highly established scholars in their field.

25:16And so that could introduce quite some tension and difficulties for the PhD students. They also very much doubted their own results first. so they checked and rechecked a lot. So there were also some sort of problems arising from that, that some PhD students were really very much under pressure and, yeah, had a lot of burnouts. One thing that Stephanie thinks may be able to help with some of these problems is to rethink replications. Rather than using them just as a way to find errors, be more curiosity-driven and try to discover what may have led to the differences. So there was this one study where they partially failed to replicate, and instead of saying, well, there must have been an error in the original, they basically went into the details of how the data had been generated in the two studies, and they realized that there were differences, and these differences might very well explain the difference in the replication results.

26:19And I think that is maybe a way forward of taking this more on a scientific level and seeing also that a replication is by necessity done in a new setting, in a new time, in a new... So it's always in new experiments, in fact. So we should maybe take more care in thinking that we have always contextual results in the sense that we have done the experiment in a specific lab and under specific circumstances and there has been this one experimenter and not another one. and that is not per se bad. It's just that we need to be aware of this. That process may also help open dialogues with the original authors by finding a way to collaboratively investigate the differences.

27:09Stephanie and Abel also point out that replications aren't just about detecting errors in others' work. Doing them can also make people better scientists. This is what researchers told Stephanie when she interviewed them. Through the process of doing such a replication, they got a much better understanding of what it means to do science, and they started thinking about their own approach, and they said they'd much more critically read other people's papers. And that, I think, is a very important point that was actually a very positive aspect coming out of the replication work for a lot of the people we interviewed and we followed.

27:51Reproducing science is hard, but it doesn't have to be. There are ways to make it a bit easier. Looking at it as a constructive way to move science forward, institutionalising some of the harder bits, and creating dialogues with original researchers could be ways forward. Replications are also research in their own right, which can help us learn more about the truth of the world. But replications may not be suitable every time, and for some things, like clinical trials, they would be incredibly complex and expensive to do. So what else can be done to prevent errors? Well, many researchers are working on better ways to find errors in the first place.

28:36But everyone, even scientists, make mistakes. So how would that work? well maybe you could pay people to find them we are going back and doing the second round of much deeper error inspection and based on what we find we pay out a bounty

28:58if you've learned one thing from listening to this episode so far is that people make mistakes errors slip into the scientific record perhaps this is revealed with a replication but most replications aren't published, at least at the moment, and not every study will be replicated. Another approach could be to create more ways to pick up errors. If errors can be identified and corrected more often, then we could have more trust in the scientific record. This is something that Malte, who you heard from earlier, is quite passionate about, as he feels that errors are not detected often enough. Well, they are, but they are through a process that depends entirely on coincidence.

29:43You have to have the person who has the knowledge and the skills to detect errors, and that person has to read a paper or multiple papers with errors. And then they need to be in a state of mind, and they have to have the resources to write this up, and then they have to bring to the public. All of this depends entirely on coincidence. And that's just the discovery part. That's not even the correction part, right? This frustration led Monte and a team of scientists to come up with estimating the reliability and robustness of research, or error, a program where people are paid to find errors. Here's Ian Hussey, another member of the error team, to explain.

30:24So if no errors are found, the author of the article themselves gets paid an extra bonus. And if just minor errors, the sort of normal human error levels of problems are found, we pay an extra bonus to both the author and the reviewer because we think everyone should be rewarded for normal behavior. And then the more severe errors we find, the larger a bounty we pay to the reviewer themselves, up to about two and a half thousand euros. The aim of the error programme is to assess 100 papers for errors over four years. The team nominate papers that they feel are important, based usually on them being highly cited, and so mattering most if they were corrected, as maybe a lot of research has been built on these studies.

31:15Then they contact the authors to tell them they would like to do this analysis. Only with the author's consent do they actually go forward with it. Partly for practical reasons, like access to the relevant datasets, but also as they're not trying to be antagonistic. They want to incentivise doing the right thing, correcting the scientific records. Then they find reviewers who have the necessary expertise to go through the paper. And finally, they deliver a report with their findings and recommendations. And so far, they think it's going well, even if there are some roadblocks. I think it's going very well as a process, but not all of those processes are visible yet.

32:00So our goal is to inspect about 100 papers. That's what the fund will cover. We're a year in, we have four reports out and two more just about to come out. So six in a year needs to build up to 104 years, which is a bit of a challenge. But I would say that what we accidentally did here was sort of establish what's basically a journal in the sense of needing to have a lot of internal processes, have a set of reviewers that are willing to work for us and a set of authors who are willing to sign up. So there's a lot of internal documentation for workflows to guarantee the sort of integrity of the process itself.

32:42And a lot of the last year has been put into getting that in place. Now that it is in place, I'm confident that we can scale over the next three years. Will we hit 100 papers exactly? I'm not sure, but we will do a lot better than the first year. Of the four papers where errors reviews have been completed, the team concluded that two had minor errors that didn't need a correction as they didn't affect the core conclusions. One had moderate errors that didn't affect the core conclusion, but would benefit from a correction being posted on the journal website. And the other had a major error that affected the core conclusion, and so would require a correction to make this clear and to change some of the text to update their findings.

33:27And whilst you might expect that having errors pointed out in your work may be a bit uncomfortable, perhaps surprisingly, one of the best experiences that Ian and Malte had was with the author of the paper that had the biggest errors. I think possibly the best case of this is the paper led by Eric Heyman. And Eric's reaction was exemplary, right? He took to Blue Sky and he said that this was a little bit ouchy, Nobody likes errors to be found in their work. And it is one of his most cited papers, so this matters. And at the same time, he said he would recommend others engage in this error process, right?

34:08That it was a professional process, it was an enjoyable process, that he learned from the process, and that ultimately it contributes towards doing better science. For one of the other papers, once errors review was sent over, the authors decided to, completely voluntarily, review the entire dataset. and in doing so found even more errors which they then corrected. However, these are the people who were contacted by Error's team and were willing to go through the process. Getting people to do that hasn't always been easy. I think we have sent out roughly 150 invitations and of those, only roughly a third even replied to the invite.

34:52So most people that get the request... Ignore it, or don't see it, or don't consider it worth replying, whatever the reason might be. And then of this third that did reply, only another third ends up actually participating in this process. As Morte says, a lot of time, researchers just didn't get back to them. So it's hard to know exactly why they may not have been interested in working with error. Of those that did get back and said they didn't want to be involved, some just didn't want to be first. They wanted to see how error progresses and builds its reputation before taking part. While others simply thought their studies lacked errors.

35:36Some also felt a bit singled out and wondered why their study was being targeted. Perhaps feeling as though they're being accused of something. Ian and Malte are undeterred though, and they point out there are similar response rates when people just ask a research group for data, never mind going through their paper with a fine-tooth comb to check for errors. One possible criticism of error is it's still after the fact. In this case, research has been out there for a while containing possible errors, being cited, and maybe affecting policy or the broader world. One recent example of an error with that sort of impact came last year.

36:21There was a paper about the harmful chemicals in black plastic spatulas. It sparked a number of news articles and widespread concern, with people being encouraged to throw the utensils away. It was later revealed, though, that the paper behind this had a mathematical error, which greatly exaggerated the risk. In this case, the authors realised their mistake and issued a correction. However, they also say it doesn't change their overall conclusions. Nonetheless, it shows what can happen when errors are out there.

36:58In fact, this mistake was so newsworthy, it sparked an interest in using AI tools to catch errors. We had actually, I talked to my wife and we got all the plastic out of our kitchen. as I think a lot of people did. This is Steve Newman, a software engineer and founder of a non-profit working on AI policy. And then later it turned out that the result in that paper was overstated due to a simple math error, just getting a digit wrong and a multiplication, which was right there in the paper if you looked at it, but all the reviewers had overlooked it. And then I saw a tweet where Ethan Malik, a professor at Wharton, had just thrown that paper at, I think it was OpenAI's O1 model and asked it, check the math in this paper.

37:44Now he did that deliberately because he had, you know, someone else had discovered the error and he knew about it, but he was just curious to see, would an AI catch this? So he just gave it that very simple prompt, carefully check the math in this paper, and the AI found the error. And he tweeted about this. And my immediate thought was, well, that's an interesting result, but would it replicate, you know, how many errors like that are floating around in the literature and, you know, how many of those good AI find. And so I just casually tweeted, someone should try this with another thousand papers and see what happens.

38:15And that was my one and only lifetime viral tweet. This viral tweet led to Steve becoming that someone and starting what's known as the Black Spatula Project. So I created a WhatsApp group for people to gather in and then someone else set up a Discord channel, our Discord server, and pretty soon just hundreds of people were gathering to discuss the idea like we just seem to have struck a bit of a nerf now steve emphasized that the project is still in its early stages but in essence people are using online ai tools to assess papers for errors maybe checking for mathematical errors checking the statistical analyses or comparing the paper with others on the topic they're still figuring out the best way to go through everything, developing pipelines to make the process as smooth as possible.

39:05So far, the project hasn't published any of the errors it's found in the 500 papers or so it's looked through, instead opting to approach the authors of the papers directly, much like Ian and Malte. Part of the hesitation for this is that these AI tools are not exactly error-free themselves. The errors or reports may not be real, and there is a significant rate of that, so So certainly at this stage of the technology, you need a human expert to review any alleged errors that are reported by the AI. And you want to be careful about how you, if you just start publishing on the Internet, look at all these errors and all these papers.

39:43Obviously, that's bad if what you're reporting isn't true, if the errors aren't really errors. Flagging errors that are not really errors to begin with is a known issue with AI tools. One other project called Yes No Error has analysed more than 37 ,000 papers in two months, according to its founder. It has then published those errors on its website, but many of them are not verified by humans and errors have been listed that are not real. So some researchers have raised concerns about these tools generating more work for humans to go through and check. Steve and others are working to make such AI checking more error-free as right now this may not be something you want to roll out at scale.

40:30Another issue is exactly how to raise the errors with an author. As Ian and Malte have found, people can be sceptical when contacted about a potential error in their study. One potential solution to this issue could be to create a tool that the original paper authors can use to quickly check for errors before they publish their work. That's probably the first and best application of this technology, in part because it gets around a lot of these challenges of, OK, we have an error report, but we don't know whether it's valid. And even if it is valid, we want to be careful about how we present that.

41:05But all those problems go away if you're working with the paper author. And so, yeah, as someone has put it, you know, like we had spell check and then we had grammar check. And, you know, now we should have, you know, scientific validity check as a tool for the author. Error correction should be a normal part of science, and it often is, but not enough according to the researchers I spoke to. There are ways to institutionalise it and promote it. In the future, there may even be automated tools that researchers can use to find errors before they publish. There are challenges though. Getting researchers involved in the process is a big one.

41:44A lot of them are scared of errors and the damage they believe they could do to their careers. But the people I spoke to were optimistic. Part of the reason for Malte and Ian's optimism is because the aim of error is not really to correct 100 papers. It's to find where and how errors are made in the first place and help create systemic change to tackle them. It's to change culture. If they can nudge things just enough to make people a bit more careful and think about areas a little bit more, they'll see the project as a win. But science is a huge conglomeration of institutions, agencies and individual scientists.

42:26How do you go about changing the culture of such an enormous beast? Well, it might start with being a bit more honest about our mistakes. So we call this f***ed lights. So then you talk about error, everyone shares something which you normally more try to hide. Then of course this helps in normalising what we perceive as an error.

42:54One common theme from a lot of people I spoke to was that more replications or better error detection are only part of what needs to be done. Really it's the culture around errors in science that needs to change. However, many researchers fear the damage errors being revealed would do to their careers. Scientists seem to believe that they should be in some ways infallible with the weight of the scientific method on their shoulders. Being rigorous and attentive to errors should be part and parcel of being a good researcher. At the same time, scientists are also human, and in reality, it's unrealistic to expect them to be somehow superhumanly flawless.

43:41Helping researchers come to terms with these two at-odds beliefs is a key challenge of shifting this culture. But there is reason to believe that errors aren't actually the death knell for their careers that many researchers fear that they are. There have been studies that show that if a paper has been retracted because of an honest error, then people don't see a fall in their future citations. Whereas if it's for something more nefarious, they do see a fall in their citations. And there are signs that perceptions are already changing. Pre-registration, where researchers detail what they're going to do and how they're going to do it before they conduct the actual research, has grown in popularity.

44:27The idea here is that it allows peer reviewers and other researchers to check that people really did what they set out to do, as another check and balance on the robustness of research. This and many other research practices have grown over the past few years, as many scientists have been working to help science self-correct. But there's still more to be done to change culture.

44:55at the workshop in switzerland i spoke to maximilian frank a phd student of metascience who's collecting ideas from other researchers to try and figure out how to do just this the error in science is somewhat also entangled with the concept of blame you are responsible for these errors and you are failing in a way and this for us is like one of the indicators we have like a bad error culture we need to improve. The error is like viewed as a stigma in a way. To de-stigmatize errors, one of the things that Maximilian has been trialing are f*** up nights. So these are events at conferences, let's say at the later part of the evening, where the idea is to share error stories.

45:34So it's not about you share your greatest research success or you share your greatest nature paper. So you really say like, okay, this is something where I wanted to do A, and for whatever reason, I ended up with B, then of course this helps in normalising what we perceive as an error. Maximilian has tried this out at a couple of big meetings, and he says the response has been positive. By normalising the fact that researchers are human and do make mistakes, it can help people be less fearful of finding errors, both their own and errors of others. Maximilian also suggests including training for new PhD students on error culture and other practices to increase the robustness of science.

46:17Otherwise they're just reliant on their supervisor to train them on how to be a good scientist, which can be very variable depending on the supervisor. And ultimately this is not a challenge that has only ever faced scientists. Other fields have come up with strategies to cope with errors. So Maximilian also suggests there may be things that researchers can learn from the world of open source software development. In the realm of open source software, it's like really common you have open source code. There's maybe like a main developer, like a team developing it, but there's also like a lot of other contributors just maybe like contributing one bug fix, but it's also like the community doing something together.

47:00In science, it's still like I need to build my own career. I need to have my publications to get a PhD, then I need to have like certain amount of publications in some prestigious journals who are getting a good chance of tenure track. And this is very individualized. Then of course, it makes more sense that people react in a bad way when you're pointing out possible errors in their work. And that's why I think what they're doing in the open source community, this would be like a good role model to look at errors, because they're building together something and not for the individual career. Science is a collective endeavor.

47:32But all too often, it can be hard to remember that. If we want to make science more self-correcting, then we need to develop ways to work better together, knowing that we are trying to help each other find the truth rather than tear each other down. Errors happen, but together we can stop them doing damage. The first recommendation I would have is be aware that errors occur, Be aware that they are likely far more prevalent than you think they are. Be willing to acknowledge them where they occur without terror that this is going to be a career ender. The negative consequences are likely, we believe, not nearly as severe as you might fret that they are.

48:19And the positive consequences of being seen in a very public sense to be a scientist with great integrity who corrects errors where they are found. I think there are explicit career benefits of doing so. So be more error aware and error embracing.

48:42Errors can be damaging. So we should replicate where appropriate and build a communal dialogue where science improves in a collective way. At the same time, we can build systems to detect errors and prevent their spread. For any of this to really take hold though, it will need a cultural change. Scientists need to admit that they're human and make mistakes, and know they'll have the support of their colleagues and institutions when they do.

49:19But what happens when errors aren't so honest? I have come to this interview from a morning of looking at a paediatric treatment that I'm almost certain is supported by fake evidence that is probably killing children. And that's just Thursday. That's coming up on the next episode of Self Correction.

50:06This episode was written and produced by me, Nick Petridge-Hell. Dan Fox was my editor. Thanks to Ivan Oransky of Retraction Watch, Mota Aildren and Ian Hussey from the University of Bern, Stephanie Meermans from Amsterdam UMC, Abel Brodeur from the University of Ottawa Steve Newman from the Black Spatula Project and Maximilian Frank from Ludwig Maximilian University of Munich for taking the time to talk to me for this story.

50:51Back to school is busy enough. The last thing you need is shopping that makes it harder. Whether you're moving into a dorm, your first apartment, or just getting organized for the semester, Wayfair makes it easy to get everything you need in one place. Furniture, bedding, storage, decor, study essentials, all at prices you'll love. With fast shipping before move-in day. Get everything you need for back-to-school season. Shop Wayfair.com today. Wayfair, every style, every home. I'm Serena Williams. And I'm healthier on Roe. I've lost 34 pounds in a year with GLP-1's Diet and Exercise. On Roe, you can access GLP-1 options, including the first FDA-approved GLP-1 pill for weight loss.

51:33Go to roe.co slash journey to see if you qualify. 14 to 20 % average weight loss in one year in non-diabetics with obesity or overweight with a weight-related medical condition versus 2.2 % to 3.1 % in placebo arm. Rx only. To stay informed about serious side effects, go to roe.co slash safety.