How does scientific peer review work?

23:58I'm Munga Shatigular, and I'm back with a new season of the podcast Skyline Drive. This time I'm diving into a rabbit hole of peptides, organoids, blood boys, blue zones, and brain replacement to try to understand what this longevity obsession is all about and what it really means to live forever for all of us. I learned about some rad science. I can make a brain for you, and then we can test what draw is the best for your brain, as opposed to his brain. Here are some hard truths. I would expect Indians to age faster, but I did not expect it to be almost a four to five year acceleration. And get myself into a world of trouble.

24:43I'd say probably start bone smashing. That doesn't work. Make it look more defined. They say it works. I don't know. Listen to Skyline Drive, How to Live Forever on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts. We've all been there. We've had seasons that have shaken everything. The question is, what remains when the shaking stops? Hi, I'm Gary Valenciano, and on my podcast, Unshaken with Gary V., I sit down with amazing personalities and we talk about the seasons that nearly broke them, the lows that carried them to their highs. Listen to Unshaken with Gary V. on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts.

25:32This is Michael Rappaport and my podcast, the I Am Rappaport Stereo Podcast, is unlike anyone you've ever heard. We're a variety show and if you're looking for strong opinions, funny opinions about sports, entertainment, politics, pop culture, and whatever else catches my attention, then subscribe now. This kid Jafar Jackson is as good as Rami Malek, as Freddie Mercury, and it's as good as Timothy Chamillai, as Bob Dylan. And I say that with love and respect for both of those actors. And I don't know how many Oscar nominations they give out. I don't know if it's five, six for best actor. 150 % this kid Jafar Jackson should absolutely positively get nominated for his portrayal as Michael Jackson.

26:20Listen to I Am Rappaport on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts. I Am Rappaport Podcast. Bill, why is my eye twitching? Does this mean I'm in perimenopause? Maybe I have adult ADHD. I need to look this up. Where's my phone? Juliana, your phone is hot to the touch. I think it's asking you for a break. You know what? Maybe I should just ask chat GPT. Or maybe we can ask an actual human. Yes, like a couple sex therapist. Whoa, that escalated quickly, but okay. Because I have questions. We have questions. And I bet everyone has questions. Like, is it normal to sleep in separate bedrooms?

27:00We do that. How about this one? Is bribing your kids bad parenting or just negotiating? Oh, and I still do need to know, why was my poop green that one time? Hypothetically, right? Okay, well, instead of letting the internet guests, we've got actual people answering these exact questions. And laughing with us has got to be better than spending three hours down a rabbit hole online. Listen to Bill and Juliana, the podcast on the iHeartRadio app, Apple podcasts, or wherever you listen to your podcasts.

27:42All right, we are back and we are talking about peer review. Thanks to a question from a listener and we dug into the process of peer review for grant proposals and grant writing. And now let's talk about what it's like to review an experiment while it's running before the paper is even sent to the journal. Kelly, I mostly do research on particles that don't have rights and don't have institutional review boards protecting them. What's it like to do an experiment on living creatures with emotions that can feel pain and have people looking over your shoulder? Oh, it can be pretty stressful. So say you get that National Science Foundation grant.

28:19They won't actually release the funds to you until you've shown that you've acquired certain permits and protocols so that your institution is giving you permission to do the research. So my Ph.D. work required collecting some fish out of estuaries in California. So first I had to talk to the state government and fill out permits to get permission to go collect those fish. So I had to convince the California government that I wasn't going to take too many, that there were plenty of these fish out there, that what I was going to do to them was asking a worthwhile scientific question. Kelly, what are you going to do to them?

28:53That sounds so aggressive. Well, the questions I'm going to ask of them, that they will be contributing in a meaningful way to science. And so first you have to get that permission to take the animals out of the wild. And then you need to fill out a protocol through the Institutional Animal Care and Use Committee, which is IACUC. It includes five people, and they include folks like veterinarians and outside members, so like somebody who's part of the community who is just sort of going to weigh in on what the general public feels about the work that's being done. Sometimes you'll also have like an ethicist or a lawyer in there, and often you'll have like a faculty member on there also.

29:31And what do these people want? Do they want to say yes? Do they want to say no? Are they just totally disinterested? Like, what are their motivations? Why is like a rando from the public doing this anyway? So if I'm being completely honest, my field in the past did some uncomfortable things to animals. And, you know, they didn't use, for example, anesthetic when they were doing some surgeries and they should have things like that. And so this committee is trying to make sure that you are treating the animals as nice as possible and using the fewest animals that you need to get answers to your questions.

30:05So you need to convince them that you have read up on the most effective anesthetics for whatever animal it is that you're using. You also often have to take a bunch of online training courses where you show I am well trained in the best way to treat these animals and I know how these anesthetics work. You often have to prove to them that you have teamed up with someone who can make sure you're doing the process correctly. And you need to convince them that you've thought really hard about the exact number of animals that you're going to use for these experiments. Their job isn't to stop science, and they're not necessarily trying to figure out if the question that's being asked is good or not.

30:42They assume that if you have funding from the National Science Foundation, that has already happened. But their goal is to just make sure that nothing unethical happens and that the animals who are contributing their lives to science have the best life possible. Oh, that sounds nice. It is. And it's nice to see the process sort of self-correcting. Like, okay, we trust scientists. It's actually maybe they need some more eyeballs on them because they have conflicts of interest. And so it's good to have other people who don't have those conflicts looking over your shoulder and making sure you're doing things the right way.

31:12That's nice. It is. And I also appreciate that there's a veterinarian on there. The veterinarian will like check in pretty regularly. I have training in parasitology, but I don't necessarily have training in, you know, care of fish or something. And I've done a bunch of reading, but it's nice to have a veterinarian check in every once in a while and offer their expertise to make sure that these animals really are being treated as well as you can in a lab setting. And I have never worked on humans, but if you are doing things with humans, including just like sending out a survey to humans that might ask questions that would make people feel uncomfortable, you have to pass those procedures through an institutional review board.

31:47So there's a similar procedure for working through humans. So I guess you don't have anything like that for particle physics because we don't care what you do to the particles. No, we don't. But I did one time come home from work to find Katrina collecting samples for an experiment from our children. Oh. And I was like, hmm, shouldn't you be asking the other parent for approval before experimenting on your children? Like, you have a conflict of interest here. And she was like, it's just a saliva sample. And I'm like, the children can't consent. So we have a lot of fun joking about that. Did you give permission?

32:20I did, yes. Okay, that's good. That's good. So Daniel, there's this new thing that's getting sort of big in science that my friends have been talking about. I haven't had a chance to do this yet, but pre-registering a scientific study. Have you done this yet? We don't actually do this in particle physics because this is another way that particle physics is weird, is that we publish our studies even when we get a negative result. Nice. Like in lots of fields of science, you might say, I have an idea for how we might learn something. And you do the study and then it didn't work. Or you learned that there's nothing new there.

32:52Like, okay, bears eat salmon. We already knew that, you know, like, or whatever. And often those studies don't get published if the result isn't statistically significant or you didn't learn something new. And there's a statistical issue there, which is that this can lead to a bias in our data and our understanding. The effect is called p-hacking. And it means that sometimes things that are not real can appear to be significant just due to random fluctuation. Say, for example, I'm testing a bunch of coins to see if they're fair. I flip each one 100 times, then like most of the time I'm going to get 50 heads.

33:26But occasionally just due to random fluctuations, I'm going to get one that gets like 70 heads, right? And the more I do this experiment, the more likely I am to have one that's a fluctuation. And so if I only publish the ones that seem to be significant, that like cross some statistical threshold to be weird, it's going to look like these coins are not fair when in reality they are. That's what p-hacking is. P refers to the probability for the alternative hypothesis that the coin is fair to fluctuate randomly to look like it's not fair. And so the way to counteract this is to publish negative results, to say, look, I did all these experiments and the coin came out fair.

34:05We already knew that. Yawn. But it's important that we include this context so that if one in a thousand studies says, look, I saw a one in a thousand times effect, you would know what it means. And often we do this by pre-registering studies by saying, I'm going to go out and do this study, and I don't know the results yet, but I'm going to publish it either way, right? And so that's a way to counteract p-hacking. So in my field, what you've just described, I think we would call the file drawer effect. And so the idea here is that, yeah, when you get a null result or a result that's less interesting, you might try to publish it somewhere.

34:39But you're less likely to put in the effort to try to publish it in a lower tier journal that you're going to get less credit from your institution for having published in. And that could make it look like an effect is there because when you randomly get a positive effect, it gets published and otherwise it doesn't. For us, p-hacking is when you didn't get the result that you wanted, but you're looking at your data and you're like, oh, I wasn't actually asking a question about this other thing. But if I look at my data, there's actually a statistically significant result there. And so I'm going to publish on that.

35:09And maybe I'll mention that this wasn't the initial thing, but I found this significant relationship that's positive. So I'll publish on that. And the deal there is, you know, like you said, randomly you would expect to get results that look like something is really going on. And if you just are searching through your data set, you're more likely to find something that's significant. And that wasn't the question you were originally asking. And so the reason we do pre-registering a scientific study is you'll say ahead of time, I am specifically looking at the relationship between, I don't know, parasites and how often a fish darts.

35:42And so if I go on and I publish a paper about parasites and how active a fish is, you could say, hey, did you just find that result and that you changed your paper to be about that because it looked interesting? And so anyway, this is how we make sure that you're not looking for significant results and just publishing whatever you find. And in particle physics, we're lucky enough that we always publish because a negative result is still interesting. Like if you look for a particle because you think it's there and the universe would make more sense for it to exist, like the Higgs boson, then if you see it, great, you publish it.

36:16If you don't, that's still interesting. You still want to know, oh, there isn't a Higgs boson. Because if we smash particles together often enough and we don't see it, we can say something about the likelihood of it not existing. Because if it existed, we would have seen it. Now, we still are susceptible to p-hacking because we will sometimes see fluctuations. like data will just like look like a new particle sometimes and to counteract that we have a very stringent criteria it's called this five sigma threshold which means that we only claim a particle is discovered if it's very very very unlikely to come from a fluctuation but we still in principle could get fooled and that's why we always have duplicate experiments like at the large agent collider we have two of these big detectors and they're totally independent and that's why you expect to see the same physics in both.

37:03And if you don't, then you know something is squirrely. Yeah. And in my field, we're getting better where if you get no results or negative results, as long as you can show that you did a sound experiment and the answer is no, then you can still publish it somewhere. And so Public Library of Science or PLOS is a journal that encourages folks to submit their null results as long as the science was good and you can defend what you did to try to make sure that you're not getting this file drawer effects stuff. So you get the no answers just as often as the yes answers. And that's helpful, right? It saves people time because if Kelly had a great idea and it turned out the answer is no, and then she just throws it in the file drawer, then Sam across the country is going to try the same thing someday and waste their time.

37:44So it's good to get this stuff out there. Right. And you know, in my field, wasting time also often means that some animals may have died in the process. And so to me, I feel like there's this additional ethical requirement that you get anything that you learned from these animals out so that nobody has to go and repeat it and it, you know, more animals might have to pass away for science. So anyway, good to get it out there. All right. So then let's talk more about what happens when you think you have an interesting result and you've written it up and you want to publish it. And we're going to leave everybody on a ledge for just a second.

38:16And when we get back from the break, we'll talk more about the fascinating world of peer review.

38:30I'm Munga Shatigular and I'm back with a new season of the podcast Skyline Drive. This time I'm diving into a rabbit hole of peptides, organoids, blood boys, blue zones, and brain replacement to try to understand what this longevity obsession is all about and what it really means to live forever for all of us. I learned about some rad science. I can make a brain for you and then we can test what draw is the best. for your brain, as opposed to his brain. Here's some hard truths. I would expect Indians to age faster, but I did not expect it to be almost a four to five year acceleration. And get myself into a world of trouble.

39:14I'd say probably start bone smashing. That doesn't work. Make it look more defined. They say it works. I don't know. Listen to Skyline Drive, How to Live Forever on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts. We've all been there. We've had seasons that have shaken everything. The question is, what remains when the shaking stops? Hi, I'm Gary Valenciano, and on my podcast, Unshaken with Gary V., I sit down with amazing personalities and we talk about the seasons that nearly broke them, the lows that carried them to their highs. Listen to Unshaken with Gary Vee on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts.

40:25is as good as Rami Malek as Freddie Mercury, and it's as good as Timothy Chamillai as Bob Dylan. And I say that with love and respect for both of those actors. And I don't know how many Oscar nominations they give out. I don't know if it's five, six for Best Actor. 150%, this kid Jafar Jackson should absolutely, positively get nominated for his portrayal as Michael Jackson. Listen to I Am Rap Report on the iHeartRadio app Apple Podcasts, or wherever you get your podcast.

41:25Because I have questions. We have questions. And I bet everyone has questions. Like, is it normal to sleep in separate bedrooms? We do that. How about this one? Is bribing your kids bad parenting or just negotiating? Oh, and I still do need to know, why was my poop green that one time? Hypothetically, right? Okay, well, instead of letting the internet guess, we've got actual people answering these exact questions. And laughing with us has got to be better than spending three hours down a rabbit hole online. Listen to Bill and Juliana. The podcast. on the iHeartRadio app, Apple Podcasts, or wherever you listen to your podcasts.

42:13All right, Daniel, you have a significant result. You've got a result that looks good, or at least even if it's no, you're sure it was no because the experiment was done well. What do you do next? So we write it up, and when all of our co-authors are happy with it, the first thing we do is we post it online. So particle physics is a very, very online science, and we invented this thing called the archive, where scientists post their papers, and it's called a preprint because we post it before we submit it to the journal. So for example, I finished a paper this week and yesterday it appeared on the archive and nobody's reviewed it.

42:50Nobody's confirmed it. It's just out there for other scientists to look at. And in a week or so, if we don't get like screaming fits from somebody saying you stole my idea or this is deeply wrong, then we'll submit it to a journal. So the process for particle physics is first posted online, then submitted to a journal where it's going to get reviewed. And that's going to take months and months. But it's so slow that particle physics just read the preprint server. and read those papers. And nobody really cares when it comes out in the journal later, because we already have seen the paper months earlier.

43:21But what if the peer-reviewed version catches an error? Do you go back and update the archive version? Oh, okay. You can update it. And usually papers in the archive are like version three, version four. And there's comments in between. You can compare them and you can see the mistakes getting fixed. Absolutely. And some papers appear in the archive and then are never published. And that's fine. I actually had one paper like 10 years ago put in the archive, It took like more than a year to review. And then they rejected it because the paper already had like 30 citations. And so they were like, why publish this?

43:52It's already out there. People are reading it and using it. And I was like, that sucks. That's such a stupid reason. Yeah. It was really dumb. But then we just gave up. And now it's like one of my more cited papers. And it was never published. The system is silly. It is silly. But I know that in other fields, it's different. Like my friends in astronomy, they never put papers on the archive until after they've been reviewed. And they think that if they send it to a journal and it's already out in the archive, those journal referees are going to find that disrespectful, like that they didn't wait for them to chime in.

44:23So it's definitely a field-by-field culture kind of thing. How does it work for you guys? Yeah, so y 'all were definitely the trailblazers. We subsequently came up with BioArchive. And I remember when I was initially wanting to submit to BioArchive, I was hesitant to do it because some of our journals did say, your paper should not be available anywhere else because we want to be the only place where you can find it. And that's because journals make money off of people who download their articles. And so they didn't want someone else to be able to get another version for free somewhere else. And on BioArchive or Archive, all of the articles are free.

44:59But it seems like it's now become acceptable that you can put papers in these preprint servers like BioArchive. And so now a lot of people will do that. And they do it for a variety of reasons. And some of it is they want to get feedback early. They want to get the results out there early. But sometimes it's also like if you've got a PhD student who wants to go looking for a job, on their resume, if they finished a study, they'll write in preparation next to the manuscript. But in preparation could mean I've got the data and I still need to do all the stats and I haven't even started writing. Or it could mean I'm about to submit it to a journal.

45:34And if you've put it on bioarchive, you're saying, look, it's done. I just need to start the peer review process. or even it's in peer review because that process can take a long time. So anyway, it's becoming a lot more common to put your papers on bioarchive just to make them easier to access for other people and just to show that you really have actually finished that study finally after a decade. Or however long it took. The clock is still ticking on your thesis there, Kelly. We'll see. I'll get it. In preparation. In preparation. So then you send it to a journal and you pick a journal. Like if you think it's a really exciting paper, you send it to nature or science.

46:08If you think it's less exciting, you send it to a journal with a smaller impact factor. So in particle physics, we often publish in like physical review or in journal of high energy physics or these kinds of journals. And it goes to an editor and an editor finds people to review it. These are the peers and they reach out to folks they know who are experts in the field, who aren't your supervisor or your brother or this kind of thing, and ask them to review these things. And Katrina is a senior editor on a journal. So I see this process all the time. I mean, she has to read the paper, think about who might know about it, and then find people to review it.

46:42And usually you have one to three people read this paper and give comments on it. But finding reviewers can be tough. So for our grants that we talked about earlier, you have to make sure that the person reviewing the grant doesn't have a conflict of interest. They're not your brother or your supervisor or something like that. And in my field, the same holds true for when you're reviewing manuscripts. Is that true for your field as well? Yeah, absolutely. Yeah. You have to say, oh, I can't read this one because that was my student or something like that. Yeah. How long does review usually take for your field?

47:13Yeah, I'd say it's, you know, like four to eight weeks or something like that. Wow. How about you guys? I'd be happy four to eight weeks. That sounds good. But it can be sometimes six months. And at six months, you start writing the editor being like, come on, man. This is crazy. But yeah, it can be hard to find reviewers. Because reviewers are just other scientists busy writing their own grants and writing their own papers and doing all their work and picking up their kids from daycare and stuff like this and trying to get through that mountain of laundry. You know, I think that people forget that, like, science is just people, right?

47:45And people are busy. Yeah. And if you're out there and you've written some treatise on the fundamental physics of the universe and nobody has read it, it's not because we're gatekeeping. It's just because we're busy, you know, and there's lots of those. And I often say no to editors who ask me to review stuff because I'm just too busy to do it in a timely manner. Yeah, no, me as well. And I think it's worth noting that while we're doing this review work, we're doing it for free. Like when you're on a grant review panel, you can lose weeks of time to reviewing these things and you don't get reimbursed for that time.

48:15And the same thing goes. So when I review a manuscript, I read it for one day. It takes me a couple hours to read carefully. And then I sit on it for a few days. And that's the main thing that is occupying my brain in the background for those days. And that could be like I could have been thinking about the introduction to my next book or something. But instead, I'm thinking about the methods to make sure it made sense. And then I spend another three to four hours reading it again and writing in detail my comments. And if I think there's some literature they missed, I go and I find it. And it's a multi-day process when I commit to reviewing a paper.

48:46And reviewers don't get paid for that, nor do the editors. How long does it take you? Well, I wish that all of my papers were reviewed by somebody as thorough as you. But my process is similar, like an initial read, some thoughts, and then let it sit. Sometimes I'll discuss it with people in my group. We'll read it together. And then I read it again in detail. And then especially if the review is negative, I wait and I sit on it and I come back later. I'm like, was this fair? And also most importantly, was I nice enough? Are my comments constructive and helpful and not like just negative or harsh in any way?

49:22I'd like try to take a sandpaper to anything that's negative and smooth it over. Because I've been on the other end of harsh reviews and it doesn't help anybody to get zingers in there. And like, especially because a lot of my papers are written by young people and it's often their first paper. And then I have to show them this review that's like, this guy's a jerk. This isn't necessary. I have to explain to them that the anonymity here is protecting them. But, you know, most reviewers are fair and are thoughtful and are courteous. And so often we get like helpful feedback, like, what about this?

49:53Or have you thought about this question? Or what would you say to this concern? Do you find your feedback to be mostly useful? Yeah, most of the time the feedback I get improves the paper. I did have one reviewer when it was one of my first papers who told me they were very disappointed in me. I was like, you're a jerk. Very disappointed in you. Thanks, Dad. And that was because my supplemental materials were too long. I was like, God, give me a break. But anyway, so in my field, we're moving towards double blind review. So the reviewers are not supposed to know the names of the authors, and the authors are not supposed to know who reviewed their paper.

50:30And that's a really great idea so that you feel like you can honestly tell someone if their paper was good or not. You don't have to worry about someone getting mad at you in particular. Though in practice, there's usually like four people who could review your paper. So, you know, like, you know, it was Joe or Francine and Francine tends to be meaner. And so but anyway, so yeah. What about in your field? Is it blind or double blind? It's only blind in one direction. The review is anonymous, but you see who the authors are. I know in some other fields, like in top computer science conferences, for example, it's double blind and that's helpful, but also it's not that hard to figure it out.

51:07If you wanted to know who these people are, you could figure it out. But you might also be wondering, like, what is the job of the reviewer exactly? Does the reviewer have to make sure the experiment was right? Like, if I'm checking Kelly's work on the reviewer, do I have to go out and do the experiment and make sure she was right? And that's the thing is the peer review is not replication. The job of the reviewer is to ask, do the claims of the paper, are they supported by the evidence provided? Is there logic there? Is there mathematical tissue between the work that's been done and the conclusions that are being claimed?

51:38And also, is it well-written in the citations? And finally, is it interesting? Is it important? Does it play a role in the scientific conversation? Which is a little bit subjective, of course, but science is by the people and for the people. but if you're working on something nobody's interested in and nobody's going to read your paper, then a journal might say, this is solid work, but nobody cares. It's a boring question or you didn't learn anything interesting. The reviewer's job is not to say, hey, I did this experiment also and I know that this is a feature of the universe. That's not the task of the reviewer.

52:11It's not their job to do that for you. It's also not the task of the reviewer to say like, hey, this is a cool idea. I would have done this differently. And so you now need to go back and do all these additional studies that I think are important. And you see this a lot in peer review that reviews come back and say, this is cool, but also do X, Y, Z. And like, that's frustrating to me because that's not the job of the reviewer. Yes. Yep. Nope. That's super frustrating. And I think it's worth noting that sometimes bad papers get through this procedure. And one of the ways that bad papers get through is that reviewers aren't required or, and this makes sense, but they're not responsible for going through, for example, the codes for the statistical models that you ran.

52:53And so if somebody, you know, checked their codes a bunch of times, but they forgot a minus sign somewhere, the result could be wrong. And maybe nobody knows. And I have known a couple people who afterwards have like gone to use those models again for some other question and caught their mistake. And then a good scientist will contact that journal and say, I have to retract my paper or I have to submit an errata and let everybody know, I forgot the minus sign. It's a different result. And that's really painful, but at least they're being honest and that's great. And I think that's really important.

53:26Or some people are dishonest and there's no way for the reviewer to know that. My field had somebody where they were one of the biggest names in the field. And there's this new requirement in our field. Well, it's a couple of decades old now, I'm old, But where you have to put the data that you use, that you collected as part of the experiment, online somewhere in a public place where people can download your data. And again, you know, in a field where animals are being used to collect data, it's great to know that those data could be used by other people to ask questions. You can get more information out of them.

53:59But it also means that if someone gets suspect results, you can pull their data and rerun the models. And when somebody was looking at the data, it was clear that the numbers just didn't make sense. Like one column was always the prior column times three. And anyway, after more scrutiny, it became clear that this person was making up their data. But, you know, we have this new check where your data have to be available to everyone else. And that has helped us sort of tease out people who are being dishonest. So the system is evolving and getting better over time. But still, sometimes you get stuff through.

54:32You certainly do. and there are folks out there who are like combing through papers to find this stuff. And there's a scientist, for example, her name is Elizabeth Bick, and this is her passion. She combs through old papers and finds evidence of like Photoshopping in biology. You know, you take a picture of your gel and it's supposed to have this blob and she finds, oh, this blob is the same as that blob or it's been reverted or whatever. And so there are definitely lots of ways that you could find this stuff now that we couldn't have done beforehand. But peer review is not a guarantee that this hasn't been done.

55:06Like, reviewers should look for this stuff and call it out if you see it. But stuff definitely can get through. It's not a perfect system. You know, it's one of these, like, it's a terrible system, but it's also the best that we have so far. Just an aside here, I am a massive wimp. If I, like, purposefully fabricated data in this era where everything is, like, public and people can peek behind the curtain, I would never sleep again. I'd be like, someone's going to find me out. And it would be like, I would be miserable. the rest of my life. I'd rather have a bunch of low impact papers than worry that I was going to get called out for lying.

55:40But anyway, I am a wimp. Yeah. And so the highest standard really is not just that something has been peer reviewed, but that something has been independently replicated. Like you might have a scientist who's doing totally solid work and see some effect in their lab, but doesn't realize that it's an artifact due to some condition, the humidity or the local gravity or the trains or something are causing some influence on their experiments. And so you want people on the other side of the world who built it differently, who made different assumptions, who sensitive different stuff to reproduce it.

56:11You might remember this excitement about high temperature superconductors a couple of years ago, LK99. Korean scientists claimed to have created this room temperature cheap superconductor, which would revolutionize the industry. And so very quickly, people were out there trying to replicate it. And people were excited, but until another independent group built it and showed that it was real, nobody really accepted it and thought we were in a new era. And personally, for example, when we were discovering the Higgs boson, I saw the data accumulating around a mass of 125. We were looking at it constantly.

56:45It was building up and up and up. But until I heard that my colleagues around the ring also saw an effect at the same place, I didn't personally think, okay, yeah, we found this thing. And so this sort of independent replication is really sort of the highest standard. What do you think, Kelly? I think so. And to me, this is one of the current weaknesses in science, at least in my field. So replicating data is absolutely critical. But if you write to the National Science Foundation, for example, and you're like, oh, I just want to do the same experiment that Maria did, but I'm going to do it on another continent, it's going to be hard to get money for that because it's not a new idea.

57:23And it's also probably not going to get published in a top tier journal. And so if you are training PhD students who are going to want to get jobs and they're just quote unquote, just replicating somebody else's work, that's not going to help them get a job as much as following up on their own exciting new idea. And so we've got this incentive structure that's actually not super great for encouraging people to replicate each other's studies. But I do think it's absolutely critical. And I would love to see us sort of work on that incentive structure to make replication just as important as the initial thing that got done.

57:56That's interesting. In particle physics, there's maybe a slightly healthier environment. We have a few signals of new physics that we've seen in experiments that we're all curious about, but nobody really accepts because it doesn't quite make sense. And then there's been another generation of experiments to follow up on those. So for example, there's a very significant signal of dark matter, an experiment in Italy called DAMA. And nobody really believes it because we've never seen it in another experiment. And people have set up other experiments very similar to DAMA, but in another part of the world with different conditions in a different cave, for example, and not seeing the same effects.

58:31And so those were experiments definitely inspired by this signal to test this in other conditions. And in the last few years, there was this quote unquote discovery of a fifth force in this experiment in Hungary. And folks in Berkeley are trying to replicate it. And there's an experiment in Italy trying to probe it as well. And so there's definitely like follow-up work. But usually those follow-up experiments are a little bit broader and they try to not just check this one new result, but also learn something else along the way to make it so they're also covering new ground. But I agree, we should definitely have replication.

59:04But it comes back to funding, right? Like if you're a reviewer, what would you rather fund? Like let's check Kelly's study or let's do this brand new thing that could tell us something new about the universe. Yeah. And when I write a grant, I try to work my own replication in there. Like as part of asking a new question, I'm going to repeat the experiment, but maybe put a little tweak, but then I can at least make sure that I'm still getting the same results in, you know, a different space or with slightly different lighting or something like that. And so, and, And, you know, especially when, to defend my field a little bit, especially when animals need to be euthanized as part of the experiment, you know, you might be a little bit hesitant to be like, well, this is just for replication.

59:41Well, if you already got an answer, do animals need to? So anyway, so we do also try to ask additional questions while trying to replicate. But it would be nice if we had more incentive for that. And there's folks out there just doing this. Like you might have heard of the replication crisis. That comes out of people finding papers in the literature and saying like, all right, let's reproduce this. Let's see if it holds up. And this is one reason that like p-hacking is a thing people talk about because we've discovered that some of the results in literature are just statistical artifacts that were selected in order to get a paper out there.

1:00:14And so I think what you're seeing in a broader sense is science is self-correcting. Just like we saw that we needed to add external reviewers and members of the public when we're talking about the experiments you do on animals. Now we see like, okay, we need some sort of way to protect against this kind of abuse as well. And so, you know, the process is a living thing. It's not like science is a crisp philosophy. What we call science has changed over the last 10 years, 50 years, 100 years, and it will continue to evolve. And I hope keep delivering great truths about the universe. Yeah, me too.

1:00:47And so Stephen asked, how can you know if something is good science or not? I feel like that was the big push behind the question. And so what do you think? When you read a paper for the first time, what do you look for? Well, you know, science has to stand on its own. So the thing I look for is like, does the paper make sense? Does it lay out an argument? Do the conclusions follow from the evidence presented? That's the most important thing to me. But before I'm going to personally believe that something is real part of the universe, yeah, I need to see it reproduced by another group. I've often reproduced papers myself, especially if they're like statistical, just to make sure I understand like, how is this calculation being done exactly?

1:01:24How do you get from step three to four? Because I really want to understand what are the assumptions being made and think about whether those are broad enough. So I would say that yes, peer review papers can be wrong, but that's part of science. And the highest standard I think is independent replication. What do you think, Kelly? Yeah, no, I agree. I mean, when I read a paper, I'll look to see what journal is it in. So lately there are, in this day and age, there are some new what we call predatory journals where they will encourage people to submit, but they publish just about anything they get.

1:01:57And peer review isn't so much about peers having a chance to turn down a study, but they'll like give a little bit of input, but the paper gets published one way or another. So you need to look to see what kind of journal it was published in. And then, you know, if it's a field I know about, I'll look to see, did you cite the papers that I would expect you to be citing? Was your literature search deep enough? And were your experiments designed well? What else could those results have meant? And then who funded the study and stuff like that? But, you know, if you are a member of general public and you can't do all of that, and you're just reading a pop-sci summary, I would look for, again, like, where is this being published in?

1:02:33You know, like, if it's The Atlantic by Ed Young, it's probably great. And if the science journalist had other scientists weigh in on what might have been wrong about the study, that's a good sign. And so you look for where you're getting the information, how critical they seem to be, and stuff like that. What do you look for in a pop sci article? Yeah, in a pop sci article, I definitely look to see whether they have talked to people who are not authors, you know, other people in the field who know this stuff. And is it just a press release from the university? Or is it a journalist who's actually thought about this stuff and written something up?

1:03:09Those are definitely the things I look for in pop sci press releases, because there's the temptation and the sort of marketplace of ideas to over inflate the meaning of an incremental study. Yep, yep. All right. So in general, I would say that, you know, we have this process in place to try to make sure that the best ideas are the ones that move forward and that we're all checking to make sure no one missed anything important along the way. Some stuff gets through either by mistake or because some people are unscrupulous, but hopefully that doesn't happen that often. But, you know, the system is evolving at all times.

1:03:40And if you have a study that you're interested in, but you don't know if you can trust the press release or whatever, you can send it to us. And if it happens to be in our wheelhouse, We're happy to weigh in and tell you what sort of, you know, set off our alarm bells in our heads or what we liked about the study. And we're happy to help people figure out what was done well and what was just squeaking through. All right. Thanks very much, Stephen, for asking this question and for shining a light on the inner workings of science. And if you'd like to reach out to us, send us an email at questions at DanielAnnKelly.org.

1:04:14And let's see what Stephen had to say about our answer. Hi, Daniel and Kelly. It's Steve here. Thanks so much for answering my question. I think you shed a lot of light on some topics that most of the general public are not aware of. And I really appreciate that. It's actually really fascinating to learn that just because something is peer-reviewed doesn't mean it's 100 % fact. And it's definitely a lot to take away here. So I appreciate you guys diving into the topic and looking forward to the next episode.

1:04:53Daniel and Kelly's Extraordinary Universe is produced by iHeartRadio. We would love to hear from you. We really would. We want to know what questions you have about this extraordinary universe. We want to know your thoughts on recent shows, suggestions for future shows. If you contact us, we will get back to you. We really mean it. We answer every message. Email us at questions at danielandkelly.org. Or you can find us on social media. We have accounts on X, Instagram, Blue Sky, and on all of those platforms, you can find us at D and K Universe. Don't be shy. Write to us. I'm Mange Shatigular, and I'm back with a new season of my podcast, Skyline Drive.

1:05:35This time I talk to scientists, biopunks, curmudgeons, blues owners, super seniors, and Goa's top cryotherapy lab to try to understand this obsession with living forever and what it means for all of us. And I get into a bit of trouble along the way. I'd say probably start bone smashing. That doesn't work. Make it look more defined. They say it works. I don't know. Listen to Skyline Drive, How to Live Forever on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts. This is Michael Rappaport and my podcast, the I Am Rappaport Stereo Podcast, is unlike anyone you've ever heard. If you're looking for strong opinions about sports, entertainment, politics, pop culture, and whatever else catches my attention, then subscribe now.

1:06:19This kid, Jafar Jackson, should absolutely, positively get nominated for his portrayal as Michael Jackson. Listen to I Am Rap Report on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts. Can superstars even exist the way they used to? 2016 was sort of that last era of monoculture where we still consumed things in community. Everybody wanted to be Beyonce at that point. I don't think we'll ever see another Rihanna. What does it mean to be Black and eat in America? You will never make me feel bad for being a Black girl, for being a Black American girl, ever. From music to food to the conversations shaping Black culture right now, Therapy for Black Girls is bringing it all to the mic.

1:07:02Listen to Therapy for Black Girls on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts. Hey, it's Nora Jones, and my podcast Playing Along is back with more of my favorite musicians. Check out my newest episode with Phineas. So I went home and I asked Billie if she wanted to sing it. She immediately made it her own thing. You really know how to make me cry when you give me those ocean eyes. And I'm scared. So come hang out with us in the studio and listen to Playing Along on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts. This is an iHeart Podcast. Guaranteed human.