Did two honesty researchers fabricate their data?

0:00Support comes from our 2025 lead sponsor of Planet Money, Amazon Ads. Less than half of customers fully trust their financial institutions. Amazon Ads helps financial brands build that trust through relevant ads using first-party signals to reach the right customers. More at advertising.amazon.com. This is Planet Money from NPR. About a decade and a half ago, this one idea started showing up everywhere. The nudge. The idea of the nudge came from behavioral economics. Basically said, if you really understood how people think, their psychology, you could make a huge difference in the world just by doing these little tweaks, just by nudging people in the right direction.

0:48Like if you want people to use less electricity, use social pressure. You can send them a letter saying, hey, you're using way more electricity than your neighbors. Or organ donation. How do we get people to sign up to be organ donors? Turns out the secret has to do with the form at the DMV. This is one of the most famous behavioral economists around, Dan Ariely. He's giving a TED Talk that's gotten more than 10 million views. And yeah, he's right. Some European countries require people to opt out of being organ donors instead of opting in when they're getting their driver's licenses. And making people opt out, that gets a lot more people on the organ donor list.

1:25Ariely, here in a different TED Talk, he's given about a dozen. So what have we learned from this about cheating? is talking about some of his own research, focused on how you can get people to be more honest, with simple reminders. A lot of people can cheat. They cheat just by a little bit. When we remind people about their morality, they cheat less. Ariely has had some blockbuster research into the subject, why people cheat and how to make them more honest. In one study, he asked people to recall the Ten Commandments, You know, like, thou shalt not this, thou shalt not that. And then he had them play this game where they would be tempted to cheat.

2:04The moment people thought about trying to recall the Ten Commandments, they stopped cheating. In fact... Another one of Ariely's big deal papers was about this one simple trick that would make people way more likely to tell the truth when they're filling out forms or paperwork. For this one, he collaborated with a bunch of researchers, including another huge star of behavioral science, Francesca Gino. Here's Gino talking about that trick in 2015 on Planet Money. We should sign at the top of the form. That's it. Instead of signing at the bottom, we should sign at the top. Exactly. It's that simple.

2:39Makes sense. When you fill out a form and then at the very end you're asked to swear that everything you wrote was true, uh, was it? Close enough. But if you swear to tell the truth ahead of time, your brain is primed for honesty. Gino and Ariely, they became these academic superstars for all this nudge research, especially for their papers on how to get people to be more honest. And nudges? Well, nudges became all the rage. In the UK, the government created an entire organization. It's called the Behavioral Insights Team, or as they were nicknamed, the Nudge Unit. That's Michael Sanders. Back in 2014, he was the Nudge Unit's chief scientist.

3:21It was his job to test out nudges all around the world. Yeah, and Michael still remembers the moment this famous sign-on-the-top idea was brought before his team. He was at his office in London, and one of Gino and Ariely's co-authors came in to give a presentation on the research. What was the mood like in the room? I think, you know, we're British, so we don't have electrified rooms. People don't sort of, people don't buzz like they do in America. But, you know, I exchanged glances with a couple of senior colleagues. And it's like, we could do this in this area. We could do this in this area.

3:56There are so many opportunities for this. This sign on the top idea? It was the quintessential nudge. So clever and so easy to implement. And so surprisingly powerful. It's just so simple. It is. Many lies are simple. Allegedly. That quintessential nudge idea was clever and was powerful. And also, it was a lie. Hello and welcome to Planet Money. I'm Nick Fountain. And I'm Jeff Kuo. Dan, Ariely, and Francesca Gino have played this central role in taking nudge theory mainstream. And they've done really well for themselves. Gino's this big shot professor at Harvard Business School with all these speaking gigs and book deals.

4:42Arielli's got this fancy job at Duke with multiple best-selling books. One of them's even being turned into a TV show by NBC. But now their famous sign-on-the-top paper we were talking about before, that paper is at the center of a massive scandal. And these two superstars famous for showing how to nudge people to be more honest, they are being accused of dishonesty themselves. Today on the show, the truth about dishonesty, dishonesty about truth, and the spreadsheet detective who cracked the case wide open.

5:22Support comes from our 2025 lead sponsor of Planet Money, Amazon Ads. What do financial customers want more than low fees, promo offers, or premium perks? Trust. It's the number one factor that drives loyalty. Amazon Ads helps financial brands build that trust through relevant ads that reach customers at key decision moments, whether they're planning retirement, buying a home or managing debt. Using rich first party insights, Amazon Ads connects brands with the right audiences where they browse, buy and stream. Learn more at advertising.amazon.com.

6:03It was a lot of work to condense more than a century of contract law into a single 20-minute episode. There was an article that you sent me that had like an old English font at the top of it. And I was immediately like, I'm not reading this one, Jeff. I'm sorry. You gotta be more selective. It's called Research. How we did ours for our recent story on the rise of internet contracts and how they got so sneaky, that is in our latest bonus episode for Planet Money Plus listeners, whose support helps make this show possible. Back in 2014, before two titans of behavioral science were engulfed in scandal, and back when nudges were still becoming the hotness, Michael Sanders was the chief scientist at the British government's nudge unit.

6:55It was his job to take some of these nudges and test them out in the field. You guys were sort of like special economics agents bringing behavioral econ to the far-flung corners of the world. That was sort of your mission? I mean, that sounds very grand, doesn't it? Yes, I'd say that was our mission. Which is why when the government of Guatemala had a problem getting its citizens to pay their taxes, Michael was like, this is perfect for us. Guatemala has among the, if not the lowest level of tax compliance of any country across Latin America. And so the opportunity for up, the opportunity to improve things was just enormous in that context.

7:37You felt like you could make a real difference there? Yeah. The Guatemalan government was losing hundreds of millions of dollars every year to tax evasion, money that could go to building roads and schools and whatever else. Maybe Michael, armed with all his little nudges, could help fix that. So Michael and a small team flew to Guatemala City to meet with tax authorities there and asked them how they collect taxes. And they said, well, one of the ways is called the value-added tax, which is kind of like a sales tax. Every month, businesses are supposed to fill out this form where they declare how much they owe in VAT.

8:12And most people fill in the form. Most people who do fill in the form say, oh, I don't owe any. I don't owe any sales tax. And that's just not really credible, Frank. I feel like in places like the UK and in Europe, the VAT is just part of the sales price, right? I mean, I'm not going to take lectures from Americans about how you should run a tax system. Michael, as you might expect, immediately thought back to that famous paper by Dan Ariely and Francesca Gino. The one where if you make people promise to be honest before they fill something out, they're much more likely to tell the truth than if they promise after they fill it out.

8:54Michael and his colleagues realized they could use this idea on the Guatemalan tax website. Before people entered their taxes, they would first have to pledge that they were going to tell the truth. And Michael tells the tax authorities, there's real research behind this, done by professors from Harvard and Duke. And also to prove to you that this works, we're going to run an experiment to test it. Some people will get the new honesty pledge at the start, and others won't. And what'd they say? They said, let's do it. Let's go. So we were in Guatemala for three days, and we went from, hi, my name's Michael, to we're running this study, and we're going to launch it and we're going to run it for three months with three million tax returns and we'll call you back in four months with the results.

9:39Four months go by, Michael gets the data set and he stays up late crunching the numbers. Did the intervention work? It worked not at all. So we were... Not at all? Not at all. There are few things in my life that I am in a statistical sense more confident of than that this didn't work. But it should have worked. The original research behind the whole sign on the top of the form thing had been done by a team that included two of the biggest stars in behavioral science. And Michael's thinking, I can't even repeat their research. I can't even copy them right, right? It's not like I'm trying to come up with something new and innovative and I'm struggling at that part.

10:20I just can't even copy them in a way that creates the right results. But here's the thing. Michael wasn't the only one having trouble getting the right results. Around the time he published his paper describing his failure in Guatemala, other researchers started noticing their experiments were failing too. Eventually, one of Gino and Ariely's co-authors on that original sign-on-the-top paper convinced them that they needed to revisit their findings too. So they tried to redo one of their experiments from the paper. And this time, it straight up didn't work. This was a big deal. One of the most famous nudge studies couldn't be replicated.

11:00But that didn't necessarily mean anyone had done anything wrong. This is how science works. Sometimes you just get fluke results. In 2020, a bunch of researchers, including Ariely, Gino, and the other original co-authors, published something that says, yeah, maybe this sign-on-the-top intervention doesn't actually work. But more importantly, they also published for the very first time all the original data, all those Excel spreadsheets that they had used. Which, if you had some suspicions about what was going on here, might be very interesting. I'm Yuri. I'm a professor of behavioral science at a started business school in Spain.

11:40In addition to teaching behavioral science, Yuri Simonson has a little side project called Data Collada, where he and some colleagues evangelize to other behavioral scientists about how they should conduct and analyze their experiments. But what the Data Collada folks are really famous for are their investigations into research that they think smells a little fishy. Maybe a researcher juiced his regressions a little bit to get the results he wanted. Or maybe he played fast and loose with his Bayes factors. Those tricky Basians. Anyways, Yuri in particular loves digging through people's spreadsheets for clues on how exactly they were put together.

12:17And why? It's like a puzzle that nobody knows a solution to and then you solve it. I mean, it's like true crime. That's why true crime is so interesting to people and trying to figure it out, right? And it makes for good podcasts, too. Hopefully. Because of Data Collada's reputation, some tipsters reached out to Yuri and said, hey, you know how the original sign on the top paper had three different experiments in it? You might want to take a closer look at experiment number three. The one about car insurance. Yeah, the car insurance experiment. You know how your car insurance company occasionally asks you what the mileage is on your odometer?

12:53That is a sneaky way of asking how much you've been driving. Because car insurers want to charge people who drive more, more. So drivers have an incentive to lie. Which is where this sign-on-the-top intervention comes in. One of the superstar researchers, Dan Ariely, worked with a car insurer. Ariely said he asked the company to send out different versions of the same form to around 13 ,000 customers, requesting them to update the odometer readings the company had on file for them. Half the people got forms with honesty pledges right at the top, and half of them got them on the bottom. And according to the paper, the ones who signed at the top, they were way more honest.

13:34But the tipsters who reached out to Yuri said, there's lots of weird stuff going on in the data. Like, even though when people originally reported their odometer readings, they often rounded to like the nearest thousand or 500. They did not do that when they updated their results. And the tipster said, look at the fonts. Why are half the cells in this column in Calibri and the other half are in Cambria? We were puzzled. Like, why would this happen? Yuri's like, fine, fine, fine. But immediately he hones in on this one thing. This was like piece of evidence seven. And I was like, I don't care about piece of evidence one through six.

14:15Like, this is it. Piece of evidence number seven, the distribution of miles driven. Okay, so usually if you took a bunch of people, what you'd expect is that most of them are driving like a normal number of miles in a year, right? Like, say, 12 ,000 miles. And a few people might be driving way less, only like 1 ,000 miles a year, and maybe a few people might be driving way more, maybe like 50 ,000 miles. But overall, the data would look like a bell curve with a big hump of people in the middle. But when Yuri looked at the distribution of miles driven in Dan Ariely's experiment, he did not see a bell curve at all.

14:53He saw basically a flat line. It just had an impossible shape. Like there were just as many people who drove 1 ,000 miles as there were who drove 2 ,000 miles, as there were who drove 10 ,000 miles as there were 40 ,000 miles, all the way to 50 ,000 miles. And that's just crazy. Yuri says anyone who works with data would immediately be like, there's no way. Like I showed up with people, people started laughing hysterically looking at the screen. Like it was just like a literal reaction. So it was just self-evidently faked. This data set was faked. And as if it could not get any more obvious, even though there were lots of people who drove nearly 50 ,000 miles, it all cuts off right there.

15:35Not a single customer apparently drove more than 50 ,000 miles. When he saw that, Yuri immediately knew the exact spreadsheet command whoever faked the data had used. Round between. So give me a random number in between zero and 50 ,000. And if you do that, you get exactly this spreadsheet. So it's kind of remarkable. So you know how they did it. Yeah. Yeah, we figured out every step of the way, like how the spreadsheet was generated. Yuri had done it. He cracked the case. It's just so I've never seen anything so blatant in my life. It's just incredible. As far as smoking gun evidence goes, this for you is like, this is it.

16:16Yeah, nothing will ever match this. But there was another unsolved mystery here. Who was responsible for this fake data? Yeah, what I wanted to look for was things that would point to who did it. And there was not a great way for him to figure that out. Before Yuri and his colleagues published what they had found, they reached out to the authors of that famous paper to give them a chance to respond. Usually when they do this, he says it can get kind of contentious. But this time, everyone was pleasant. That was very unusual because we were saying, hey, you have some fake data there. And they were like, oh, we should do.

16:49Sorry. And so usually we get, I mean, we were not expecting this. So almost immediately, Dan Ariely replied and said, just to be clear, I was the only person responsible for this in this team. This meaning Ariely was responsible for getting the data from the insurance company. But Dan Ariely was not taking responsibility for forging the data. He basically blamed the insurance company. He said, quote, The data were collected, entered, merged, and anonymized by the company and then sent to me. This was the data file that was used for the analysis and then shared publicly. So, Uri knew the data had been forged.

17:27But without access to the original data from the insurance company, there was no way to prove who did the forging. The case was at a dead end. But not for long. In another experiment in that same paper, Yuri would find another smoking gun. And this time, there were fingerprints. Also, after the break, we uncover a little evidence ourselves.

18:05This message comes from Vanguard. Capturing value in the bond market is not easy. That's why Vanguard offers a suite of over 80 institutional quality bond funds, actively managed by a 200 person global team of sector specialists, analysts, and traders. They're designed for financial advisors looking to give their clients consistent results year in and year out. See the record at Vanguard.com slash audio. That's Vanguard.com slash audio. All investing is subject to risk. Vanguard Marketing Corporation Distributor. So here's where we're at. After Yuri and Data Collada pointed out that the car insurance experiment in that original sign on the top paper had fake data in it, the authors of the paper retracted it.

18:52If you look up the paper online, right there on the top, there's now a warning in red that basically says this paper is no good. But it's not like anyone lost their jobs over it. Now, Yuri and his team had actually gotten another tip that there might be something wrong with the work of one of the paper's co-authors, Francesca Gino. And they start looking at a lot of her research, including one of the experiments she worked on for the sign on the top paper. So they start digging in. Maybe this time they can do more than just show that the data was obviously wrong. Maybe they can find definitive proof of of how the data had been manipulated.

19:30So the experiment in question asked people to solve some math problems. They were told they would get paid a certain amount of money for each one they got right. They did their math, and then they got to say on this official looking form how many problems they'd gotten right. The ones who signed on the top of the form, the paper said, were way more honest. But, unlike with the odometer data, there was nothing obviously wrong with the math problem data. Yuri looked at it. He looked at it again. And then he thought back to this one secret little quirk about the spreadsheet program Excel. Excel remembers.

20:03Excel remembers. And when I first heard about this, I could not believe this was true. But if you use formulas in Excel, and everybody uses formulas in Excel, Excel essentially keeps track of where you move stuff. And it records all your moves in a secret file called CalcChain. Calc chain is something that I think 99.9 % of people have never heard of before. But Yuri, of course, is part of the 0.1 % who has heard of it. He loves it. Yeah, it wants to figure out it's very addictive. Like you get a data set and you start just checking the calc chain file, you know, like immediately. So Yuri pulls up the calc chain file for experiment number one and realizes that a bunch of those spreadsheet cells, they have been moved by somebody.

20:49And all of those cells that have been moved, every single one happens to strengthen the finding that signing on the top increases honesty. Like, how convenient, right? Yes. And this time, Yuri and his colleagues at Data Collada could do more than prove that the data was fishy. They could prove that somebody involved in the research had gone in and actually futzed with the data. And that really sort of wraps it up in a very compelling way, because you have like confirmatory evidence from a complete different source of information. And by all accounts, this was Francesca Gino's experiment. Run under her supervision, the results analyzed by her.

21:32So just to reiterate, two experiments by two different superstars of the field, two researchers who are famous for showing how to make people more honest, Their own studies about honesty are based on lies on fraudulent data. Like, this is bananas. And it's even bigger than that. Yuri's team has identified three more suspicious experiments done by Gino, pointing out what they call, quote, evidence of fraud. She's on administrative leave from Harvard, and two more of her papers have already been retracted. Dan Ariely's story is also getting more complicated. Remember, back in 2021, Ariely more or less blamed the insurance company for the fabricated data that had appeared in his odometer study.

22:19He said, quote, the data were collected, entered, merged and anonymized by the company and then sent to me. This was the data file that was used for the analysis and then shared publicly. Now, for years, the insurance company, which is called The Hartford, hasn't said much of anything about all this. But a couple of days ago, we convinced the company to finally go on the record. The company told us in a statement that they'd pulled the original data set they sent to Dan Ariely. And it looked dramatically different from the published data from Ariely's experiment. The company said they'd only given Ariely data for about 3 ,700 insurance policies.

22:57But in Ariely's paper, he claimed he had data for more than 13 ,000 policies. That's a difference of almost 10 ,000. In their statement, the company told us, quote, though some of the data in the published study data is originally sourced from our data, it is clear that the data was manipulated inappropriately and supplemented by synthesized or fabricated data. The company basically confirmed everything that Yuri and the Data Colada people had said about Ariely's numbers. And the company made it clear that whatever had gone wrong with the data set had gone wrong after they had already given it to Ariely.

23:36We shared parts of the insurance company statement with Ariely, and he responded in an emailed statement. Quote, as I said two years ago, I was responsible for the relationship with the insurance company that provided the data for the paper. I got the data file from the insurance company in about 2007, and I can't tell now who had access to it. Getting the data file was the extent of my involvement with the data. We also reached out to Francesca Gino, but she didn't make herself available to be interviewed for this episode. Nor did representatives from Harvard or Duke. The co-authors of the original paper declined to be interviewed.

24:08But the rest of behavioral science, they are commenting aplenty. So this is a disgrace. This is Michael Sanders again. He's the researcher who tried to bring the sign on the top thing to Guatemala, who left thinking that he had been the one who somehow messed up. Now, he feels like the whole field was duped, not just him. Sometimes it does make me feel like we are stupid when you see Dan Ariely, who, you know, famously says, everybody lies a little bit. And Francesca Gino says, here's my book on how you can succeed at work if you don't follow the rules. And it's like you walk in with an eyepatch and a tricorn hat and a cutlass.

24:58Then a hundred pages in the book later, we're like, oh my God, they were a pirate. I never saw it coming. But Michael says the problem with all these fake studies is bigger than the feelings of a bunch of academics like him. In Guatemala, there was this big investment. Hundreds of thousands of people were put through this experiment that was never going to work. All of that time and money could have been spent on something different, a different intervention with a better chance of working if we'd known. This is not a petty academic squabble about he said, she said. This has real impact in the real world.

25:33So I'm pissed off about that. I'm also pissed off by how stupid it is. so when you see like eminent harvard professor has committed research fraud what you want is really sophisticated clever ways of cheating not oh i thought i'd copy and paste these observations because they were higher and i'll make more result come together this is this is dumb and that annoys me but it's also it's also a source of terror because that means there'll be people out there who have cheated in a really sophisticated way who we haven't caught and who we may never catch. Where Michael is, that's where his whole field is right now.

26:20Because of the questions around Dan Ariely and Francesca Gino's work, this cloud of uncertainty now kind of hangs over all of behavioral science. Like, what else might not be true? Ariely and Gino's research offered this really tempting story. That if you're just clever enough, you could come up with all these powerful nudges. One simple trick after another that could solve all the world's problems. But now, that story is starting to look like just that. A story.