Do multiple choice questions make us biased?

0:00This BBC podcast is supported by ads outside the UK.

0:30at Whole Foods Market. Picture this. You're with your spouse at a seaside botanical garden. The moss hangs low, but your spirits are high and you've never felt more in love. This is when it hits you, that Myrtle moment. It's the second everything clicks when you realize Myrtle Beach is more than a getaway. It's right where you belong. Don't just daydream about it. Find your own Myrtle moment at visitmyrtlebeach.com.

1:06Right, which one are you picking this time? Number two. OK, you're choosing number two. In the office of a mathematician, I'm sat in front of a series of upturned cups, all of which are carefully labelled. All right, one, two, three. I'm going to pick number three. I'm here to learn about the mathematics of probability, but there's been a slight deviation. Another sweet! I'm trying to win sweets. You chose the right one first that time, so no sweet for you, sweet for me that time. Honestly, if maths at school had been like this, I think I'd have been top set and ended up as a prize-winning mathematician.

1:44Might be a bit old, by the way, there's not really props, though. A prize-winning mathematician with terrible teeth. Oh, wow, that is... Is that really rank? That's like a pear drop. Why am I putting my dental health on the line? Well, this is CrowdScience from the BBC World Service, the programme that peers under the cups of scientific mystery to uncover the sweet truths beneath, with the help of questions from our listeners, and they're never stale. I'm Alex Lathbridge, and today I'm hunting down answers to a question from Listener Griffith in Accra, Ghana. here's the thing listener griffith isn't just a crowd science listener he's also a listener to another show here at the bbc world service science headquarters maybe you've heard of it as well it's called unexpected elements and if they're known for one thing it's the halftime quiz hello i'm alice and it's time for the unexpected elements quiz this week we've been inspired by Basically, I realised that the answers to the quiz questions were quite reliably B and C and not A.

2:53So the question was, was the bias for the B and C intentional or unconscious? And then are there ways as a student to explore this bias to make better choices in exams? Have a think about that and I'll be back later with the answer. Science is a wonderful thing. It gets us to the moon and back, helps us comprehend the mysteries of the cosmos and decipher the depths of our DNA. More importantly, it can help you cheat on your exams. I remember when I was also in school, I came up against many multiple choice exams. I remember there was a joke, a running joke, that if you don't know the answer, choose B or C or D and it's almost never A.

3:38And so sometimes I used to do that. I don't know how successful I was, but I just used to do that when I didn't know the answers. You never picked A. You always went B, C or D. Yeah, yeah. Why? Well, I guess, like, if I want to hide something from someone, I wouldn't put it on the surface. like if I was a lecturer I wouldn't want the answer to jump out at them immediately if they don't know you know a is the first answer that will come and b and c and d go deeper and deeper and deeper that's my logic behind it well unfortunately for the unexpected elements team they sit on the same row of desks as me and there's no escaping the eye of scientific scrutiny so I grabbed the first two Unexpected Elements producers I could find to answer for it.

4:30Hi, my name's Lucy. I'm one of the producers on Unexpected Elements. I've been working on it for about six months. And my name's Alice. I'm also one of the producers on Unexpected Elements and I've worked on it for about a year and a half now. What makes a good quiz question? It's got to be surprising. Okay. Yeah, I feel it can't be too hard either. You can't do something so obscure that no one's got any chance of getting it right. Right. And also at least one decent science pun, I think, per quiz. Yeah, I think puns are essential. So our listener, Griffith, he's noticed the answer to the quiz.

5:02It's hardly ever A. It's always B or C. And so the real question I have is what have you got to say for yourselves? Is this corruption at the heart of unexpected elements? It's anti-A bias. I think it's an acceptable form of corruption. the way I write the questions it just ends up being C like say it was like how tall is like this mountain I want the answer to be the most surprising so it's generally going to be the biggest one it just said and naturally ends up being C and I've tried to write it differently but it just feels wrong whereas Lucy tends to go for C I'd like to go for B so I'd pick like something nice in the middle you know that might be the more logical answer but I have to say when I sort of realised I was going for B more often, I then thought, oh, you know, I'm going to throw in some A answers now because, you know, I've just been biased towards B.

5:50And I have to say, I barely ever go for C, which is the one that Lucy is a seagull. So, you know. See, I've been trying to throw in some A's and sometimes it does feel just sort of impossible. Do you think then that making option A be the answer is somehow too obvious? Yeah. And I think if you got the answer straight away, if you knew it, you're going to be like, oh yeah, that's it. And then you might turn off the rest of the quiz. Yeah, we want people to listen to the whole quiz. I don't want people to switch off after option A, because they're like, oh, I've got it now. I want them to listen to my voice for the whole minute.

6:19I've put the effort in. It is a lovely voice as well, so, you know. Are the unexpected elements producers unique? Or can we all fall foul of this pattern-picking behaviour in our day-to-day lives? Well, this is the point where I needed to enlist the help of an actual mathematician. Hi, I'm Kit Yates. I'm a professor of mathematical biology and public engagement here at the University of Bath. So our listener Griffith wrote in because he spotted that the answer to the unexpected elements quiz is always B or C, but hardly ever A. Now, you're a mathematician. What's going on? So I think if you were genuinely choosing these answers, placing them at random, sometimes it would be A, sometimes it would be B, sometimes it would be C.

7:04but we tend to exhibit a number of biases which mean we're not very good at being random it might be that we're being influenced by something called middle bias which is our propensity to choose the middle most option when we've got a bunch of options to choose from so you see middle bias in sat tests where students get a choice between four options a b c and d and if they genuinely don't know the answer they tend to choose b and c with a higher propensity than a and d You also see it in sort of games like battleships. People tend to put their battleships disproportionately towards the middle and away from the edges.

7:39You even see it in people's choices of toilet cubicles, apparently. The thing is, this middle bias might skew our instincts in multiple choice questions, but it actually exists for a good reason. Kit says that in normal life, if you don't know something, it's actually a great tactic to guess towards the middle of your options. For example, if I had to guess a stranger's height with no other information, it would make sense for me to guess somewhere around the average human height. Some things are distributed according to what's called the normal distribution or the bell curve. So-called because when you map it onto a graph, it does kind of look like a bell, small on one end, large in the middle, and small at the other end.

8:22Men's heights or women's heights, for example, are distributed according to this bell curve, where genuinely more of them do sit towards the middle than towards the outer edges. So it's an okay assumption in those cases, but when we're trying to genuinely uniformly, so equally choose between different options, that's when we really fall foul of middle bias. So why are we susceptible to fallacies like these? I think the experiences that we have had in the past, both in our lives, but also evolutionarily have primed us for spotting patterns and we think we see patterns in even random data this is this phenomena called pareidolia where we we seem to spot a pattern in in a random stimulus like um seeing jesus in a piece of burnt toast back in the day when we were out hunting or gathering we're out in the woods and we see a sort of stripy pattern in the trees it could have been just the sun shining through the leaves or it could have been maybe a tiger.

9:19It was important for us to be able to spot those patterns, but sometimes we get it wrong when we interpret what those patterns mean. The brain's amazing ability to spot patterns amongst the noise has come at a cost. We struggle to deal with data that is patternless, true randomness. That's why the unexpected elements producers are so reluctant to make the right answer A. A just doesn't feel very random, like the vibes are off. And this is true for everyone, even the really clever people who understand maths, like Kit, they are biased in favour of answers that feel right. I'll give you an example, the birthday problem.

10:02So the birthday problem basically says how many people do you need to have in a room before it becomes more likely than not that two people in that room will share a birthday. So the probability of two people sharing a birthday goes above 50%. Okay. 365 days in a year. Let's say you need 182. That is the answer that most people would give, right? So you divide 365 by two and you get to about 180 something. But actually the answer is surprisingly small. It's just 23. How? It's crazy, isn't it? It's totally mad. So by the time you've got 23 people in the room, there are 253 different pairs of people.

10:46Assuming all those pairs are independent... This is the point where things became exponentially more complicated and I completely zoned out. To the power of 253 or multiply it by itself 252 at a time. The point is, with 23 people in a room, the chance is just over 50 % that two people will share a birthday. Even though that's true, it just feels wrong, you know? If you want a proof of that, you can go to the World Cup, right? In the World Cup, they have squads of 23 players. And if you so desire, as I do, to go onto Wikipedia and to download all the birthdays of all the squads of all the teams that play, you can then match up and see how many of those squads of 23 players have two players that share a birthday.

11:29And actually, in the Men's World Cup in 2014, it was exactly half of them. It was 16 out of the 32 teams. So you see it in the real world. So, taking it back to multiple choice questions. One of the most famous examples of the way that our instincts about probability can let us down is in a problem devised in the 1970s, known as the Monty Hall problem. It comes from a game show which was called Let's Make a Deal, and the host of that game show was a guy called Monty Hall. There were three doors that you had to choose, numbered one, two and three. and behind one of the doors was a car and behind two of the doors was a goat.

12:07So you're potentially going to end up with a goat two times out of three, but you might win this car, okay? The way it worked was you got to choose a door and then the host, Monty, he would open another door which had a goat behind it and then you would be left with these two doors, one with the goat behind it, one with the car behind it and you had to choose whether you wanted to stick with your original choice or whether we wanted to switch and go with the other alternative. What should you do? Should you stick or should you twist? What do you think? It's a 50-50 chance at that point. Yeah, that's the intuition, but it's actually not a 50-50 chance.

12:40Actually, if you switch two times out of three, you're going to win the prize and only one time out of three are you going to end up getting the goat. It feels like a 50-50 chance, but it isn't. So I'm sat in Kit's office completely baffled. Fortunately, Kit came prepared. Tell you what, let's play the game. And because mathematicians hate small sample sizes, we're not going to play it once. We're going to play it ten times. We've got ten sets of three cups, numbered one, two and three. So under one of the cups is a sweet and under the other two cups there's nothing. OK, let's begin. So I'm going to start with cup number one.

13:22Okay, then I am going to show you cup number two. I pick a cup, but I don't look under it. Then Kit removes one of the empty cups from the game. And then I switch my choice to the other remaining cup. So you switch. All right, I'm going to pick number three. According to the maths, this should be a solid suite winning strategy. There's a suite underneath. Very good, next round then. I win a suite, then another. I win four suites in a row and I begin to realise that I am a genius. Another suite! Oh, smashing it. Take that, statistics. Right, next one that you're going to choose. Then my luck turns.

14:05So that time you picked the right one first, so you didn't end up winning when you switched. In total, using Kit's switching tactic, I win the game six times out of ten. So it's close to what we would expect, which is 66%, right? Two thirds of the time that you would expect to win. Okay. So basically, because there's more wrong options, there's only one correct option. There are two wrong options. So 66 % chance you'll pick a wrong option. And then you switch. You always get the right one. Yeah. And that look on your face when it was like, oh, it's about what you choose first. And then you swap, you always get the opposite.

14:42That's it. That's the magic of understanding the probability. It's a beautiful thing when it comes. Yeah, yeah, it's great. So our pattern spotting brains have given us these tricky cognitive biases, which skew our decision making. But if you think that multiple choice questions are the only way that these biases trip us up, you're very much mistaken. More brain boggling examples for you coming up next. This is summer at its peak. Whole Foods Market Summer Fruit Fest is your invitation to eat the season. Fresh, organic, and bursting with flavor. Start your day with peaches and organic blueberries and yogurt.

15:22Build a grazing board with fresh fruit, prosciutto, and artisanal cheese. Then fire up the grill with no antibiotics ever proteins and fresh produce. Savor the season. Shop Summer Fruit Fest at Whole Foods Market. Picture this. You just got a romantic beachfront massage. The fun is flowing. The relaxation is real. and each drink tastes like a well-earned reward. This is when it hits you, that Myrtle moment. It's the second everything clicks. When you realize Myrtle Beach is more than a getaway, it's right where you belong. Don't just daydream about it. Find your own Myrtle moment at visitmyrtlebeach.com.

16:12you're listening to crowd science from the bbc world service the show that's gambling with our own curiosity and trying to win the glittering prize of scientific knowledge i'm alex lathe and i'm trying to unravel the unconscious cognitive biases that live deep within our brains thanks to a question from listener Griffith in Ghana. They can really trip you up in unexpected ways, particularly when we're playing games of chance. So my husband and I were in Las Vegas at a very low-end, inexpensive casino. We were playing a little bit and we were watching people playing. This is Rachel Croson, queen of the roulette table, master gambler of the Las Vegas Strip and professor of economics at the University of Minnesota.

16:59I do research in experimental and behavioural economics, how people make economic decisions, what mistakes they might make and how you can help them make better decisions. Speaking of bad economic decisions, back to Vegas and the 10 cent roulette wheel. There was one guy there who had a piece of paper and he was very carefully keeping track of all the numbers that had come up. And he had data for hundreds and hundreds of spins, so he must have been there for hours. If you're not familiar with roulette, let me paint you a picture of what's going on here. A roulette wheel is a sort of spinning disc laid flat on a table, and there are little slots all around the edge of it.

17:39Each slot is numbered from 0, or sometimes double 0, through to 36. And to play, the dealer spins the disc around, and then throws a tiny ball into the middle, where it bounces, almost chaotically for a while, and then eventually comes to rest in one of the little slots. You can bet which of the numbered slots that you think the ball is going to land in. And if it lands in your slot, you win. Of course, with so many numbers on the wheel, it doesn't land on your one very often. Probably around one in every 38 times. Or at least that's how it should work. So we look at his data and there is indeed one number, I think it was 18, that had come up way more times than you would expect.

18:29And so we talked to him and we said, so what's going on with number 18? And it's possible that you would actually have a biased wheel in a very low level casino. Of course, as the wheel was biased in favor of 18, you should be betting 18. But that's not what this man was doing. He said, it's come up so often, it's done. And he said, I'm not going to bet 18 anymore. So this was so amusing and frustrating to us because he had collected all the right data and he had come to exactly the wrong conclusion. That number is done. It's over. It's never going to come up again. What was going on in his head?

19:04Yeah, that's a great question. Of course, it's always hard to tell what's going on in any one individual person's head. And so that's really what led me to the gambler's fallacy. The gambler's fallacy, just like middle bias, is one of those tricky little cognitive biases that trip us up when maths doesn't work the way that we think it should. Say you and I are playing a game, flipping a coin. Heads, you win. Tails, I win. Let's say it comes up tails twice, then three times, then four times in a row. If you find yourself thinking, oh, four tails in a row, it's got to be heads next, then I'm afraid that you just fell victim to the gambler's fallacy.

19:44Because the truth is, the chance is still exactly 50-50. The wheel has no memory of what came before. So the previous spins, the previous outcomes, have no predictive value about what's going to happen next time. And the gambler's fallacy, in contrast, says, I believe that once something has come up in the past, it is less likely to come up in the future. Rachel also studies another common fallacy, which pulls us in the other direction, called the hot hand fallacy. Let's say that we flip the coin over again, and this time it comes up heads. Again, let's say four times in a row. Now you're winning, over and over and over.

20:23If you find yourself thinking, oh, I'm on a winning streak, it's got to be heads again for sure. I'm on a roll, I'm on a streak, I can't lose, right? Everything I do wins. I'm afraid that this time you've fallen victim to the hot hand fallacy. So the gambler's fallacy is a biased belief about the outcome of a gamble. The wheel's going to come up red, the wheel's going to come up black. The hot hand, in contrast, is a biased belief about whether I'm going to win or lose. It's a belief about me. So if I'm hot, whether I'm betting on red or black, I'm going to win. It doesn't matter. When it comes to researching the gambler's fallacy and the hot hand fallacy, are you able to do it at the university or are you having to go down to Vegas?

21:07So yes and yes. You do a lot of it at the university. You do it with students in labs. You do it with coin flips and dice rolls. But at some point, you're going to want to look at people making real significant gambling decisions in the context that they would normally make those decisions in. So when we did our experiments with the hot hand, the gambler's fallacy and roulette, We originally had planned to send a PhD student to go and stand behind the roulette table and observe all the bets people made and write them down. And we were all set to do this. And then the individual at the casino we were working with said, oh, no, no, don't do that.

21:44We'll just send you the videotapes. And so our poor PhD student who had been hoping to go to Las Vegas and spend about a week collecting data instead was in the basement in Philadelphia watching videotapes frame by frame and coding up everybody's bets. So in that study, what were you doing apart from making this boy very, very sad? So we were looking for how betting patterns changed when particular outcomes happened one time, two times, three times, four times, more than five times in a row. And we saw whether people changed their bet to the other one. So that's kind of a very clean test of the gambler's fallacy, right?

22:24If after reds come up four times, you switch to black or you stop betting red, that means you think red's less likely. Then the second thing we did for the hot hand is look at how people's bets changed after they won versus after they lost. And if after they won, they increased their bet, that was a signal that they thought they were hot. They're betting more. And what did you find there? So generally speaking, those beliefs are correlated with each other, but that correlation is not 100%. And they are absolutely people who believe in both the hot hand and the Gamos fallacy. So they're related, but they're not identical.

23:01And do we see it in sort of real world scenarios outside of the casino? Absolutely. I also teach a class on behavioral finance. And one of the regularities in how people deal with their investment portfolios is called the disposition effect. And the disposition effect means that people tend to sell their winners, their stocks that have appreciated, and hold on to their losers. And this is exactly what you would expect if people had a gambler's fallacy belief. Because this stock has gone up, it's due to go down, and so I should sell it. Or the stock has been going down, it's due to go up, and I should hold on to it.

23:37Why do these fallacies happen? Is it something intrinsically human in our brains? Yeah, that's a great question. There's been a lot of speculation about that. Certainly, there's a body of research that suggests that while roulette wheels and coin flips are indeed autocorrelated, the probability of what happens next is independent of what happened before, most of the world probably is autocorrelated. If you're a hunter-gatherer and you're looking for where the wheat is, the most likely place that the wheat is going to be growing is where the wheat grew last year, right? So you can absolutely imagine situations where having biased beliefs could be a positive rather than a negative.

24:19So there are all sorts of cognitive biases that can trip us up in games of chance and probability. But of course, some of the best games are less about chance and more about skill. So what kind of cognitive biases might we be vulnerable to when playing a game that mixes chance and skill? Poker is a game of skill with a chance element. In poker you can, and this is the absolutely crucial thing, in poker you can win while holding the worst hand. You can outplay other people and win. Now this is someone who knows a thing or two about games of skill. My name is Maria Konnikova. I am a writer, a psychologist.

25:04I worked on self control and risky decision making. And I also happen to be a poker player. How did this come about? How did you first come to poker? That is a wonderful question, Alex, because I did not grow up playing games at all. I had no idea what poker was. And I actually got into it from the perspective of game theory. So after my second book came out, which was the confidence game, the book was doing incredibly well. I was on book door, everything was great. And then I got very, very sick. And nobody could diagnose what was wrong with me. And just everything started going wrong all at once.

25:45You know, my mom lost her job, my grandmother died, my husband lost his job. And I realized that I wanted to write about the role that chance plays in life. And one of the books that was recommended to me was John von Neumann's Theory of Games. And I was absolutely floored to find out that one of the geniuses of the 20th century was a poker player. And that von Neumann believed that solving poker would basically solve the most complex strategic decision making in the world. And so I basically thought, OK, you know, what is this poker thing? Why don't I learn this game from zero, literally? When I started, I did not know how many cards were in a deck.

26:29And why don't I use poker kind of as a metaphor for life, as a way of looking at decision making in a very practical way? What started as a thought experiment? Well, let's just say it kind of took off. These days, Maria is a pretty significant figure in the poker world. Although she wouldn't describe herself that way. I have won one World Series of Poker bracelet and I've won some pretty big titles. But honestly, I am a mediocre poker player who's just gotten very lucky. When you came to poker, you came as a complete novice. Yes. Well, background in psychology and really understanding the mind.

27:09So what were some of the cognitive biases that you had to contend with? Did your instincts lead you astray in any way? Oh, I mean, I think that you have to contend with all of your cognitive biases all the time. I mean, it doesn't matter how good of a psychologist you are, you're still going to exhibit them. I came with all of them. I mean, you name it, I had it. A sunk cost fallacy, hot hand fallacy, gambler's fallacy, risk aversion, loss aversion. And poker is a kind of laboratory environment where you can clean up some of those biases because there is money on the line. When you act in a biased way, you lose.