Are polls misleading Americans?

0:00This BBC podcast is supported by ads outside the UK. If you are currently overpaying on software to run your business, remember this number. 10 ,000. That's the number of new businesses that join Odoo per month. Join Odoo today at odoo.com. That's O-D-O-O dot com. This is Will Friedle from Podmeets World. Let's be honest, the 90s gave us some great things. The music? Totally rad. The fashion? So fly. Home decor? It's a different story. In HGTV's new competitive series, Totally 90s House, two teams of iconic TV stars take on two homes frozen in time. Paired with HGTV experts, they'll turn dated spaces into fresh modern homes and compete for the ultimate bragging rights.

0:46Goodbye, honey oak cabinets. Hello, 21st century. Totally 90s House, special series premiere starts August 26th at 9, 8 central on HGTV. All right, Sumi, we are about two months away from the midterm congressional elections. Everyone is looking about who might win, who might lose. They're looking at polls in particular and what polls might be telling us about the chances for Democrats to respect control of Congress. Yeah, everyone is trying to read the tea leaves by looking at these polls. And if you look at them right now, they indicate that Democrats are leading in a lot of these races, that they have a good chance to win back the House of Representatives, maybe even an outside chance to take the Senate back as well.

1:30But we have to keep in mind that polling is not an exact science and they give us a snapshot, an idea of where things are heading. But we cannot definitively say what they're telling us. And of late, there have been a few polls that have not been spot on. Yeah, I think people will think about 2016 and the polls indicating that Hillary Clinton might have an edge going into that race against Donald Trump in 2020 when the polls suggested that Joe Biden was going to comfortably win the presidential race. And it ended up being a lot closer than we thought. So all of this has people wondering, are polls still accurate?

2:03What's wrong with them? are the polls that we have seen in recent races and primaries this year that have been pretty remarkably off. Is that a sign of things to come? So as polling season, Anthony, gets underway, we're going to, in this episode, give you all you need to know when you hear broadcasters like myself shouting numbers at you. Welcome to AmeriCast. AmeriCast. AmeriCast from BBC News. You hear that sound? Oh, I think when I hear that sound, it reminds me of money. We didn't start this war, but under President Trump, we are finishing it. This is a big cover-up, and this administration is engaged in it.

2:42This guy has Trump derangement syndrome. I have four words for you. Turn the volume up.

2:54Hello, this is Sumi in Washington, D.C. And it's Anthony here with Sumi in Washington and the BBC Bureau. So, Anthony, we're actually going to have a pollster on in a moment who will kind of pull back the curtains on the mysterious world of polling. But explain why we're talking about this right now, Anthony, why we're kind of taking a closer look at the process behind polling and the numbers that we're seeing. Well, one of the big reasons is that there have been some pretty remarkable polling misses so far this primary election season, one in Michigan where Abdul El-Sayed, the progressive candidate for the Democratic nomination for Senate, looked like he had a big lead.

3:30And then when it came down to election day, he only won by a few percentage points over Haley Stevens, the congresswoman from Michigan. And people started to wonder, why were the polls off on this? And then just a week later in Wisconsin, there was an expectation that Francesca Hong would win the Democratic governor's nomination. She was a Democratic socialist. In fact, so much so that Mandela Barnes, one of the rivals of hers, dropped out saying that she was going to win. And then it turns out she didn't win. And a different candidate, David Crowley, ended up pulling it off. So there's a couple of instances now where we've seen polling tell us one thing and either the results are a lot closer than we expected or they were just flat out wrong.

4:14So that has people second guessing themselves. That has raised all of these concerns we discussed about whether polling is accurate or not dating back over a decade. And we need to figure out what can polls tell us and what can't they tell us? Yeah, that has all given me a deep sense of unease looking ahead to the midterms. And to put our worries to rest, we have our pollster with us, Stephen Hawkins, Global Director of Research at the Pollster, more in common. Stephen, thanks for joining us. Pleasure to be here. Thanks for having me. So what is going on? Great question. Well, let's contextualize this by focusing on the type of elections that we're talking about here, which are primary elections, not general elections.

4:54And that creates a whole bunch of uncertainty that makes it much harder to predict what's going to happen. So what you do with a poll is you have to first decide who do we think is going to vote. And then you have to go find them and talk to them and ask them who they're going to vote for. And then you make a prediction about what's going to happen. Now, that is much, much harder in a primary election because you're talking about a small percent of the population. You're talking about maybe 15 percent of the population. And it's highly variable between cycles. So I was just speaking this week with somebody who was doing get out the vote efforts in Georgia and Texas, and she was saying that her efforts were yielding between a 50 percent and 138 percent increase in voter turnout in the specific areas where she was working.

5:42Now, that's a huge amount of variability that pollsters are going to have a hard time accounting for. The other thing that's happening with primary elections as opposed to general elections is people are making decisions in the last few days or even the day of the election. Historically, we know that between 30 and 40, sometimes as much as 50 percent of the people who go to vote in primary elections are making the decisions in those final moments. So let me just provide a quick analogy here, just maybe it'll be helpful for the conversation. polling is doing statistical inference which is taking information that you do have to try and make a prediction about something that you don't know about so what you do using what you know to try and figure out what you don't know so i think people want polling to work kind of like it's the night before christmas and you go under the christmas tree you pick up a present it's wrapped and you shake it and you use the information about how it feels and what it looks like and how it sounds to say, I think this is what's inside.

6:41With primary polls, it's kind of like trying to make that prediction before your wife has gone shopping for your Christmas present, right? The people have not decided who they're going to vote for. And so, let's just, the last thing I'll say is on this Wisconsin race, the winning candidate had dropped out of the race and then come back into the race and then in the final days was endorsed by the governor. So there was a lot of variability in the real world that likely is changing how people are actually going to vote as opposed to it being a measurement error. Right, right. And I think Hong was at 42, 43 percent in the polls, which is about what she got in the end.

7:18It's just that all of this, as you mentioned, people made up their mind. All this big undecided block of voters all broke towards Crowley. So in the end, that was what happened with the difference in why Crowley won. It still was a narrow margin. So we have general election polls where we have a better idea, I guess, of who's going to vote, who typically votes in these. But these primaries have a lot more variables that you're trying to figure out. Well, that's right. And so think about it this way. In the general election, you know, based on historical precedent, roughly how many Republicans, Democrats, independents are going to show up.

7:56And then that's a very good predictor for if they're going to vote for the Republican or the Democrat. In a primary, you don't have those cues. It's all from within the same party. You know, what I try to always remind viewers on our programs, for example, is that polls are also snapshots in a moment of time. As you said, there's a lot of variability, things that can change. But also you have to always look at the statistics behind a poll, so the sample size, when it was conducted. And that's really important. I should say at the BBC, we don't headline or lead a story with the result of a poll.

8:26And it's really the aggregate of polls that kind of drive our coverage or certainly we make sure that we are trying to put a focus on an aggregation of polls so that we get a better picture. But what I've always been interested in, Stephen, is it seems that parties or campaigns have their own internal polling that's a lot more accurate than, let's say, the general public polls. Why is that? What's the difference? Yeah. So let me take a step back to just explain the challenging of polling right now. So when polling was invented, you had three beautiful conditions in place. One is you had a ubiquitous technology in the landline telephone.

9:04Everybody had one. Two is you had very nearly complete information about who had what phone lines because you had a telephone book. And then three, you had people who were honored on average to be called during dinner and asked to give their opinion. So you had really high participation rates. Now you have no ubiquitous technology where you can reach people for surveys. People generally don't want to give away their time and you don't have good information about how to reach everybody. So you're dealing with, instead of the majority of the population being willing to participate, you have maybe a small percent, 1 to 5 percent of people who are generally wanting to participate in surveys.

9:45So that's the general problem that survey research has to address. So the challenge that this presents is that you have something called non-response bias. And non-response bias is you know that maybe 90 or 95 percent of people aren't responding to your polls, but you don't know exactly how the people who do participate are different from the people who don't participate. And one of the things that happened, you mentioned the 2016 results where President Trump's support was systematically underestimated, is we know that there was something going on where Trump supporters weren't as excited to participate in polls or as willing to share their support as other people were.

10:28And what the parties do is the parties have a very big data set that is comprehensive. They have a file on effectively the entire electorate by household. So they know when they do their polling, they can match people back to this massive data set. Why do they have that massive data set? because it's used as a big piece of infrastructure for campaign mobilization, for targeting, for advertisement, and for modeling. So they have a big infrastructure where they're trying to figure out who is going to show up to vote and who might show up to vote and how can we find those people, persuade them, mobilize them, et cetera.

11:04So they're working within this big infrastructure. And then when they do polling, they're connecting the dots between what people say on the phones and say via email and then this big infrastructure. So they're able to figure out who's not responding to the surveys and do more accurate projections as a result of having that information. Is it just more expensive to do that too, the media outlets and polling outlets to have that sort of resources compared to a political campaign that's just apples and oranges? It's more expensive to do that type of polling, and there's not the same incentive around it.

11:34The political parties want to know much more about voters than people who are just doing polls for the media, for instance, are. So in 2024, when Republicans in the Trump campaign were saying they thought they could win in a blowout, they felt like they were ahead by a lot more than the public polls were telling us in the general public. There was something to that. There may have been something to that, but there's another dynamic at play here, which is that some of the people who are contributing, pulling information into the discourse and into those averages have motivated reasoning. They have a benefit to gain from their participation.

12:11And so one of the things that the American Association for Public Opinion Researchers, APOR, has found is that parties, this won't surprise you, and candidates selectively release polls when it has an advantage to their campaigns. So it shows their surging, shows their support is underestimated, et cetera. And they withhold information when their internal polling does not benefit them. And so that may have been what was happening in 2024 as well as there is an advantage to sharing information when it might make your candidate look better. Just on cue, as I was preparing for this episode, I got this kind of big, loud email from CPAC, the Conservative Political Action Committee, that says, for a limited time, we have a 2026 midterm election poll open to top patriots live poll.

12:57And so it got me thinking, as you said, there's no ubiquitous technology, but how are pollsters then going about getting their data? Yeah. So for these types of primary elections that we've been talking about, the typical method is a blended approach. And that's to answer this challenge of there not being a single ubiquitous technology where you can reach everybody. And so what they're doing is they're likely using some combination of landline telephones, which are great because you can talk to a real human, but they're not great because that human is probably going to be retired. Those are the only people who tend to have landlines and want to talk to bolsters.

13:35Then you also have text message to online approaches. So it's called MMS to Web where you get a text message. Can you please fill out this form? And then you also have online panels, sometimes called convenience sampling, where what you're doing is basically people getting compensated a little bit for joining a list and then getting emails where they are invited to take part in a survey. So to get the right modeling approach, you tend to have to blend those approaches. The most common approach to doing survey research in the country today now is these convenience panels where there's recruitment processes that happen and a company will build up a list of 50 ,000, 200 ,000 people and can help find a representative sample for you.

14:17Now we talk about AI changing things elsewhere in this country. Is AI having any kind of influence on polling as well? Is it possible to use AI to increase the survey sample or do some of the menial things that are more costly for polling companies in the past? You've put your finger on a very live subject in the polling world. Yeah, there are concerns and opportunities with AI in the survey world right now. One concern is that the online panel survey research that I just mentioned, it's opaque, right? So as a pollster, I don't see the individual on the other end taking the survey. And that creates an opportunity for bad actors to effectively deploy AI agents who represent different human beings.

14:59Maybe they specifically try to represent parts of the population that they know are hard to reach. So they get a lot of invitations to do surveys. And so it's one thing that now we as pollsters are screening for is, does this survey response look human? Does it have some randomness in the way that responses happen? Does the selection of answers follow a kind of stochastic approach or does it have a systematic approach? How good are you guys at spotting that? We're pretty good at it. The social science research that's come out in recent years shows that it's not zero. There is some AI in the sample, but it's probably around 2 % to 3%.

15:36And the panels themselves are very much motivated to try and get this stuff out of their data. So they're also working to screen it out using IP addresses and other sorts of markers. So that's the threat. The opportunity is also very interesting because there are companies, there's one out of New York called Aru, which just got a billion dollar valuation. And they're trying to model the entire population. So they're trying to use AI to simulate an entire city or entire country so that you can pose a question to their system and get a sort of simulacrum of a public opinion poll based off of their synthetic data.

16:18What that's not going to allow us to do is exactly the challenge that we're talking about here, which is to figure out who's going to win an election when 30 or 50 percent of electorate has not yet made up their mind. AI is not going to be able to help us get ahead of that. It's always going to be working on training data, so it'll have this sort of lagging effect. When we're looking at polls, are there ones that are better than others? And how, say, if our listeners out there, and they see three or four polls all in the same race, and they see names maybe they haven't heard of, maybe some that they have, New York Times or Quinnipiac or something, how can they make a judgment about whether this poll is reliable and accurate or not just as a kind of a layperson?

16:59So brand name does matter. For your listeners in the UK, there is a British polling council, which is a kind of regulatory authority. It's not very tough, but it does exist and it does have some transparency requirements and you have to join in order to be considered an official pollster in the UK. The US doesn't have the same sorts of constraints. And so reputation does matter. If you've heard of the pollster, that's a good sign. If you haven't heard of the pollster, do a little bit more digging because we have seen, even in recent weeks, that there are fraud. This is Will Friedle from Podmeets World.

17:35HGTV is bringing back the 90s in a big way with Totally 90s House, a brand new renovation competition series that reunites some of your favorite television stars from the era for a showdown unlike anything you've seen before. Let's be honest. The 90s gave us some great things. The music, totally rad. The fashion, so fly. But the home design choices, well, that is a different story. Hosted by Jaleel White, this throwback competition sends two teams of your favorite 90s TV stars across America in search of homes frozen in time. Their mission, find the two most dated houses they can and transform them into modern dream homes.

18:12Jody Sweetin, Beverly Mitchell, and Brian Austin Green team up against Matthew and Joey Lawrence, Melissa Joan Hart, and Keisha Knight Pulliam. Paired with HGTV mentors, they'll transform those outdated spaces into fresh, modern homes while keeping just the right amount of dope, nostalgic charm. Goodbye, honey oak cabinets, sponge-painted walls, and glass blocks. Hello, 21st century dream homes. Totally 90s house. Special series premiere starts August 26th at 9, 8 central on HGTV. This is Troy Millings from Earn Your Leisure. On Earn Your Leisure, we're all about smart money moves. And here's one most people skip.

18:45And it starts with Insurify. Most of us never shop our car insurance. The policy renews every year, the rate creeps up a little at a time, and nobody calls you to tell you. You stay with the same company because it's easy, and that convenience costs you real money. That's what Insurify fixes. Real quotes from more than 120 top insurance companies side by side. About two minutes, free, no phone calls, no pressure. And if there's nothing better for you right now, Insurify keeps watching and tells you when that changes. so you're never stuck overpaying. Millions of people have used Insurify to shop for their car insurance.

19:20So insure with Insurify. Go to Insurify.com. That's I-N-S-U-R-I-F-Y.com. Insurify.com. Hey, I've got the campaign brief ready, built from last quarter's data and the competitive landscape. Approved? Oh, agents, where would we be without you? Somewhere with a lot more tabs open. Create your first Monday agent in minutes at Monday.com. There are fraudulent pollsters out there who are fabricating data. So do look at the reputation of the firm. Look at the transparency of the methodology. Look at the sample size. Generally, it's best if you have a sample size of at least 1 ,000. Notice if they report their margin of error.

20:04Notice if they report their data collection method. All of these are relevant to assessing the quality of a poll. and then also take a look at their fieldwork methods. If they are doing a blended sampling method, it's probably more reliable. Sumi, have you heard of median strategies? No, and I think a lot more people might be kind of listening now to this. What you just said is what you need to look at if you should know whether to trust a poll or not because there was indeed a widely shared fake poll from Wisconsin and California for median strategies. Where did this come from? It's a bit of a mystery, isn't it?

20:42It's something I have never seen in polling before, but this mysterious organization was publishing data and they claimed to have insight into what was going on in these contested races that you just mentioned. And then after the elections occurred, they said, actually, this was a social experiment. We were trying to figure out how polling data would move within an information system. Now, it could be that that's true, But I am also curious whether, as we already spoke about, there are beneficiaries when data is released. And it could be that they were connected to some political cause or political candidates that would benefit from the sorts of distortions that they created by publishing their data.

21:28there's also a theory which I don't know how much credence to give this but there are now betting markets on just about everything in the country right now and being able to put your finger a little bit on the scale towards a different outcome could be financially beneficial for people who know what's going on yeah that's the crazy thing is you know you watch a commercial or cable television news television here in the u.s and you will see betting market predictions put up alongside polling, saying there's a 70 % chance that this candidate will win. And they show that kind of with the same reliability or the same sense of authority as a poll from the New York Times or elsewhere.

22:07I mean, how are you folks dealing with that? Because that's not polling. That's just the general public making bets for, as you mentioned, whatever reasons they may be doing it. It's a kind of remarkable new development just in politics in the past few years. Yeah. So it doesn't affect our work directly. So it's not a major concern for us, but it is an interesting data source because it's just kind of like a stock value. It's a good way of synthesizing what the public thinks about something's probability. and it sort of averages out a lot of different views there. So it's something we look at, but it's not going to affect the quality of our work.

22:47Yeah, it raises the question, of course, that we've all talked about. What should people be betting on? And is politics and races and elections, is that one of them? But, you know, it also brings up a question that I've been wondering, Stephen, which is the money behind polling. You know, is it a big business? How do they get funded? And are there pollsters that are able to get a lot more money for their polls than others? So it's a relatively big business. A lot of the overlap between political polling happens with market research. So interestingly, we're here in Washington, D.C. A lot of the polling firms work for presidential candidates, Senate candidates, and then they're also working for Toyota or Walmart.

23:26And the reason for that is that the operation is very much a similar process where you're trying to model a population, develop a sample, collect data, and then make predictions about how a population is going to move. And that's useful if you're trying to sell toothpaste or trucks or if you're trying to sell a candidate. And it's very much the same process. So my start when I started in D.C. about 15 years ago was we would move between projects that were political and corporate very easily. And so there is a lot of money to be made in public opinion research. What's happened in terms of media is because of these challenges around credibility of polling, a lot of major media firms have brought internal polling operations into their own operations in-house so that they have a polling expert so they're not so reliant on external people.

24:20And even sometimes they collect the data themselves. So there's the media operation, there's market research, and then there's social science. And in the social science world, to get the gold standard polling, you have a very, very high ticket price. So I just want to mention one thing about the pricing here. So the range that you could spend on interviewing a thousand Americans could be as low as, say,$500 and as high as$150 ,000. There's a huge, huge range and has everything to do with data quality. If you wanted to spend the most amount possible on a poll of Americans, you would be doing what's called probability sampling, where what's happening, this is a group called Amerispeaks, Newark, connected to Chicago and other universities.

25:15They're actually mailing out at random laptops and invitations to participate on a panel to Americans selected at random all across the country and saying, will you please participate in our polling? It's very expensive. It's very slow. And for some social scientists, it's really worth it to get that granular, that precision in the data. So they're spending as much as$150 ,000 on a poll. Right. So all of this sets the stage for the big event, which is the midterm elections and what the polling is telling us about this right now, because I think, Stephen, you can agree it looks good for Democrats.

25:53We keep seeing these prediction analysis sites like Cook Report and Sabado's Crystal Ball, which uses polls and then makes forecasts based on what they're seeing, saying Democrats are likely to pick up the House of Representatives. They actually might be able to get the U.S. Senate. They've been moving more and more races from Liens Republican to toss up from toss up to Liens Democrat or even safe Democrat. Do we think at this point that the polling we're seeing for the midterm elections is reliable, that this is actually happening? Or is there a risk that we're getting some sort of an illusion that Democrats are ahead?