After the Reform scandal - does anyone still trust pollsters?

0:00Bite into a stacked sandwich made with Hero Bread, and the only thing you'll think is delicious. You won't think it has up to 19 grams of protein, 11 to 32 grams of fiber, or just 0 to 5 grams net carbs. But it does. Hero Bread. No compromises. Just loaves, buns, tortillas, bagels, and noodles packed with flavor. Right now, get 10 % off at Hero.co with code iHeart. That's H-E-R-O dot C-O, code iHeart. All figures per serving of Hero Bread see nutrition facts on Hero.co. This is a global production. Nigel Farage's former aide, Dan Dukes, went on an all-expenses-paid trip courtesy of the polling company JL Partners.

0:46There's only so much distance that Nigel Farage is able to put between himself and Dan Dukes. Dukes was asked, what does a reformed government want from a polling company? And he replied, showing that we're winning. Well, if JL Partners are up to it, are others up to that sort of thing as well? This will raise questions in people's mind. What polling is for, who it's for and whether we should trust it. I think there is a little bit of like informal rule vending that we did with our last one. So like, you know, when we publish those results, we should have disclosed to the plan for this. That is James Johnson from the polling company JL Partners.

1:25And it was taken from a Channel 4 documentary two weeks ago, a scoop in which the pollsters admitting to bending the rules to conceal that the Reform Party were behind the polling that was then splashed all over the newspaper front pages. Reform deny they were involved, but an inquiry is now underway. This is just one company, one pollster. But today we want to look more widely at polling and public trust. Polling is everywhere. How much does it tell us and how much does it decide our own behaviour? Welcome to The News Agents.

2:12It's John. It's Maitlis. And we're going to start today's episode on the Red Sea Resort of El Cusir. That's nice. December 2025. And the reason we're starting there is that Nigel Farage's former aide, Dan Dukes, went, we now understand from Politico, on an all-expenses-paid trip there, courtesy of the polling company JL Partners. This is an internal email that Politico's seen. Now, why did he go? He went to give a talk for the company, which subsequently released opinion polls without then disclosing they were financed by, you know, understand, supposed donors to reform. This is part of that channel for Verbatim Scoop that came out a couple of weeks ago.

3:04and what we now learn from that trip, Dukes and Jail Partners, was the extent of the business relationship. Somebody with knowledge of the trip said that the company took a limousine from the airport to the hotel, also at Jail Partners expense and during that trip, Dukes was asked, what do you want? What does a reformed government want from a polling company? And he replied, showing that we're winning. We know what subsequently happened to Dukes. He was essentially chucked out of the party. And what Farage has said is that was nothing to do with reform. That trip was nothing to do with us. It was, you know, in a different capacity.

3:54Well, we know that Nigel Farage thinks that Dan Dukes was just a contractor that was working for reform when, in fact, he was the de facto chief of staff to Nigel Farage and had been at his right hand for absolutely years. So, I mean, so much, there's only so much distance that Nigel Farage is able to put between himself and Dan Jukes. But I think one of the secondary questions it asks is not just about Dan Jukes or even reform, but about the polling company at the centre of this story, JL Partners, which was pretty well established. It was, I think, owned or half owned by a former age, Theresa May.

4:35It had come out of sort of, you know, the political establishment, essentially. There was nothing in what they were doing to suggest that they were taking off members of a party and sort of chauffeuring them by limousine and then saying, what do you want your polling to show? And yet, I think inevitably, this will raise questions in people's mind about what polling is for, who it's for, and whether we should trust it as much as we quite often do. I think this is such an interesting and controversial area. All through the 2016 presidential election in America, anytime there was a poll that was unfavorable to Donald Trump, he would say, this is a fix.

5:19They're working for the Democrats. They're against me. Donald Trump has sued polling companies, you know, in America for producing polls that he felt were unfavorable. And when I think it was in Iowa. Des Moines, the Des Moines Register. There was a very well respected pollster there. And she was sued by Donald Trump for producing a poll that suggested that the Republicans were going to lose Iowa. By a substantial margin. By a substantial margin, when in fact Trump carried it. And so Donald Trump's charge was always you can't trust polls. They're fake news. They're part of the same. They've got a political agenda, just as in Britain, you could say that the Daily Mirror has a political agenda for Labour or the Daily Mail has an agenda for the Tories and you can't trust what you read.

6:03And I have always taken the view that actually polling is really helpful to us. And think how often we refer to polling as a sort of guide in the dark to where political trends might be moving. And I think what JL Partners have done is put a light that is deeply uncomfortable on polling. Because you think, well, if JL Partners are up to it, are others up to that sort of thing as well? And you've seen that the Daily Telegraph and the Times have said they're not going to be taking JL Partners polls. Which is now being investigated by the British Polling Council. Yeah. What's going on? Because what they've done is they've undermined confidence in a really serious way in what we should believe and what trust we should place in polling.

6:53Yeah. And today on The News Agents, we're going to try and widen that debate. It is not just about reform and it's not just about JL partners, but I guess it does ask about the place that polling has now in our political conversations, because we're going into the conference season. We will be seeing a lot of polls coming at us suggesting this is how people felt before the Conservative conference, this is how people felt after the Labour conference, this is where we think we are with reform, this is why we think we're going with Ed Davies' leadership of the Lib Dems or whatever it is. and polling has actually undergone some really interesting revolutions recently, not least because of the way we ourselves communicate.

7:38Very few people have landlines anymore. Very few people pick up the phone to strangers anymore. A lot of polling depends on you being quite politically active, sitting down, filling in a form. How many single mothers with three kids are you going to have that suddenly has 20 minutes to talk to a pollster or wants to go on the computer in the middle of the day and start filling in online forms. So there are a lot of shifts now which are changing the way that data is received. And we want to talk about this more widely, more generally, with our old friend of the podcast, Luke Trill, from More in Common.

8:13Luke, I know you don't want to talk specifically about the reform question and jail partners, but let's kind of start from the beginning, because you're in the business of presumably commissioning polls and listening to parties. And if a party wants polling done, how does that conversation start? Well, normally when a party wants polling done, they will approach you and say, you know, we're interested in finding out about X. And most of the time that polling will be to inform their strategy privately. So most of the polling that political parties are doing is not public because and because of the British Polling Council rules, whoever commissions you for a poll, if it's a public poll, if it goes into the public domain, you have to declare that client.

9:02And so let's assume the Labour Party commissioned a pollster. If they sold a poll into a newspaper, they would have to say in the data tables that you have to put up, this was commissioned by the Labour Party. So it's actually quite unusual for political parties to do public polling in the UK. Most of it is private and behind the scenes. And would you ever ask a question like, what do you want this polling to show? Would you frame it in a sense that said, if the Labour Party comes to you, what do you want us to show from our polling? Is that a normal thing to do? No, I think the conversation you have is, what are you trying to find out?

9:43That is the key question in polling. And I often say there are two types of polls. There are polls to know, which is where you are genuinely interested in either finding out where the public stand on an issue or actually more often, more interesting, message testing, what might shift where the public stands. Lots of people, when they criticise polling, will say, look, are we just taking public as baseline? There's no leadership. Actually, a lot of the polling, particularly private polling, is around what messages might make someone more amenable to rethinking the triple lock. How do I land it in a way which goes down well with voters?

10:22So that is a big part of it. That's polling to know. On the other hand, you've got polling to show, which is, you know, very often, you know, 70 % of people agree that, I don't know, I'm trying to think, you know, we should give out free. Well, not necessarily, but, you know, we should give out, you know, we should give a Cadbury bar to every child in school when they finish school. It's that kind of motherhood and apple pie stuff that no one can really disagree with. And I guess my, you know, there is always polling to show. But my thing is, what is that telling you? You know, actually, I often when I'm talking to my team.

10:59So you say you mean nothing. Yeah, it doesn't really. Because the other thing is, you know, when we're talking as a team, we're always thinking about, you know, so let's, I don't know, a wealth tax. You know, if you poll headline wealth tax, you get very high support for it. More interesting, I think, in polling is polar wealth tax. What if it causes some rich people to leave? What does that do to support, you know, putting trade-offs in? And actually, as it happens on lots of wealth tax, people still support a wealth tax. And I think that's even more interesting that it's robust to that. And I think the challenge with polling is to try and get into trade-offs, to try and avoid agree-disagree statements as much as possible, because we know there are things like agreement bias.

11:41so to give you one example um nationalization should water companies be nationalized one way of phrasing that question would be do you think you know water companies should be nationalized strongly agree somewhat agree neither somewhat disagree strongly disagree that is quite a bad way of phrasing that question because of agreement bias the better way to phrase it is to give a paired statement which is which comes closest to your view water companies should be nationalized water companies should not be nationalized because there you're forcing people to pick between alternatives rather than ending up with agreement buying so there's a lot of thought and actually you know our work um a lot of uh the you know clients and partners that we work for are in civil society a lot of the time there's quite a robust exchange between a pollster and the organization commissioning them with us saying you can do that but it's not going to tell you anything interesting.

12:38Let's get into the hard stuff about what it means in terms of trade-offs and how you land this. Okay, stay with that thought. If you are revealing something to a party that they don't like, take me inside those conversations. Would you ever find the party said, I don't want to pay for this, or we decide we don't want to publish it, even if it was going to be a public poll? What happens then? So look, I mean, we, because we're cross-party, we brief all parties we don't actually do paid polling for um the political parties we take our public work and then we go and brief them but i've had a number of conversations with parties where they've gone are you sure about this luke are you right because we're dead committed to this and i guess part of it is you know again you're there to communicate with p you know with the parties you're there to you know help them reach the public better and where parties and politics go wrong is where you get a bit of your stick your finger in your ear we're right you know this isn't an accurate representation of uh the public and you know we're always very frank with that it is obviously up to and again this mostly i think would apply to when you're working with an organization rather than a political party you know they will get some polling back and decide that they don't want to publish it that is that that is up to them they are uh the the client but But if they publish part of it, this is really important, this is in the rules, that the full data tables have to go up.

14:09And one of the rules is that anything in the poll which might contradict a finding that you have put out, you also have to publish. And so that's really important. So you either keep it entirely private or anything that's relevant has to go in. So to give you another example, let's say there was an organisation campaigning for the£2 bus cap, which is very popular. If you put that in a poll and they run that, but in the same poll, you've got a question which is, I don't want to pay for the£2 bus cap if taxes have to rise. You have to publish both of those because it might contradict. You also, there's something else which is called priming, which is, you know, you put a series of questions about climate change first, and then you ask people, would they vote for the Green Party?

14:55And because, you know, people are being primed in the polls. So not for nefarious reasons, but just as polls, there are so many things you have to think about in terms of, do I randomise the question order to avoid that? what question do i put when you know do i put you know if it's about ukraine do i put that at the top or do i put it after questions where you've asked about worries about international security and defense so there's so much work that goes into good survey design and i've said you know posts across the industry are constantly challenging debating you know how you get that right and the public outputs the x percent support y that you see is only a tiny fraction of what's gone into the thinking there.

15:40So do you think that we have become too dependent on polling? Do we, as journalists or as public consumers, depend too much on polling? Because there is so much of it about, but I'm guessing that the data itself is much harder to come by now Because we don't have landlines. We don't have as much time. We know exactly who fills in an online polling survey as opposed to somebody who you might reach at sort of three o 'clock in the afternoon in their home. Right. So how has that changed in terms of how you pull in the material that actually forms the basis for your data? So getting your sample right, I think, is almost the most important part of polling now.

16:32And you're absolutely right that in the past, you could have relied on landlines, you would call people up, you would reach them, and, you know, jobs done. now you have to be much more reflective or you have to be much more careful about how you construct yourself so a lot of polling a lot the polling that we do use as a big online panel provider we have to be sure that you know this is working with the power provider that they are doing proper data checks on the people that they are getting into the poll we have to make sure that they have proper representation of over 75s, for instance, who might be less likely to use an online panel.

17:17We've got a real potential issue, which is already here in the sector with AI. Because to your question, in order to get people who, you know, bear in mind, most people aren't that interested in politics to fill out polls, you have to offer an incentive. So that might be working towards a voucher in some cases to get um you know reach more middle class uh respondents who might not be incentivized by that you do things like avios or things like that in order to get people is that right so what is it so you're doing so who gets the voucher who gets the avios so there are different you can you can actually pick on some of the platforms which one um you are going for but there's obviously clearly different things as an air miles as in points Yeah, there's some, there's other pollsters do it as an online lottery.

18:03So you get entered in and then you pick it. So lots of different ways. So nobody gives their opinion for free. Nobody gives their opinion because you don't want that, right? What you don't want is the sort of person who would want to sit and fill out a survey for nothing because they are, by their nature, unusual, they're politically engaged. And so you've got to do that. So the harder you have to work, probably the more useful they are to you. Completely. Right. And the risk is, and this is a risk which the whole industry is taking very seriously, is that therefore it could potentially, I don't think the economics are quite there yet, but it could become quite lucrative to have a series of AI bots filling out polls.

18:44So you just unleash them, they're doing it. People often assume they're doing it to try and skew poll results, I don't know, for a foreign actor, and that might be a risk. But actually the bigger risk is just that people might end up wanting the incentive. And so we constantly have to think one step ahead of how are we stopping these AI bots from filling out polls. So, you know, one thing which has been used but isn't enough on its own anymore is you put one question in white text. And so a human wouldn't fill it in, but an AI bot would fill it in because it can see it. You know, that's why you're constantly having to think.

19:18That's the way the young are doing their CVs now. They sort of put these subliminal messages through their CVs to get past the… They're doing it. Exactly. And so that's one thing. And it may be that we end up weirdly at some point having to regress and go back to more phone calls face to face. There's some innovation. We're doing it on apps, on phones and online gaming and getting people at the end of that. But it's why I, you know, I'm such a big believer in not just relying on the numbers. It's why I think the qual and the focus groups are so important. And actually, you know, I believe in the Bezos maxim that when, you know, the quant and the qual disagree.

20:02Just talk us through, what do you mean quant and qual? So the quant is the poll. So that's the 2 ,000 people, nationally representative survey, goes into the field, comes back. uh yeah x percent want to trade welfare for defense yeah the focus for the qual is the focus group this is where you then what you do is you get eight real people that you're sat around and if folk if polling can tell us what people think focus groups usually tell us why people think it in their own words and i you know look i'm biased other people in the industry would disagree i as i say i get a lot more out of the qual because it's real people and there are times when you're doing the qual and you realize actually people misinterpreted that question in a poll um they just didn't understand they didn't understand it and actually so we will sometimes before fielding a survey do a focus group first to check people yeah understand what we mean give me an example of something people get wrong so like i mean a lot of our work we as well as looking at sort of political attitudes we look at some social psychology questions you have to try and understand more kind of upstream what people might think and when we were designing our segmentation we had a lot of questions about you know things about like would you rather have a more equal pie or a bigger part you know or a bigger pie even if it's like would you like everyone to get richer or would you like everyone to be more equal and actually phrasing a question like that is really important and you have to use a focus group to try to understand to basically make sure you're testing the right thing but also i think focus groups you know will throw up those trade-offs more because you're in a dialogue with someone for 90 minutes 75 minutes and you know we've talked before about that tony blair kind of you know 30 second three minute 30 minute conversation people who might start off in one place actually when they consider all the nuances end up so miles or you just find out there are nuances they're thinking so immigration i think levels of immigration you're very kind and you say nuances but and i speak as a voter here i would say just massive inconsistencies with i mean the way human brains work is random we you know we slag off a politician and then we admit that we voted for them last time and might do so again right and so there's something i mean the thing that is deeply human is that we're completely inconsistent.

22:35Oh, completely. We are. And inconsistency, I actually think, is such a wonderful thing. And you only get that from talking to people and finding out those inconsistencies. And it's quite a weird, unusual thing. Basically, people in politics have much more of what you call stacked identities. That is, if I know what you think about climate change, I can probably say what you think about immigration, what you think about criminal justice, what you think about the economy. Not always perfectly. Voters are not like that at all. You can have someone who, you know, thinks channel crossings are the biggest issue facing the country and also thinks the climate crisis is a national, you know, emergency, you know, and vice versa.

23:19And again, I think you have to do both the quant and the qual in order to get to that and to get to that nuanced picture. I mean, what you're really saying, I think, is it takes an awful lot of just man hours of time to make any poll actually stack up, right? And we don't necessarily get that. In a moment, we're going to talk about what happens when polling goes wrong. Stay with us.

23:52Hey man, you hungry? Let's order delivery. Yeah, I was thinking... Italian. Burgers? Oh. Game night just got tense. Two bros on different sides of the fence. Burgers or pasta tonight. Why can't we have food without the fact? With Wonder, you can combine 20 restaurants in one delivery, so no one has to settle. Use code MEALTIME for 50 % off your first Wonder order. Terms apply. See wonder.com slash new customer. Hi, it's Marcus Allen, Pro Football Hall of Famer, NFL MVP and Super Bowl champion. I played football for a long time. One thing I've learned is nobody really knows what's going to happen next.

24:33Before the snap, you can study the defense, you can know the play, and you can see the opening. But eventually, you have to make a decision and go. For me, that was the fun of football. The moment between getting the ball and finding out whether you made the right read, I loved it. And it's why I understand what Underdog brings to sports fans. Because once you make your pick, you're not just watching anymore. Every snap matters, every yard matters, every decision matters. Everybody thinks they know what's coming next. Well, Underdog gives you somewhere to put that opinion into play. Pick them, money lines, totals, spreads, games, and more across your favorite sports.

25:09So trust your read, make your call, and see what happens. Download Underdog and get in on the action today. Underdog, unleash your dog. Trading involves risk, including total loss on positions. Content is promotional and not a recommendation to trade. Terms apply.

25:30I want to take you back, Luke, to a moment. I mean, we discuss a lot of sort of polling moments in the way obsessives do, I guess. If I take you back to the US presidential election of 2024, And on November the 2nd, just days before the Trump-Kamala Harris presidential election, there was a poll released from Iowa, from the Des Moines Register, by a very, very well-respected pollster, Anne Selsa, who got the 2016 readings of Trump right. She got the 2020 readings of Trump right. She kind of understood that people don't always do what they say. And she released a poll which showed Kamala Harris winning by enough of a margin, I think it was 47 % in her poll to Trump on 44, that put Iowa in the hands of the Democrats.

26:28And we all went, and it felt like a game changer, because once you see it on Iowa, you assume that all the swing states would also follow that pattern. And it started to look at that particular moment. I remember us recording a podcast that that morning as if Harris had the whole thing in the bag. It was not just wrong. It was incredibly wrong. In fact, Iowa was won by Trump on 56 percent, like a 13 percent margin. And as it turns out, he won all the other swing states as well. And Anne Seltzer, curiously, I think, I mean, pulled back from Pony at that point and has never really been able to explain what went on.

27:08I mean, just take us inside the mindset of publishing a poll where, I mean, A, I want to hear your thoughts on that. But B, that sense of you're going out to the public with something that your gut tells you feels wrong, but your data tells you you should say out loud. I mean, I honestly get cold sweats just thinking about that and Seltzer poll and the equivalent happening to us. Because whilst there have been recent UK polling misses, they've tended to at least be in the right direction. Although some exceptions we can come on to talk about. But it is that challenge. You get the data back. And we had, to a much more minor degree, we obviously worked with the news agents on our MRPs in the run up to the 2024 election.

27:55and we were one of the few pollsters that still had the Tories getting 120, 130 seats, I think, in our final. But some others had them behind the Liberal Democrats. And I remember us being sat with the team being like, we're outliers here on this. Are we confident in the numbers? And of course, you go back and you, particularly with those models, you rerun stuff, you check you've got all of the factors which could shape the model. Do you push them closer together? or? So I think herding is a really bad thing because the advantage, you know, some people complain there are too many polling companies.

28:31The advantage of having so many polling companies is that you can experiment, you can test, but it is your reputation on the line. There is safety in numbers. What you will do is if you have an outlaw poll, you will look at it and you will want to make sure that you have stress tested. And I think the Anne Seltzer example is just, I feel so bad for her because I mean not only in general uh elections in presidential election she was really good at primaries too which is really American primaries as we've seen this season are really hard to poll and then one wrong poll and everything goes wrong I think one of the specifics actually for Ann Seltzer is that she is a big fan of what is called um just getting a really good sample and leaving it and not touching it.

29:16Whereas most pollsters now wait quite aggressively. So there's this kind of friction, isn't there, between the sample, which is like the data, and the modelling, which is how you extrapolate. Exactly that. And, you know, so, you know, I think all UK pollsters use quite a few weights to make sure, because, you know, you won't get enough non-voters very often in polling. You know, in the US, education has They weren't getting enough graduates, so you would weight some up. There's a danger to weighting up, though, because what if the white, non-college, educated, Camilla supporter is unusual and you're timesing them by seven in the polls?

29:58You've got to have weight caps. Anselt was the other way. She's a big fan of just get a good sample, leave it alone. And I think that was actually the undoing in that particular poll. Amazing. So actually the raw data was what let her down. Yeah, it was. I think it was that. There was also this really interesting emergence of a polling company called Median Strategies, which came out in August. And this was about the time of the primaries, the Democrat primaries. And there was one in Wisconsin, which had Hong winning by a huge amount. And Median Strategies was one of the companies that was pushing it out there.

30:36And everyone was saying, God, they got it wrong. It was terrible. It turns out that Medium Strategies wasn't a genuine polling company at all. It was a 21-year-old college grad who was basically kind of fucking with us, who was having a go to see what happens. He called it a short-term social experiment designed to test whether bogus polling could enter the political information system without any verification. I mean, the human equivalent of your AI bots. And sure enough, it did, right? People just sort of lapped it up. I mean, it was extraordinary. And the one thing I would say is that for any criticisms people might have of polling in the UK, in the US, it is much more of a Wild West.