Campaign pollsters clashed over methodology with their media-researcher counterparts during a series of 2020 post-mortem discussions this week.
The University of Chicago Institute of Politics and the Cook Political Report’s look back at what happened on the research front in '20 opened with a panel featuring Trump and Biden campaign pollsters Tony Fabrizio of Fabrizio, Lee, & Associates and John Anzalone of ALG Research, who both took aim at media researchers’ methodologies.
“You will see some [public polls] that are done totally by IVR [interactive voice response], or online. It is not often that you see mixed-mode surveys that are being reported. The best way of doing surveys these days is mixed mode,” said Fabrizio.
Through his multi-modal research, Fabrizio said the Trump camp was able to pick up differences in how 18-34-year old voters were responding when reached by landline and by cellphone: “If you’re not watching the changes that are occurring in the different modes, you can easily fall prey to producing surveys that are not reflective.”
Anzalone said media firms often aren’t using correct sample sizes and they don’t spend the money to employ tactics such as text-to-web: “You’re not seeing the media firms do that in any way,” he said.
He added that IVR surveys miss cellphones, which means that voters of color are underrepresented in the sample. “[It’s] real expensive to go and do it right,” he said.
Still, Anzalone readily acknowledged that campaign pollsters had their own problems getting their methodology right in 2020. Specifically, how to deal with the higher response rates from voters during pandemic-induced lockdowns.
“Our response rates went through the roof during COVID. It was like 1990 all over again,” he said.
The problem was that Democratic voters tended to be the ones home and available to pollsters during in June and July. “It actually turned out that you had a bigger miss in public polling and private polling because of COVID,” Anzalone said.
These voters were polled so many times, “they’re like professional interviewers.”
Media pollsters, who joined the University of Chicago event later on Thursday, had their own criticisms of the campaigns’ methodology.
Public Opinion Strategies Partner Bill McInturff, who conducts the NBC/Wall Street Journal poll, wondered why, if Anzalone’s numbers were so spot on, the campaign sent Biden to Iowa for an Oct. 30 event, and Ohio for an election eve rally on Nov. 2. The Democrat ended up losing both states by more than 8 points.
“Given the margins in those states, I can’t image the Democratic polling was showing a 6-8 point Trump margin,” he said. “[It’s] a little convenient after the fact.”
McInturff, who switched to using the voter file as opposed to random-digit dialing in the WSJ/NBC surveys, also noted the parties behind each campaign pollster have “spent a fortune on data analytics” that give them another tool to work with.
Moreover, he said President Trump’s attacks on the media and pollsters could have influenced how some voters responded to surveys. Non-respondents, together with the difficulty of modeling turnout in an election with completely new voting rules, made research in 2020 particularly challenging.
Meanwhile, J. Ann Selzer, of Iowa-based, Selzer & Company, questioned why pollsters were even trying to model the electorate’s makeup in their surveys.
“It’s so hard, why would you try? Why not let your data show you the size and the shape of the coming electorate? I’m at a little bit of a loss for the details of how one would go about deciding, based on science, what a future electorate might look like.”
She added: “If you’re looking backward, you’re going to miss a freight train that might be coming right at you.”
She noted that some campaign pollsters use a propensity score for voters in their sampling. But she pointed to how in 2016, there was a drop in turnout among Democratic men compared with 2012.
“Having seen that, it alerts you to pay attention to the idea that things are not set in stone and if you lock yourself into a methodology that’s based on what has happened in the past, you’re preventing yourself from observing.”
In her methodology, she explained, “we have demographics for a general election sample and we weight to the Census for those elements and then we extract the likely voters from that so we’re balancing everything at the population level.”