Shop Talk: Margin of Error
This issue’s shoptalkers: Glen Bolger, partner and co-founder of Public Opinion Strategies; Harrison Hickman, founder of Hickman Analytics; Alex Lundry, vice president and director of research at Target Point Consulting; and Matt Hogan, vice president at Anzalone Liszt Research.
C&E: Is the cellphone-only problem the biggest challenge for pollsters going forward?
Glen Bolger: You want the sober answer or the drunk answer?
Harrison Hickman: The cellphone problem gets worse over time and here’s why: someone living in Northern Virginia gets a cellphone and then they move to suburban Atlanta. So even if you’re sampling suburban Atlanta cellphones, you’re going to miss that person. The one thing I think we could all agree on if we were kings of the world is that telephone exchanges would all be contiguous.
Matt Hogan: It’s a real problem.Cellphone-only households are now a third of the country and we’re continuously shocked by how many pollsters are still not using cellphones in their samples. Maybe you could get away with it in 2010, but not next year. Thankfully it’s getting a little cheaper, but it’s still tough for campaigns to pony up. If you want an actual representative sample you need to spend the money on this.
Bolger: And that’s the real issue. Matt hit on it directly. It’s more expensive to interview cellphones. Doing it on the national level is obviously easy. But on the statewide level it’s tougher. Some statewide campaigns have enough money and realize they need to do it, but that’s not all campaigns. The other thing is that you can’t do cellphone interviewing below the statewide level on a fairly rigorous basis. If you call someone in Northern Virginia and reach them on their cellphone, you can’t ask what congressional district they live in. They’ll say, “Beats the hell out of me.”
Alex Lundry: The problem gets even worse when you look at cell-mostly households like mine. We have a landline in the house, but nobody knows the number and whenever anyone calls it we look at the caller ID to see if we want to pick it up. This is another 15 to 20 percent of the electorate.
Hickman: And you can’t make up for it by weighting. You used to be able to just adjust the number of people who are under 35 and weight them more. But the people who are landline only have a fundamentally different lifestyle and attitude, so you can’t make the traditional adjustment.
Lundry: The agency that actually tracks cellphone versus landline is the Center for Disease Control. They have an interest in tracking it because one of the first initial differences between the cellphone-only population and the landline population was on health issues. People with cellular service only were more likely to binge drink, use illicit drugs, have STDs…
Bolger: Vote Democrat. (laughter)
Lundry: Right. As that cellphone-only population has grown, though, we’re finding there are real partisan differences between the groups, too. We’ve been getting away with not doing this for a while. Now, I think it’s malpractice for a pollster to not offer a quote that includes cellphones.
Hickman: These public polls that rely exclusively on computer assisted interviewing are going to start making huge errors because they can’t reach the cellphone-only population.
C&E: How frustrating are public polls, especially when a bad poll gets attention in a race you’re working?
Bolger: They’re mostly background noise. The issue for me is when they’re automatically presumed above it all and the campaign pollsters are the ones presumed suspect. The case I make is the exact opposite. If our polls are wrong and we’re giving bad data to our campaigns, we don’t get hired anymore and we don’t get to feed our families. If the media poll is wrong they just say, “Well, this ad clearly changed things in the past two days.” The idea that media polls are somehow superior is the one thing that gnaws at me. I’m not going to give happy data to a client and then after they lose say, “Sorry, our data was totally wrong.”
Hickman: And media organizations are more budget constrained than campaigns are. It’s sort of difficult for people to believe sometimes, but a TV station may have $12,000 set aside for this kind of project. A campaign would be paying $25,000 for the same poll. So of course we can do it better.
Hogan: It’s just frustrating how many of them don’t release their demographics. They’ll release some poll that shows a big shift and really drives the narrative for a couple of days. In reality, there probably wasn’t any shift at all and their sample is just more Republican or Democrat than it was in the previous poll. Reporters are getting a little better about covering that and picking up what the sample is, but too few of these media pollsters release all their data.