On the eve of the 1916 presidential election, Republican challenger Charles Evans Hughes was so heavily favored that President Woodrow Wilson contemplated resigning during his planned concession speech.
Confident that he would be elected president, on election night Hughes retired early to his suite at the Hotel Astor in New York. His advisors assured him that the polling data in the West—particularly California—would guarantee his election. Unfortunately for Hughes, his pollsters got it wrong. The majority of western states went instead for Wilson.
When it became clear after midnight that Wilson would secure his reelection, a New York reporter called Hughes to get a quote about his defeat. One of Hughes’ aides, unaware of the reversal of fortune, answered the call. Presumptively, he said to the reporter, “The president can’t be disturbed.”
“Well,” the reporter said, “when he wakes, tell him he’s no longer president.”
A century later, pollsters are experiencing many of the same prediction problems — and for good reason.
There are serious challenges facing traditional polling. One of the main ones we face is the ever-skyrocketing conversion to mobile phones. Now, 47 percent of American households are cellphone only, while 42 percent have a cellphone and a landline. Only 8 percent have a landline and no cellphone, according to a recent Centers for Disease Control and Prevention tally.
In my experience, depending on the region, we regularly complete more than three-quarters of our interviews on mobile phones.
But when pollsters call cells, it’s expensive — largely because the FCC has ruled that autodialing mobile phones is against the law. Instead, each mobile phone number has to be manually dialed.
In order to control costs, we’ve witnessed many pollsters place an artificial “cap” on cellphone completes – often as low as 30 percent. This results in the sample of respondents being surveyed inaccurately reflecting the target population, in turn regularly delivering poor – even epically awful – predictions.
Looking for ways to cut corners and reduce costs is common in any industry. So many strategists, pundits and news organizations are turning to online polling which is far less expensive than traditional telephone polling.
And online polls are making the news.
This past week Team Trump has been breathlessly trumpeted online post-debate “poll” results. Yet CNNMoney reported that one such poll outlet, Fortune, stated: “No one should confuse” the magazine’s own online survey "with with a scientific poll.”
The fact is that online polling generally faces obstacles that are even more difficult to overcome than those faced by traditional polling. Those who choose to respond to online surveys generally don’t accurately reflect the voting population.
For instance, they’re disproportionately younger and more inclined to embrace technology. Meanwhile, most online polls are visible only to potential viewers of one outlet, such as a news website, so the potential audience is artificially narrowed yet again by their online viewing preferences.
And online surveys don’t have a way of determining whether or not people will actually vote. Forecasting voting behavior is difficult, and demands a knowledgeable use of the voter file to develop accurate forecasts from actual behavior. There’s no effective way for these kinds of polls to leverage the voter file.
In other words, online polls, such as Fortune’s, are only entertainment. They’re as much scientific research as “Real Housewives of the Orange County” depicts genuine relationships. Their sole value appears to be that they’re fun to talk about.
And yet this isn’t entirely fair. There’s more to the picture, and if we view all forms of online surveys solely through the above lens we would be doing ourselves a genuine disservice.
Online polling does indeed have a place in the world of “real” research.
Let’s not forget that online polling, as spectacularly proven by Google, can be dead-on-accurate in extremely large universe, single-question horse races. In 2012, the search engine was only a fraction of a point away from predicting the exact results of the presidential race, beating out all of the pollsters using variations of traditional telephone polling.
And online polls can be useful — not to mention kind to the bottom line — when simply looking for large changes in patterns, as opposed to the specific accuracy of the numbers themselves. Online research can also be highly effective such for qualitative purposes such as focus groups when targeting especially hard-to-reach groups (such as, say, cardiologists or attorneys).
Moreover, online polls are getting better, especially when used in conjunction with the voter file, rather than just existing on a news outlet or aggregator’s website.
In California, for instance, 50 percent of new registrants supply an email with their voter registration form — a change in behavior that will surely continue to improve online polling’s accuracy and expand its use.
This allows researchers to connect directly with those registrants not only by phone, but also by email and online. This isn’t so useful with large universes, but can be incredibly helpful in small universes, such as a small town, with fewer than 10,000 voters who are likely to vote in a particular election. In these cases, a so-called “mixed mode” methodology, leveraging calls to mobile phones, landlines and augmented with online polling via e-mail invitations, can deliver directional results that can be relied upon to make reasonably informed decisions.
So where does that leave us?
Public opinion research using live telephone interviewers, when done right, is the most accurate way to predict outcomes. Using live callers, calling mobile phones and land lines even though it's more expensive, calling in the respondents’ main language – whether its English, Spanish or Vietnamese — and utilizing the voter file to capture an appropriate sample delivers the most reliable, accurate forecasting.
Just like any other tool, using online surveys in the wrong situation or in the wrong manner will continue to deliver misleading or even flat-out inaccurate results.
But online research is evolving for the better – and when thoughtfully folded into a research program can be a unique and useful component of delivering a robust, accurate aspect of public opinion.
Justin Wallin is pollster and COO of Probolsky Research, a full service public opinion research and strategy firm.