Chapter 3. How Technology Is Changing the Polling Industry
Patrick Ruffini is a cofounder of Echelon Insights, a survey research and analytics technology company. He has been applying data and technology to political campaigns for more than a decade. Ruffini helped establish one of the first full-fledged digital operations in Presidential politics for Bush-Cheney ’04, led digital strategy for the Republican National Committee, and later founded and grew Engage, one of the leading digital agencies on the right.
In recent elections, high-profile examples of polling missing the mark have people asking whether traditional polling is as reliable as it used to be. These questions reached a fever pitch after the UK’s recent vote to leave the European Union, with immediate pre-referendum polls pointing to a win for remaining in the EU; the 2015 UK general election, where nearly every pollster missed the Conservative Party’s comfortable win; and the stunning 2014 primary loss of House Majority Leader Eric Cantor, who had led by as many as 30 points in pre-election polling.
These questions come at a time when it is getting much harder and more expensive to reach a representative sample of adults over the telephone, the dominant way of reaching survey respondents for decades. Response rates have declined from 36% in 1997 to just 9% in 2012, according to the Pew Research Center. Additionally, respondents increasingly only use cell phones, which are twice as expensive to reach, since the government mandates that these numbers be dialed manually. The increased cost of calling mobile devices disproportionately affects access to young people between 18 and 29 years old and Hispanics, two groups where more than 60% of the population doesn’t even have a landline telephone, according to statistics compiled by the Centers for Disease Control.
These difficulties are leading consumers of polling to look for alternatives, from Internet surveys to digital sentiment analysis to big data analytics. The polling industry as we know it today will look very different ten years from now, and we already see the transformation underway in this year’s elections.
Despite Challenges, Polling Isn’t Dead Yet…
First, the good news: Despite highly publicized “misses” and the challenges facing the industry, there is little evidence that polling has gotten less accurate over time. This primary season, polls have largely been on the mark. The Republican polls showed a strong lead for Donald Trump throughout. When analysts strayed from the raw polling numbers, conjecturing that Trump would wither under a barrage of establishment attacks, they were usually wrong. On the Democratic side, polls showed a stable and consistent national Hillary Clinton lead with few surprises in state-level contests.
The polling industry has also adapted, conducting more interviews via cell phone or online. Internet polling in particular has proven very popular as a way to counteract the rising costs of dialing cell phones. In the Republican primary, Internet polling rose 257% over 2012, counteracting a 15% drop in live telephone polling and contributing to an overall 37% increase in national polling, according to an analysis by The Huffington Post. Nearly half of all national polls in the 2016 primary race were Internet polls.
…But Understanding How to Poll Is More Essential than Ever
Recently, we’ve been seeing a rise in state-level media polls conducted using phone numbers collected from state voter rolls, a best practice used by campaign pollsters. Calling phone numbers at random, or random-digit dialing, has been the traditional way of ensuring that polls are representative of the general population. This method is time-tested and is still often used in national surveys, but can fall short in low-turnout primaries when knowledge of the respondent’s past participation matters more. Polls were most off the mark this year on the Republican side in Iowa, where just 6% of Iowa adults voted, when many pollsters dialed randomly or didn’t know who had attended caucuses in the past.
Data from voter files is being used to update traditional methodologies for assessing who is and isn’t likely to vote. Rather than only asking voters if they are likely to vote, we can see whether they actually have in the past. Then, rather than discard data from unlikely voters in a “likely voter” survey as pollsters traditionally do, we simply weight according to the share of sporadic voters traditionally in the electorate. In our pre-primary survey in South Carolina, we found that those with less than a 50% chance of participating supported Donald Trump at 36%, while those 95% likely to turn out supported him at 26%. In no turnout cohort did Trump lose. By appropriately weighting based on a voter’s likelihood to participate, we were able to accurately project Trump’s winning margin within 1%, while also releasing multiple sets of expected results based on high, medium, or low turnout. This approach gives consumers of polls a more accurate, scenario-based understanding of the interplay between turnout and election outcomes.
The Shift to Online Interviewing: Possibilities and Pitfalls
The new paradigm of online polling has led to exciting successes, and inevitably, novel challenges. Many of the reasons behind the shift to online polling are pragmatic: they cost less, typically have larger sample sizes, and have proven they can be accurate in forecasting election results. For example, the online survey technology company SurveyMonkey correctly forecast the 2014 US midterms and 2015 UK elections—which other pollsters missed—by randomly sampling users of its survey tools, people doing everything from filling out customer satisfaction surveys to choosing times for their next book club meeting.
Some online surveys are conducted of “non-probability” panels, meaning that respondents opt-in to take surveys and that every member of the public does not have an equal probability of being included (a basic tenet of traditional survey research). Yet, even those Internet panels performed better than most traditional pollsters in 2012. In a review compiled by Nate Silver, four of the seven most accurate pollsters in 2012 conducted interviews online. By contrast, only one of the bottom ten performers used online polls exclusively, while another used online panels as a supplement to automated landline surveys.
While online polling is getting better every year, it isn’t perfect. The number of people who take online surveys is rarely large enough to complete a reliable survey in Congressional or local elections. There are also vast differences in the composition of panels on lines of educational attainment and political engagement. Some panels are skewed heavily in the direction of more engaged citizens (who have actively volunteered to take surveys). This is true even if they’re representative of the demographics of the country. A recent Pew Research Center study found that one company in particular (YouGov) was able to achieve significantly reduced bias and increased accuracy by weighting to education and civic participation in addition to traditional demographic variables.
For firms like Echelon Insights, the reasons to use online polling extend beyond capturing younger demographics or electoral accuracy. Much of the value in online surveys is in opening up new possibilities for the types of questions we can ask, and how.
Most polling is not about elections, and even the political polling done for election campaigns is not primarily about knowing who’s winning. Campaigns and issue organizations poll to know how to win the argument. They poll to understand which messages will persuade voters. Properly breaking down and framing the argument—and wording questions accordingly—is critical to this task.
Additionally, the political industry has a “big data, small content” problem. Campaigns are ever-more precise in their targeting, but not always as rigorous as they should be at validating message effectiveness. Voter files that have been matched to consumer databases often come with hundreds or even thousands of variables that we can use to build target segments. And while much effort has been expended on discovering new niche audiences, campaigns often fail to close the loop by proving that a given message or ad actually works in the field. New techniques made possible through online surveys are closing this gap.
In fact, Internet polling may be uniquely suited to understanding what resonates with different populations. Telephone surveys can easily fatigue respondents forced to sit through lengthy 20-minute interviews designed to tease out every angle of an issue. The same questions can often be asked more efficiently online. Survey respondents can also rate campaign ads second-by-second online, in a mechanism similar to dial tests in focus groups, except that technology allows us to show this message to hundreds of people rather than the dozen or so in a physical focus group. At our firm, we’ve applied this same concept to any piece of text, letting respondents highlight text they like and strike through text they don’t. Methods like these—which aren’t possible over the phone—help organizations surgically hone their message before committing limited resources to TV ad production and paid media.
The Future as a Hybrid
We’ve heard the case for gloom and doom in the polling industry: response rates are plummeting, and new techniques aren’t reliable enough yet. But upon closer inspection, this narrative doesn’t hold water: as the industry’s methods have shifted, accuracy hasn’t suffered. If anything, these shifts have been a necessity to defend against the declining accuracy of what the industry used to do: call voters at random on landline phones.
Furthermore, the scope of what we can learn about public opinion and human behavior from data is growing daily, and what we think of as “polling” is only a tiny part of it. Voters exhibit behaviors we can learn from even when not in a voting booth or talking to an interviewer. Often, the outward signs of this are digital: typing their questions into the Google search box or sharing what they think on Twitter and Facebook.
Social media and search trends data is not truly representative of the general population, sure, but it can help us go places traditional methods can’t. Day-of-the-election Google search share caught late swings to surging candidates and was accurate to within a few percentage points in early Republican primary contests. Language used in online conversation can help us craft better surveys by making sure our questions are reflective of how people talk about the issues. Media analytics, measuring the rate at which both traditional and new media sources mention candidates, were the canary in the coal mine for Donald Trump, with a surge in media coverage and online interest preceding his initial rise in the polls and his improbable march to the nomination.
The polling industry in the future will be informed by the time-tested methods of the past, but will increasingly bring in newer disciplines, from modeling and statistics, to capturing and making sense of mountains of real-time digital data. Technology disruption in the polling industry isn’t just coming. 2016 shows it’s well underway.