Washington Post polling director Scott Clement and his colleague Emily Guskin explain “How polling works and how it’s changed since 2016.” OTB readers are likely up to speed on the basics of survey methodology, sampling error, and the like but the nuance is constantly evolving.
First off, while any competent polling firm can pretty accurately portray public opinion, predicting behavior is much harder. So, while we have a great idea about how Americans feel about Donald Trump’s performance as President, it’s harder to know who will actually show up to vote. Second, because of the odd way in which we elect Presidents in particular, the problem is compounded by having to predict how people in a handful of key states will behave.
A 2017 report by the American Association for Public Opinion Research (AAPOR) found that while national polls had an average error of about two points in presidential elections since 2000 and were fairly accurate in 2016, the error in state polls was double that, at four points. The level of state polling error ranged from 3.2 points in 2004 to 5.1 points in 2016.
It is almost impossible to predict the direction in which polls will miss in a given year. In 2012, some supporters of Republican nominee Mitt Romney criticized surveys during the campaign for overrepresenting Democrats. But final state polls actually underestimated Barack Obama’s vote margin by 2.3 percentage points on average, according to the AAPOR.
The 2012 errors were largely forgotten because most polls showed Obama leading, just by a smaller margin. Yet the 2016 election demonstrated how systematic errors in a variety of states can lead to a surprising outcome. National polls missed by just a single percentage point on average, yet state surveys underestimated Trump’s vote margin by three points in Michigan and Pennsylvania, as well as seven points in Wisconsin.
Even though my late first wife was an executive with Public Opinion Strategies, the leading Republican polling firm in the country (they’re also half of the team that does the NBC/WSJ Poll and the CNBC All-America Economic Survey), I don’t fully understand why polling at the state level is harder. Part of it, though, is simply that it’s essentially as expensive to conduct a state poll as it is a national one and there are therefore many fewer state-level polls available to the public. Additionally, since the Presidential election hinges on a handful of “swing” states, it’s quite possible that those particular contests are intrinsically harder to predict. (Meanwhile, I can confidently predict which party will take Alabama in 2024 without conducting a single poll or having any idea who the party nominees will be.)
Pollsters have made two significant changes this year aimed at improving accuracy over 2016.
One is simply conducting more polls, a costly decision but one that may contribute to greater precision overall. From the start of September to last week, RealClearPolitics tracked 105 polls in Pennsylvania, Michigan, Wisconsin and Arizona, nearly double the 54 polls over the same period in 2016 in these states. The number of nonpartisan polls conducted with live telephone interviews — a more expensive but historically more accurate method — grew from 24 in 2016 to 36 in 2020.
In addition, more polls also appear to be weighting samples by educational attainment, something that wasn’t done in 2016 and that the AAPOR’s post-election report found contributed to polls underestimating Trump’s support.
Polls are routinely weighted to match estimates of population demographics including race, age and other factors that are correlated with voting. In 2016, there was an especially strong correlation between education and support for Trump and Clinton in key states, with Trump winning by wide margins among White voters with some college or less. Current polls show that correlation remains strong in this year’s election between Trump and Biden. College graduates have long been more likely to participate in surveys, and states that did not weight samples to correct for that — by weighting down the share of responses from people with degrees to their accurate share of the population — were at greater risk of underestimating Trump’s support.
But, of course, that’s dangerous. Any weighting takes what is a purely mathematical exercise and adds human judgment. That’s required; indeed, the difference in a “good” poll and a “bad” one is the likely voter screen. If those with college degrees and those without vote at the same levels, then it obviously makes sense to correct one’s sample to reflect that fact. But my guess is that college-educated folks are more likely to turn out. If so, making an adjustment based on what may well be a one-off in the previous election could well skew the polls. (Although, in this case, it would likely understate Biden’s margin if 2020 turns out to be a reversion to the norm.)
They close with the standard disclaimers:
Polls in 2016 were more accurate than many people think, and in the long run, there’s reason to have confidence in them. Still, there are enough wild cards this year — the coronavirus pandemic, extraordinary political interest, an election like none before and adoption of new voting methods — that could affect the accuracy of pre-election polls.
Take the pre-election vote estimates as advertised: a tally late in the game, with voters in charge of the final score.
That’s both good advice and a cop-out. I’d be willing to bet a good deal of money on Biden winning the popular vote. Further, I’m reasonable confident that he’ll win in enough states to get to 270 Electoral votes given the steadiness of the polls. But I’m not at all confident in how Pennsylvania or Wisconsin will turn out.
