538 senior elections analyst Geoffrey Skelley is back from the seminal conference for professional pollsters and reports that “Polling isn’t broken, but pollsters still face Trump-era challenges.” His essay is long and worth reading in full but I will excerpt some key takeaways.
Sampling: Meet people where they are
One of the conference’s consistent refrains was to “meet people where they are” when trying to reach potential respondents — that is, by using different modes of communication to contact different groups of people, based on their personal preferences. “There’s no way anymore to get a representative sample of the U.S. population or the voters from a single mode,” McPhee observed. “There are too many technologies out there. There are too many beliefs and preferences and attitudes and feelings about responding in general and responding by different modes.” Ensuring no group is underrepresented because of low response rates is critical for pollsters to gather a representative sample of the population they’re interested in — such as all voters in a given state — particularly if potentially underrepresented groups hold notably different views from those who are overrepresented.
McPhee told me the industry has moved toward agreement that multi-mode approaches are the best way to get a more representative sample. She stressed that it’s not that one mode is better, but rather a combination is “better than the sum of its parts.” For instance, one SSRS survey experiment saw improved response rates for state-level surveys that recruited respondents by various means, including postcard or SMS text message, and gave respondents six potential access points to respond: URL, QR code (directing them to the survey), text, email, a phone number for respondents to call (inbound dialing) and SSRS reaching out to them by phone (outbound dialing).
Texting in particular has become a common means to engage respondents across all demographic and partisan groups, often as part of a mixed-mode approach. Kevin Collins is the co-founder of Survey 160, a firm that focuses on SMS text-based surveys in its work with Democratic pollsters and nonpartisan organizations, largely via text-to-web sampling (text messages that link to a survey on a web browser) or live interviews over text. “We do text message surveys, but really what we believe in is mixed-mode surveys,” Collins told me. “Texting offers a very important additive benefit and cost savings over phones.”
[…]
Weighting: The known unknown of who will vote
Beyond gathering as representative a sample as possible, election polling presents another specific challenge: trying to gauge what the electorate will look like in a given election cycle. This raises questions about how to appropriately weight sample data — that is, how to adjust survey datasets to account for differences between the sample population and the overall population — to reflect who is most likely to actually vote in November.
In its post-2020 report, AAPOR cited nonresponse bias as one of the potential reasons for the cycle’s larger polling error, possibly because Democrats were systematically more likely to respond to pollsters than Republicans, and because the Republicans who did answer may have differed in important ways from the Republicans who didn’t.
[…]
But even the most accurate weighting by past votes can’t answer the ever-present question of which voters will turn out this year. How to handle the many voters who cycle in and out of the electorate is no minor consideration: As Nate Cohn at the New York Times recently noted, historically, roughly one-quarter of presidential election voters didn’t vote in the last presidential contest. That churn results in part from younger voters entering the electorate and older ones leaving it, but also reflects the participation of less engaged voters who bounce in and out of the electorate from year to year. “Good pollsters have to be thoughtful about how they account for those people,” McPhee said, as weighting by recalled vote isn’t an available option with new voters.
“Asking who is a likely voter is the wrong question,” Collins said. “It’s easy to know who the likely voters are — they’re the people who regularly vote. But the challenge is identifying which unlikely voters will end up voting.” And he stressed that this question becomes much more difficult if likely and unlikely voters have divergent vote preferences — as some national surveys have suggested is the case this year, finding that registered voters who didn’t vote in 2020 are slightly more Trump-leaning in 2024 surveys than those who did vote in 2020. That makes a pollster’s estimates of just who will show up on Election Day even more important in determining the outcomes of their polls.
Leaving aside those polls run by hack organizations to spread a particular ideology, those who do survey research have every incentive to get it right. Polling professionals are constantly adjusting their methodologies to changing technologies and habits.
While I have strong confidence that the polls accurately reflect the opinions of the American public on President Biden, former President Trump, Congress, the Supreme Court, and various longstanding policy issues, the same is not true for election outcomes. It’s always been very hard to know which fence-sitters will show up to vote in a given election. And it’s gotten much harder in the Trump era.
