Steven Taylor’s “Split Polls” post from this morning, based on a NYT/Sienna poll, got me digging into the poll’s methodology and a related FAQ. Given the pattern of commenter reaction to any post on polling, I thought some of what I found there useful to highlight.
Whom do we talk to, and how do we reach them? Do you call cellphones?
The New York Times/Siena College Poll is conducted by phone using live interviewers at call centers based in Florida, New York, South Carolina, Texas and Virginia. Respondents are randomly selected from a national list of registered voters, and we call voters both on landlines and cellphones. In recent Times/Siena polls, more than 90 percent of voters were reached by cellphone.
One of the most common questions we get is how many people answer calls from pollsters these days. Often, it takes many attempts to reach some individuals. In the end, fewer than 2 percent of the people our callers try to reach will respond. We try to keep our calls short — less than 15 minutes — because the longer the interview, the fewer people stay on the phone.
This has long been my understanding of the practice. But, of course, if only 2 percent of those contacted will respond, we’re left with a self-selected sample rather than a representative one. To the extent that the kind of person who will volunteer to take the time to participate in a ~15 minute survey are different than the other 98 percent, that’s a problem.
Is a phone poll really still a good way to reach people?
Phone polls used to be considered the gold standard in survey research. Now, they’re one of many acceptable ways to reach voters, along with methods like online panels and text messages. The advantages of telephone surveys have dwindled over time, as declining response rates increased the costs and probably undermined the representativeness of phone polls. At some point, telephone polling might cease to be viable altogether.
But telephone surveys remain a good way to conduct a political survey. They’re still the only way to quickly reach a random selection of voters, as there’s no national list of email addresses, and postal mail takes a long time. Other options — like recruiting panelists by mail to take a survey in advance — come with their own challenges, like the risk that only the most politically interested voters will stick around for a poll in the future.
In recent elections, telephone polls — including The Times/Siena Poll — have continued to fare well, in part because voter registration files offer an excellent way to ensure a proper balance between Democrats and Republicans. And perhaps surprisingly, a Times/Siena poll in Wisconsin had similar findings to a mail survey we commissioned that paid voters up to $25 to take a poll and obtained a response rate of nearly 30 percent.
That telephone surveys remain the most efficient means available to conduct survey research is not the same thing as telephone surveys being an effective means of conducting said research. What does “fare well” even mean here? Presumably, they’re useful only insofar as they’re predictive of actual behavior.
How can you be sure the people you reach are representative of the entire country?
Our best tool for ensuring a representative sample is the voter file — the list of registered voters that we use to conduct our survey.
This is a lot more than a list of phone numbers. It’s a data set containing a wealth of information on 200 million Americans, including their demographic information, whether they voted in recent elections, where they live and their party registration. We use this information at every stage of the survey to try to ensure we have the right number of Democrats and Republicans, young people and old people, or even the right number of people with expensive homes.
On the front end, we try to make sure that we complete interviews with a representative sample of Americans. We call more people who seem unlikely to respond, like those who don’t vote in every election. We make sure that we complete the right number of interviews by race, party and region, so that every Times/Siena poll reaches, for instance, the correct share of white Democrats from the Western United States, or the correct share of Hispanic Republicans in Maricopa County, Ariz.
Once the survey is complete, we compare our respondents to the voter file, and use a process known as weighting to ensure that the sample reflects the broader voting population. In practice, this usually means we give more weight to respondents from groups who are relatively unlikely to take a survey, like those who didn’t graduate from college.
So, again, this comports with my understanding of how a nonbiased survey is conducted by professional pollsters. But it doesn’t solve the problem I mentioned earlier: How do we know that those who opt in to surveys are representative of their subgroups?
If so few people actually answer their phones, how do we know their responses aren’t skewed?
In 2022, we did an experiment to try to measure the effect nonresponse has on our phone polls. In our experiment, we sent a mail survey to voters in Wisconsin and offered to pay them up to $25 to respond. Nearly 30 percent of households took us up on the offer, a significant improvement over the 2 percent or so who typically respond by phone.
What we found was that, overall, the people who answered the mail survey were not all that dissimilar from the people we regularly reach on the phone, on matters including whom they said they would vote for. However, there were differences: The respondents we reached by mail were less likely to follow what’s going on in government and politics; more likely to have “No Trespassing” signs; and more likely to identify as politically moderate, among other things.
But the truth is that there’s no way to be absolutely sure that the people who respond to surveys are like demographically similar voters who don’t respond. It’s always possible that there’s some hidden variable, some extra dimension of nonresponse that we haven’t considered.
So, that’s somewhat comforting. Phone surveys are a lot cheaper, easier, and faster to conduct than mail surveys—let alone those that pay $25 a pop for responses. If the results are essentially the same, then it’s obviously preferable to a mail survey. Still, that’s not the same thing as saying they’re good.
Granting that one’s stated preference and one’s behavior aren’t necessarily going to align—a lot of people who state a preference for either Trump or Biden won’t bother to vote at all and, I suspect, many who say they prefer RFK Jr. will either stay at home or vote for their least unfavorite of the major party candidates—the main value of election polling is to gain an understanding of the mood of the electorate to predict election outcomes.
How accurate are polls in general, and The Times/Siena Poll in particular?
In the 2022 midterm elections, Times/Siena poll results were, on average, within two points of the actual result across the races we surveyed in the last weeks of the race. That helped make The Times/Siena Poll the most accurate political pollster in the country, according to the website FiveThirtyEight.
At the same time, all polls face real-world limitations. For starters, polling is a blunt instrument, and as the margin of error suggests, numbers could be a few points higher or a few points lower than what we report. In tight elections, a difference of two percentage points can feel huge. But on most issues, that much of a difference isn’t as consequential.
Historically, national polling error in a given election is around two to four percentage points. In 2020, on average, polls missed the final result by 4.5 percentage points, and in some states the final polls were off by more than that. In 2016, national polls were about two percentage points off from the final popular vote.
Being that close in midterm elections is an indicator that polling remained valuable. Midterms have always been harder to predict than general elections precisely because of the preference-behavior gap addressed previously. In the main, people are far more likely to vote in a general election than in a midterm but the degree to which that is the case varies considerably.
