How To Read The Polls: General Election Edition
Time to prepare yourself for the incoming deluge of polling.
Nate Silver is out with a very useful guide to reading, understanding, and putting into context the plethora of national and state-level polls that will be hitting us on nearly a daily basis from now until Election Day. His final point is perhaps the most prescient:
There have been only 16 presidential elections since World War II. That simply isn’t a lot of data, and overly specific conclusions from them, like “no recent president has been re-elected with an unemployment rate over 8.0 percent” or “no recent incumbent has lost when he did not face a primary challenge,” are often not very meaningful in practice and will generally not carry much predictive weight.
Another point to keep in mind is how seriously to treat the numbers for demographic subgroups:
The sample sizes on subpopulations in a poll — like Hispanics, young voters or evangelical Christians — are much smaller than for all voters as a whole and therefore contain much larger margins of error. For instance, a poll that surveys 600 respondents, of whom 75 are Hispanic, has a margin of error of about plus or minus 11 points on that subgroup. And that is under ideal circumstances; in practice, some subgroups (including Hispanics) are harder to get on the phone than others.
It’s easy to write the “Candidate X has problems among Group Y” stories, but very often they are just weaving narratives from statistical noise. Unless the demographic patterns are clear and consistent across several different polls, these stories are usually worth ignoring.
There are other things to keep in mind, such as paying attention to whether a poll result is among likely voters, registered voters, or just “all adults,” of those three groups “adults” probably has the lease predictive value since it is likely capturing people who have no intention of voting in November, or aren’t even registered to vote. Among all three groups, it’s also important to keep an eye on the sample weighting among Republicans, Democrats, and Independents. With a polling population of a few hundred, or even as few as a thousand, respondents it’s fairly easy for a Romney v. Obama result to be thrown off by a sample that is overly represented by one party or another, or that doesn’t contain enough independents. Of course, determining what the right sample balance isn’t always an easy thing to do because it requires at least some guess work in figuring out what turnout on Election Day is likely to be. Typically, though, a poll that has a D/R/I breakdown that is wildly different from previous elections and other polls should be looked at skeptically at least. Of course, some pollsters don’t release complete crosstabs for their polls while others, like Rasmussen, keep that data behind a paywall and charge extra for those who want to access it. If the data isn’t available and the poll results seem out of the norm, then it’s probably a good idea to be at least a little skeptical.
That leads to what might be the best piece of advice that Silver gives:
This ought to be obvious, but you should generally be looking for a trend to show up in several different polls from several different polling firms before you start to view it as newsworthy. Again, this differs a little bit from the primaries because there is less of a premium on recency in the general election; you’re usually better off waiting for another (or better yet two or three more) data points.
The easiest way to do this is to take an average of recent polls, as sites like Real Clear Politics do. The technique that FiveThirtyEight uses is a little fancier, taking a weighted average of polls based on their past accuracy as well as their methodological standards. However, the gains from doing this are modest as compared with the simple average method. In contrast, taking even that simple polling average provides for considerable gains in accuracy over any one poll taken alone.
Four years ago, RealClearPolitics proved itself to be invaluable to political junkies because it was the first site that introduced the concept of the polling overage, and kept track of it on a daily basis. Now, in addition to RCP, we’ve got Nate Silver’s incomparable date crunching, along with tracking that you can keep track of at The Huffington Post and Talking Points Memo. The cool thing about the HuffPo and TPM trackers is that they include options that allow you to control what data is showing in the charts in generates. You can restrict the chart to a certain date range, restrict the types of polls that are included in the average, and restrict the specific pollsters that are included. RCP remains the go-to place but it’s chart’s aren’t manipulable in this manner, at least not yet.
The other part of Silver’s comment here is worth taking to heart too. It’s very easy to get caught up in the excitement of a new poll, especially if it’s one that shows something dramatic (Obama up! Obama down! Romney losing support among Hispanics!). I’ll admit to getting caught in that trap more than once myself. However, one poll doesn’t necessarily tell us much of anything, especially given some of the caveats I noted above. What matters is the trend, especially since we’re talking about an election that is still seven months away. There will be ups and downs between now and then. Both candidates are likely to get a bounce from their respective conventions, for example. And, of course, the economy will continue to be the major influence on the attitudes of voters. So, let’s all try to give these polls the attention they deserve, and none that they don’t.
Update: Here’s another useful tool, this from The New York Times, an interactive chart of Exit polling for all Presidential elections going back to 1980
All polls reported will be to foster the belief that the “horse race” is close.
…The reality is that it is not particularly close given the electoral college advantage of team Obama.
…Really the polling reported will mean next-to-nothing until the inevitable debates in the fall and the re-engagement of the vast majority of Americans who, unlike us, start to pay attention.
…The the polling will have some bite.
Around here, it appears to be from right to left.
Thank God for polls, without them we’d have to talk about issues.
@michael reynolds: Issues are passe. Too bourgeois.
How have the pollsters adjusted their numbers to reflect the fact that land line phones are becoming more and more rare, especially for anyone under the age of 40?
It seems to be any phone poll (and as far as I know most of them are phone polls) would be skewed by the types of people who have land lines, and even more by those that would answer said phone call from an unknown number.
@J-Dub:
Pretty much every pollster now includes cell phones in the numbers they call. Most of them will have data on land line v. cell phone respondents as part of their crosstabs
@Doug Mataconis: Pew article on that here:
http://www.pewinternet.org/Reports/2008/Polling-in-the-age-of-cell-phones.aspx
What’s more boring than reading polls? Reading articles about polling methodology.
Single polls should always be taken with a grain of salt, because, statistically, there are bound to be outliers published over the course of Presidential Campaign.
Ah, but what is reading the polls compared to interpreting the trends? That’ll be where the fun bullshit pops up.
the way to tell what inside polling shows is where the candidates are spending their money. If Obama is making appearance and spending large amounts of money in Virginia and North Carolina, then Obama has a huge lead and the result will be a rout. If both candidates are spending a lot of time in Ohio or Florida, then the election is close.
Since the Republicans are planning are spending funds in Virginia and North Carolina, it should be obvious that Romney will lose in a rout. If President Obama makes more than one appearance in Arizona, then one should realize that he will be reelected by more than 54% of the popular vote.
I remember that after Gingrich took his hits at Romney and Bain Capital, polls came out that showed him doing not so badly, so people thought the comments ultimately had no effect.
Except that they did, but it didn’t happen right away. About a week and a half after his final comments on the matter, Newt started dropping in the polls and never regained his footing. Around the same time, Santorum gain a second wind in the polls after dropping a bit a week after Iowa.
@superdestroyer:
If they both don”t spend a lot of time in Ohio, they are guilty of electoral malfeasance. As I read the electoral college, I cannot see Romney winning unless he wins Ohio. I do not see a way for it to happen. So both candidates will be in Ohio, Romney because he needs it to win and Obama to do all he can to deny Romney any shot at winning.
Romney’s chances in Ohio are looking pretty dismal right now, but a lot can change between today and November.
@Doug Mataconis: Gallup doesn’t…..which is why I dismiss them.
@KariQ:
If President Obama if far enough ahead in Ohio, he will put less resources there and more resources into Virginia, North Carolina, Arizona, Missouri.
What everyone should admit is that the Republicans are just not competitive at the national level. Instead of readings about presidential polls, why not focus on the Senate and House races. The more uncertain question is whether Nancy Pelosi will be back as Speaker of the House.
@Jr:
Actually, they do:
@Doug Mataconis: reaching a cell phone number does not guarantee that number belongs to a “cell-phone only” household. Most people who have land lines also have cell phones. According to the article I linked to, people who have cell phones but no land lines respond much differently than those who have both. I would argue that people who only have cell phones are going to be younger, more likely to live in an urban environment, and are probably more liberal than the land-liners.
Suggestion for a future post topic (or if someone can provide a good link): Sampling in general.
It seems like one of the key things that pundits use to immediately ignore polls is the over/under sampling argument. It would be great for a quick primer on what constitutes “representative” sampling.