Charles Franklin takes issue with Stan Greenberg’s contention that media polls are “criminally bad” because of a variety of reasons, especially that “They are not carefully weighted and, as a result, show wide swings in voter preference that the media interpret wrongly as voter fickleness.”
I’d be curious what constitutes careless weighting. Almost all pollsters weight the data to demographic distributions derived from the Current Population Survey (a huge monthly government survey with over 90% response rate and therefore considered particularly reliable.) Pollsters might differ on some technical issues here but it is hard to believe that media polls are that different from Greenberg’s own methods. None of the weighting techniques are in any way secret– just buy a textbook on survey sampling or read the journals or attend panels at AAPOR (the pollsters conference) and the variety of options are all right there in the public domain. So there is little reason to think that variation in weighting practice is due to either secret knowledge that Greenberg and colleagues have that is unavailable to others, or that “media pollsters” systematically choose to be reckless by using poor weighting schemes.
[…]
My bottom line is that polling techniques and methodology are “open source.” Take classes in grad school and you can learn all the theory. Work in a polling firm and you’ll also learn a lot of practical wisdom. Survey professionals all have access to this. Greenberg may well think that his polls are superior to those conducted by others, but I’d disagree that this has anything to do with secret knowledge or methods.
I’m dealing with an excerpt of Greenberg’s argument, which is made in a book I don’t have access to, so it’s hard to assess his claim. Certainly, media polls conducted early in cycles tend to be rather sloppy, in that they seldom apply “likely voter” screening. They also tend to vary wildly from one another, which would seem to indicate that either 1) the people running them didn’t attend graduate school or AAPOR conferences or 2) polling is really hard and simply applying the basics isn’t sufficient.
Even late in cycles, when most of the national polling outlets switch to likely voter models, there’s wide variation in the numbers. That’s not a big deal in races, such as the most recent presidential contest, where the margin is wide. In close races, though, they’re virtually worthless.
Full disclosure: My wife, as I’ve mentioned more than once, is the COO of Public Opinion Strategies, the major Republican polling firm, which partners from time to time with Greenberg on, oddly enough, some media polls (for NPR and MSNBC).
I’d be interested to see the comparisons between the polling of Greenberg, POS, Gallup, and the various major media polls and electoral outcomes. My strong suspicion is that the first two are closer in alignment with the thing they’re trying to project than the others, especially on a sustained basis.
Link via Political Wire.
