Gallup is one of the oldest names in polling, but it’s 2012 polling turned out to be among the most inaccurate of all the professional pollsters. On more than one occasion, its Daily Tracking Poll claimed to show the race tightening and movement in Mitt Romney’s direction that was not being reflected in any other poll with the exception of Rasmussen, which was also among the least accurate polls of the 2012 cycle. Ever since then, Gallup has apparently been conducting an internal review to attempt to figure out what went wrong, and they recently shared some of their findings with the public:
The review, led by Mr. Newport and Michael Traugott of the University of Michigan, was conducted by a team of statisticians, methodologists and analysts from Gallup, assisted by outside consultants. After identifying more than 20 possible elements that could have been related to the Republican skew in Gallup’s polls, the team isolated four factors as the likely causes.
Gallup’s model for identifying those most likely to vote — a series of seven questions — seemed to have failed in 2012, and the organization is re-evaluating its formula for ranking voters who will turn out.
Just as technology has changed the way campaigns work, it has altered the way survey researchers gather data. A change in the way that Gallup called respondents on land lines may have been a contributing factor that led to its sample to be older and more Republican. Half of their respondents, however, were reached on cellphones — a proportion that is at or above industry standards.
In addition, there were too many respondents from certain time zones. In the South and the Midwest, there were more respondents from the Central Time Zone, which tends to be more Republican, than the Eastern Time Zone, which skews Democratic.
Finally, the way Gallup asked callers about their race overrepresented some groups.
Individually, none of the four problems is considered to have been a significant factor in the results, but taken together, they helped lead to an overestimation of Mr. Romney’s share of the vote. Gallup plans to use the governor’s races this fall in New Jersey and Virginia to test new election polling methods.
Given that polling is what Gallup does to earn money, it’s in their best interests to make sure that the models that they are using are as accurate as possible. Gallup doesn’t really make money off the political polling that gets released to the public, but they do make money off of the private polling that they do, mostly for corporations, trade associations, and similar organizations. If the accuracy of their public polling is called into questions, that’s likely to raise concerns among private clients that could cut into revenues. Therefore, this kind of review is in their interests. Interestingly, though, I have yet to hear of a similar review being done by Rasmussen.
