You may recall that a couple of months ago, there was a “shocking survey” circulating around which showed that only 23% of Oklahoma students knew that George Washington was the first President of the United States. At the time, my colleague James expressed his extreme skepticism of this result, a skepticism shared by most of the commenters. Well, as it turns out, this skepticism was quite justified, as Nate Silver points out:
[Oklahoma State Representative Ed] Cannaday therefore had little difficulty setting up an experiment: he arranged to have all the seniors in the 10 secondary schools in his district take the Strategic Vision/OCPA survey. Cannaday tried to replicate the Strategic Vision survey to the greatest extent possible. The same exact questions were used, and as in the case of the original survey, the answers were open-ended rather than multiple choice. The survey was administered to a total of 325 seniors, including special education students.
Cannaday’s survey however, found his students doing just fine: They answered an average of 7.8 out of the 10 questions correctly. By comparison, the high school students that were purportedly surveyed by Strategic Vision had gotten just 2.8 out of the items correct. 98 percent of the students on Cannaday’s survey — not 23 percent — knew that George Washington was the first President. 81 percent — not 14 percent — knew that Thomas Jefferson had written the Declaration of Independence. 95 percent — not 43 percent — knew that the Democrats and Republicans are the major political parties. There was just no comparison between the two.
[…]
There is no reason to think, in other words, that the students in House District 15 should have gotten such profoundly superior results to the “students” in Strategic Vision’s survey. Nor could Strategic Vision’s results have been the result of any sort of mathematical or methodological oddity. Consider their claim that literally none of the 1,000 students they surveyed were able to answer more than 7 of the 10 questions correctly — lower than the average score achieved in Cannaday’s test.
There are, rather, only two possibilities. Either the Strategic Vision survey was entirely fabricated — or Cannaday’s was.
I would put every dollar to my name on Cannaday, who has kept the surveys and is happy to show them to them to anyone who comes asking.
So would I. Cannaday’s results appear to have a sounder methodology, and also make more intuitive sense.
