A new study seems to show that student evaluations of teachers are something other than a popularity contest. Sam Dillon for NYT:
How useful are the views of public school students about their teachers? Quite useful, according to preliminary results released on Friday from a $45 million research project that is intended to find new ways of distinguishing good teachers from bad.
Teachers whose students described them as skillful at maintaining classroom order, at focusing their instruction and at helping their charges learn from their mistakes are often the same teachers whose students learn the most in the course of a year, as measured by gains on standardized test scores, according to a progress report on the research.
Financed by the Bill and Melinda Gates Foundation, the two-year project involves scores of social scientists and some 3,000 teachers and their students in Charlotte, N.C.; Dallas; Denver; Hillsborough County, Fla., which includes Tampa; Memphis; New York; and Pittsburgh. The research is part of the $335 million Gates Foundation effort to overhaul the personnel systems in those districts.
Statisticians began the effort last year by ranking all the teachers using a statistical method known as value-added modeling, which calculates how much each teacher has helped students learn based on changes in test scores from year to year. Now researchers are looking for correlations between the value-added rankings and other measures of teacher effectiveness.
Research centering on surveys of students’ perceptions has produced some clear early results. Thousands of students have filled out confidential questionnaires about the learning environment that their teachers create. After comparing the students’ ratings with teachers’ value-added scores, researchers have concluded that there is quite a bit of agreement. Classrooms where a majority of students said they agreed with the statement, “Our class stays busy and doesn’t waste time,” tended to be led by teachers with high value-added scores, the report said. The same was true for teachers whose students agreed with the statements, “In this class, we learn to correct our mistakes,” and, “My teacher has several good ways to explain each topic that we cover in this class.”
I’m quite skeptical of these findings. That’s not surprising, in that I’ve developed rather strong opinions on teaching evaluations from personal experience with them as a college professor. My own ratings with survey courses were all over the place — sometimes wildly different in different sections of the same class in a given semester with open format questions delivering precisely opposite conclusions. In the upper sections, populated mostly by political science majors, my evaluations followed a curve nearly identical to the student grades.
Granted, that’s anecdotal, with a very small “n.’ And maybe elementary school students are more insightful and honest than their college peers.
But three other questions jump immediately out at me.
First, which direction does the causality flow? Maybe the type of students who perform well on tests are more likely to give their teacher credit, rather than the teaching style influencing both the ratings and the test performance?
Second, how do the correlations hold up longitudinally? That is, do the same teachers get similar ratings and test performances every year? Otherwise, this may be a function of a teacher getting a randomly generous allocation of bright, well behaved students in a given rating period and it being taken as a sign of good teaching.
Third, what other variables are at work here? Scanning the PDF of the Gates Foundation report, it appears as if the research team proceeded as if teaching styles and test outcomes were the only factors. But we know that parental involvement, parental education, home life, child nutrition, and all manner of other things really, really matter. Maybe the “good” teachers mostly taught in wealthy schools with involved parents who ensure their kids went off to school with a good breakfast, a decent night’s sleep, and had mastered their homework. If there’s no control for this sort of thing, the study is completely worthless.
