Can students evaluate teachers? (part 1 of 2)

I've been looking at the results from our annual large-scale survey to our 5500 families. As you would expect, there are a range of opinions - and it can be hard to know how to interpret the conflicting 'please do more X' and 'please do less X' comments.

One theme that emerged very clearly, however, was that it's teachers who make the difference to students, and we were grateful for the parents who told us about the immensely positive transformations experiences that their children are undergoing with inspirational teachers.  But there were also many comments that asked us to formally involve students in teacher appraisal and evaluation - sometimes with the implication that there's a lot of stuff going on that school leadership does not know about.

Teacher Evaluation
A Hot Topic in many Schools and Countries

In a busy organisation it's impossible for anyone person to know everything that's happening - but we do have to try.  And I guess the notion is that students are the ones receiving the teaching, so shouldn't they get a say?

There are two points here.  Firstly, there is the moral need to treat students with respect, and to hear their perspectives about how things are going.  Secondly, there is the analytic need to know how to interpret data that comes from students (given that getting this second point wrong will damage student education, that's also close to being a moral need too).

In this post I want to suggest that what we know about the second point should directly inform the first point; and that we should not simply ask students to evaluate teachers.  To prevent this post from being too long, I'll follow up next week with some details about how, despite this, we still can and should involve students.

So why is student evaluation of teachers a bad idea?

What we know is that using students to evaluate teachers can be very problematic for many reasons - often because there is confusion between a teacher's popularity (he's a nice or funny or charming person) and a teacher's effectiveness (students learn a lot).  The two are not entirely independent of course, but there is a clear distinction.  We should therefore be troubled by, for example, Uttl et al's 2016 analysis that shows the correlation between students' opinions of their teachers and what they  actually learn is close to zero. If this is right, then it's a central flaw because it means we can learn nothing reliable from the evaluations.   But it is at least better than Rodin and Rodin's classic paper which suggests that it might even be worse than that - and that in fact students rate most highly the teachers from whom they learn least, and furthermore that students don’t learn more from teachers with high ratings - so the evaluations may actually be inverted! Perhaps many students will like a teacher who goes easy on a class and gives high grades for mediocre work, thus leading students to feel good about themselves and consequently give high ratings to that teacher.  The teacher who sticks to high standards, gives tough love and honest un-inflated grades, by contrast, will likely not be so popular, and may receive lower ratings.  Rodin and Rodin speculate 'perhaps students resent instructors who force them to work too hard and to learn more than they would wish'

This is not surprising to experienced teachers who can distinguish between the long- and short-term effect of what they do, based on years of experience.  These teachers worry that if the evaluations are high-stakes, there will be systemic pressure on them to be popular rather than effective.  I think we can all see where that has led to if we look at the global political stage.  Nothing serious should be decided solely on popularity.

A further reason to doubt student capacity to judge teachers is that judgement like this is subject to serious bias.  Hessler et al found that teachers who provided chocolate cookies during a course got significantly higher evaluations than those that did not.  No surprise there either!  Other biases are more serious.  MacNell et al undertook an ingenious online study, where teachers told one class they were male, and one class they were female.  They found that 'students gave professors they thought were male much higher evaluations across the board than they did professors they thought were female, regardless of what gender the professors actually were. When they told students they were men, both the male and female professors got a bump in ratings. When they told the students they were women, they took a hit in ratings. Because everything else was the same about them, this difference has to be the result of gender bias'.

So we know that student evaluations are problematic.  So what?  Let me return to and affirm the first point - that there is a moral need to involve students; and to hear their voices.  So how can we do it? 

It is possible to do; but it involves different approach, away from an evaluative consumer model to a collaborative one.  I'll write about that in some detail next week.

  

References

Hessler et al (2018) Availability of cookies during an academic course session affects evaluation of teaching. Medical Education 52(10):1064-1072

Kirschner, P and Neelen, M (2017) Student Teacher Ratings: Males are Brilliant, Females are Bossy

MacNell L., Driscoll A., Hunt A. N. (2016) What’s in a Name: Exposing Gender Bias in Student Ratings of Teaching Innovative Higher Education Aug 2015, Volume 40, Issue 4, pp 291–303

Marcotte, A. (2014) Best Way for Professors to Get Good Student Evaluations? Be Male. Slate Dec 9 2014

Miller, C.C, (2015) Is the Professor Bossy or Brilliant? Much Depends on Gender New York Times 6 Aug 2015

Rodin and Rodin Student Evaluation of Teachers Science 29 Sep 1972:Vol. 177, Issue 4055, pp1164-1166

Uttl, R, White, C.A. adn Wong Gonzales, D (2017) Meta-analysis of faculty's teaching effectiveness: Student evaluation of teaching ratings and student learning are not related Studies in Educational Evaluation Volume 54, Pages 22-42

Erik Voeten Student evaluations of teaching are probably biased. Does it matter? Washington Post Oct 2 2013

Addendum
Just came across this - 

Carrell, S. E and West, J.E (2010) Does Professor Quality Matter? Evidence fromRandom Assignment of Students to Professors 

Excerpts:
We compare metrics that capture these three different notions of instructional quality and present evidence that professors who excel at promoting contemporaneous student achievement teach in ways that improve their student evaluations but harm the follow-on achievement of their students in more advanced class...

Results show that there are statistically significant and sizable differences in student achievement across introductory course professors in both contemporaneous and follow-on course achievement. However, our results indicate that professors who excel at promoting contemporaneous student achievement, on average, harm the subsequent performance of their students in more advanced classes. Academic rank, teaching experience, and terminal degree status of professors are negatively correlated with contemporaneous value-added but positively correlated with follow-on course value-added. Hence, students of less experienced instructors who do not possess a doctorate perform significantly better in the contemporaneous course but perform worse in the follow-on related curriculum.