Student Feedback (SF) as an illustration of the problems

Student Feedback (SF) illustrates some of the archetypal problems in teaching evaluation. SF provides information that allows us to make reasonable inferences about teaching quality, but not a direct measurement of it. SF tells us whose teaching students consider to be excellent/good/poor, based on their value system; this may not include awareness of the need to develop independent learning, independent inquiry, and critical thinking. Their value system therefore may not be consistent with that of the educational institution itself. Hence, while it is safe to take poor rating from students as indications of a problem (discussed below), we cannot conclude that the higher the SF rating, the higher the teaching quality within the University’s value system.

Furthermore, if a teacher is a truly outstanding educator, his/her emphasis on critical thinking, independent learning, independent inquiry, and creative problem solving would create anxiety and resistance at least among some students. Such reactions can cause a lowering of SF rating. It is only in rare cases that a teacher stretches students to their limits and yet consistently receives a rating of 9 and above on a ten point scale. Hence one must be watchful of extremely high ratings.

If a teacher is a truly outstanding educator, there is also chance that his/her emphasis on critical thinking, independent learning, independent inquiry, and creative problem solving would create anxiety and resistance at least among some students. Such reactions can cause a lowering of SF rating. It is only in fortunate combinations of truly motivated students and highly accomplished teaching that a teacher stretches students to their limits and yet consistently receives a rating above 9 on a ten point scale. Hence one must be watchful of extremely high ratings from very large classes. If we accept that the many NUS students are

1. motivated by pragmatic considerations such as high grades,
2. not interested in learning for its own sake, and
3. would like to have modules that offer high grades with less work, it would follow that students would be favourably disposed towards easy options and negatively disposed towards tough options.

From this, it would follow that at least some students will mark down teachers who offer challenging modules, which would lower the overall grades that teachers get from students. In other words, if we accept the possibility of (i)-(iii), then it is important to be cautious about extremely high rating from all students in a large class.

To turn to a different kind of problem, the existing method of ranking teaches is based on the mean score in SF. If we look at the distribution of scores in SF, however, we get a different picture. For instance, imagine teachers X and Y whose overall ratings from a cohort of 100 students have the following distribution:

In terms of the mean score, we see students regard teacher Y as a better teacher. If we take the number of students who regard the teacher as excellent, however, teacher X is regarded as better. Which of these should we go by in ranking teachers in terms of student opinion?

Despite these problems, however, it is possible to make reasonable inferences on what the students must have learnt as a result of the teacher's intervention on the basis of the entire spectrum of pedagogical activities. Given below are a few samples of such inferences.

Rating in Student Feedback (SF) and Peer Review (PR)

Quantitative ratings in SF far below average are indications that students are negatively disposed towards the teacher/module. This can be problematic as negative reactions impede learning. However, that one teacher's rating is 9 and another's is 8 on a ten-point scale in SF does not legitimize the conclusion that the former is a better teacher than the latter. Similar remarks apply to the quantitative rating in PR as well. Given the variability across peer reviewers, it would be unwise to conclude that a candidate who receives a 9 from one reviewer is superior to another who receives an 8 from a different reviewer. Given that not all peer reviewers would have absorbed the value system advocated by the university, it is unwise to make this inference even when 9 and 8 are assigned by the same reviewer.

Qualitative comments in SF and PR

If the qualitative comments in SF or PR repeatedly mention some learning outcome (e.g., critical thinking, problem solving) or means of teaching (e.g., challenging tasks, interactive teaching and learning), it is likely that the module has facilitated that outcome or has that property. The absence of such comments, however, does not legitimize the inference that the relevant outcomes/properties are lacking.

Assessment tasks

Given the pragmatic culture of our student community, if the assessment in a module over a period of 3 or 4 years does not test a particular learning outcome, it is unlikely that the module could have led to the development of that outcome among students. If the exam questions in a module rely heavily on testing a particular outcome, and the student response to the module is not negative, it is highly likely that the module would have triggered the development of that outcome. This conclusion is not legitimate, of course, if there is evidence to show that answers to these questions could have been prepared by students in advance.

Methodology

If independent evidence indicates that a module facilitates a particular learning outcome, and the teacher employs sound pedagogical strategies such as active learning, interactive teaching and learning, problem based learning, pedagogically appropriate use of IT, and so on, then it is legitimate to infer that the module would have facilitated that learning outcome effectively. However, in the absence of independent evidence for the learning outcome, the use of sound pedagogical strategies by itself does not legitimize inferences on the outcome.

Mastery of the subject

A teacher who has a poor knowledge of the subject is unlikely to have helped students acquire sound knowledge of the subject. If independent evidence (from the curriculum/ syllabus, learning exercises, examinations, etc.) indicates that a teacher helps students acquire sound knowledge, then it is legitimate to conclude that the teacher's mastery of the subject would have helped students acquire sound knowledge of the subject.

Communicative ability

If independent evidence indicates that a module facilitates a particular learning outcome, and the teacher has a high degree of communicative ability, then it is legitimate to conclude that the learning outcome was enhanced by the communicative ability. But the mere presence of high communicative ability by itself does not entail the accomplishment of any learning outcomes. If the teacher’s communicative ability is poor, of course, chances of his/her being an excellent teacher/educator are not high.

Personal qualities

If a module facilitates a particular learning outcome, and the teacher has desirable personal qualities such as dedication, passion for teaching, approachability, sense of humour, and so on, then it is legitimate to infer that the inculcation of the outcome is enhanced by these qualities. However, in the absence of independent evidence for the learning outcome, the presence of these qualities by itself does not legitimize inferences on the outcome.