With the introduction of 'commercialisation' in higher education, we are often left baffled by the myriad of processes and feedback exercises the university solicits from students. Increasingly, universities around the world recognise that students are our customers and they are free to chose where they want to study. In the same manner as collating market surveys, these institutions assiduously collect feedback from its students with the aim to improve its 'service' as well as to serve as additional 'customer satisfaction' indicators for potential students and stakeholders. In NUS, quantitative student feedback data is also often used to rank faculty according to their 'teaching ability', and such rankings are often the main indicator used to shortlist faculty for both Faculty and University Teaching Excellence awards. Moreover, student feedback scores often have a direct or indirect impact on the faculty member's annual performance bonus, and may influence his/her chances of tenure and promotion, especially if the scores are particularly low.

However, "it is not always clear how these views collected from the students fit into institutional quality improvement policies and processes. To be effective in quality improvement, data collected from surveys and peer reviews must be transformed into information that can be used within an institution to effect change" (Harvey, 2001: 2). But before this can be worked into the system effectively, we need to know what the student feedback system can tell us.

Information from student feedback should help create a profile of each lecturer being evaluated. At a FTEC (Faculty Teaching Excellence Committee) briefing (May 20, 2005) organised by the Centre for Development of Teaching and Learning, Professor K P Mohanan described the two typical profiles of 'good' teachers:

• Profile A: "Good at lecturing, clear presentations and explanation, passion for subject, interesting, dedicated, approachable, organised/systematic, knowledgeable";
  
  
• Profile B: "Encourages students to think, clear presentation and explanation, does not spoon-feed, knowledgeable, encourages independent learning, asks probing questions, challenging".

If we believe that education involves higher order learning outcomes that distinguish the quality of the mind of a well-educated person from those of a well-trained person, then we should be aiming to identify lecturers with profile B as the best ones. Our current student feedback gives us both quantitative and qualitative information. Quantitative data are the feedback scores and percentage of nominations for teaching awards. Qualitative data, which are often overlooked, may be derived from students' written comments. We can identify repeated comments versus sporadic ones to identify the majority views of students. From such repeated comments, the lecturer's teaching profile should emerge, which would complement his/her qualitative scores.

First, let us address the quantitative student feedback scores. Some key questions that we need to ask to correctly interpret the quantitative scores include:

• Is class size important?
  
  
• Do larger classes get fewer nominations for teaching awards?
  
  
• Is there a correlation between the response rate and the class size?
  
  
• Do poor scoring modules have lower response rates?
  
  
• Are feedback scores higher for more advanced modules?

The following graphs and interpretations are based on the National University of Singapore's Faculty of Science student feedback results for all modules taught in semester 1 of AY 2004/2005, using the following standard feedback questions:

1. The teacher has enhanced my thinking ability.
   
   
2. The teacher provides timely and useful feedback.
   
   
3. The teacher is approachable for consultation.
   
   
4. The teacher has helped me advance my research (if applicable).
   
   
5. The teacher has increased my interest in the subject.
   
   
6. The teacher is able to demonstrate cross-disciplinary relationships in relevant topics and has taught us to draw interconnections between different areas in Science.
   
   
7. The teacher is able to illustrate some actual or potential applications of knowledge covered in the syllabus.
   
   
8. Overall the teacher is effective.
   
   
9. Average of Question 1-7.

Is Class Size Important?

As we can see from the plots in Figure 1, it is more difficult to obtain higher feedback scores for larger classes. This is despite the fact that good teachers are often assigned to teach these large classes, which are often general education (GEM) or first year modules.

Figure 1. Feedback score versus class size.

Do Larger Classes Have Less Percentage of Nominations?

Figure 2. Percentage nominations for teaching awards versus class size.

Figure 2 shows that the percentage nominations for teaching awards is indeed lower for larger classes, consistent with feedback scores in Figure 1. This means that it could be more difficult to obtain a teaching award if you are assigned to teach a large class, even if you have been an excellent teacher consistently.

Hence it is important to ask what incentives we can provide to encourage good teachers to teach large classes.

Is There a Correlation Between the Response Rate and the Class Size?

Figure 3. Student feedback response rates versus class size.

In order to assess whether there is any bias in feedback due to poor response rates in some modules, Figure 3 plots students' response rates against class size. It can be seen that the response rate is consistently high (>80%) if the class size is not too small, in which case the statistics could be strongly affected by individual non-respondents. Hence, we can infer that there is no clear correlation between the response rate and class size. However, it would be interesting to then ask whether poor scoring modules have a lower response rate.

Do Poor Scoring Modules Have Lower Response Rate?

Figure 4. Overall feedback score versus response rate.

Looking at the data scatter in Figure 4, there is no conclusive evidence to conclude that poor scoring modules have lower response rates. In fact, there is a cluster of low scoring modules (circled in Figure 4) which have received very enthusiastic student responses, suggesting that many students are willing to provide negative feedback if they feel that a module is not well taught.

Are Feedback Scores Higher for More Advanced Modules?

Figure 5. Feedback scores plotted according to module level.

There is a common perception that it is easier to achieve higher student feedback scores when teaching higher level modules. Figure 5 shows the overall feedback scores plotted according to module levels (1 to 6). It is clear that the minimum feedback scores improve significantly for higher level modules, while the maximum remains approximately constant (the average score increases with level). This could in part be attributed to the smaller class sizes at higher module levels. Nevertheless, this result reinforces the notion that teaching higher level modules is preferable, and teaching evaluation committees should exercise some degree of score normalisation when comparing scores of modules taught at different levels.

The major trends observed so far can be summarised as follows:

• Implications of class size:
  
  - It is harder to achieve higher scores for large classes;
  
  - Percentage nominations for teaching awards is lower for larger classes.
  
  
• The response rate is not a clear indicator of feedback scores.
  
  
• Higher level modules do not suffer from low feedback scores.

Due to space constraints, further details of this quantitative study cannot be presented here. However, we have found that teachers who promote higher order learning outcomes do not necessarily get nominated for teaching awards. This can be seen, for example, from the results shown in Figure 6, which shows the score on question 1 ("The teacher has enhanced my thinking ability") versus percentage nomination.

This somewhat surprising result which shows no correlation indicates the limitations of using quantitative student feedback in identifying excellent teachers, in particular the lecturer with profile B who "encourages students to think, gives clear presentations and explanations, does not spoon-feed, is knowledgeable, encourages independent learning, asks probing questions, and is challenging". There could be many factors influencing the scores of such questions but these will not be dealt with here. Nevertheless, it is clear that there is a need to probe specific qualitative student feedback comments to build more accurate lecturer profiles.

Figure 6. Student feedback scores on Question 1 ("The teacher has enhaced my thinking ability") versus percentage nomination for teaching awards.

Creating a Profile Using Qualitative Repeated Student Comments

To give specific case studies, we have collated selected repeated comments from the feedback of Lecturer X that reflect higher order learning outcomes:

• "He encourages us to have open discussion during lectures."
  
  
• ". encourages students to think for themselves ."
  
  
• "Made us think and do things on our own."
  
  
• "He stimulates his students' thinking."
  
  
• ". made me want to understand the concepts even before I came to lecture."
  
  
• ". stimulated students' thinking when attending to students' questions because he liked to get students think over their question ."
  
  
• "It is very refreshing to be expected to learn for yourself and not be spoon fed all the time."
  
  
• "His mode of education is really different. He makes the students think for themselves and cultivates independence during the process."
  
  
• "He taught students how to use their brain and think, and not spoon feeding and focus on memory work! (sic)"
  
  
• "He motivates me to learn, to challenge myself and work out problems on my own."

Similarly, the following are repeated comments for Lecturer Y from the same department:

• "explains very clearly. very cheerful"
  
  
• "ability to go in depth of the topic. have lots of knowledge/ experience for that subject (sic)"
  
  
• "talks patiently and slowly for students to most of the concepts (sic)"
  
  
• "notes are detailed and clear delivery of lecture"
  
  
• "He is humorous."
  
  
• "Approachable for consultation and willing to answer questions beyond the course materials as well."
  
  
• "Always punctual for lecture."
  
  
• "He is clear, and does describe the processes well. Uses good examples."
  
  
• "lecturer can explain the module content to student very clearly."
  

It is useful to note that both lecturers have similar and reasonably good feedback scores, though both would not normally qualify for teaching awards based on feedback scores alone. However, it is evident that Lecturer X's teaching promotes higher order learning outcomes, making him a better educator and a good candidate for a teaching award.

In conclusion, this simple study shows that whilst quantitative student feedback has its uses, it may fail to give an accurate profile of the lecturer concerned. One effective way of doing this is to examine student feedback more carefully and to compile repeated, consistent comments to build up a more accurate profile of the lecturer. The Faculty of Science FTEC has indeed started to do this from AY2005/06 in its evaluation of lecturers for teaching performance and awards. Peer review and module folders also provide additional inputs, but various problems in peer reviewing have been noted and will be addressed in the new peer review system to be implemented in AY2006/07².

Endnotes

¹ This article is based on the talk given by Professor Wee at the Faculty of Science Teaching Workshop, July 2005.

Back to article

² This new Peer Review System will be presented at the Faculty of Science Teaching Workshop, August 2006.

Back to article

References

Harvey, L. (2001). 'Student Feedback: a report to the Higher Education Funding Council for England'. Centre for Research into Quality. http://www/uce.ac.uk/crq/publications/student-feedback.pdf. Last accessed: 6 July, 2006.