CDTL    Publications     Mailing List     About Brief

 

   
This issue of CDTL Brief is the first of a two-part Brief that features the teaching practices of the 2005/2006 Annual Teaching Excellence Award (ATEA) winners.
August 2007, Vol. 10 No. 3 Print Ready ArticlePrint-Ready
The Empirics of Teaching Quality
 
Associate Professor Willie Tan
Vice Dean (Academic)
School of Design and Environment
 

Introduction

In 2005, I was asked by a persuasive colleague at CDTL to do a 'simple' preliminary study on whether class size (n) affects teaching student feedback scores (F). Intuitively, the answer is yes. I have taught both large and small classes for many years, and know for sure there is a difference. But the issue was how much of a difference, which is an empirical rather than a logical question.

Naively, one could model the relation as:

F = α + ßn + η;      ß < 0             (1)

or more generally, F = ƒ(n) + η where ƒ(.) is a nonlinear function, α is the intercept and ß is a parameter. Since only one independent variable is involved, the error term η will be too large, needs to be extended to include three vectors of characteristics that is:

y = ƒ(e, t, s) + η             (2)

where y is student performance (not to be conflated with quality of student learning), e is a vector of environmental or structural variables (e.g. class size, type of model, level and field of study), t is a vector of teacher characteristics (e.g. age, gender and qualifications) that may possibly reflect teaching quality and s is a vector of student characteristics (i.e. academic ability and socio-economic factors). As before, ƒ(.) may be a nonlinear function to improve the fit and exploratory data analysis suggests that ƒ(.) is nonlinear.

In this paper, I shall elaborate on how this model measures teaching quality. However, I shall sidestep the discussion on complex technical issues so that non-technical readers may hopefully obtain an intuitive grasp of the problem.

Design

Technically, nothing in equation (2) compels us to adopt a particular research design. However, the large number of variables rules out an experiment designed to control a subset of variables. In other words, η will be large, and attempts to control variables by matching or otherwise still leaves a large noise.

Hence, a regression model with important modifications was adopted. First, the dependent variable, student performance, was replaced by student feedback scores (F) because there was no systematic data on student performance or the quality of student learning attributable to a teacher. A better way to find out whether there were any value-added inputs from a teacher is to employ a before-and-after measure using the standard scores of student performance. This proxy raises a number of important issues of which three are hotly debated partly because of unintended consequences:

  • F may measure populism or 'personalism' rather than teaching performance because students tend not to or are unable to evaluate beyond teaching style. For instance, there are doubts that students can adequately assess the quality of teaching materials and assessment methods (e.g. quality of examination papers)

  • teaching quality may shift (dilute) towards the 'median voter' to reduce the gap (i.e. welfare loss) between students' expectations and actual teaching quality. For example, if there are only two ice-cream sellers on the beach, both will set up near the centre to maximise profit. The same reasoning applies to political strategies. Left and right wing political parties tend to support 'middle way' policies to maximise votes. The choice is not between the state and market, but among what other political parties offer and

  • "measurement by fiat" (Cicourel, 1964, p. 33) where even F (as measured by student feedback forms) is mismeasured, resulting in low correspondence between concept and measures.

Second, mismeasurement of the dependent variable is only part of the story. Equally serious is the absence of data on student characteristics (s) so that the model becomes:

F = g(e, t) + h(s, η)             (3)

where the unknown function h(.) is the new noise term comprising random disturbances and a vector of student characteristics. Note that s and η are unobservable. In some studies, e is presumably 'held constant' by conducting intrainstitution research to reduce the complexity in equation (3), but this approach is not persuasive at the university level because of large variations in environmental variables (e.g. class size).

Finally, teaching quality as measured by teacher characteristics (t) is likely to correlate with student characteristics (s) and the ordinary least squares estimator is inconsistent (biased) even for large samples. Better teachers tend to attract better students-this is what parents already know and why parents send their kids to good schools. At the university level, there is reason to believe that students look to seniors for information before opting for their modules.

Empirics

Based on equation (3) above and a sample of 1983 data points, I have found that three basic determinants of F are worth noting, namely:

  • class size
  • field of study and
  • type of teaching activity.

This regression result confirms our intuitive judgment that class size matters (see also Ehrenberg; Brewer; Gamoran & Willms, 2001), F differs across faculties (i.e. disciplines), and the type of teaching activity (e.g. lecture or sectional teaching) has experiential effects.

Some variables not found to be important included:

  • the type of module [whether a module is a General Education Module (GEM)]
  • module level (1-6)
  • teacher's gender and
  • number of years of teaching.

The finding on the number of years is consistent with the research by Goldhaber and Brewer (1995), and Rivkin, Hanushek and Kain (2005) who said that certification and number of years of teaching do not matter beyond the initial years. Perhaps one is jaded by teaching the same thing year after year, and is no longer young and energetic or fails to bridge the generation gap. To be sure, this is just a tendency.

References

Cicourel, A.V. (1964). Method and Measurement in Sociology. New York: The Free Press.

Ehrenberg, R.G.; Brewer, D.J.; Gamoran, A. & Willms, J.D. (2001). 'Class Size and Student Achievement'. Psychological Science in the Public Interest, Vol. 2, No. 1, pp.1-30.

Goldhaber, D. & Brewer, D.J. (1999). Teacher Licensing and Student Achievement. In M. Kanstoroom & C.E. Finn, Jr. (Eds.), Better Teachers, Better Schools. Washington, DC: The Thomas B. Fordham Foundation, pp. 83-102.

Rivkin, S.G., Hanushek, E.A. & Kain, J.F. (2005). 'Teachers, Schools, and Academic Achievement'. Econometrica, Vol. 73, No. 2, pp. 417-458.

Acknowledgements

I thank Professor KP Mohanan for useful comments on an earlier draft.

 
 
 First Look articles





Search in
Email the Editor
Inside this issue
Talking the Talk and Walking the Walk: Teaching History at NUS
   
Teaching: Share Your Passion and Have Fun
   
Taking Charge of Learning— Ownership, Learning and a Conducive Environment
   
Can Computer-aided Instruction Effectively Replace Cadaverbased Learning in the Study of Human Anatomy?
   
A Perspective on Medical Education
   
Excuse Me, Are You an Excellent Teacher?
   
The Empirics of Teaching Quality