Chapter 5
Evaluation of Teachers and Courses for Judgment and Quality Assurance: Summative Evaluation

5.1 Overview

Although this chapter deals primarily with the summative evaluation of individual teachers and individual courses, the issues raised apply also to the summative evaluation of teaching at the departmental level.

One key difference between formative and summative evaluation is that the latter may lead to significant administrative and/or personnel decisions made by third parties. Formative evaluation may indeed lead to decisions about how a course may be improved but summative decisions are of another order and may lead, for example, to:

These are weighty matters which directly affect the lives of staff and indirectly the lives of other stakeholders in the education process. In conducting the evaluations on which such decisions are based, it is crucial that justice be done and be seen to be done.

While formative evaluations can be reasonably relaxed affairs (the teacher is simply seeking a range of opinions for their personal use), the planning and administration of summative evaluations must be much more rigorous. Thus, although summative evaluations may be used for formative purposes (i.e., they can provide feedback for improvement), formative evaluations should never be used summatively. Failure by senior academics and administrators to take account of this distinction can lead to quite warranted opposition to the very idea of summative evaluation.

The section below lists and explicates principles of good practice. The principles apply whether the evaluation is of a teacher or of a course.

5.2 Good Practice

5.2.1 The Evaluation Should be Comprehensive

Formative evaluations can be quite limited if the teacher only wants feedback about one or two aspects of their teaching/course. But summative evaluations should study all relevant teaching roles and/or dimensions of a course in order that comprehensive judgments can be made. Table 5.1 lists the minimum number of dimensions which should be considered.

Summative Evaluation of Teachers: Roles

Summative Evaluation of Courses: Dimensions

Course design: learning outcomes, content, teaching and learning methods, assessment methods*

Course design: learning outcomes, content, teaching and learning methods, assessment methods

Course management: course information, administration

Course management: course information, administration

Student nurture: availability, helpfulness

Student nurture: availability, helpfulness

Classroom activities: lectures, tutorials, laboratories etc.

Resources: textbooks, reading lists, teaching materials, equipment, library, classrooms

Assessment: quality and quantity of feedback

Classroom activities: lectures, tutorials, laboratories etc.

Extracurricular activities: nature and extent, professional development, scholarship in teaching etc.

Assessment: quality and quantity of feedback

Table 5.1: The minimum number of dimensions which should be considered in summative evaluation

It will be noted that the two lists are almost identical. It is, however, important to distinguish between evaluations of teachers and evaluations of courses (although it is possible to combine the two, with a great deal of care being needed when more than one teacher is involved in the course).

5.2.2 The Evaluation Must be Multi-faceted

Summative evaluation should use multiple sources of evidence to ensure conclusions are valid. Different stakeholders have different perspectives, as many as possible of which should be taken into account.

5.2.3 The Evaluation Must Only Use Appropriate Sources of Evidence

All stakeholders may have a perspective on all aspects of teaching, but not all perspectives are valid for summative evaluation. Thus, for example, students should not be asked for their opinions about the appropriateness or otherwise of course content—their knowledge and experience will be insufficient for them to take an informed view. They are however best placed to provide valid opinions on the helpfulness of assignment feedback.

Table 5.2 matches teaching roles/course dimensions, sources of evidence and types of evidence. The following sections comment in more detail on some of these sources and types of evidence.

Roles/Dimensions

Appropriate Sources of Evidence

Types of Evidence

Course Design

Teacher (see 5.2.3.1 )

Profile, Self-report
(see 5.2.3.2 )

Discipline Peers

Report (see 5.2.3.3 )

Academic Developers

Report

Course Management

Teacher

Profile, Self-report

Peers

Report

Head of Department

Report

Students

Questionnaire
(see 5.2.3.4 )

Student Nurture

Teacher

Profile, Self-report

Students

Questionnaire

Graduates

Questionnaire

Classroom Activities
(see 5.2.3.5)

Teacher

Profile, Self-report

Students

Questionnaire

Graduates

Questionnaire

Resources

Teacher

Profile, Self-report

Discipline Peers

Report

Assessment (feedback etc.)

Teacher

Profile, Self-report

Students

Questionnaire

Graduates

Questionnaire

Extracurricular Activities

Teacher

Profile, Self-report

Discipline Peers

Report

Table 5.2: Teaching Roles, Sources and Types of Evidence
5.2.3.1 Teacher

At some stage during the evaluation process, the teacher being evaluated must be given the opportunity to comment on each teaching role and/or course dimension. His/her opinions are a necessary but not sufficient source of evidence for the evaluators.

5.2.3.2 Profile—Self Report

We have used the term ‘profile’ rather than the better known ‘teaching portfolio’ because the latter has two commonly used meanings which are frequently confused. The first type of portfolio is a long term reflective and scholarly document intended primarily as a means of self-development. Lyons et al. (2003) provide excellent advice about how to construct such a portfolio and present a number of illustrative case studies. This type of portfolio, however, is not normally used for summative purposes and should not be used as the primary evidence for, say, promotion, if only for the practical reason that most are too long to be read by busy evaluators. They have, however, been used successfully as evidence for teaching awards where the number of candidates is small.

The second type of teaching portfolio (sometimes called a teaching profile) is a much shorter document specifically designed for use in summative evaluations in that there is an emphasis on evidence which demonstrates quality teaching. Such a portfolio might include:

  1. An outline of teaching responsibilities and activities over a defined period.
  2. A statement of teaching philosophy and approaches and goals.
  3. Sample course materials and where relevant examples of innovations in teaching.
  4. Samples of students’ work (projects, essays, laboratory reports etc.).
  5. Relevant statistics such as pass rates, staff-student ratios.
  6. Evidence of scholarship in teaching.
  7. Rewards/recognition/honours received

An alternative to the portfolio frequently employed by evaluators in the promotions process is the use of a detailed application form which requires the candidate to provide the information listed above. We have named this process ‘self report’. An example is provided in Appendix F .

5.2.3.3 Report

There are frequent complaints that the traditional referee reports are ‘useless’ in that they do not provide the information needed by the evaluator. The response to this problem is quite simple—use structured report forms which ask the questions the evaluators want answers to. An example is provided in Appendix G.

It is, however, essential that teachers provide the peer reviewers with all the materials they will need to complete the report.

5.2.3.4 Questionnaires

To obtain the degree of validity and breadth of opinion required for summative evaluation, questionnaires rather than focus groups or other methods should be used to obtain information from students and graduates. It may, in practice, be impossible to obtain representative information from graduates because of an inadequate alumni data base.

Particularly useful example question banks may be found in Webb (1994) and in the Queens’ University of Canada Inventory of Possible Items to be Selected by Course Instructors for Surveys of Student Assessment of Teaching.

5.2.3.5 Classroom Activities

While peer observation of classroom activities can provide the teacher with useful feedback, such observation should not be used for summative purposes. There are a number of reasons for this prohibition:

5.2.4 Staff and Students Should Share in the Design of Summative Systems and Instruments

A summative evaluation system can be threatening to the extent that major stakeholders withhold full cooperation. This effect may be reduced if they are given a hand in the design of the system and the instruments used. There is a further benefit, namely that the system is less likely to be flawed if those who are to use it have been involved in its planning.

5.2.5 Where Possible, Standard Instruments Should be Used

This principle is of particular relevance where people are being compared against each other or rated against a set of criteria e.g., evaluation for academic promotion. The student questionnaire is an example in point. It is feasible to use a standard set of questions relating to lecturing across the university but it is however necessary to make allowances for faculty-specific teaching methods relating to, for example, clinical teaching, laboratory teaching and fieldwork.

5.2.6 Great Care Needs to be Taken in the Management of Summative Evaluation

In the interests of validity and reliability, the administration of questionnaires and peer reports and the compilation of data from such instruments should be managed by a neutral third party such as a Quality Office. Particular care should be taken that:

5.3 Summary

[Picture]


Figure 5.1: Evaluating Teaching for Promotion

Issue

Summative Evaluation

Primary purpose

Judgments of performance leading to administrative and/or personnel decisions

Timing

Retrospective

Evaluators

External to programme

Confidentiality

Need to know basis

Process

Formal

5.4 Further Reading

Notes

*If the teacher is not responsible for course design, evaluation instruments must not probe these areas in relation to that teacher.

Similarly, if the teacher is not responsible for course management, evaluation instruments must not probe these areas in relation to that teacher.

If more than one teacher is associated with the course, the evaluation instruments must distinguish between them.