Chapter 4
Evaluation of Teachers and Courses for Feedback and Quality Improvement: Formative Evaluation

4.1 Overview

Strictly speaking, formative evaluation is only concerned with feedback to the teacher about their performance or about the courses designed and taught by them. It is the teacher who considers information received from various sources—usually the opinions of peers and students. In short, the teacher is both the subject of the evaluation and the evaluator.

Evaluations of performance or of courses which are conducted at the request of third parties such as heads of department or the results of which are seen by third parties such as departmental reviewers and promotion committees should follow the rules set out for summative evaluation (as described in Chapter 5) even if the primary purpose is improvement. The reason for this restriction is that one of the main purposes of formative evaluation is to uncover/highlight areas for improvement rather than to give the balanced picture required by third parties responsible for administrative and personnel decision-making. Thus the results of summative evaluations may be used for feedback and quality improvement but the results of formative evaluations must not be used for administrative or personnel decision-making.

In one sense, there are no rules for formative evaluations apart from those relating to good ethical and methodological practice. The teacher wants certain information to help improve teaching and will employ appropriate methods to obtain that information. Choice of method and choice of information source will be largely dictated by the questions asked. For example, if a teacher wishes to find out whether a course reading-list is up to date, the most appropriate method may be to request a literature search from the Subject Librarian. More broadly, the teacher may wish to get feedback on their performance as a lecturer. Obviously, students will be a prime source of information but peer observation of lectures may also be useful. Videotaping a lecture and watching it later will certainly be salutary. Where resources are limited the use of check-lists of good teaching such as those produced by Brown and Race (1995) and Gibbs et al. (1989) can be very efficient and effective tools for personal reflection and development.

Timing of data collection and analysis for formative purposes also depends on the questions asked. The teacher may have tried an innovation early in the course but is unsure of its effectiveness. In this instance, early evaluation is essential in case remedial activity needs to be undertaken. Generally however, early evaluation of standard teaching methods such as lectures, tutorials or practical sessions should be avoided as students need time to reach an informed opinion.

The following section provides suggestions for good practice in some common forms of formative evaluation.

4.2 Good Practice

4.2.1 Peer Feedback

Most often peer feedback is seen as classroom observation but peers can provide feedback on other teaching roles particularly in areas such as course design where students are rarely in a position to provide informed opinion. The use of peer feedback can be a powerful mechanism for developing a departmental ethos which values and promotes dialogue about teaching through which colleagues provide mutual support and learn from each others’ successes and failures. See Table 4.1 .

Teaching Role

Examples

Appropriate Peer

Course design

Appropriateness of learning outcomes, content choice and currency, teaching and assessment methods, match between all of the above.

Departmental colleague, Discipline colleague*, Educational development professional, E-learning professional.

Course administration

Course Handbook comprehensiveness and currency, awareness of and adherence to University policies and regulations governing teaching—e.g., assessment, equal opportunity, health and safety, course organisation.

Departmental colleague—both academic and administrative.

Learning activities

Structure of sessions, provision of useful examples, degree of active learning encouraged, clarity of presentation, use of AV/IT resources and other learning materials.

Departmental colleague, educational/staff development professional, E-learning development professional.

Assessment

Appropriateness of tasks/questions set, appropriate level of difficulty, appropriate and consistent marking standards, clarity and appropriateness of grading criteria.

Departmental or Discipline colleague.

Table 4.1: Examples of Appropriate Peer Feedback

4.2.2 Feedback is More Effective When…

It is advisable for teacher and peer to agree a set of ground rules before proceeding, one of which should be that all information considered by the peer is confidential to and remains the property of the teacher. In addition, the peer needs to know what information the teacher is seeking. The following list, though not generalisable to all teaching situations is offered as a guide to giving and receiving effective feedback (Brinko1993):

4.2.3 Peer Observation of Teaching

Having an observer in the classroom can significantly change the dynamics of the environment to the point where valid feedback becomes difficult. What can be done to minimise the so-called ‘observer effect’?

It is always necessary for a peer to be informed about the information the teacher is seeking. In the context of class observations, this is best done by providing the peer with a checklist to be completed during the class. A number of examples of such lists can be found in the literature (e.g., Brown et al. 1993) and one is provided in Appendix E . The items in the lists may be changed at will according to the needs of the teacher but care should be taken that the lists are neither too long (difficult for the observer) nor too short (inadequate for feedback). The checklist may focus on certain strategies (getting discussion going in a tutorial) or on certain sections of the class—e.g., the opening phase.

Alternatively, the observer can be asked simply to provide a chronological record of the class for later discussion.

As with all feedback, that from classroom observation should provide information rather than value judgments. Thus, ‘you scratched your nose 15 times during the lecture’ is better than ‘you shouldn’t scratch your nose so often’. Remember that, in formative evaluation, it is the teacher, not the observer who is the evaluator.

One useful tool is the video camera. Giving feedback is much more effective when the observer can point out behaviours as seen from the camera’s (i.e., the students’) perspectives.

Whatever the method used, both teachers and observers should be careful not to confuse effectiveness with style. There is no one ‘best’ personal style where teaching is concerned—one teacher may wander about, another may remain stationary behind the lectern. One teacher may be gruff while another may keep the class laughing. All may be equally effective in assisting their students to learn.

4.2.4 Student Feedback

Any comprehensive formative evaluation of teaching should include feedback from students. Their contribution is legitimate and essential but needs to be considered in conjunction with other sources for a valid evaluation to be made. Perhaps to labour the point, ‘student evaluations’ of teaching are no such thing. Students opinions provide one, but only one, perspective and others are needed. Student opinions are legitimate because they are the learners who are the sole purpose of teaching. They are essential because for certain areas of teaching, they can be the only and/or best source of information.

In general, students can provide useful information on the following matters:

The best known method of obtaining student opinion is the questionnaire but others have been well tested. The two most common are focus groups and classroom assessment (an American term which will be somewhat misleading for Irish readers).

Questionnaires

Like all methods used in evaluation, questionnaires have their advantages and disadvantages. On the positive side, they can be both valid and reliable and provide useful and objective evidence for evaluation. They are—or should be—anonymous so that students can express their opinions freely. On the other hand, the set of questions is restrictive and the format (except for open-ended questions) limits the nature of the student response. Of course, formative evaluation, by definition, seeks answers to certain questions defined by the needs of the teacher but there is always the possibility that students have useful things to say on a matter but are unable to do so because they haven’t been asked the ‘right’ question. The references in Appendix A provide many examples of useful questions. The validity and reliability of questionnaires depends largely on their design, a matter which will also be discussed further in Appendix A but a few points may be usefully listed here:

Focus groups

Focus groups may take a number of forms but essentially they are established to enable structured discussions about courses and teaching. They overcome some of the disadvantages of questionnaires by permitting students to comment on matters which are of major interest to them and, in addition, students are able to explain the reasons for their opinions in a way impossible with questionnaires. In addition, focus groups may be used to explore issues which have been uncovered using other means.

On the other hand, focus groups are not anonymous and some students may feel inhibited in expressing their opinions freely and they will consist of only a small proportion of the class and hence may be unrepresentative. Finally, teachers may be inexperienced in leading structured discussions and may find it difficult to elicit useful responses from the students.

Maximising the advantages of focus groups and minimising the disadvantages requires some care. The following points may be useful:

Classroom assessment

Through close observation of students in the process of learning, the collection of frequent feedback on students’ learning, and the design of modest classroom experiments, classroom teachers can learn much about how students learn, and, more specifically, how students respond to particular teaching approaches. Classroom Assessment helps individual college teachers obtain feedback on what, how much, and how well their students are learning. Faculty can then use this information to refocus their teaching to help students make their learning more efficient and more effective.

Angelo and Cross (1993)

This approach to formative evaluation is relatively common in the United States and very rare in this country. But it makes a lot of sense. We have defined teaching as the creation and sustaining of an environment which promotes effective learning. Therefore, if we wish to know whether our teaching is effective, then let us discover whether the learning has been effective. In an assessment system dominated by the final examination, realisation that learning in some areas has not been effective is rather too late for remedial action with that particular cohort of students. And examinations cannot cover the whole syllabus. The remedy is to continually assess learning during the course taking immediate remedial action where necessary. Angelo and Cross (1993) list a number of advantages to such continuous assessment:

One point of clarification. ‘Continuous assessment’ in the Irish context usually means that students’ grades depend, not only on the results of a final examination but also on those of a number of projects of various kinds. Readers (and students) may well baulk at the prospect of radically increasing the number of the latter to be completed and to be marked.

Angelo and Cross (1993), however, mean something very different. Their book contains an extraordinary number of classroom assessment techniques and case studies spread over a range of disciplines and it is impossible in this manual to attempt to summarize them. They do recommend that teachers should start by using the simplest techniques possible. The following examples will illustrate their point.

Note that the amount of time involved in preparation and collection of information is minimal, although analysis may take longer. Angelo and Cross (1993) provide many more sophisticated examples of the technique together with pros, cons and caveats.

4.3 Summary

Issue Formative Evaluation
Primary Purpose Provides feedback. Suggests improvements.
Timing During course.
Evaluators Internal to course.
Focus Processes.
Confidentiality Restricted to teacher.
Processes Formal or informal.

4.4 Further Reading

Notes

*Where the teacher is the only departmental expert in a particular area of scholarship it may be necessary to seek peer feedback from a colleague in another institution.