Chapter 3
Evaluating Teaching at the Faculty and/or Department Level

3.1 Overview

The Irish Universities Act, 1997 requires each university to ‘establish procedures for quality assurance aimed at improving the quality of education and related services provided by the university’ (Sec 35.1).

While not precluding evaluation of teaching at the institutional level or that of individual academic staff, the Act does require regular evaluation of Departmental (and where relevant Faculty) activities. Further, such evaluation will include ‘assessment’ by stakeholders including students. Generally, however, this section of the Act is remarkably non-prescriptive.

All universities have met the minimum requirements of the Act by the implementation of regular departmental reviews. These reviews evaluate research and administration as well as teaching. Generally, the procedures listed below are followed:

The review process is both formative and summative—formative in that the department receives feedback on its operations which should lead to improvements and summative in the sense that judgements are made about performance which may lead to administrative decisions.

In principle, departmental reviews as described meet a number of the objections to other forms of audit such as those in place in the United Kingdom. Responsibility for the process lies with the universities rather than some kind of government inspectorate. Detailed procedures reflect institutional culture and strategic aims and the emphasis is on quality improvement.

So is teaching in the higher education sector satisfactory and is it improving? Are our quality assurance and improvement programmes working? Certainly not as of February 2003, according to the then Minister for Education and Science:

On the issue of quality, I must admit that I have a concern that insufficient attention is being paid to the quality of teaching at third level. It seems to me that there are too many instances where lecturers are standing up in front of a class without ever having shown an aptitude for teaching. It’s an issue that requires attention and, much more than that, it requires action.

Extract from Minister’s speech at the official opening of a new fitness suite at the Institute of Technology, Athlone on 10th February, 2003.

This statement does not constitute prima facie evidence of poor teaching or poor quality assurance systems but the fact that the Minister perceived poor quality is highly significant. Thus we find that the terms of reference for the subsequent OECD Review of Higher Education in Ireland included an examination inter alia, of ‘How institutions in the higher education sector might best respond to the needs of their students through the use of appropriate systems of quality assurance to support the highest quality of teaching and learning…’ (DES2003) It would seem that there is some danger that new systems may be established which lack the advantages of the current one and which share some of the disadvantages of those in the UK such as inflexibility and excessive detail.

Universities may, however, be able to forestall inappropriate intervention by government by broadening their current focus/practices to include:

3.2 Good Practice

All universities provide departments undergoing review with a guide to self-assessment. These guides vary considerably, which is not a problem in itself. At one end of the spectrum, some universities require departments to provide massive amounts of information. At the other end, one or two guides largely fail to address certain quality issues for which the department has prime responsibility (for example, the choice of assessment methods used).

Few guides demonstrate a simple and coherent framework, although University College Dublin’s Guidelines for Self-assessment, Review, Follow-up comes close. There, the process:

… comes down to answering four questions:

The discussion below will follow this structure, which, however, is not intended to be prescriptive. The points made under each heading, however, remain valid whatever the structure of the self-assessment document.

3.2.1 What Are We Trying To Do?

There is considerable research evidence to suggest that one of the key features of good teaching is providing clear goals and intellectual challenge (Ramsden1992). Thus, a primary feature of all departmental self-assessment documents relating to teaching should include statements of desired learning objectives or outcomes both at the programme (degree) and course (module or unit) levels.

If departments are not clear about what it is their graduates should know and what skills they should have, then there can be no rational basis for choice of content, teaching, learning methods and assessment. If learning outcomes are not specified for each course, students may have difficulty in determining what is important to learn

Further, under such circumstances, external reviewers may be left with little choice but to use their own programmes as a benchmark for evaluation on the assumption that the programmes under review and their own share similar goals. Where such assumptions are incorrect, comments by external reviewers regarding teaching methods, content and assessment will lack validity.

In other words, ‘what we are trying to do’ in teaching is to support our students in the achievement of certain learning outcomes. Valid evaluation of a department’s teaching requires that those outcomes be made specific.

Programme outcomes can probably be best expressed as a series of attributes to be achieved by all graduates of the programme while course outcomes should consist of a set of statements beginning with the words, ‘By the end of this course, students should be able to....’ Outcomes at both programme and course levels should include knowledge of discipline content, subject-specific skills, generic skills (e.g. communicate effectively) and appropriate attitudes and values.

3.2.2 How Are We Trying To Do It?

We assist students to achieve the outcomes we have specified by selecting:

Thus, content should be chosen on the basis of the extent to which it assists students to achieve certain outcomes rather than because it reflects the interests of individual academic staff. There is a continuing danger with the latter approach that, reflecting an effort to remain up to date, syllabi become overcrowded as new material is added and little removed. Overcrowded syllabi are one of the major reasons why many students adopt a ‘surface’ approach to learning and seek to memorise material rather than understand it (Ramsden1992, p. 81).

Again, much thought needs to be given to teaching and learning methods. Many academics will maintain that fundamental learning outcomes at both the programme and course levels include the acquisition of skills such as problem solving, critical thinking and evaluation. At the same time, they will adopt the lecture as the primary teaching method despite the results of over five decades of research (Bligh1998, Chapter 1) which indicates that such skills are not necessarily acquired either efficiently or effectively through this medium. Acquisition of skills is best supported by active modes of learning.

Finally, there should be a very close match between assessment and learning outcomes. It is sad but true that most students will only put effort into learning those parts of a course which are to be assessed. A typical mismatch occurs when course learning outcomes stress acquisition of higher level cognitive skills and the assessment consists of tasks which require no more than recall of factual information.

In summary, a departmental self-assessment document should make the case for choice of learning outcomes, content, methods and assessment. They should not simply be listed—leaving it to the review panel to guess the collective intentions.

3.2.3 How Do We Know It Works?

One of the major complaints about the departmental review process is that it can be extremely time consuming in that once every five years or so, a massive amount of data must be collected for the self-assessment document at the expense of the routine tasks of the department. If this is, in fact, happening, then the complaint is justified but it should not be happening. Quality assurance and improvement should be an ongoing activity and regular monitoring of a department’s teaching should be undertaken using a number of performance indicators. If this is done, then annual updating and preparation for a departmental review should not involve much extra work. At the very least, there should be no nasty surprises.

Selecting which performance indicators to use is a matter of some delicacy. One temptation is to use only quantitative indicators but most would agree that not every aspect of good teaching can be actually measured. Again, an excessive number of indicators can impede monitoring while too few makes analysis difficult. The following list is intended to be a guide and university requirements should be flexible in order to allow for differences between departments.

In relation to the departmental review, the department needs to demonstrate that it is monitoring the quality of its performance and should include the indicators in its self-assessment document, particularly noting any trends.

3.2.4 How Do We Change in Order to Improve?

Recording a set of performance indicators may (or may not) satisfy the requirements for quality assurance but departments also need to demonstrate how they use this information to improve the quality of their teaching. One problem is that teaching is often seen as a very private activity and academic staff can be reluctant to admit that improvement is needed. In addition, many of the difficulties faced by individual staff can only be tackled at the departmental level.

For example, an academic may find it very difficult to involve his/her students in active learning strategies. The reason, however, may not be due to his/her poor teaching but to the fact that everyone else in the department is content with lecturing as the primary teaching method and the students are getting mixed messages. This kind of problem can only be resolved by discussion and a coherent approach at the departmental level.

There are several mechanisms through which departments can improve teaching at the collective level.

In relation to departmental reviews, however, the problem of individual poor performance is not a major issue because what is being evaluated is the collective teaching of the department. What is essential is that the self-assessment document demonstrates the quality improvement measures taken.

3.3 Conclusion

Current departmental reviews do meet the minimum requirements of the Universities Act but could be improved along the lines suggested above. One matter remains to be considered—how are review recommendations to be implemented? The issue is one of resources. In particular, external reviewers from the UK frequently note the high quality of the work produced in departments despite the relatively low resource base. They then proceed to recommend that the appointment of more academic, administrative and technical staff as a means to further quality improvement. Unfortunately, in the contracting Irish economy, such recommendations are rarely acted on which can be extremely demoralising.

On the other hand, significant improvements to teaching can be achieved with few additional financial or staffing resources, particularly in the areas of curriculum reform, teaching methods and assessment.

It must be recognised, however, that such improvements do come at a price—the price of time.

Finally, Appendix D provides a checklist for evaluating Teaching at the Departmental Level.

3.4 Further Reading

Notes

1The Department will need to show that student feedback for formative purposes has been sought between reviews (see Chapter 4 ). In the year of the review, however, summative data should be collected (see Chapter 5 ) and included in the self-assessment document.