The context in which this new form of assessment was introduced is a Bachelor of Science (BSc) in the Information Technology (IT) programme presented by Oscail—the National Distance Education Centre in Dublin City University (DCU). This is a modular programme in which students have to successfully complete 14 modules (in subject areas as diverse as Communications Technology, Computing, Management and Human Sciences) in order to qualify for a degree.
Initially, the programme was presented by traditional distance education methods, i.e., specially written self-learning module texts to convey content with tutorial support provided at a range of study centres around Ireland. Traditionally, assessment on each module was a mixture of continuous assessment and examination with, for most modules, continuous assessment accounting for 50% of each module’s mark and the examination accounting for the remaining 50%. The continuous assessment usually consisted of three written assignments per module, which students submitted (by post) to their tutors.
Over the last four years, (i.e., since 2002) the programme has been converted into an online programme with the majority of tutorial support now being provided online. The move to converting the IT programme to an online programme allowed the Course Team (Programme Board) to explore a number of avenues aimed at improving teaching and learning on the programme using a variety of online pedagogical techniques and concomitant alternative forms of assessment. The Course Team for the IT programme has responsibility for the academic direction of the programme and has members drawn from a number of universities and Institutes of Technology around Ireland as well as members from industry and the public service.
A primary focus of the efforts to improve student learning and assessment was the Human Science A (HSA) module. HSA is a module on the Cultures of Technology and is one of the four compulsory degree level modules. Other new online pedagogical techniques were implemented in other modules of the programme. However, the techniques used in the HSA module and the allied new forms of assessment are the subject of this case study.
The move to alternative forms of assessment on the HSA module was aimed primarily at improving student online engagement. Given the discursive nature of the content of the HSA module, improving the online engagement of students was seen as critical to improved student learning. In other words, in line with a number of social constructivist learning theories, it was considered that the key to improving student learning was to encourage students to engage online in in-depth discussion of the content of the HSA module. A number of attempts had been made to encourage student online engagement in various modules while keeping the old forms of assessment. However, with some notable exceptions, these attempts were not successful in maintaining sustained online engagement. The techniques introduced in the HSA module made online engagement an integral part of the summative assessment of the module.
In addition, for reasons related to encouraging engagement but also to attain other learning outcomes, the Course Team wanted to introduce elements of group work into the IT programme. Before the programme went online, logistical problems caused by the geographical dispersion of the students had prevented the incorporation of group work methods into the IT programme. Therefore, group work was also incorporated into the pedagogical techniques used in the online teaching of the HSA module.
The students on the BSc in IT programme are practically all ‘second chance’ students, ranging in age from 19 to 62 with a median age in the early thirties. They are practically all currently in full-time employment and undertaking the IT programme on a part-time basis. On average, these students would take just over two modules per year (i.e., roughly equivalent to half a year of full-time study). The students reside throughout Ireland with a small proportion resident abroad.
As the learning outcomes being assessed are directly linked to the online pedagogical methods used, these will be detailed first before the learning outcomes are adduced.
The academic year for students taking the HSA module is broken into three periods. During each of these periods a different online pedagogical method is used as follows:
The students are presented with four topics relevant to the content of the first section of the HSA module. In as far as possible, controversial—or, at least, debatable—topics are used. For each topic, the students are told the relevant sections of the HSA module text; they are also given the relevant sections of their textbooks (if appropriate) and a small number of relevant journal articles. The full-text of all articles is available online, usually from the online journal databases provided by DCU Library. This is particularly important for distance education students, as they do not have the same ease of access to academic libraries as full-time students. The students were given two weeks in which to research the topics. For the following three weeks, they had to debate four topics online with the fellow students and tutors. The students have to make a minimum number of online contributions per week (with most students making significantly more than the minimum). They are given guidelines on what is considered a good contribution.
During this period, the students were divided into groups of three. Each group is given a topic/question relevant to the second section of the module text. As in the first period, each topic/question is accompanied by relevant resources, in particular, relevant articles which are available online. Each group is given two weeks in which to research the topic. After two weeks, each group has to post a 200- to 300-word synopsis of their topic. For the following three weeks, the students are required to post questions to the other groups and to answer any questions asked by other students (or tutors) on their topic. At the end of the three weeks, each group posts an amended synopsis, which incorporates answers to questions asked over the three weeks of ‘peer tutoring’.
The students are divided into groups of five or six. All groups have to produce a report on a topic relevant to the third section of the module text. The report always incorporates an in-depth analysis of the empirical evidence available on a topic. The students do not carry out empirical research rather they gather and analyse information available on the report topic. They are given guidelines in how to organise their groups but it is up to each group to decide on the methods that best suit the group members. Each group has to post online regular reports of their progress.
Collectively, the above three pedagogical methods are, what we call, Task Oriented Online Learning (TOOL) methods.
In order that students are aware of exactly what is required from them in each period, they are given a detailed Instructional Schedule. Because it contains a number of ancillary resources, this document is some 80 pages long. However, the detailed work schedule, which informs students what is required from them on a week-by-week basis, is still over 20 pages. This level of detail and exposition is required because students are geographically dispersed and therefore there is no guarantee that all students can be got together for briefing purposes (or, indeed, one cannot rely on word of mouth from other students informing students of the course requirements).
Learning outcomes being assessed are as follows:
For the first assignment period, the students have to submit to their tutor their online contributions. The students are given guidelines on how their contributions are assessed and each student receives an individual mark based mainly on the quality of their contributions.
For the second assignment period, the students submit their initial synopsis; the questions and answers posted online to their topic, and their final synopsis. The marks are a group mark, i.e., each member of the (usually three person) group receives the same mark. The level of the mark is based on synopses quality and the timeliness and quality of their contributions.
For the third assignment period, each student submits (1) the group report; (2) their online progress reports; (3) personal evaluation of the collaborative learning process in which they are asked to evaluate their own contribution, i.e., a self-reflective piece and (4) a peer evaluation in which they are asked to assess the contribution of each of their group members both descriptively and numerically on a number of characteristics. Fifty percent of the available marks are allocated to the group report, while the remaining marks are allocated to the process oriented elements. The first two parts are group marks while the second two parts are individual marks. The students can decide not to put a group member’s name on the group report; however, this must be the unanimous decision of the remaining group members.
Before the marks awarded to each student are returned to them, a senior academic monitors a selection of the work submitted and the marks awarded by the tutor. The monitor can modify marks. This ensures equitable marking (and feedback) between the tutors and also mirrors the monitoring of examination marking that would be carried out in the traditional examination process.
In addition to engaging in the three pieces of online/group work, the students also complete three individual assignments. The assignments account for 50% of the module mark and the remaining 50% is allocated to the three pieces of online/group work in the proportions 10%, 20% and 20%.
Each of the three TOOL methods outlined above has particular strengths. The use of asynchronous communication (permitted by the online forums) in the online debate method allows the students to think about what they read and to research their responses. This led to good reflective online discussions. Using the peer tutoring method gave students the experience of having to clearly explain their thoughts and to refine those that were not clear to their fellow students. The collaborative group projects provided the students with the experience of forming and maintaining an online group. In addition, the online community, which developed over time, helped to reduce the feelings of isolation often experienced by distance education students.
The key limitation was flexibility. Distance education/online courses permit students a high degree of time and place independence. The use of the methods outlined above still gave students place independence but limited, somewhat, their time independence in that they have to devote particular weeks to their online work. However, it should be noted that (a) the use of asynchronous—rather than synchronous—communication as the main means of communication increased this time independence considerably even within the designated weeks and (b) compared to face-to-face courses, the level of time (and place) independence afforded by the use of the techniques outlined above is still substantial.
The students were surveyed on completion of the HSA module. Of the 63 students who undertook the HSA module in 2005, 29 students responded to the survey (a response rate of 46%). The students expressed the opinion that a lot of extra time and effort was required to successfully complete this module relative to other modules assessed with the traditional assessment methods. In their view, the use of the TOOL methods leads to less flexibility than modules delivered using traditional methods (see limitation mentioned above). However, a number of students did see that the quality of their learning increased, while others saw the benefits of group work. Some of the comments were:
I feel that the online assessment during the year was probably a fairer way of assessing a person’s knowledge on a subject than getting them to learn off course material and spend 3 hours writing everything they know.
I thought that the ‘learning’ experience was of a Higher Quality as the groupwork was in my opinion superior to the type of ‘knowledge’ required for an examination.
The group work was rewarding … It was a good experience and challenge to build up trust and teamwork.
When asked if the TOOL methods should be extended to other modules, 45% of the students who responded said that they should. This is interesting as practically all students said that the TOOL methods took more time and a sizeable proportion considered it a lot less flexible.
After reviewing a sample of the material produced by the students for each of the three pieces of online/group work, the External Examiner stated:
I was impressed with the sample of online work I received, and the calibration procedures look rigorous. The [Instructional Schedule was] helpful in understanding the allocation of marks … The different assignments appear to encourage students to develop good study and other transferable skills in addition to motivating them to engage well with the subject material.
It probably does not need to be made explicit that there was a high level of effort required to set up the TOOL methods. However, as we now enter our third year of using these methods, it should be noted that updating does not require too much effort. We have ‘tweaked’ the methods each year but this effort is minimal in comparison to the initial work required to set it up. Once the students are ‘up and running’, a certain amount of effort is required to tutor and monitor the students’ contributions but this is not excessive in comparison to what would be required in other modules.
The TOOL methods outlined above could be adapted for use in the classroom or, better still, for use with the ‘extended’ online classroom. For example, a debate could be started in class (possibly based on topics, readings, etc. distributed in advance) and then continued in an asynchronous online environment. This would ensure that (a) every student would get an opportunity to contribute—not just the extroverts! and (b) the undoubted advantage of asynchronous online discussion groups—noted by a number of researchers—would be achieved, i.e., online discussion groups encourage reflection as students can take time to research and consider their replies before responding online. The peer tutoring method could also be adapted for classroom delivery. The students could be given their topics and split into groups as with the online version. Class time could be used for groups to give presentations on their topics and then, in the online environment, students could post questions for the presenters.
One major advantage of the classroom use of the TOOL methods would be that the highly explicit instructional schedule that we had to write for the online students would not be required and changes could be made that adapt to issues that emerge during the academic year.