Virtually effective: the measure of a learning environment

David Jennings
University College Dublin

Printer friendly version in PDF


This chapter examines the preliminary findings of a recent user survey of the staff use of the virtual learning environment (VLE) BlackboardTM in University College Dublin (UCD), Ireland. Many Higher Education Institutes (HEI) are in the early stages of adoption and implementation of such systems. The results from this small survey serve to reflect a portion of the current dilemmas facing the individual academic and institute.

One of the key aims of the UCD survey was to quantify the level of usage of Blackboard within faculty and to attempt to qualify its usage as a means to aiding student learning. Further issues considered what particular ‘drivers’, both institutional and personal were behind the uptake of e-learning, what types of teaching materials were being employed and how were these integrated within the traditional face to face teaching environment, also considered were the effects on organizational change that the presence of a VLE might bring to faculty.

The results of the user survey are the first part of an overall strategy aimed at evaluating the effectiveness of Blackboard within the university and to develop a means by which to offer an evaluation tool to our users so that they may measure and develop their own usage of the VLE.

This paper does not propose to evaluate the technological merits of a VLE but instead offers a means by which we may assess its impact and allow us to improve on our own interactions and methodologies of usage.

What is a virtual learning environment?

“VLE’s are learning management software systems that synthesise the functionality of computer-mediated communications software and on-line methods of delivering course materials.”1

They allow the academic (often referred to as the facilitator) and learner a means by which to participate in online interactions using an array of tools such as email, databases, discussion threads, web resources etc. Ideally they offer an all in one system where a single interface allows access to a range of learning materials and communication tools. The academic may use the system to provide enhanced resources (bibliographies with live links to electronic journals, simulations expanding practice) or initiate online collaborative projects to stimulate and develop ideas and theories beyond their face to face meetings A student may access these learning materials and engage in online interactions (via email or chat tools) with peers and facilitators at a time and location that suits their personal needs.

From an administrative point of view a VLE offers the opportunity of integrating with a dedicated management information system (MIS) that may be responsible for student enrolment, registration and exams. Once again the idea of an all in one system allows for mass communications to be immediately effective and the possibility of tracking student progress and results.

The current market place for VLEs in HEIs is dominated by two products BlackboardTM( and WebCTTM (, each offering a variety of tools and functionality. Many other products are available and in wide use such as FirstClassTM ( (the VLE of choice for the Open University) and LearnWiseTM ( More recently the advent of open source has offered the academic community Moodle ( This VLE is fast superceding the proprietary products as being the most popular and easy to use system of choice (see McMullin 2005).

Pedagogical Context:

One of the fundamental issues of engagement with any new educational technology is the need to place pedagogy first, using this as the catalyst and choosing the product/s that best suit individual requirements. Best practice indicates that this is key to establishing the effective integration of any technology into the core curriculum.

Current practice is trying to adapt and transform ready-made curricula to ‘fit into’ the VLEs that are being established throughout the academic world. In the traditional education system currently employed, students learn by a means of transmission (Ramsden 1992) ‘interacting with and transforming received knowledge so as to own it and make it personally meaningful.’ They do this ‘... by actively constructing or reconstructing information.’ (Nicol et al. 2002). What a VLE can do is to encourage and place a greater emphasis on the active engagement of the material rather than the predominantly adoptive delivery approach. A VLE should act as a facilitator for both student and teacher, providing them with a set of tools to accommodate a wide range of learning styles and goals, to encourage collaborative, inclusive and student centred learning, and to act as a resource for shared experience.

The first step on engaging a VLE is to adopt an approach for course integration that will allow academics to present their current working practices sympathetically and inclusively. Each discipline will have to be handled subjectively as each will have its own distinctive pedagogies, involving characteristic learning activities, teaching and assessment methods. It follows that though there may well be a campus ‘digital signature’, as part of an ICT strategy, each faculty and department will still maintain its academic independence.

With current student numbers ever increasing, the labour intensive support once provided by academics has been greatly reduced. Through the use of a managed learning environment (MLE) [often seen as the combination of a VLE and MIS] a more flexible teaching and learning situation can be adopted to facilitate these student requirements, with both mentoring and tutoring services provided online.

All of this requires that a pedagogically sound model must be used to develop curricula for e-learning, a programme must activate and stimulate the cognitive and reflective component of the students psyche to promote a deeper research led path of learning. Furthermore academics should be encouraged to build research-based courses in an effort to facilitate the latter and to maintain their own field of excellence. This will fall in line with the mission statements of HEIs in ‘...achieving scholarly excellence through research, publication and excellence of teaching.’ (UCD Staff Manual 2001)

The Survey Design:

The Chickering and Gamson (1983) guiding principles for effective learning were integrated into the design of the survey as a means to consider if effective practice was taking place and being encouraged within the context of the VLE. These principles offer a premise from which to consider ones own pedagogy in practice and may act as a check list that one can apply to any area of curriculum design. It was hoped to prove that the use of a VLE offers a means by which to reconsider and improve ones own fundamental pedagogical beliefs. The use of any educational technology often invites a close examination of how best one might choose to implement new methodologies into day to day practice. Can the use of a VLE stir the need for pedagogical enlightenment?

Guiding Principles for Effective Learning:
  • Establish clear learning objectives and learning outcomes
  • Provide learning grounded in effective, i.e. contextual, authentic, case-based examples
  • Provide a manageable workload
  • Emphasize time on task
  • Encourage contact between students and faculty
  • Develop reciprocity and cooperation among students
  • Encourage active learning
  • Encourage deep learning
  • Make the assessment relevant to the task
  • Reward critical thinking in the assessment process
  • Provide prompt feedback
  • Provide feedback commensurate with performance
  • Respect and accommodate diverse ways of learning
  • Communicate high expectations

Background to the Survey:

Many institutes over the last number of years have chosen a proprietary VLE that has been licensed and released over their local area network (LAN) for use amongst the academic community. Training and various levels of support are available to those wishing to develop an online presence within the framework of this dedicated VLE. Most institutes are now in their third to forth year of such schemes and a great deal of feedback has already come to light in relation to initial best practice and implementation (Browne and Jenkins 2003Tearle and Prosse 2004).

BlackboardTM has been in use at UCD for over two years replacing the former online environment of TopClassTM and more recently the Online Classes system. UCD currently has two live Blackboard servers in production running version 5 (and one test server for staff training). Version 6 of Blackboard will be released campus wide in September 2005. The primary server is housed by Computing Services and is open to all faculty, this currently holds in the region of 400 courses run by c.300 instructors. Another production server dedicated to the Michael Smurfit Graduate School of Business is also in operation.

The user survey was jointly developed by the Centre For Teaching and Learning (CTL) and the Computing Services Department. The Centre is responsible for the promotion of excellence and innovation in teaching and learning within UCD. The Computing Services Department is responsible for the IT infrastructure of the entire college and also manages and administers the Blackboard system.

New users of Blackboard are provided with a half day induction to familiarize them with the main functionality and to set them up on the test server so that they may engage with the system in their own time. All staff are recommended to take part in a two day induction course which is jointly run by the Audio Visual Centre, the Centre For Teaching and Learning, Computing Services and the Library and provided through the system itself. This course covers a breadth of material including basic HTML and image manipulation, promoting active and deep learning, key tools in Blackboard and copyright issues. The majority of current Blackboard users would have undertaken both these courses.

Results from the Survey:

The key cohort of c.300 registered instructors were notified by means of an internal ListServe and by notices published in the University’s E-Learning Forum and within Blackboard itself shortly after the 2004 summer exam period. A 15% response rate was achieved, an open forum will follow and individual interviews have yet to take place.

Those that responded were across the breadth of user experience, representing both the novice and advanced adopter. From this small cohort of users who replied it is important to note the number of issues and concerns raised that are mirrored in many other HEI.

A broad range of usage occurs throughout faculty with a core element residing in the Sciences (19%) and Medicine (37%). The types of courses in place are primarily undergraduate and range from Archaeology and Accountancy to Veterinarian Pathology.

In relation to who is using the system the majority are academic members of staff (76%) but a key cohort of administrative members (16%) are being drawn in to maintain courses, post notices and in one department act as a ‘gate-keeper’ to all content postings. Thereby providing a consistent and harmonious structure to both the design and layout in the content and communications.

Part of the survey asked ‘why’ individuals decided to engage with Blackboard to ascertain what personal or institutional drivers may be at play. Fundamental to this question was the ‘perception’ of what the VLE was and therefore how it might be used. Was it a seen as a content delivery mechanism (acting as a resource for student notes, bibliographies etc.). Or was it recognized as a multi functional tool with which to offer an array of opportunities to both learner and teacher?

A sizable cohort (28%) are utilizing the VLE as a means for providing supplemental materials to face to face (f2f) sessions. This is integrated with the day-to-day activities by providing learning materials (such as slides) prior to and after traditional lectures, practicals and tutorials. This is often seen as a first step in the use of a VLE, allowing individuals a means with which to distribute content effectively. What is not noted is that a subtle change may occur in the interactions at the f2f level. For example by providing content online prior to a session the learner may be more likely to move beyond the didactic intake to a deeper level of learning by engaging more in discussion and reflection.

A number of responses (23%) cited that the reason why they engaged with the VLE was to offer the opportunity for student centred learning (see O’Neill and McMahon 2005). This shows an awareness of the potential laid out by the many tools within the VLE. By using these, one may offer the learner the choice of how it is they engage with the content, and allow them to self direct their own learning.

The promotion of scholarly intent and development of teaching portfolios within HEI acts as a driver for many individual academics who continuously strive to develop their own teaching. The use of educational technology such as a VLE, offers the opportunity to develop new methodologies and practices. It is rewarding to note that many (23%) engaged with the VLE as way to improve their teaching and be personally innovative.

Within UCD the use of Blackboard has been offered as means by which to utilize e-learning methodology within the traditional curriculum, although integrated in the ICT strategy, it is not a fundamental requirement. 15% of the users however have a departmental or faculty policy with which to promote the use of the VLE within their teaching.

Other reasons cited for using the VLE included the need to place student notices, to enhance the use of existing websites and one reason in particular found resonance ‘ other choice was provided by the academic institution’. We will return to this later.

An interesting by product noted by the survey occured when users were asked what further training they might like to undertake. As expected there were a number of requests for developing quizzes and multiple choice questionnaires (MCQs) not a strong feature of the Blackboard VLE and a series of requests for management style themed training. These consisted of general site management, project management and online time management skills - all of which are being developed in response to this.

What was particularly interesting was the amount of requests for fundamental training in the area of Blackboard itself, particular reference was made to the tools such as the group functions, communications, surveys, the electric blackboard etc. One of the key reasons UCD acquired Blackboard was for its ease of use and yet here we have a cohort of users, some familiar enough with Blackboard to be running courses for over two years, that are unable to use some of its basic functions.

Why is this? Basic training has been provided through the induction courses. Perhaps an underlying reason for this is quite simple. Although these tools appear easy to use, the way in which they are designed is often un-necessarily complicated. This makes it harder for the novice user to structure and plan how it might best be integrated into their day to day practice.

Other training requests included courses to aid in the development of interactive resources, such as Flash animations, digital video clips, audio files etc.

In UCD, although training and basic support are provided pro grata there is no specific e-learning support team provided to those who wish to develop e-learning materials (e.g. digital video, Flash, MCQs etc). Such resources can only be developed with additional finance (e.g. Teaching Grants and Awards), departmental initiatives and the release of time enabling the end user to develop the framework upon which such content will sit.


Figure 1: Current ‘File Type’ usage within Blackboard.

Analysing the material content within the VLE (see fig. 1 ) one was immediately presented with the dominance of the use of Microsoft (MS) Office tools. This was noted in the majority of text files used online. Further analysis found that these are almost all Microsoft Word Documents, and despite guidelines very few have been saved in either Rich Text Format (.rtf) or plain text (.txt). Although MS Office is almost ubiquitous in its use in HEI one must realize that the end user, the learner, may not be able to afford such an application and be looking towards low cost or open source examples such as Star Office ( http// Although MS Office files may be opened in alternate packages, formatting and layout often goes astray. Furthermore assistive technologies may encounter difficulties with reading the additional coding often placed in MS Word .html files.

It has been noted that there is a wide divergence of the ICT skills base within the UCD Blackboard community, a certain cohort are akin to web and Flash designers and the majority are at an introductionary level of providing supplemental materials online. This case of extremes needs to be remedied. The presence of champions (those who have pioneered and/or developed methodologies and content of an exceptional or innovative nature) offers an insight into how the VLE may best be used, but unless their endeavours are transferable and scalable across the whole community it may represent to the novice user a stark reminder of the burgeoning gap in their own VLE usage. It is interesting to note that where collaboration occurred in developing course materials particular technological innovations were often required. Thus it is here that a ‘champion’ might best share their expertise and experience.


Figure 2: How materials/content was developed for implementation into Blackboard


HEFCE (Higher Education Funding Council for England) has required all HEI in the UK to have a Teaching and Learning closely aligned with an Information Strategy demonstrably in place, together these attempt to ensure that the use of ICT is appropriately embedded in an institution’s core running.

The Dearing Report (NCIHE 1997) has amongst its recommendations some key areas that HEI needs to address in the implementation of ICT strategies for the immediate future. A key element is to ensure that staff and students receive appropriate training and support to enable them to realise the full potential of any ICT initiatives2 . The change in future student interactions recognises the need for all students to have access to their own portable computers.3

A major barrier in instigating e-learning has been the lack of coherent institute wide strategies. This has been compounded, where strategies exist, by the conspicuous absence of support at management level in attaining this end goal. A recent report (Studies in the Context of the E-learning Initiative: Virtual Models of European Universities, (2004)) identified a cluster approach to the integration of e-learning in European universities. Four clusters prevail with the first cluster ‘The Front-Runners’ far in advance of the other groupings providing a fully integrated approach, self-financed, with an explicit ICT strategy, and a leader in co-operative development across the community. A worrying trend occurs that can be seen repeating itself at the academic level. Although cluster two ‘The Co-operating Universities’ are not that far removed in many areas, full integration of e-learning in the university itself does not exist. The lack of an internal strategy suggests that it will not be possible to catch up with the front runners in the near future.

The student perception of online learning is often problematic. In a traditional face to face environment there may well be a misinterpretation of its purpose if not appropriately delivered, the students may see the online content as a mere collection of revision aids. Laurillard (1993) offers a series of guidelines for the effective use of the internet that apply equally to the VLE concept.

  • Is it clear to the students why they are using this new way of learning?
  • Is any prerequisite knowledge needed to use the material?
  • Is there sufficient support (eg hardware and software; training; access to experts)?
  • Has the assessment for the course being redesigned in the light of the introduction of Internet-based materials?
  • Have the students been made fully aware of the importance of the course: eg is it essential, important, or simple optional?”

Source: (Laurillard 1993)

These questions provide a valuable prompt with which to begin the consideration of implementing an elearning initiative. If complied with they should ensure that an appropriate structure is in place to allow for student engagement with the content of a VLE course.

The Evaluation Tool

A key element of the survey was to present an opportunity for the CTL and Computing Services to gain an insight into how users were engaging with Blackboard and to provide them with a way in which to assess and develop their own methodologies and interactions with the system.

A basic premise has been developed for an evaluation tool that allows the user to create a systematic development plan that will enhance the use of Blackboard from both the learners point of view and the academic (or administrative) facilitator who wishes to increase and promote the active engagement with the online content.

The evaluation tool will be developed for use within Blackboard and will utilize the ‘Survey Tool’ function. This will allow all users access to the tool and the ability to apply it to their own particular courses.

A three step approach has been taken in an attempt to capture a snapshot at key phases in the life cycle of an online course (Mason 1998) and to lay down milestones that will act as a guide and point to resources for future growth.

The first Phase will concentrate on the students point of view and ascertain what their level of experience is with ILTs and ICTs in general. It will then enquire as to what the learners expectation of the online course may be.

Phase two focuses on highlighting what additions to the online environment may be added during the lifetime of a course, akin to a Mid-Term Evaluation this model offers an insight into when and how best one might attempt to increase the online activity and enhance the learning experience.

The final phase is to assess the course on the merit of its achieved learning outcomes and ascertain if these may have been better served by further student interactions and the promotion of active and deeper levels of learning.

Evaluation Model:


The use of ICT to support teaching and learning has increased dramatically over the last decade (Conole 2002Browne and Jenkins 2003). This use has been seen as a catalyst to revisit fundamental teaching and learning issues. With a wealth of technological resources at hand one is prompted to question how best these tools may add to the learning experience. Is it possible to enhance particular elements of the curriculum by the use of ICTs or can we develop the social and communications skills of our learners. Has ICT enabled us to once again look at the prospect of HEI offering a genuine holistic development for our learners?

The title of this chapter hoped to capture the virtual element of learning environments and assess their true effectiveness but on reflection it appears a pun has been created. Is the VLE Blackboard proving to be virtually (almost) effective rather than wholly effective?

It is apparent that the majority of Blackboard users in UCD are only just beginning to tap into the potential on offer, and are using the system as an effective means of delivering and managing an array of multimedia content. Our VLE has become a CMS (Course Management System). As time goes on users will become more familiar with the tools and attempt to blend them into the day to day process of teaching and learning. However, those that are already familiar have begun to look elsewhere to enhance the environment by including outside sources of interactivity in the guise of digital video or Flash files (Thakore and McMahon 2004). This anomaly only further raises the question of the true effectiveness of the VLE in being the answer to all our e-learning needs. It is clear that there is no overarching solution, and more often than not new technologies on the world wide web (www) fast supercede what any company can produce and develop under strict market requirements.

Concluding Statement

Tim Berniers-Lee (considered by many as the father of the www) considered that the basic concept of the Web was “...that it is an information space through which people can communicate... communicate by sharing their knowledge in a pool... The idea was that everybody would be putting their ideas in, as well as taking them out.” ( This evocative statement provides hope for a new dawn in the use of ICT in education. What fast became a consumer market place has only now just begun to offer a way to claim back the www for what it was originally intended - to share information (particularly with the advent of tools such as VLEs and innovative collaborative tools such as wikis and bloggs (see McMullin 2005)). We as academics have a wonderful opportunity to bring this to fruition. Tempered with the foresight to heighten our pedagogic needs we can begin to use the VLE as yet another tool among many in forwarding teaching and learning.


The author would like to thank Brendan Dixon (UCD, Computing Services) for his invaluable contribution in designing and delivering the questionnaire. And to thank all those staff members who contributed to the findings.


   Browne, T. and M. Jenkins (2003). VLE Surveys, A Longitudinal Perspective between March 2001 and March 2003 for Higher Education in the United Kingdom. UCISA,

   Chickering, A. W. and Z. F. Gamson (1983). Seven Principles for Good Practice in Undergraduate Education.

   Conole, G. (2002). Review of JTAP Projects: Perspectives on teaching and learning. University of Southhampton.

   Laurillard, D. (1993). Rethinking University Teaching: a framework for the effective use of educational technology. London: Routledge.

   Mason, R. (1998). Models of Online Courses. ALN Magazine 2(2). Accessed 09.03

   McMullin, B. (2005). Putting the learning back into learning technology. In S. Moore, G. O’Neill, and B. McMullin (Eds.), Emerging Issues in the Practice of University Learning and Teaching. Dublin: AISHE.

   NCIHE (1997). National Committee of Inquiry into Higher Edication - the Dearing Report. Higher Education in the Learning Society, HMSO/NCIHE.

   Nicol, D., N. Kay, G. Gordon, and M. Coen (2002). INSIGHT: A Model for Evaluation the Costs and Benefits of ICT in Teaching and Learning. University of Strathclyde.

   O’Neill, G. and T. McMahon (2005). Student-Centred Learning: What does it mean for students and lecturers? In S. Moore, G. O’Neill, and B. McMullin (Eds.), Emerging Issues in the Practice of University Learning and Teaching. Dublin: AISHE.

   PLS RAMBOLL Management A/S (2004). Studies in the Context of the E-learning Initiative: Virtual Models of European Universities (Lot 1). Draft Final Report to the EU Commission, DG Education & Culture.

   Ramsden, P. (1992). Learning to teach in Higher Education. Routledge.

   Tearle, P. and D. Prosse (2004). Use of Virtual Learning Environments for teaching and learning in higher education: Pedagogic models. Paper at ALT-C 2004, Exeter University.

   Thakore, H. and T. McMahon (2004, 6-8 September). Developing a Pilot Web-based Interactive Tutorial System for Medical Students. Association for Medical Education in Europe Annual Conference.

   UCD Staff Manual (2001). Mission and Vision Statement.˙acc.htm


1JTAP Report -041 A Framework for Pedagogical Evaluation of VLE’S

2Recommendation 9

3Recommendation 46