Evaluating the Contribution of DesktopVR to Computer Aided Learning

Abstract

Over the last three years we have developed a range of Computer Aided learning (CAL) packages that use desktop virtual reality (desktopVR) techniques. This paper describes some of the problems that have arisen during the evaluation of these applications.

Keywords:

1. Introduction

DesktopVR techniques, including Virtual Reality Mark-up Language (VRML) and QuicktimeVR, have been introduced into a wide range of CAL interfaces (Johnson, 1998). However, it is difficult to determine whether these presentation techniques actually support users' needs. Valid empirical measures are hard to identify because desktopVR often supports many different users with a wide range of learning needs (Johnson, 1998a). The users of the Hunterian Museum system, described below, were identified as school children, undergraduates, curators and archaeologists. More qualitative techniques, such as think alouds, focus on the superficial appeal of the technology if they are not supported by longitudinal studies (Sutcliffe and Kaur, 1998).

This paper uses two case studies to illustrate the practical problems that arise when attempting to validate desktopVR. The first case study focusses on a range of CAL tools that were developed to support the training activities of a regional fire brigade. The second case study focusses on the use of desktopVR within a visitor information system for the Hunterian Museum in Glasgow.

2. The Fire Brigade Case Study

Fire Officers are required to possess practical skills. For example, they are frequently called upon to use breathing apparatus during fires. They are also increasingly expected to have specialised technical knowledge. Officers must know how to apply the latest foam technology to combat a range of different fires. CAL tools are perceived by many in the Fire Brigade as a cost effective means of delivering technical knowledge and practical skills. They are particularly appropriate for an organisation whose members are scattered amongst many different stations.

Fire fighters are often characterised by activist learning styles. It has been argued that they learn more effectively through direct experience than through the mediation of books or lectures (Johnson, 1998). DesktopVR techniques, therefore, provide important benefits for the development of CAL in the fire brigade. Fire fighters can learn by interacting with objects in virtual environments rather than by passively listening to conventional lectures. Figure 1 shows how this approach can be applied to a Heavy Rescue Vehicle (HRV). The photo-realistic facilities of QuicktimeVR provide a three-dimensional representation of the storage area inside the HRV. Fire fighters can use a conventional keyboard and mouse to move inside the QuicktimeVR image (Johnson, 1998). Equipment is located through exploration using the desktopVR view, shown in the middle panel of Figure 1, or by selecting an item from the list on the right.

Figure 1: The Heavy Rescue Vehicle Training Package

The HRV package also provided detailed information about the equipment on the vehicle. Hypertext was used to provide electronic access to existing technical notes. Video clips were used to show the equipment "in action". Figure 2 shows how QuicktimeVR enabled fire fighters to manipulate individual items of equipment.

Figure 2: An Object Rotation of Lucas Cutters on the Heavy Rescue Vehicle

3. The Hunterian Museum Case Study

Figure 3: VRML Model of the Hunterian Museum, Glasgow.

Physical, or rather virtual, locations provided information about the relationships between objects. Roman tools could be held in one area of the gallery, information about weapons would be held in another and so on. As the user moves through the gallery, then these collections change to reflect different time periods. This strategy reflects the careful planning that goes into the layout of `real-world' museums. Within each section, the VRML gallery could contain QuicktimeVR movies as well as video clips and more conventional presentations. Figure 4 presents an excerpt from one of these videos.

Figure 4 : Excerpts from the Hunterian's Video on Roman Armour

4 The Problems of Validating DesktopVR

4.1 Problem 1: Benchmarking

Figure 3: Perceived "Ease of Learning" in the Fire Brigade Case Study

Figure 4: Perceived "Effectiveness of Learning" in the Fire Brigade Case Study

Unfortunately, the subjective responses to the previous questions do not provide detailed criteria against which to evaluate the contribution of desktopVR in any new CAL tool. Tables 1 and 2, therefore, present the results of questions that were intended to identify the reasons behind the assessments in Figures 3 and 4.

Table 1) What do you enjoy most about fire training?

Table 2) What do you hate most about fire training?

These results helped to guide the formative evaluation of our desktopVR systems. Think alouds were used to ensure that the new interfaces exploited the positive features of Table 1 and avoided the negative aspects, such as repetitiveness, in Table 2. Unfortunately, this approach did not provide benchmark criteria for summative evaluation. Our intention had been to issue the questions to assess the existing situation. We would then introduce the desktopVR systems into the fire stations and re-issue the questionnaires. Ideally, they crews would then have reported high levels of subjective satisfaction with the new systems. there might even have been some positive effects on standard lectures as Instructing Officers transfered more teaching duties to the CAl application. Such an evaluation methodology is flawed for a number of reasons:

1. the Hawthorne effect. The very fact that this project had support from the highest levels of the fire brigade made it likely that we would receive biased responses to any direct questions about the utility of desktopVR within a CAL tool.
2. desktopVR is an embedded technique. It made little sense to evaluate desktopVR on its own. These presentation and interaction techniques were integrated into a much larger CAl tool, as described in Section 2. The contribution that these techniques made to the overall system was clearly less important for the fire brigade than that the overall application was effective.
3. organisational factors. This is related to the Hawthorne effect. Not all stations had their own PCs at the time that we were developing the project. Those stations that were included in the evaluation were, therefore, seen to be pioneers. This must have had an effect on individual responses to the introduction of new technology.
4. longitudinal effects. The average length of service amongst the users of our CAL system was fifteen years. The responses to the first questionnaire reflect the fire fighters experience over many years of lectures, videos and real incidents. Simply issuing questionnaires about subjective attitudes to new technology would have provided an extremely poor impression of the long term effectiveness of these systems.

5. The Hunterian Case Study

Evaluator:      Do you prefer pictures or the models?
User:           The models are great.
Evaluator:      Why?
User:           Um, it just feels different.

Figure 3: Questionnaire Responses to the Introduction of QuicktimeVR within Muse um Web Pages

The questions in Figure 3 were not intended to obtain statistically significant and replicable results. They were simply intended to provide some insight into whether or not the users supported our introduction of desktopVR. The large number of uncommitted responses to such fundamental questions can be e xtremely disappointing. Gibson provides some insight into the source of this disappointment:

`` Not only do we perceive in terms of visual information, we can also think in those terms. Making and looking at pictures helps us to fix these terms. We can also think in terms of verbal information, as is obvious, and words enabl e us to fix, classify and consolidate our ideas. But the difference is that visual thinking is freer and less stereotyped than ve rbal thinking: there is no vocabulary of picturing as there is of saying.'' (Gib son, 1971).

3 Applying Laurillard's Converstaional Model of CAL

Figure 1: Laurillard's Conversational Model for Effective E ducation.

Figure 1 illustrates how both the student and teacher contrib ute to the learning process. The right-hand components of the diagram represen t the iterative process by which students modify their view of the concepts that they are being taught. They adapt their descriptions (activities 4 and 9) and modify their actions (activities 8 and 10) as they learn from their tutor's fee dback. The left-hand components of Laurillard's model describe the iterative p rocess that informs the teacher's interaction with the student. The tutor modi fies the way that they present key concepts in response to a student's initial a ttempt to express those concepts (activity 3). They also modify the tasks that they set in response to their student's initial attempts to fulfil those tasks ( activities 11 and 12). This notion of dual contribution explains Laurillard's view that education is essentially a conversation in which both the student and the teacher are attempting to maximise their common ground about the topic in qu estion. Most of the tutor's tasks in Laurillard's model can be perfor med by either a human instructor or by an appropriate CAL tool. For example, s tage five involves the tutor in establishing task objectives for the student. This could take the form of an essay title set by the teacher. Alternatively, it might be implemented as a computer generated construction task embedded withi n a virtual reality simulation. This paper, therefore, sets out to determine w hether Laurillard's model can help designers to identify appropriate allocations of training tasks within a complex and dynamic organisation (Montgomery, 1997). There is considerable evidence that such domain characteristics help to determ ine appropriate function allocations between human trainers and CAL applications (Plowman, 1996). However, most of this evidence focuses on educational practi ces in Universities or in schools. This paper, therefore, looks beyond these c onventional settings to examine the impact of function allocation within the tra ining mechanisms of one of Scotland's regional Fire Brigades.

3.1 The Strengths: The Fire Brigade Case Study

3.2 The Weaknesses: The Hunterian Case Study

4. Conclusion

Acknowledgements

References

H. Beyer and K. Holtzblatt, Contextual Design: Defining Customer Centred Systems, Morgan Kaufman, San Francisco, 1998.

S. Benford, J. Bowers, L. Fehlen, C. Greenhalgh, D. Snowdon, User Embodiment in Collaborative Virtual Environments, CHI'95 Proceedings, 242-249, 1995.

See www.dcs.gla.ac.uk/~johnson/papers/mobile/HCIMD1.html

D. Roberts, IBM RealThing Guidelines, see www.ibm.com/ibm/hci/guidelines/design/realthings, 1997.