Desktop virtual reality (desktopVR) provides a range of benefits for Computer Aided Learning (CAL). Users can exploit conventional keyboards and mice to navigate through three dimension models of virtual environments using the Virtual Reality Markup Language (VRML). This, in turn, supports learning through exploration. Other approaches, such as QuicktimeVR, enable users to manipulate photo-realistic images of real-world objects in three dimensions. Such techniques support activist learning styles because they provide a direct impression of the objects being studied. This paper describes some of the problems that arise when designers attempt to validate the claimed benefits of desktopVR for CAL systems.
DesktopVR techniques have been introduced into a wide range of CAL systems (Johnson, 1998). However, it is difficult to determine whether these presentation techniques actually support users' needs (Kaur, Sutcliffe and Maiden, 1998). Valid empirical measures are hard to identify because desktopVR often supports many different users with a wide range of tasks (Johnson, 1998a). More qualitative techniques, such as think alouds, focus on the superficial appeal of the technology if they are not supported by longitudinal studies. The following sections extend this analysis by focussing on the specific problems that arise during the validation of desktopVR within CAL systems:
Fire Officers are required to possess practical skills. For example, they are frequently called upon to use breathing apparatus during fires. They are also expected to have specialised technical knowledge. Officers must know how to apply the latest foam technology to combat a range of different fires. CAL tools are perceived by many in the Fire Brigade as a cost effective means of delivering technical knowledge and practical skills. They are particularly appropriate for an organisation whose members are scattered amongst many different stations.
Fire fighters are often characterised by activist learning styles. It has been argued that they learn more effectively through direct experience than through the mediation of books or lectures (Johnson, 1998). DesktopVR techniques, therefore, provide important benefits for the development of CAL in the fire brigade. Fire fighters can learn by interacting with objects in virtual environments rather than by passively listening to conventional lectures (Sutcliffe and Patel, 1996). Figure 1 shows how desktopVR techniques were applied to a Heavy Rescue Vehicle (HRV) training package. The photo-realistic facilities of QuicktimeVR provide a three-dimensional representation of the storage area inside the HRV. Individual items of equipment can be found by exploring the desktopVR view, shown in the middle panel of Figure 1, or by selecting an item from the list on the right.
Figure 1: The Heavy Rescue Vehicle Training Package
The HRV package also provided detailed information about the equipment on the vehicle. Hypertext was used to provide electronic access to existing technical notes. Video clips were used to show the equipment "in action". Figure 2 shows how QuicktimeVR also enabled fire fighters to manipulate individual items of equipment.
Figure 2: An Object Rotation of Lucas Cutters on the Heavy Rescue Vehicle
Physical, or rather virtual, locations provided information about the relationships between objects. Roman tools could be held in one area of the gallery, information about weapons would be held in another and so on. As the user moves through the gallery, then these collections change to reflect different time periods. This strategy reflects the careful planning that goes into the layout of `real-world' museums. Within each section, the VRML gallery could contain QuicktimeVR movies as well as video clips and more conventional presentations. Figure 4 presents an excerpt from one of these videos.
Figure 3: Perceived "Ease of Learning" in the Fire Brigade Case Study
Figure 4: Perceived "Effectiveness of Learning" in the Fire Brigade Case Study
These results helped to guide the formative evaluation of our desktopVR systems. Think alouds were used to ensure that the new interfaces were perceived to compare favourably against existing training techniques. In particular, we strove hard to use interactive desktopVR techniques to ensure that the HRV package avoided the criticims of more passive lectures. We had originally intended that the same questionnaire might then be re-issued to compare the performance of the new CAL applications against the existing techniques. However, this raised a number of objections:
It can be difficult for designers to identify suitable evaluation criteria even if they have a clear idea of the user groups that CAL tools must support. For instance, the Hunterian gallery was designed to help local schools with their project work. We, therefore, decided to conduct an evaluation to determine whether the existing web pages or the VRML model provided more support for the comprehension tests that are frequently set by school teachers. Figure 5 presents part of this comparison. One class of primary children was set a number of questions about the Romans in Scotland. Another group was simply encouraged to browse the site. Their attitudes to retrieval delays were assessed because preliminary studies indicated that this was a critical factor in the successful adoption of the web based systems in local schools. It was hypothesised that the VRML gallery would provide greatest support for the browsing users because they were not attempting to complete a specified information retrieval task. However, the children had the same reaction to the interface whether or not they were set a task by their teacher (Johnson, 1997).
(a) Browsing User
(b) Task User
The problems of obtaining empirical results for the contribution that desktopVR makes to user tasks are exacerbated by the barriers to more qualitative evaluation techniques. For example, Figure 6 presents the school childrens' opinions of QuicktimeVR object rotations for Asante gold weights in the Hunterian Museum (see http://www.gla.ac.uk/Museum/HuntMus/asante/HuntPages.html). The proportion of children who preferred not to state an opinion is striking. Users found it difficult to express the contribution that desktopVR made to their interaction with the CAL tools (Robertson et al, 1997).
Figure 7: Laurillard's Conversational Model for Effective E ducation.
Laurillard's model is important because it shows how the success or failure of desktopVR systems can depend upon a much wider range of educational activities. Activity-implementation charts provide a practical application of this idea. They can be used to identify the support that CAL tools provide for the various stages in Laurillard's model (Montgomery, 1997). Table 1 applies this approach to the fire brigades' technical training. A blank in the Teaching Mode column indicates that a learning activity is not supported by the current allocation of educational tasks. As can be seen, the desktopVR techniques in the HRV package provides minimal support for many of the learning activities identified by Laurillard.
|1. The learner listens to a teacher's exposition.|| Human
| Officer in charge gives a lecture
Fire fighter uses HRV package.
|2. The learner describes the conception as they understand it, in the form of an essay or verbally.||Human||Fire fighter asks question at end of lecture|
|3. The teacher re-describes the conception to the learner based upon activity 2 and provides feeback.||Human||Officer in charge explains answer to question.|
|4. The learner attempts activity 2 again.||Human||Fire fighter confirms their understanding of Officer in charge's response.|
|5. The teacher sets a goal for the learner to complete.||_||_|
|6. The learner attempts the goal set in activity 5.||_||_|
|7. The teacher provides feedback regarding the learner's attempt at the task described in activity 6.||_||_|
|8. The learner modifies their actions in the light of feedback provided by the teacher.||_||_|
|9. The learner reflects on the interaction in order to modify their grasp of the concepts.||Other||Call-out may provoke reflection on technical training.|
|10. The learner modifies their actions in the light of reasoning at the "public" level of descriptions.||Other||Fire fighter may alter behaviour in practical exercise on basis of technical material.|
|11. The teacher modifies the task set to address some need revealed by the learner's descriptions or actions.||_||_|
|12. The teacher examines the learner's actions and modifies their description of the original conception.||_||_|
Table 1: Activity-Implementation Chart for Technical Training in the Fire Brigade
In retrospect, the HRV tool failed to address many of the fundamental concerns raised by Laurillard's model. It supported the presentation of material, embodied in stage 1 of the model. It did not support stages 4-8. These activities focus on the use of tasks and exercises to provide students with feedback about their understanding of key concepts. These stages also provide the tutor with feedback about the need to improve their delivery techniques. Figure 8 presents a self-assessment tool that was, therefore, developed to accompany the HRV package. This screen implements a photographic multiple-choice question. Fire fighters are provided with feedback after each selection and are encouraged to provide further input if they make an incorrect selection.
Figure 8: A Photographic Multiple Choice Question
The meta-point here is that DesktopVR is only one technique amongst the many that are used in CAL tools, such as the HRV package. These systems, in turn, form part of a range of techniques that support an operators overall training. To focus an evaluation on the contribution of desktopVR is, therefore, to largely miss the point about the effective application of CAL. The usability of a particular interaction technique is less important than the overall coverage that an organisation provides for an individual's range of educational needs.
Group dynamics have been studied for immersive forms of collaboration in virtual environments (Benford et al, 1995). They have not previously been studied for desktopVR in CAL applications. It is important to emphasise, however, that non-immersive interfaces pose different evaluation problems from those studied in previous research. Group interaction is not mediated by the system itself. In other words, designers cannot simply analyse traces or logs to view the social interactions that take place around the keyboard. Contextual enquiry techniques provide one means of analysing these group dynamics during the use of non-immersive desktopVR. This is the subject of a research project that will be starting later this year, in cooperation with the Hunterian Museum and Art Gallery.
There is another level at which contextual factors complicate the evaluation of desktopVR. During the later stages of the fire brigade project, we became aware of the complex criteria that were being applied to assess the success or failure of our systems. Some people were interested in the financial benefits that might be derived from allocating training duties away from Instructing Officers towards CAL tools. Others saw this as an opportunity to provide Fire Fighters with more varied material and, in particular, to support activist learning styles. Some individuals wanted to ensure that the Fire Brigade investigated "leading edge" technology and were less worried about establishing the "effectiveness" of that technology. Finally, there was a concern that the introduction of new technology might distance fire fighters from the first-hand experiences of their human instructors. Given such diversity, there seems little prospect of ever identifying single measures that might be used to effectively validate the introduction of desktopVR into CAL tools.
S. Benford, J. Bowers, L. Fehlen, C. Greenhalgh, D. Snowdon, User Embodiment in Collaborative Virtual Environments, CHI'95 Proceedings, 242-249, 1995.
J. Carroll and J. Thomas, Fun, SIGCHI Bulletin, 19(3)21-24, 1988.
C.W. Johnson, Ten Golden Rules for Video over the Web. In J. Ratner, E. Grosse and C. Forsythe (eds.) Human Factors for World Wide Web Development. 207-224. Lawrence Erlbaum, New York, United States of America, 1997a.
C.W. Johnson, The Problems of Validating DesktopVR. To appear in People and Computers XIII: Proceedings of HCI'98, Springer Verlag, Berlin, 1998.
C.W. Johnson, Why `Traditional' HCI Design and Evaluation Techniques Fail With DesktopVR. In IEE Colloquium Digest 98/437, 1998a.
R.S. Kalawsky, New methods and techniques for evaluating user performance in advanced 3D virtual interfaces. In IEE Colloquium Digest 98/437, 1998.
K. Kaur, A. Sutcliffe and N. Maiden Improving Interaction with Virtual Environments. In IEE Colloquium Digest 98/437, 1998.
D. Laurillard, Rethinking University Education: A Framework for the Effect ive Use of Educational Technology, Routledge, London, 1993.^M
M. Montgomery, Developing a Laurillardian Design Method for CAL. In Proceedings of Ed-Media '97. Association for the Advancement of Computing in E ducation, 1997.^M
G. Robertson, M. Czerwinski, M. van Dantzich, Immersion in DesktopVR, In G. Robertson and C. Schmandt (eds) UIST'97, ACM, New York.
A. Sutcliffe and U. Patel, 3D or not 3D: Is It Nobler in the Mind? In M.A. Sasse, R.J. Cunningham and E.L. Winder (eds.), HCI'96, 79-94, Springer Verlag, London, 1996.