Evaluating the Contribution of DesktopVR to Computer Aided Learning

Chris Johnson

Department of Computing Science, University of Glasgow, Glasgow, Scotland, G12 8QQ.
johnson@dcs.gla.ac.uk

Abstract

Desktop virtual reality (desktopVR) provides a range of benefits for Computer Aided Learning (CAL). Users can exploit conventional keyboards and mice to navigate through three dimension models of virtual environments using the Virtual Reality Markup Language (VRML). This, in turn, supports learning through exploration. Other approaches, such as QuicktimeVR, enable users to manipulate photo-realistic images of real-world objects in three dimensions. Such techniques support activist learning styles because they provide a direct impression of the objects being studied. This paper describes some of the problems that arise when designers attempt to validate the claimed benefits of desktopVR for CAL systems.

Keywords:

Computer Aided Learning; DesktopVR; Training; Evaluation; Human Computer Interaction.

1. Introduction

DesktopVR techniques have been introduced into a wide range of CAL systems (Johnson, 1998). However, it is difficult to determine whether these presentation techniques actually support users' needs (Kaur, Sutcliffe and Maiden, 1998). Valid empirical measures are hard to identify because desktopVR often supports many different users with a wide range of tasks (Johnson, 1998a). More qualitative techniques, such as think alouds, focus on the superficial appeal of the technology if they are not supported by longitudinal studies. The following sections extend this analysis by focussing on the specific problems that arise during the validation of desktopVR within CAL systems:

Two case studies are used to illustrate the impact that these problems had upon the development of `real-world' applications. The first focuses on a range of CAL tools that were developed to support the training activities of a regional fire brigade. The second case study centres on the use of desktopVR within a visitor information system for the Hunterian Museum in Glasgow.

2. The Fire Brigade Case Study

Fire Officers are required to possess practical skills. For example, they are frequently called upon to use breathing apparatus during fires. They are also expected to have specialised technical knowledge. Officers must know how to apply the latest foam technology to combat a range of different fires. CAL tools are perceived by many in the Fire Brigade as a cost effective means of delivering technical knowledge and practical skills. They are particularly appropriate for an organisation whose members are scattered amongst many different stations.

Fire fighters are often characterised by activist learning styles. It has been argued that they learn more effectively through direct experience than through the mediation of books or lectures (Johnson, 1998). DesktopVR techniques, therefore, provide important benefits for the development of CAL in the fire brigade. Fire fighters can learn by interacting with objects in virtual environments rather than by passively listening to conventional lectures (Sutcliffe and Patel, 1996). Figure 1 shows how desktopVR techniques were applied to a Heavy Rescue Vehicle (HRV) training package. The photo-realistic facilities of QuicktimeVR provide a three-dimensional representation of the storage area inside the HRV. Individual items of equipment can be found by exploring the desktopVR view, shown in the middle panel of Figure 1, or by selecting an item from the list on the right.

Figure 1: The Heavy Rescue Vehicle Training Package

The HRV package also provided detailed information about the equipment on the vehicle. Hypertext was used to provide electronic access to existing technical notes. Video clips were used to show the equipment "in action". Figure 2 shows how QuicktimeVR also enabled fire fighters to manipulate individual items of equipment.

Figure 2: An Object Rotation of Lucas Cutters on the Heavy Rescue Vehicle

3. The Hunterian Museum Case Study

The second case study focuses on an information system that was developed to provide visitors with a range of multimedia exhibits in the Hunterian Museum, Glasgow. An important objective in the design of this interface was that it should motivate further exploration by a wide range of potential visitors; including both school children and adults. Figure 3 shows how VRML was, therefore, used to place these multimedia exhibits within a three dimensional model of the nineteenth century Museum buildings.

Figure 3: VRML Model of the Hunterian Museum, Glasgow.

Physical, or rather virtual, locations provided information about the relationships between objects. Roman tools could be held in one area of the gallery, information about weapons would be held in another and so on. As the user moves through the gallery, then these collections change to reflect different time periods. This strategy reflects the careful planning that goes into the layout of `real-world' museums. Within each section, the VRML gallery could contain QuicktimeVR movies as well as video clips and more conventional presentations. Figure 4 presents an excerpt from one of these videos.

Figure 4 : Excerpts from the Hunterian's Video on Roman Armour

4 The Problems of Validating DesktopVR

The previous section briefly described two tools that used desktopVR as part of more general CAL applications. This section goes on to describe the problems that arose when we tried to assess the "usability" of these systems.

4.1 Problem 1: Establishing Benchmark Criteria for DesktopVR

The first problem in evaluating any system is to determine the criteria against which to assess "usability" (Kalawsky, 1998). This immediately raised problems for the first case study because we were developing in a green-fields environment. At the start of the project, the fire fighters did not have access to any CAL applications. None of them had used or even heard about desktopVR. This created considerable problems because we had no means of establishing the "benchmark" criteria against which to evaluate the new interfaces. The best that we could do was to issue a questionnaire to assess the fire fighters' attitudes about existing training techniques. Figures 3 and 4 presents the results for 27 fire fighters from two different stations within the same region.

Figure 3: Perceived "Ease of Learning" in the Fire Brigade Case Study

Figure 4: Perceived "Effectiveness of Learning" in the Fire Brigade Case Study

These results helped to guide the formative evaluation of our desktopVR systems. Think alouds were used to ensure that the new interfaces were perceived to compare favourably against existing training techniques. In particular, we strove hard to use interactive desktopVR techniques to ensure that the HRV package avoided the criticims of more passive lectures. We had originally intended that the same questionnaire might then be re-issued to compare the performance of the new CAL applications against the existing techniques. However, this raised a number of objections:

  1. the Hawthorne effect.
    The fact that this project had support from the highest levels of the fire brigade made it difficult for us to judge whether we were receiving unbiased responses to questions about the utility of desktopVR within a CAL tool.

  2. the problems of measuring embedded effects.
    The HRV package embedded desktopVR techniques within a much larger CAL tool. It was less important for the fire brigade to show that these novel presentation techniques were effective than it was to ensure that the overall application satisfied their training objectives.

  3. the problems of measuring longitudinal effects.
    The average length of service amongst the users of our CAL system was fifteen years. Any subjective assessment after a few months access to the new technology would provide a poor impression of the long term effectiveness of desktopVR.

4.2 Problem 2: How to Identify Users and their Educational Tasks?

It was much easier to identify suitable benchmarks for the Museum case study. The Hunterian already provided its visitors with access to multimedia resources through a number of "conventional" web pages (see http://www.gla.ac.uk/Museum). Evaluations could, therefore, compare user performance with the VRML model against their performance with the hypertext pages. In neither case was their organisational pressure to exploit the new technology. Users were free to walk-up and use the systems; or alternatively to walk-away and ignore them. Unfortunately, a number of further problems complicated the formative and summative evaluation of the Hunterian's VRML exhibition. In particular, it was extremely difficult to predict the range of educational tasks that users expected to accomplish with the new CAL system. These users ranged from school children to curators, from historians and archaeologists to casual browsers, from computing scientists to graphic designers. Each group had their own set of tasks and, therefore, each had different criteria with which to measure the success or failure of those tasks. This task diversity typifies many interactive 3D systems which are open to public access. It also contrasts with the Fire Brigade case study where there was a much more tightly defined set of learning objectives.

It can be difficult for designers to identify suitable evaluation criteria even if they have a clear idea of the user groups that CAL tools must support. For instance, the Hunterian gallery was designed to help local schools with their project work. We, therefore, decided to conduct an evaluation to determine whether the existing web pages or the VRML model provided more support for the comprehension tests that are frequently set by school teachers. Figure 5 presents part of this comparison. One class of primary children was set a number of questions about the Romans in Scotland. Another group was simply encouraged to browse the site. Their attitudes to retrieval delays were assessed because preliminary studies indicated that this was a critical factor in the successful adoption of the web based systems in local schools. It was hypothesised that the VRML gallery would provide greatest support for the browsing users because they were not attempting to complete a specified information retrieval task. However, the children had the same reaction to the interface whether or not they were set a task by their teacher (Johnson, 1997).


(a) Browsing User


(b) Task User

Figure 5: Attitudes to retrieval delays for task-directed users and casual browsers.

The problems of obtaining empirical results for the contribution that desktopVR makes to user tasks are exacerbated by the barriers to more qualitative evaluation techniques. For example, Figure 6 presents the school childrens' opinions of QuicktimeVR object rotations for Asante gold weights in the Hunterian Museum (see http://www.gla.ac.uk/Museum/HuntMus/asante/HuntPages.html). The proportion of children who preferred not to state an opinion is striking. Users found it difficult to express the contribution that desktopVR made to their interaction with the CAL tools (Robertson et al, 1997).

Figure 6: Questionnaire Responses to the Introduction of QuicktimeVR

4.3 Problem 3: Assessing the Contribution of DesktopVR to Learning

There is a danger that by focusing on particular educational tasks, designers may overlook the broader context of CAL applications. This point can be illustrated by Laurillard's (1993) conversational model illustrated in Figure 7. The right-hand components of the diagram represent the iterative process by which students modify their view of the concepts that they are being taught. They adapt their descriptions (activities 4 and 9) and modify their actions (activities 8 and 10) as they learn from their tutor's feedback. The left-hand components of Laurillard's model describe the iterative process that informs the teacher's interaction with their students. Tutors modify the tasks that they set in response to the student's initial attempts to fulfill those tasks (activities 11 and 12).

Figure 7: Laurillard's Conversational Model for Effective E ducation.

Laurillard's model is important because it shows how the success or failure of desktopVR systems can depend upon a much wider range of educational activities. Activity-implementation charts provide a practical application of this idea. They can be used to identify the support that CAL tools provide for the various stages in Laurillard's model (Montgomery, 1997). Table 1 applies this approach to the fire brigades' technical training. A blank in the Teaching Mode column indicates that a learning activity is not supported by the current allocation of educational tasks. As can be seen, the desktopVR techniques in the HRV package provides minimal support for many of the learning activities identified by Laurillard.

Activity Teaching Mode Example
1. The learner listens to a teacher's exposition. Human
CAL/desktopVR		 Officer in charge gives a lecture
Fire fighter uses HRV package.
2. The learner describes the conception as they understand it, in the form of an essay or verbally. Human Fire fighter asks question at end of lecture
3. The teacher re-describes the conception to the learner based upon activity 2 and provides feeback. Human Officer in charge explains answer to question.
4. The learner attempts activity 2 again. Human Fire fighter confirms their understanding of Officer in charge's response.
5. The teacher sets a goal for the learner to complete. _ _
6. The learner attempts the goal set in activity 5. _ _
7. The teacher provides feedback regarding the learner's attempt at the task described in activity 6. _ _
8. The learner modifies their actions in the light of feedback provided by the teacher. _ _
9. The learner reflects on the interaction in order to modify their grasp of the concepts. Other Call-out may provoke reflection on technical training.
10. The learner modifies their actions in the light of reasoning at the "public" level of descriptions. Other Fire fighter may alter behaviour in practical exercise on basis of technical material.
11. The teacher modifies the task set to address some need revealed by the learner's descriptions or actions. _ _
12. The teacher examines the learner's actions and modifies their description of the original conception. _ _

Table 1: Activity-Implementation Chart for Technical Training in the Fire Brigade

In retrospect, the HRV tool failed to address many of the fundamental concerns raised by Laurillard's model. It supported the presentation of material, embodied in stage 1 of the model. It did not support stages 4-8. These activities focus on the use of tasks and exercises to provide students with feedback about their understanding of key concepts. These stages also provide the tutor with feedback about the need to improve their delivery techniques. Figure 8 presents a self-assessment tool that was, therefore, developed to accompany the HRV package. This screen implements a photographic multiple-choice question. Fire fighters are provided with feedback after each selection and are encouraged to provide further input if they make an incorrect selection.

Figure 8: A Photographic Multiple Choice Question

The meta-point here is that DesktopVR is only one technique amongst the many that are used in CAL tools, such as the HRV package. These systems, in turn, form part of a range of techniques that support an operators overall training. To focus an evaluation on the contribution of desktopVR is, therefore, to largely miss the point about the effective application of CAL. The usability of a particular interaction technique is less important than the overall coverage that an organisation provides for an individual's range of educational needs.

4.4 Problem 4: Assessing the Contribution of DesktopVR in Context

Contextual factors have a profound impact upon the quality of interaction with desktopVR systems. These factors are difficult to account for within many existing evaluation techniques. For example, our analysis of download times with the Hunterian gallery relied upon a carefully controlled environment. Rather than measuring user responses to the VRML interface "on the Museum floor", school children were observed using the system in a quiet area away from the main exhibits. This was necessary because initial tests provided unsatisfactory results. Other children would "join-in" by issuing requests for other resources if they disagreed with their friends' selection. Few requests were completed as the users battled for control of the tracker-ball. This form of social interaction or play varied between groups of children. An individual's subjective experience with the VRML model was often determined by the amount of interference or cooperation that they experienced from the other members of their group (Carroll and Thomas, 1988).

Group dynamics have been studied for immersive forms of collaboration in virtual environments (Benford et al, 1995). They have not previously been studied for desktopVR in CAL applications. It is important to emphasise, however, that non-immersive interfaces pose different evaluation problems from those studied in previous research. Group interaction is not mediated by the system itself. In other words, designers cannot simply analyse traces or logs to view the social interactions that take place around the keyboard. Contextual enquiry techniques provide one means of analysing these group dynamics during the use of non-immersive desktopVR. This is the subject of a research project that will be starting later this year, in cooperation with the Hunterian Museum and Art Gallery.

There is another level at which contextual factors complicate the evaluation of desktopVR. During the later stages of the fire brigade project, we became aware of the complex criteria that were being applied to assess the success or failure of our systems. Some people were interested in the financial benefits that might be derived from allocating training duties away from Instructing Officers towards CAL tools. Others saw this as an opportunity to provide Fire Fighters with more varied material and, in particular, to support activist learning styles. Some individuals wanted to ensure that the Fire Brigade investigated "leading edge" technology and were less worried about establishing the "effectiveness" of that technology. Finally, there was a concern that the introduction of new technology might distance fire fighters from the first-hand experiences of their human instructors. Given such diversity, there seems little prospect of ever identifying single measures that might be used to effectively validate the introduction of desktopVR into CAL tools.

5. Conclusions

This paper has identified a number of problems that complicate the evaluation of desktopVR within CAL systems: The paper has, therefore, moved from the specific problems of assessing particular desktopVR interfaces to the general issues of evaluating desktopVR within complex organisations. It is discouraging that we are faced by so many problems and so few solutions.

Acknowledgements

Thanks are due to the members of Glasgow Interactive Systems Group (GIST). In particular, Brian Mathers, Pete Snowdon, Alan Thompson all helped with the design, implementation and testing of the systems described in this paper.

References

S. Benford, J. Bowers, L. Fehlen, C. Greenhalgh, D. Snowdon, User Embodiment in Collaborative Virtual Environments, CHI'95 Proceedings, 242-249, 1995.

J. Carroll and J. Thomas, Fun, SIGCHI Bulletin, 19(3)21-24, 1988.

C.W. Johnson, Ten Golden Rules for Video over the Web. In J. Ratner, E. Grosse and C. Forsythe (eds.) Human Factors for World Wide Web Development. 207-224. Lawrence Erlbaum, New York, United States of America, 1997a.

C.W. Johnson, The Problems of Validating DesktopVR. To appear in People and Computers XIII: Proceedings of HCI'98, Springer Verlag, Berlin, 1998.

C.W. Johnson, Why `Traditional' HCI Design and Evaluation Techniques Fail With DesktopVR. In IEE Colloquium Digest 98/437, 1998a.

R.S. Kalawsky, New methods and techniques for evaluating user performance in advanced 3D virtual interfaces. In IEE Colloquium Digest 98/437, 1998.

K. Kaur, A. Sutcliffe and N. Maiden Improving Interaction with Virtual Environments. In IEE Colloquium Digest 98/437, 1998.

D. Laurillard, Rethinking University Education: A Framework for the Effect ive Use of Educational Technology, Routledge, London, 1993.

^M

M. Montgomery, Developing a Laurillardian Design Method for CAL. In Proceedings of Ed-Media '97. Association for the Advancement of Computing in E ducation, 1997.

^M

G. Robertson, M. Czerwinski, M. van Dantzich, Immersion in DesktopVR, In G. Robertson and C. Schmandt (eds) UIST'97, ACM, New York.

A. Sutcliffe and U. Patel, 3D or not 3D: Is It Nobler in the Mind? In M.A. Sasse, R.J. Cunningham and E.L. Winder (eds.), HCI'96, 79-94, Springer Verlag, London, 1996.

http://www.dcs.gla.ac.uk/~johnson.