Why `Traditional' HCI Techniques Fail to Support DesktopVR

Chris Johnson,

Glasgow Interactive Systems Gro up (GIST),
Department of Computing Science,
University of Glasgow,
Email: johnson@dcs.gla.ac.uk


This paper argues that `traditional' Human Computer Interaction (HCI) techniques, such as hierarchical task analysis and iterative development, do not support the design of desktop virtual reality (desktopVR). If this problem is not addressed then users will continue to be presented with superficially pleasing but essentially useless applications of this technology.

1. Introduction

DesktopVR creates a huge opportunity for the development of innovative user interfaces. For example, Figure 1 shows how the Virtual Reality Markup Language (VRML) can be used to construct a three dimensional model of a Museum. This enables curators to use virtual space to group multimedia exhibits in the same way that phyiscal positions are used to group objects from the same time period or culture within a conventional museum.

Figure 1: VRML Model of the Hunterian Art Gallery

Similarly, Figure 2 shows how the photorealistic renderings of QuicktimeVR can be used to provide a 360 degree view of the interior of a Boeing 757. This application enables fire crews to familliarise themselves with the layout of the aircraft without grounding large numbers of planes during frequent training exercises.

Figure 2: QuicktimeVR Image of Boeing 757
(Courtesy of Strathclyde Regional Fire Brigade)

This optimism about the effective use of desktopVR must, however, be balanced against the many applications that provide only limited benefits to their users. For example, the Internet provides literally thousands of VRML models that do little more than illustrate the capabilities of the language. Many applications provide inaccurate models of physical artefacts. Most support only the most trivial of user tasks.

2. The Limitations of Task Analysis

The strong initial appeal of desktopVR interfaces will only be sustained if they provide long-term benefits to their users' tasks. Fortunately, a range of techniques have been developed to help interface designers ensure that interactive systems actually do fulfill their users' needs. Unfortunately, most of these approaches fail to support the development of desktopVR applications.

The first problem with conventional task analysis is that it can be difficult for designers to identify the intended user population for a VRML or QuicktimeVR application. The accessibility of desktopVR is often a key motivating factor in the customers' decision to exploit this technology. For instance, the potential users of the virtual Museum in Figure 1 were identified as: school children; curators; computing scientists; historians and archaeologists. The Boeing 757 system of Figure 2 was initially developed for firefighters. It was later proposed as a more general training tool for aircrews, for airline advertising and aircraft maintenance teams. These diverse groups differ in many different ways: learning style; education qualifications; previous exposure to computer systems; average age.

Even if designers agree upon their intended user population, it can be difficult to identify meaningful tasks. For instance, Figure 2 was intended to help members of the Fire Brigade to learn where exits were located on the 757. It is unclear how this high level requirement could be represented within conventional task analysis. The sub-task for browsing the aircraft might trivially be identified as browsing some component of the fuselage. This problem occurs because DesktopVR has opened up new forms of declarative, knowledge acquisition. It provides little or no support for the proceduralised tasks that have dominated previous generations of form and menu based interfaces.

3. The Limitations of Formative Evaluation

The problems of task analysis are exacerbated by the difficulty of conducting iterative development with desktopVR. It is difficult to obtain direct user feedback through `think aloud' protocols. For instance, the following excerpt typifies the responses of users interacting with the multimedia exhibit shown in Figure 1. In this case the user had been ask to compare the VRML presentation with a conventional web-page of Museum exhibits:
Evaluator:	Do you prefer pictures or the models?
User: 		The models are great.
Evaluator:	Why?
User: 		Um, it just feels different.
This analysis is confirmed by the results of post task questionnaires. For example, the following graph presents the results of asking a class of school children for their opinions of QuicktimeVR within a Museum web site. The Y-axis records the number of users responding to a particular category within a checked list. The proportion of children who preferred not to state an opinion is striking.

Figure 3: Questionnaire Responses to the Introduction of QuicktimeVR within Museum Web Pages

The questions in Figure 3 were not intended to obtain statistically significant and replicable results. They were simply intended to provide some insight into whether or not the users supported our introduction of desktopVR. The large number of uncommitted responses to such fundamental questions can be extremely disappointing. Gibson provides some insight into the source of this disappointment:

`` Not only do we perceive in terms of visual information, we can also think in those terms. Making and looking at pictures helps us to fix these terms. We can also think in terms of verbal information, as is obvious, and words enable us to fix, classify and consolidate our ideas. But the difference is that visual thinking is freer and less stereotyped than verbal thinking: there is no vocabulary of picturing as there is of saying.'' (Gibson, 1971).

If Gibson is correct then there are important consequences for the iterative development of desktopVR. Users may simply lack the vocabulary to directly participate in the formative evaluation of VRML and QuicktimeVR systems.

4. The Limitations of Summative Evaluation

Given the problems of identifying meaningful tasks with desktopVR, it follows that it will be difficult to determine whether or not an interface actually supports those tasks. For example, the ability to navigate an information space is often cited as an important high-level objective for VRML and QuicktimeVR interfaces. Summative evaluations might measure this by identifying two key locations within the virtual environment; designers might then record the time taken for users to move between these two points. Unfortunately, this assumes that an optimal route can be defined in terms of the time taken to move between two points. It completely ignores the fact that a slower, more prolonged route may introduce users to more key concepts and ideas than a shorter path.

Further problems arise if designers attempt to measure more subjective qualities of desktopVR. For example, high levels of motivation and `fun' are often cited as an important strength of this new generation of interactive systems. For example, the activist learning style of most firecrews and the low success rate with conventional `lecture-based' courses was cited as a key reason behind the development of the system illustrated in Figure 2. It can, however, be difficult to identify appropriate means of measuring `fun'. Longitudinal approach are necessary to determine whether high levels of satisfaction are sustained through repeated use of a VRML or QuicktimeVR interface. Unfortunately, previous studies have not conducted prolonged investigations into the subjective appeal of desktopVR (Johnson, 1998).

5. A Way Forward Through Contextual Design and Evaluation

Given the problems with task analysis as well as formative and summative evaluation, it is hardly surprising that there are so many poor applications of desktopVR. In the long term it seems likely that many of these problems will be resolved through the development of appropriate extensions to existing interface design techniques. For example, Mary Czerwinski and the 3D User Interface Group at Microsoft are pioneering methods that support both the formative and summative evaluation of desktopVR. In anticipation of the results of this research, designers must exploit existing methods that minimise the problems mentioned above. For example, contextual design techniques avoid the need to explicitly focus upon particular user groups, early in the development cycle. Designers are encouraged to consider the diverse and conflicting requirements of many different people. Design progresses not by establishing the primacy of one group of users but by identifying, and hopefully resolving, the conflicts between diverse user groups.

Contextual design techniques also avoid some of the problems that restrict the utility of task analysis for desktopVR. These techniques assume that designers can seldom predict the full diversity of tasks that users will find for their applications. The identification of these tasks is only possible through the observation of users operating similar tools within their daily lives (Johnson, 1998). Finally, contextual evaluation techniques focus upon the interpretation of an artefact within its working context. Designers are not forced to identify and pursue summative performance measures that provide an impoverished view of desktopVR applications.

6. Conclusion

This paper has argued that `traditional' HCI techniques cannot easily be used to inform the development of desktopVR. Task analysis fails because it can be difficult to predict and characterise the intended users of these applications. This, in turn, makes it difficult to identify and prioritise the tasks that these users will perform. Similarly, formative evaluation techniques provide only limited support for the iterative development of desktopVR. Many users find it difficult to explain what contributes to a successful 3D interface. Finally, it is hard to find valid statistical measures that support the summative evaluation of desktopVR. More subjective measures of satisfaction rely upon longitudinal techniques that have still to be applied to desktopVR.

The penultimate section of this paper has briefly argued that contextual design techniques avoid the limitations of more `conventional' approaches. They encourage designers to consider heterogeneous user populations with diverse and ill-defined tasks. They focus upon the analysis of an interface within the working environment rather than upon precisely defined statistical measures. Initial attempts to apply these techniques have proved to be very useful for the examples used in this paper. More evidence is required.

The urgent need to develop suitable HCI techniques for both desktopVR applications and their browsers is illustrated by the following excerpt from a think aloud with the Museum site illustrated in Figure 1:

Evaluator:	How are you trying to find the exhibit?

User:		I'm looking for the exhibit from the picture and trying to move 
		towards the area that seems most relevant...this area bears no 
		relation to where I wanted to go....

It is an indictment of HCI research and practice that the user could be be talking about almost any interface to a desktopVR system.

Further discussion of this work is provided here


Thanks are due to the many students who have helped in the design, construction and evaluation of the desktopVR applications that are mentioned in this position statement. In particular thanks are due to: James Birrell; Karen Howie; Anthony McGill; James Macphie; Bryan Mathers; Pete Snowden and Mike Waters. I am very grateful to the staff of the Hunterian Museum and Art Gallery, Glasgow for providing technical advice during the development of the VRML and QuicktimeVR resources that are mentioned in this paper. Similarly, Bill West and the staff of Strathclyde Regional Fire Briagde guided the application of QuicktimeVR shown in Figure 2.