Multimodal display and input modalities can reduce a heavy visual information processing load, as well as facilitate more natural and efficient human-computer communication. Our research looks into how multimodal interaction techniques can be incorporated into a variety of existing device interfaces. To date, work has primarily involved audio-enhancment of graphical interfaces for desktop machines; however, we are also developing audio and gesture-based interaction tools for 'eyes-free' communication with small screen devices (e.g., telephones and PDAs).

guidelines for creating multimodal interfaces

Multimodal interface design is in its infancy and there is little systematic research to demonstrate the best ways of combining sound, touch and gesturing with graphical displays and traditional input methods. An innovative aspect of our research is to produce a set of guidelines, and a toolkit based on it. Interface developers will benefit from these guidelines (and the toolkit) because they will simplify the design and incorporation of alternative modalities (principally audio) into existing interfaces. End-users will benefit because the guidelines will maximize the usability of multimodal interfaces.

Lumsden, J. and Brewster, S.A., (2001). Guidelines For Using The Audio Toolkit of Audio-Enhanced Widgets, Department of Computing Science Technical Report TR-2001-100, October, Glasgow University
Adobe PDF

earcons

Earcons, proposed by Meera Blattner in 1989, are the primitives of much of our audio interface design work. They are abstract, musical tones that can be used in structured combinations to create auditory interfacing messages. Click one of the earcons icons below (100K, each) to hear a sample.

These earcons were created as part of an early study to investigate their effectiveness. These earcons (and more examples in this HyperCard stack (400K)) are described in more detail here. More recently developed earcons and sonically enhanced widgets are available off our home page.

using earcons to represent hierarchies

In a separately funded project, Stephen Brewster and Gregory Leplatre have done experiments to investigate the use of earcons for representing hierarchical information. This has applications for telephone-based interfaces (such as phone-banking or voicemail) and blind computer users. A description of this project is available here. (There is a hypercard stack full of earcons to download too.)

spatial audio displays

The potential display space for auditory interfacing cues is very large. By contrast to the high-resolution, narrow-field senses of vision and touch, audition operates over a wide-field -- a 360 sphere surrounding the user's head. Moreover, due to the auditory system's natural ability to spatially segregate streams, high information rate (multiplexed) displays can be created within this sphere. We are mapping a number of different information sources onto spatial axes -- including iconic cues and speech rendered messages. In the case of the former, spatialized audio cues form an intermediary between existing cueing mechanisms such as auditory icons and earcons. Auditory icons -- being metaphorical sounds which are easy to interpret -- don't scale-up well. On the other hand, earcons -- which can be combined to form complex messages -- are abstract and require learning. Spatial cueing is an abstract (and extensible) form of mapping, but a simple one which is immediately perceptually salient.

gesture-based input

More immersive and immediate displays which employ the fuller dimensionality of the senses require companion input tools which employ more realistic motion. This is an important failing of the traditional 1D and 2D input devices (keyboard and pad-based mouse). Among mobile devices, these modes of input also severaly restrict the input rates, as they are high-precision and static. We are investigating a gesturing vocalulary for use within an egocentric 3D/spherical auditory bubble display space.

haptics

Given the availability of a PHANToM haptic device, we have been able to investigate the previously relatively un-explored and therefore not well understood area of haptic feedback - that is, to explore the use of touch at the human-computer interface. This exploration has led to the first set of guidelines for the use of haptics in multimedia user interfaces. The advantage of haptic feedback is that it allows a designer to potentially stop uesrs from making errors whereas with sound it is only possible to warn users if an error has taken place.