Crossmodal Interaction

Using Auditory and Tactile Displays In Mobile Devices

The design of a user interface for mobile devices, such as PDAs and mobile phones, is a challenging task given their small size and restricted input/output capabilities. Due to the lack of screen space, both the graphical user interface and the amount of information able to be presented are limited. This has resulted in displays with small text which is difficult to read, cramped graphics and little contextual information. Anyone who has attempted to view small icons like the clock or read a message whilst in a poorly lit environment or sitting in a moving train can confirm that such visual output can place considerable demands on the user. When attempting to interact with these small screens, the user's attention is diverted from the rest of the physical world. There are many environments and activities, such as walking, in which the user's eyes may be occupied although they are otherwise able to attend to information from the mobile device via their other senses.

Being predominantly dependent on a single sense such as vision is unnatural because, in the real world, we receive information from several modalities, as when we both hear and see someone speaking. Many of the same issues affect blind people too. Sensory substitution is used everyday by those who are unable to perceive information via a certain modality. For instance, visually impaired people regularly use alternative senses to gain the information usually obtained through vision. For example, Braille provides people with information through touch while human echolocation is a technique where people can navigate by listening to the echo of sounds. It is proposed that these crossmodal interactions can be used to influence and improve the design of mobile device interfaces. Crossmodal interfaces have the potential to significantly improve the accessibility and flexibility of interaction by offering multiple modalities through which information may be passed between the device and user.

Interaction through modalities, other than vision, is now becoming an option in mobile devices. For example, mobile phones, PDAs, and pagers all feature audio and vibrotactile output. However, the vibrations and audio alerts used in these devices usually contain limited amounts of information. So, the time is right to start thinking about ways in which crossmodal use of these features may improve interaction by exploiting the potential of both audio and vibration as methods of informative feedback.

In this project we are taking an alternative approach to interface design for mobile devices by creating crossmodal interfaces based on sound and touch which may be advantageous to mobile device users because different modalities may be more or less suitable depending on the user, the type of information received, hardware capabilities, the user's current activity, and the environment. For example, a user may not want their phone to make a noise in a quiet environment like a library or workplace so a tactile cue may be more appropriate. However, in a situation where the user is not in direct contact with their phone, tactile cues could go unnoticed so using the auditory modality may be more appropriate.

What Is Crossmodal Interaction?

Crossmodal interaction relates to both synesthesia and sensory substitution. Recent empirical research in cognitive neuroscience has given rise to the notion of crossmodal attention, a term used to refer to capacities and effects involved in the process of coordinating or matching the information received through multiple perceptual modalities [1]. Studies reveal extensive crossmodal links in attention across the various modalities (i.e. audition, vision, touch and proprioception). Unlike multimodal interaction where each modality is used to transmit a different type of information e.g. audio for alerts and vision for graphical data, crossmodal interaction uses the different modalities to present the same data. Crossmodal use of the different senses allows the characteristics of one sensory modality to be transformed into stimuli for another sensory modality [2].

The crossmodal interactions of different sensory inputs have advantages because they can help to overcome a specific sensory deprivation or situational impairment (e.g., auditory and tactile senses in darkness) and they can also reduce perceptual ambiguity (for example, tactile and olfactory senses prevent us from eating a plastic replica of food).

Audio and tactile displays are ideal candidates for crossmodal use because our senses of hearing and touch share several important similarities, in particular their temporal characteristics and their ability to perceive vibrations. Moreover, sounds are often described in tactile terms. Mursell [3] observed that tones can contain tactile values as can be seen when we describe a tone as hard or soft, rough or smooth, wooden or metallic. It has been suggested that the more properties shared between two modalities, the stronger will be the observer's unity assumption that information from different sensory channels can be attributed to the same distal event or object [4].

An attribute that can communicate comparable information across modalities is considered to be amodal. Mendelson [5] provided a scheme or list of such amodal properties. These amodal properties relate to space and time and involve points along a continuum (e.g. location), intervals within continuum (e.g. duration), patterns of intervals (e.g. form), rates of patterns (e.g. tempo), or changes of rate (e.g. texture gradients). Other crossmodal correspondences such as numerosity or intensity also have been examined.

The auditory/tactile crossmodal interaction design described here is based on the amodal attributes available in the auditory and tactile domains which include intensity, spatial location, rate, texture, and rhythmic structure [6].

Crossmodal Icons

In order to explore the possibilities of crossmodal auditory/tactile output, we have introduced the concept of the crossmodal icon. Crossmodal icons [7] are abstract icons which can be automatically instantiated as either an Earcon or Tacton, such that the resultant Earcons or Tactons are intuitively equivalent and can be compared as such. Crossmodal icons allow the same information to be accessible interchangeably via several different modalities.

For example, a set of Earcons/Tactons can be transformed into crossmodal icons by ensuring that the information represented can be encoded in both modalities so that users can move from an audio to a tactile presentation of the same information.

Potential Uses

There are many potential uses for crossmodal auditory tactile displays. This research will concentrate on possible uses in mobile/wearable devices.

Context Aware Mobile Devices : tactile perception may be reduced when a person is engaged in another physical activity, while audio cues may be masked by environmental noise. Also, given the personal nature of mobile computers, the interface must be adaptable by adjusting to the individual preferences and habits of its user. An interface incorporating crossmodal icons could present audio and/or tactile cues depending on the situation and adapt by permitting easy movement between modalities.

Widgets: as mentioned earlier, mobile devices often have cluttered displays due to the lack of screen space. Crossmodal features could be added to buttons, scrollbars, and menus, etc. so that information about those widgets can be presented non-visually. This would allow the widget size to be reduced (or even removed from the screen) and allow more information to be presented on the display.

Displays for Visually Impaired People : since mobile devices primarily provide output via the visual modality, visually impaired people have limited access to this information. Crossmodal icons could improve the interaction between visually impaired users and mobile devices by providing alternative channels through which this information may be displayed.

References

[1] Driver, J. & Spence, C. "Attention and the crossmodal construction of space" In Trends in Cognitive Sciences, 2(7), (1998), pp. 254 – 262.

[2] Lenay, C. Canu, S., and Villon, P. "Technology and perception: the contribution of sensory substitution systems" In 2nd International Conference on Cognitive Technology (CT '97), (1997), pp. 44.

[3] Mursell, J.L. "The psychology of music" W.W. Norton and Company, Inc, New York (1937).

[4] Adelstein, B.D., Begault, D.R., Anderson, M.R., and Wenzel, E.M. "Sensitivity to haptic-audio asynchrony" In Proc. 5th International Conference on Multimodal Interfaces, ACM Press (2003), pp. 73-76.

[5] Mendelson MJ. "Acoustic-optical correspondences and auditory-visual coordination in infancy" In the Canadian Journal of Psychology 33, (1979), pp. 334–346.

[6] Lewkowicz, D.J. "The development of intersensory temporal perception: an epigenetic systems/limitations view" In Psychological Bulletin, 126, (2000), pp. 281-308.

[7] Hoggan, E. and Brewster, S. "Crossmodal icons for information display" In Proceedings of CHI 2006, Canada, vol. II, ACM Press (2006), pp. 857-862.