Taking Fun Seriously:

Using Cognitive Models to Reason about Interaction with Computer Games

Chris Johnson

Department of Computing Science,

University of Glasgow, Glasgow G12 8QQ.

Fax: 0141 330 4913, Telephone: 0141 330 6053,

Email: johnson@dcs.gla.ac.uk, http://www.dcs.gla.ac.uk/~johnson

This paper extends techniques from affective psychology to show how cognitive models can be used to represent and reason about interaction with computer games. It is argued that this modeling activity provides insights into the motivational appeal that often distinguishes computer games from other forms of human computer interaction. The long-term aim behind this research is to use our improved understanding of interaction with computer games to inform the subsequent development of more general classes of interactive systems. Barnard's Interacting Cognitive Subsystems (ICS) is used throughout this paper. This decision is justified by the fact that ICS has already been applied to analyze the negative emotions surrounding clinical depression. This previous work provides a useful starting point for our investigation of the more positive emotions evoked during interaction with computer games. A further justification is that ICS has also been successfully used to represent and reason about the design of human computer interfaces.

Keywords: computer games; user modeling; fun.

1. Introduction

The last decade has seen a considerable amount of research into the psychology of play (Rieder et al, 1998) and the motivational appeal of computer games (Pausch et al, 1994). For instance, Neal (1991) identifies factors that contribute to successful computer games: a sense of control; the opportunity for strategy; the discovery of information. This research is significant for interface designers because motivation and play are becoming increasingly important aspects of human computer interaction. For example, desktopVR interfaces help to motivate users through exploration and discovery (Robertson et al, 1997). Microsoft Channel technology has created a range of Internet services that are intended both to entertain and to inform. Unfortunately, the work of Neal and others into the successful features of computer games has had relatively little impact upon interface development (Laurel, 1993). This paper, therefore, uses cognitive to apply recent research into motivation and play and support the design of human computer interfaces. This is appropriate because cognitive models have been used to both explain empirical studies of emotion and have also supported the design of human computer interfaces (May and Barnard, 1995).

2. Computer Games and Interface Design

A number of researchers have argued that successful features of computer games can support other forms of human computer interaction (Carroll, 1997). The interface on the left of Figure 1 provided users with access to a range of web pages about exhibits in the Hunterian Museum, Glasgow. The map shows the user standing in front of the exhibit that is pictured in the photograph. Both views are updated after each movement. This combination of a map and a direct image is based on id Software's Doom. Unfortunately, a number of problems prevent designers from further exploiting surface level features of computer games. For example, numerical scores often provide users with direct information about their progress with a game. It is difficult to see how this might be extended to text editors or spreadsheets (Pausch et al, 1994).

Figure 1: Learning From Successful Games

High level heuristics provide a more generic means of exploiting the strengths of computer games (Malone, 1987). For instance, Neal (1990) encourages designers to support "a sense of control" and the "opportunity for discovery" rather than surface level features, such as numerical scores. This approach is illustrated by the CAL system on the right of Figure 1. The interface supports the users' sense of control and discovery by enabling them to choose a path through multiple, redundant sources of information. However, a number of problems limit the utility of such heuristics as a means of transferring successful features of computer games into other forms of user interface. They can be difficult to apply. Requiring that interfaces support a "sense of control" is of little benefit if designers cannot identify the means of satisfying such a requirement. Further problems complicate the use of heuristics to guide interface development. High-level guidelines can obscure the underlying psychological factors that influence the success of a computer game (Rieder et al, 1998). Stating that an interface should provide a "sense of control" tells us little about the cognitive and perceptual factors that encourage such feelings about an interactive system.

3 From Affective Psychology to Cognitive Models

Cognitive modeling techniques provide means of propagating the successful features of computer games, as indicated by recent work in affective psychology (Rieder et al, 1998), into other forms of human computer interaction. For instance, Bonnie John and Alonso Vera (1992) used GOMS to analyze interaction with a number of video games. However, this approach neglects physiological factors. It also takes a functional view of interaction that ignores the role of affect and emotion during playing. Finally, the focus upon expert performance in GOMS also prevents an effective analysis of why many users do not like computer games. In contrast, Teasedale and Barnard (1993) have used the Interacting Cognitive Subsystem (ICS) to analyze the symptoms and causes of depression. This provides good clues about the negative emotions that can arise when users become disillusioned with role playing applications, shoot-em up systems and simulated environments. One of the further motivations for using ICS is that it considers both the physiological and cognitive state of the user. This is important because computer games, typically, impose both physiological and cognitive demands on their users. They, typically, require good hand-eye coordination as well as strategic thinking in order to obtain high scores.

Cognition is represented in ICS as the flow of information between a number of different subsystems. Each of the subsystems has associated with it a unique mental code in which it represents the information it receives and processes. Each subsystem transforms its data output into the corresponding mental code of the subsequently receiving subsystems. The nine subsystems can be grouped into four categories. The sensory subsystems include visual, acoustic and body-state components. The effectors deal with articulatory and limb movement. The structural subsystems include object elements controlling mental imagery as well as a morphonolexical element dealing with words and other lexical forms. Finally, the meaning subsystems deal with both propoisional and implicational encodings; these will be explained in more detail in later sections.

3.2 Using ICS to Model Depression

ICS provides a link between cognitive modeling and recent research into affection. It, therefore, provides useful analytical tools for understanding the motivational benefits that computer games provide for other forms of human computer interaction. Figure 3 shows how Teasdale and Barnard have used ICS to model the negative mode of thought that leads to the internal self-regeneration of depressive thoughts. The links between the user's body state and their implicational subsystem suggest that inactivity and a beaten posture may lead to thoughts about previous failures which, in turn, further affect the individual's body state.

Figure 3: Modelling depression in ICS (Teasdale and Barnard, 1993)

Teasdale and Barnard's work on depression can be used to explain the unsuccessful attributes of computer games. In particular, Figure 2 corresponds to a breakdown in the sense of flow identified by Csikszentmihalyi (1990) as a critical aspect of game playing activity. Flow theory provides an account of the state of happiness and satisfaction that people experience when they are "carried" by an activity that is automatic and spontaneous. This sense of flow is a frequent, everyday occurrence but requires deliberate effort and attention. It shares many common characteristics with Neal's analysis of computer games (Rieber, Smith and Noah, 1998). It is characterized by optimal levels of challenge, feelings of complete control, attention focused so strongly on the activity that feelings of self-consciousness and awareness of time disappear. As levels of skill increase, users maintain this sense of flow by increasing the level of difficulty associated with their task. If this level of difficulty and absorbtion cannot be increased then users quickly become bored and disillusioned with the game.

The previous paragraphs have shown how work in affective psychology can be used to represent and reason about the negative aspects of interaction with computer games. Work on the self-regenerative nature of depression can be used by analogy to model the cycle of interaction between low expectations and poor performance with computer games. The following paragraphs build on this analysis and move away from Teasdale and Barnard's work to analyse the more positive emotions that arise when people play with these applications.

4. Using ICS to Model Successful Interaction with Computer Games

This section demonstrates that ICS can be used to model three different cognitive aspects of successful interaction with computer games: maintenance of mood by external sensory input; mood maintenance by internal body state and the interaction of anticipation, imagination and tactical thought. This is not intended to be an exhaustive taxonomy. However, these examples do illustrate the potential power of ICS in this area. For instance, maintenance of mood by external sensory input describes the ways in which games designers exploit audio and visual feedback to maintain a sense of excitement during interaction. In contrast, mood maintenance by internal body state describes the ways in which the users' previous experiences with computer games can generate a sense of enjoyment during interaction with new applications. Finally, the interaction between anticipation, imagination and tactical thought describes the way in which skilled games players exhibit more sophisticated tactics by anticipating or "imagining" the potential consequences of their actions, and those of the game in response to their actions.

4.1 Maintenance of Mood by External Sensory Input

In the ICS model, emotions and moods are principally explained in terms of the implicational subsystem. Users form schematic representations that represent general characteristics of situations that elicit particular emotions. These situations are formed from the visual and acoustic information available in the environment, for example, when a user plays a game. They also include internal sensory input about the users' body-state; for example an alert posture may be associated with feelings of excitement. Over time, the occurrence of similar sensory inputs in similar emotion provoking situations will lead to a reinforcement of these schematic representations. Within the implicational subsystem, propositional codes will be developed to reflect the underlying interpretation or "meaning" of situations that elicit a particular emotional response. This propositional encoding enables users to find the same emotive "theme" behind situations that share very few sensory features.

ICS models can explain why games exploit visual and auditory feedback to create a direct relationship between sensory input and the implicational representations that determine the meaning of those signals. Rewards are clearly indicated and penalties are presented in a manner that is difficult to avoid. Progress is indicated by numerical scores but also through the pitch and rhythm of music and even the colors in the display. These presentation techniques have a direct effect on the emotional state of the player, increasing excitement; communicating feelings of success or failure. As the previous paragraphs suggested, the interpretation of external stimuli depends upon the repeated association of those stimuli with particular moods. However, similar moods can be associated with different stimuli. This explains why players may obtain the same emotional reaction in very different situations. It also explains why some people fail to achieve an emotional reaction to the same set of stimuli that enthrall other players.

The previous paragraph also help to explain particular feelings of success and failure during interaction with computer games. For instance, an experienced player would, over time, develop a schematic representation of the various feedback mechanisms that indicate the successful completion of a stage or level in a game. These changes in the display help to trigger the extraction of the implicational encoding whenever the user moves on to the next level. This implicational schema can lead to an associated proposition reflecting "arrival at goal". Processing of this implicational pattern will be associated with "being successful". Such feelings have profound effects upon the user's subsequent interaction through the continuation of moods.

Figure 4: Maintenance of mood by visual and auditory input.

Moods are important for any emotional analysis of games because they persist longer than the emotional reaction to specific events. A sense of success must continue beyond the individual setbacks and failures that provide the challenge and motivation for many games players. If this sense of success does not continue beyond the periodic setbacks then users may simply stop playing the game and may stop buying subsequent titles in the same series. This style of analysis creates problems for the ICS model developed up to this point. The implicational subsystem processes the propositional meaning of particular schemas. These schemas are characterized by particular patterns of external and internal sensation. This does not account for the persistence of moods across particular patterns of sensory input. In order for a mood to continue, therefore, the schematic models related to that affect have to be "regenerated" over time. In effect, there has to be some form of feed-forward between previous emotional states and the future interpretation of internal and external sensations. Figure 4 illustrates one way in which moods persist over time. Visual and auditory inputs can be sustained to continue the stimuli associated with particular emotions. For example, music can be used to indicate the level that a user has currently reached. If this theme continues playing during interaction then the user is presented with a continuing reminder of their achievement in having reached a particular point. Location and setting are used in a similar manner within most adventure games. Reaching a higher level is continually indicated by the environment, city, jungle or sea, that the user interacts with. The converse is true if the player is returned to a lower level. The feeling of success diminish as the player's mood changes.

4.2 Maintenance of Mood by Internal Body State

In ICS, the body-state encodes types of stimulation such as cutaneous pressure, temperature, muscle tension. It also encodes the location and intensity of these stimulants. Teasdale and Barnard (1993) note that this "proprioceptive sensory input is not as dependent on continuing environmental sensory input. In addition to any external taste, smell or pain stimuli, implicational elements derived from Body-state sources within the person can contribute to the internal regeneration of mood states". This regeneration is sketched in Figure 5.

Figure 5: Using ICS to Model Mood Maintenance through a Body-State Feedback Loop

The users' body state can affect the implicational subsystem and its interpretation of external and internal stimuli. The interpretive processes within the propositional subsystem, mentioned above, will in tun affect the body-state of the user. This feedback loop depends upon repeated and sustained exposure to similar affect-related schemas. In other words, the transformation process that links the body-state to the implicational subsystem must learn to generate codes that affect the future interpretation of external and internal stimuli. In particular, for moods to be sustained, this transformation process must "guide" the implicational subsystem towards interpretations that produce the same emotional responses that originally led to the proprioceptive feedback. This can be illustrated by an example from game playing. If a user were faced with multiple hazards on their display and the game emitted rapid rhythmic sounds then this would result in an implicational schema that relates to excitement. The propositional interpretation of these sensations would be one of approaching danger or threat. All of this would result in a range of proprioceptive effects including increased heart rate, a possible rise in temperature, tightening of the facial muscles and changes in the user's posture. These proprioceptive changes, in turn, make it more likely that any further changes to the display will sustain the users' feeling of excitement as their body state makes it more likely that subsequent changes to a display will be regarded as a continued threat. A common observation amongst games players is that tension is sustained for some period even after they have avoided an immediate threat or achieved a short-term goal.

Figure 5 also illustrates the key role that the VISC and SOM "codes" have in the ICS account of emotion in game playing. SOM represents the somatic output of the implicational subsystem. It captures the motor-expressive elements of emotional reaction including both facial orientation and body posture. Given an appropriate pattern for the Implicational code, ICS describes a transformation process that will produce a coordinated pattern of responses for the somatic musculature. . VISC represents the visceral output of the implication subsystem. This represents the autonomic and physiological components of an emotional response. Given an appropriate input pattern from the Implicational code, this process produces instructions for a coordinated pattern of autonomic response. Brevity prevents a detailed analysis of both of these aspects of games playing. It is important to note, however, that successful interaction with many computer games depends not only on the well considered application of strategic and tactical planning but also on the visceral reactions to perceived threats and opportunities. Similarly, initial field-studies of computer games players have shown a much higher frequency of extreme somatic effects than can be witnessed in more conventional forms of game playing. In team sports, such as football or hockey, these effects are typically concentrated around relatively infrequent attempts on goal. In games, such as chess or bridge, users may even attempt to mask any changes in facial muscles or posture during critical sections of play. If our analysis is correct then the high frequency of somatic changes in computer games both reflects the emotional impact of many of these applications and also helps to sustain those effects during more prolonged interaction.

4.3 The Role of Anticipation, Imagination and Tactical Thought

The ICS model is built around two basic types of operation. The first of these transforms the mental encoding of information. Previous paragraphs have already provided several examples of this form of operation. Figure 4 shows how external and internal sensory information is transformed into schematic representations within the implicational subsystem. These schematic representations may be transformed within the propositional subsystem to determine the meaning or interpretation of such schematic representations. In this way, a user's physical experience of interacting with a game may be "transformed" into a propositional representation that encodes danger or threat. This feeling of threat, in turn, may be transformed or associated with schemas that encode excitement at an implicational level.

The COPY operation is the second basic process in ICS. This replicates the encoding within a subsystem and stores it for future recall within that system. There is one of these specialized stores for each of the nine subsystems within ICS. The COPY operation is fundamental because it provides an episodic record of recent patterns encountered within a subsystem. It also supports specific databases of patterns that recur within each subsystem. For example, the COPY operation provides a means of storing the implicational schemas that are associated with recurring patterns of sensory input. This is the critical stage in the mechanism by which users "learn" to associate particular display characteristics with a sense of progress or achievement, see Section 5.1. The COPY processes, therefore, form the basis of subjective experience. They record the mental codes that were derived from previous experiences. These records can, in turn, be retrieved to influence subsequent transformations within the various subsystems. The COPY operation provides a means of buffering the input that is provided to the transformation processes. The impact of this buffering depends upon the particular subsystem in question. For example, Figure 6 uses a black triangle to denote the use of the buffered transformations within the Object subsystem. The Object subsystem encodes abstract structural descriptions of entities and relationships in visual space. The introduction of this subsystem has important consequences for the feedback loops that were discussed in previous sections. Its inclusion enables the configuration to draw upon specialized information processing skills that the Object subsystem has acquired over successive interactions.

Figure 6: Heightened Awareness of Visual-Object Based Imagery

The COPY operation in the OBJ-PROP transformation of Figure 6 represents particular skills that the user may have developed in the extraction of meaning from complex patterns of spatial information. In many game-playing situations, these skills provide more effective means of transforming sensory input than the information processing of the implicational and propositional subsystems, described in previous sections. For instance, if an expert user were interacting with a game of strategy, such as SimCity, then they might anticipate the consequences of their intervention by transforming the relevant propositional information into patterns of the Object code. These encodings together with the image of the existing situation can be used as input to a propositional transformation that will yield the meaning of these potential future states in the context of the existing situation. In other words, the object subsystem helps to explain why expert users are better at "envisioning" the consequences of their interaction than novice games players.

5 Using ICS to Support Design

ICS not only provides means of modeling interaction with computer games, it also provides constructive tools for interface design. For example, a cognitive model of interaction with a computer game can be compared with a similar model for a proposed interface to determine whether the interface preserves the successful features of the game. The previous analysis in the paper has identified a number of these successful feature.

  1. The link between internal and external sources of sensory information and the implicational subsystem will lead to associations being formed between sensory encodings and a particular emotion. Previous experience with computer interfaces, including games, will already have established some of these associations. It, therefore, follows that designers can use interface elements not only to convey information but also to convey mood and emotion (see 2);
  2. Games exploit visual and acoustic cues to provide direct indications about the user's progress over time. These progress indicators rely upon users have an appropriate implicational and propositional encoding. Over time, these subsystems may COPY schematic representations of these indicators that may, in turn, become associated with emotions of success or failure. Therefore, the introduction of these indicators into other aspects of a game may be sufficient to trigger the emotional response that has been learnt during previous interaction. The implications for interface development are clear. Feelings of success or failure may be triggered by the presentation of interface features that have become associated with previous successes or failures in other contexts of interaction;
  3. Given that the emotional triggers, mentioned in 2, depend upon an appropriate encoding within the implicational and propositional subsystems, these techniques will not produce the same emotional response in all users. In particular, novice users may not have formed the relevant schemas within the implicational subsystem. This has implications for more general aspects of interface design. In particular, some common schemas are acquired from an early age. Fast moving visual cues can communicate a sense of threat. Similarly, certain tones and melodies can be used to evoke more relaxed responses;
  4. It is important to emphasize that the COPY operation determines subjective experience. As this operation is available within each of the subsystems, the triggering of an emotional response is more complex than the recollection of information from long term memory. Simply introducing visual and acoustic cues can neglect the other sensory information, such as posture, smell and taste that also affect an individual's response to a particular application. Such insights have strong links with situated approaches to interface design. The emotional response to interaction is not simply a property of particular interface features. It is the product of complex internal sensations that are affected by a myriad of external, contextual factors including heating, lighting and noise levels.
  5. Proprioceptive sensory input is determined both by external sensations but also by internal implicational elements. This finding has important consequences for the design of desktopVR interfaces that share many surface features with three dimensional games. However, the initial enthusiasm for such presentation techniques is hardly ever sustained as it is in games. VRML interfaces to information retrieval systems fail to provoke the same emotional response as Doom or Riven. One explanation might be that these more general interfaces lack the feedback loops, for instance between the body state and implicational subsystems, that have been identified as a critical means of mood regeneration in affective psychology.

The elements in this list resemble the heuristics that were cited in Section 2. These were criticized as being too vague. They can also be difficult to apply to constructive interface design. For instance, it can be difficult to establish whether or not a particular interface actually does support "the users' sense of control" as Neale advocates. However, unlike these heuristics, ICS provides a clear means of validating the list given above. In particular, the nine subsystems provide a common vocabulary that can be used to precisely describe the intended effects of particular interface feature.

6. Conclusions and Further Work

This paper has applied techniques from affective psychology to show how cognitive models can be used to represent and reason about interaction with computer games. This modeling provides insights into the emotional appeal that often distinguishes computer games from other forms of human computer interaction. The long-term aim behind this research is to use our improved understanding of interaction with computer games to inform the subsequent development of more general classes of interactive systems. Barnard's Interacting Cognitive Subsystems (ICS) has used to support the argument in this paper. ICS is an appropriate choice because it has already been applied to analyze the negative emotions surrounding clinical depression. This previous work provides a useful starting point for our investigation of the more positive emotions evoked during interaction with computer games. A further justification is that ICS has also been successfully used to represent and reason about the design of human computer interfaces.

It is important to emphasize, however, that there are some drawbacks with ICS. It is well suited to represent and reason about an individual's interaction with computer games. However, ICS provides little support for analyzing the group interaction that is a growing feature of many team-based computer games. Of course, such distributed interaction could be analyzed in terms of a number of ICS models; each representing a single user's perspective. It is hypothesized, however, that approach would quickly become intractable and may even hide certain aspects of group cognition. This remains a topic for future research. ICS is a subtle and expressive modeling tool. Unfortunately, it can be too "low level" for many design activities (Busse and Johnson, 1998). Similarly, many of the examples in this paper have been grounded by reference to previous laboratory studies into the cognitive psychology of games. These studies, typically, provide insights into the more "holistic" cognitive features of game playing. They provide few clues about the effects of interaction within individual subsystems. On the other hand, it is difficult to construct a coherent account of these disparate studies unless one pitches the level of analysis in terms of the more detailed subsystems within ICS. Unless we have a detailed vocabulary with which to express and debate the cognitive features of interaction with computer games then we stand little chance of understanding why these artifacts have such appeal.

References

L. Berkowitz, Implications of the Cognitive-Neoassociationistic Perspective In R. Wyer and T. Srull eds. Advances in Social Cognition, Lawrence Erlbaum, Hillsdale, 1993.

D. Busse and C.W. Johnson, Modeling Human Error in a Cognitive Framework, In F.Ritter and R. Young eds. 2nd Conference on Cognitive Modeling, 90-98, Nottingham Univ. Press, 1998.

J.M Carroll, (D)riven, Wired, (5)9:120-127, 1997.

M. Csikszentmihalyi, Flow: The Psychology of Optimum Experience, Harper and Row, New York, 1990.

M. Igbadia, S. Schiffman and T. Wieckowski, Respective Roles of Perceived Usefulness and Perceived Fun, Behaviour and Information Technology, (13)6:349-361, 1994.

B. John and A. Vera, GOMS Analysis of a Graphic, Machine-Paced, Highly-Interactive Task Modeling the Expert User, In Proceedings of ACM CHI'92, 251-258, 1992.

C.W. Johnson, The Problems of Validating DesktopVR. In H. Johnson, L. Nigay and C. Roast (eds), Proceedings of HCI'98, 326-338, Springer Verlag, Berlin, 1998.

C.W. Johnson, Why Traditional HCI Design and Evaluation Techniques Fail With DesktopVR. In IEE Collquium Digest 98/437, 1998a.

B. Laurel, Computers as Theatre, Addison Wesley, Reading, 1993.

J. May and P. Barnard, A Cognitive Task Analysis of the CERD Exemplar, Working Paper UM/WP23, MRC Applied Psychology Unit, Cambridge, 1995.

T.W. Malone, Heuristics for Designing Enjoyable User Interfaces: Lessons from Computer Games. Human Factors in Computer Systems, 63-68, 1982.

L. Neal, Implications of Computer Games for Systems Design. In D. Diaper, D. Gilmore, G. Cockton, B. Shackel (eds.) Interact'90, Elsevier Science, 93-100, 1990.

R. Pausch, R. Gold, T. Skelly, D. Thiel, What Interface Designers Can Learn from Video Games Designers. CHI'94, Volume 2, 411-412, 1994.

L. Rieder, L. Smith and D. Noah, The Value of Play. To appear in a special edition of Educational Technology on affective psychology. 1998.

J.D. Teasdale and P.J. Barnard, Affect, Cognition and Change, Lawrence Erlbaum, 1993.