Human Error in the Context of Work Activity Systems: A Case Study of a Knowledge-Based Computer System in Routine Use in Obstetrics
Harris M1, Jagodzinski AP2 and Greene KR1
1Perinatal Research Group, Department of Obstetrics, Derriford Hospital,
Postgraduate Medical School, University of Plymouth.
2School of Computing, University of Plymouth.
Keywords: human error, activity theory, case study, context
A wider perspective of work activity and organisational context is central to understanding human errors. To help clarify the research approach required to investigate errors in context and inform design, we undertook a comparative case study of two Maternity Units where a knowledge-based computer system was in routine use. An activity theory analysis of the work activity system found the roots of errors were embedded in the wider work system rather than in direct interaction with the computer system. In one Unit, the role of the computer system was undermined as emergent, local adaptations contributed to errors, and the effects went unnoticed at an organisational level. Development of an effective relationship between computer systems and work systems, is an iterative process, which includes re-design of work activity. An appropriate research design and theoretical framework for the design of error tolerant computer systems and work systems needs to accommodate context and the process of change.
The user and the user interface, the most visible aspects of human-computer interaction, have received a disproportionate amount of "blame" for human errors. However, it is important to remember that computer systems are embedded in a broader system of activity which is purposive and social. Over the past nine years our group has developed and validated a knowledge-based computer system to assist clinicians in the management of women during labour and delivery (refs.1-3). The Medical Research Council (MRC) have supported the work over the past three years to test the system and develop the user interface prior to a randomised control trial. A wider perspective of human work activity and organisational context is a central concept for understanding errors in human-machine systems (refs. 4-6). However, we found it difficult to identify a research approach which accommodated context and informed design. We undertook a preliminary investigation of an existing knowledge-based computer system in routine use, to clarify the research approach required to investigate errors in context and its potential contribution to the design of our prototype computer system. This paper describes our experience in the design of medical knowledge-based computer systems and:
Traditional and Recent Perspectives on Human-Computer Interaction
Early studies of human computer interaction focused on cognitive models and quantitative research methods, but the artificial and unrepresentative nature of many studies limited their application (ref. 7). Numerous complex processes occur during a task, any one of which may affect performance. It is therefore difficult to generalise research findings to other tasks or systems (ref. 8). Growing criticism and dissatisfaction with traditional approaches, combined with the growth of technology to support communication and collaborative activity, have fuelled interest in social and contextual paradigms from sociology, anthropology and philosophy (ref. 7).
Cognitive models: Early studies of cognition were often undertaken in controlled conditions removed from the context of work. Cognitive science attempted to develop theory-based models of information processing and user models to match computer system and user. For example the Goals, Operations, Methods and Selection model (ref. 9) for human information processing attempted to predict human behaviour at the computer interface. While it was a significant attempt to develop theory, most work was in text editing with error free performance by experts. Thus its scope is narrow as learning, skills acquisition and errors are an important part of human-computer interaction. Applying psychological principles to computer design has not been as successful as first hoped. For example recall, recognition and working memory are well described phenomena, but this does indicate that all interactions should be menu-based or that the number of displayed items should be limited to seven (ref. 10).
Distributed Cognition: There is now an increased awareness of the importance of context. More recent studies of work systems have studied how cognition is manifest at a system level rather than at an individual cognitive level. In this view knowledge is dispersed through departments, organisations, people in their roles and in the physical artefacts they use (ref. 11-12). Distributed cognition theory highlights the social and environmental nature of knowledge and views cognition as distributed and collective.
Situated Action: Behavioural and social components of cognition have traditionally been treated as extraneous variables. Situated action offers an alternative view in which cognition, action and situation are bound together in particular social and physical circumstances (ref. 13). In this view, cognition is a social action and context an essential resource. To study only individual internal cognition removes the phenomena of interest. Situated action focuses on individuals in emergent situations where action is flexible in dealing with unexpected events. In these circumstances situated action suggests that plans do not govern human activity but are a post hoc analysis of action. However, situated action excludes the subjective and does not account for learning or enduring structures.
Computer Supported Cooperative Work: Team performance and their shared mental models have been the focus of much research, particularly in industrial, commercial and military sectors. Teams comprise of interdependent individuals with different roles, responsibilities and knowledge (ref. 14). Studies of group dynamics offer some insight into human errors. Time-constrained situations may lead to "recognition primed" decisions, with selection of the first acceptable solution rather than an optimal solution. Social factors may lead to normative errors such as the outward group acceptance of a solution although internally each member disagrees (ref. 15) and lower status members often fail to intervene when higher status membersí performance is poor (ref. 16). Despite a considerable body of research, the factors which contribute to team effectiveness and the role of technological support are not well understood. Support systems which display all available information to team members appear to improve analysis, level of participation, confidence, satisfaction, quality of decisions and reduce dominance of the group by a single member (ref. 17).
Organisation and culture: Computer systems usually do not fit neatly into pre-existing practice. They transform organisations, processes and the way in which people work (ref. 18). These issues were often neglected and consequently problems emerged too late to influence the design process. Organisational theory suggests a multilevel analysis (individuals, task and organisation) before embarking on the design process to assess the appropriate level of intervention for knowledge-based systems (ref. 19). However, many conceptual models are structural and static and fail to accommodate the constant state of flux in organisations.
Ethnography, a qualitative research approach, has been used to study organisations and elicit hidden work and its meaning, to help clarify systems requirements. This approach is gaining acceptance in both engineering and medicine (ref. 20-21) and involves researchers entering the work culture to observe and map social discourse. While qualitative research adds richness and depth of understanding, it is not a complete method and does have some practical problems. The duration of studies is usually prolonged, the amount of data generated can be overwhelming, its role in informing design is not clear and its contribution to cumulative research results is limited.
Human factors: Human activity is extremely complex and influenced by many factors; personal, social and organisational. Unfamiliarity with a task, time shortage and poor signal to noise ratio have been ranked as conditions which increased the risk of error by a factor of ten or more. Surprisingly, factors traditionally considered as major contributors to errors such as sleep disturbances, hostile environments and boredom, appear to have less impact on the risk of errors (ref. 22).
The role of human factors in adverse events has been widely studied in air transport, nuclear power stations and processing plants. Human errors take various forms and have different psychological origins (ref. 6). These include attention failure, memory failure and mistakes through inadequate planning. Mistakes may be rule-based, when a solution is available but is misapplied or omitted, or knowledge-based, where the situation is novel and no solution readily available (ref. 6). In the field of human factors there is also a shift in focus from individuals to the wider activity system. In aviation and medicine, strategies to improve team activity and communication have had considerable impact on human performance (ref. 4).
Activity theory: Activity theory is a conceptual and analytical framework, developed from Soviet psychology, which provides perspectives, tools and a common language for understanding work activity in context. Activity theory avoids dichotomising thought from action, individuals from collectives or social from technical issues. Activity theory proposes that understanding the role of artefacts and their integration into social practice is crucial to understanding human activity. The framework has been applied and developed in the study of human computer interaction to offer an alternative view of human activity by redefining the relationship between individuals, communities, technologies and actions (ref. 23).
The unifying theme of the more recent perspectives is the importance of context and need for investigative strategies which accommodate the complexity of real situations and real systems. However, this poses significant challenge for the investigation of human work activity in context while avoiding mechanistic oversimplification.
A comparative case study of a knowledge-based computer system in routine use in obstetrics
We had planned to visit other hospitals where medical knowledge-based computer systems were thought to be in routine clinical use, to see at first hand the influence of work context and the strategies adopted to study this. However, the systems identified from the literature were either limited prototypes or had been decommissioned.
An existing knowledge-based computer system (Expert DataCare) had been developed by our group for the interpretation of umbilical cord blood acid-base at the time of birth. The development of this system was driven by clinicians and a participatory design approach was adopted to ensure the usersí needs, capabilities and preferences informed the design process. The system had been in routine use in the development site for three years and is now also installed in 29 Maternity Units in the U.K. It was therefore decided to undertake a preliminary investigation of this system. While this knowledge-based computer system addressed a different and less complicated problem than our prototype system, it was routinely used in the same domain and by the same target user group.
The aims of the study were to:
Background to the system
Methods of assessing the condition of the infant at delivery have been mainly visual and subjective. Objective information can be obtained at birth from blood from the umbilical cord artery and vein. Blood samples analysed with a blood gas analyser provide information that reflects the oxygenation of the baby during labour. It is now recommended that this is undertaken routinely (ref. 24). However, retrospective analysis of data from a clinical trial in Plymouth (ref. 25) in which cord blood analysis was performed highlighted a number of problems. Approximately 25% of samples contained errors which prevented accurate interpretation. A knowledge-based computer system, interfaced to a blood gas analyser, was developed to check for data errors and provide an interpretation of the results. The aims of the system are to improve the quality of data collected, clarify the importance of the procedure, and ease the introduction of umbilical cord blood analysis into clinical practice. Immediate user feedback affords the opportunity to obtain further samples if required. The data is recorded to a database to provide accessible information for audit and risk management.
We selected two sites for study: the development site, Unit A, with three years experience of using the system and a comparison site, Unit B, with two years experience using the system. A case study approach was adopted to study the complex work processes in context and explicate interconnected levels and loops of causation (ref. 26).
The study used multiple sources of data to afford investigation of a wider range of issues. Triangulation of data and methods reduced potential problems of construct validity and avoided the weakness inherent in any single approach.
Interviews: The study began with interviews of doctors, midwives and care assistants as the initial data collection method. Nineteen participants, who had experience using the computer system, agreed to take part in the study. The aim of the interviews was to elicit information to uncover the informantís views and perceptions surrounding the use of Expert DataCare.
Unstructured interviews allowed for flexibility, but were more time consuming to analyse. Initial interviews were unstructured, but over time became more focused and structured as particular issues emerged through the process of data collection. A list of topics to cover during the interview was developed to act as a prompt for the researcher. Conversations were tape recorded, transcribed and analysed in detail. Each line of transcript was coded to enable identification of the source and sequence of material.
Observation: Informal observation provided useful additional information and the degree of researcher involvement ranged from complete observer in the comparison site to complete participant in the development site. Data were collected about specific events, social interaction, physical activities and the characteristics of the environment.
Analysis of Artefacts: The computer and artefacts associated with the computer were also included in the analysis. This included the blood gas machine, printed outputs, and the computer database. Examination of these artefacts helped to develop a broader perspective of the role of the computer system and provided insight into the technical operations concerning the application. Documents, including protocols and guidelines, were examined and prompted further lines of inquiry. These were helpful in identifying contradictions which would not have been readily apparent otherwise.
We adopted human activity theory as a conceptual framework and analytical tool. Activity theory is considered useful for studying human practice, as a development process, with both individual and social links. Activity theory focuses on the relationship between people and technology to uncover how they negotiate with the social and technological environment to solve problems and learn. This theory recognises the contingent and changing nature of activity systems and that human activity is mediated and transformed by the tools they create.
Figure 1 - A Conceptual Diagram of an Activity (ref. 27)
An activity is a form of doing directed at an object (figure 1). The transformation of the object into an outcome motivates the existence of an activity. The relationship between subject and object is mediated by tools; the relationship between subject and community is mediated by rules; the relationship between object and community is mediated by division of labour. This activity forms the minimal meaningful context for individual actions. Engestrom (ref. 28) extended activity theory to analyse work development and focused on contradiction and tensions which are present in every activity system. These tensions, incompatibilities and conflicts (collectively termed contradictions) offer an opportunity for change and development. If the community is receptive and the contradictions made visible, then change can be effected. This may lead to the development of a new activity system or to cessation of the activity. We followed a similar approach in our analysis of work activity and focused on conflicts, tensions and inconsistencies. The views that had been constructed about the activity are presented as emerging perspectives (ref. 29).
In the exploratory interviews it was found that users viewed the system in different ways between sites. Few participants were clear about the role and the capabilities of the computer system.
Emerging perspectives: Different perspectives were found between Units. In Unit A the output from the system was highly visible, used by individuals to evaluate their care and at department meetings. The output was viewed as a tool and as an important commodity. Respondents in Unit B indicated tensions and conflicts. Excerpts from the interviews illustrate the general perspective of Unit B:
"Its a technological thing. Anyone can do it. I feel I need to be with the mother and baby." Staff Midwife, Unit B
"Its a task - something else to do. It could easily be done by others - non professionals." Staff Midwife, Unit B.
In Unit B the system inserted itself in a negative way, between midwives and patients, and was perceived as conflicting with their other duties. The use of the output from the system was not visible to the midwives, it was viewed as a task and no value was placed on it.
Contradictions: In Unit A some midwives had initial concerns about the involvement of auxiliary nurses.
"I do feel it is a professional job, it should not be done by the auxiliary nurses." Midwifery Sister, Unit A.
"When we first started there were a few midwives who thought that auxiliaries shouldnít do it [cord blood gas analysis] and that was a bit upsetting to think they didn't have the confidence in us." Auxiliary Nurse, Unit A.
Lack of confidence in the ability of the auxiliary nurses was unjustified. An audit of the computer database found they had a lower error rate than midwives. Figure 2 shows the number of valid paired samples (a sample from both the artery and vein with no measurement errors) and the incidence of inadvertently taking two samples from the same vessel.
Figure 2 - Comparison of error rates of auxiliary nurses and midwives in Unit A
Comparison of the computer databases found considerable differences between the two Units. The proportion of significant metabolic acidosis (a measure of oxygen lack in the baby during labour) was found to be 0.7% in Unit A and 2.7% in Unit B. Further analysis of the Unitsí clinical computer databases found Unit A obtained samples in 98% of deliveries nd the overall sampling error rate was 14%. In Unit B less than 50% of deliveries had cord blood analysis and the overall sampling error rate was 28%.
Placing errors in context of the wider activity system: Each Unit constructed their work in different ways. In Unit A, auxiliary nurses were included as part of the team, analysis of cord blood was performed immediately after delivery, training was provided by a computerised teaching package (CAL) and on-site support for maintenance of the blood gas analyser was provided by the medical physics department. In Unit B, only midwives were involved in the procedure, the analysis of cord blood was delayed for up to 1 hour after delivery, training was "on the job" and provided by midwives who had previously undergone training (cascade training). A midwife was responsible for daily maintenance of the blood gas machine with direct support from the manufacturers.
The implicit rule in Unit B that auxiliary nurses could not undertake cord blood gas analysis resulted in the workload being allocated to midwives. Despite adapting the sampling procedure to allow a delay of up to one hour, the midwives experienced conflict with their other duties. The delay in analysis also weakened the utility of the procedure as it reduced the opportunity to take further blood samples if required. In Unit A, some midwives had initial concerns about the involvement of auxiliary nurses. This implicit rule may be embedded more deeply in hierarchy and tribalism. In Unit B, a clash with the biomedical departmentís existing rule of purchasing one particular make of blood gas analyser resulted in withdrawal of technical support. A midwife had to take on this role with direct support from the manufacturers. Consequently, this increased training demands and on occasions the system would request a technical operation when no trained staff were available. The main differences between the two Units and the impact of this on the activity are outlined in Figure 3.
Figure 3 - Differences in work activity system between sites
This work combined interviews, observation and analysis of artefacts framed within a case study. Using convergent sources of data strengthened the interpretation of the interview data. Activity theory as an analytical framework offered some insight into the roots of errors, many of which originated from broader activities, attitudes and perspectives surrounding the computer system.
The perspective of Unit A was information orientated and the activity was considered important for research, legal purposes and evaluation of care. The information from the system was highly visible in department audit meetings and at an individual level as a focus for discussion, assessment and improvement of intrapartum care. Unit Bís perspective of the work was as "just another task" to be done. There seemed to be a lack of understanding as to how it could help them, individually and professionally, if done properly. The changes to the work system in Unit B did not support attainment of all the objectives of the computer system.
While activity theory suggests that motivated actions within an activity system merge to reach the shared motive, it is not always the case. When this specific type of polymotivation occurs those who benefit are not those who do the work and the technology is likely to fail, or be subverted (ref. 30). The selective sampling in Unit B skewed the data towards a higher incidence of abnormal results. Inappropriate inferences, which could impact on organisational and management decisions could have been made from the data if this discrepancy had not been uncovered.
Ringstaff, Kelly & Dwyer (ref. 31) identified four factors which they considered an important influence on the effectiveness of technology. Psychological factors such as being comfortable with the technology and seeing it as an opportunity rather than a threat; technical support; ease of access; and colleague and institutional support. However, the origins of these factors are embedded in the wider system of activity. People use computers to achieve meaningful goals beyond the actual operation of the system. The physical, social and historical circumstances of work activity give meaning to actions and shape emerging perspectives.
Holland and Reeve (ref. 29) suggest that perspective is a place where contradictions become manifest in consciousness and affect the way work is done. In this study midwives in Unit B did not seem to recognise opportunities for development. The practical difficulties involved in training may have contributed to this lack of awareness. "Cascade" training has a number of limitations. The quality of education becomes diluted over time, particularly in areas with rapid turn over of staff and it is difficult to assess consistency and quality of training. Unit A addressed this practical problem with the use of an interactive computer-based teaching package. The improved morale associated with maintaining a commitment to staff development (ref. 32) may further shape perspectives of an activity and improve performance.
Ease of access and technological support imply institutional support through investment in the technology. However, in Unit B the Biomedical Departmentís adherence to rigid purchasing policies contributed to errors of omission and was a barrier to effective use of the system. Lack of on-site support impeded operational effectiveness and increased training demands. This type of problem has been highlighted by the Audit Commission (ref. 33) who noted that whether or not a system meets real needs has ceased to be an issue as long as the procurement procedure is followed. In Unit B, the role of the computer system was undermined as emergent, local adaptations contributed to errors. Conflicts and barriers to effective use of computer systems originated from the wider activity system including the rules, tools and division of labour. These effects went unnoticed at an organisational level.
Context has become a touchstone for a better understanding of human-machine errors. However, the study of complex phenomena is a significant challenge for study designs. In this case study we found multiple sources and methods of data collection allowed us to explicate a complex process and uncover important features in the work system. The great strength of the case study is its flexibility. Relevant issues uncovered during the progress of a case study can be explored and guide further data collection to develop a wider perspective. When a study begins with a broad question and where context is considered to influence outcome there is a real danger of attempting to collect data about everything. The case study strategy reduced this risk as it constrained data collection to relevant sources which converged on the phenomena of interest. This approach helped us to place the computer system in context and to understand why errors had occurred. The study also highlighted the need to develop a better understanding of the process of change and its interaction with and effect on context.
While the case study makes no commitment to a particular theoretical stance, it is not atheoretical. The interpretative theory which underpinned this work was considered to be the most salient as it:
The roots of human errors are often embedded in underlying organisational issues and the contributing factors are often multiple. The computer technology in this case study was robust. No errors were attributed to failures in the computer system or in the direct interaction between the operator and the computer system (i.e. what is normally called the "interface"). The wider interaction of the human activity system, rather than the human-computer interface, contributed to errors. An unfortunate synergy of individually small factors in the work system can lead to the collapse in the effectiveness of a computer system. This small exploratory study highlighted that:
This study and the use of activity theory changed our thinking about the design of medical knowledge-based computer systems. Computers are tools which mediate and transform work activity systems. The introduction of new technology changes the work system and changes to the work system change the technology. The design of error tolerant systems cannot make the mistake of separating the technical from the social or the individual from the collective.
The lessons learned in this study are being applied in the design of our prototype computer system to assist clinicians in the management of women in labour. A longitudinal case study of the work activity system in delivery suites is being used to model the work activity system. This will form the basis of models of interaction to inform the design of the computer system and suggest changes to the work activity system to reduce the risk of errors.
A MRC multicentre trial of our knowledge-based computer system, to support clinicians in patient monitoring during childbirth, is planned for the summer of 2000. This will provide an opportunity to undertake longitudinal studies of work activity in 6 UK hospitals.
We would like to acknowledge the support of the Medical Research Council, the South & West Regional Health Authority Research and Development Directorate and the Directorate of Obstetrics & Gynaecology, Derriford Hospital, Plymouth.
Maureen Harris, Perinatal Research Group, Department of Obstetrics & Gynaecology, Level 4, Derriford Hospital, Plymouth. PL7 8DH Devon. Telephone 01752 763631. Email:firstname.lastname@example.org.
Maureen Harris, a midwife, has worked with the Perinatal Research Group since 1990. In 1995 she was awarded a South & West Regional Research Training Fellowship to develop expertise in health related research. Her current Ph.D work is on integrating work systems and computer systems in association with an MRC funded project to develop medical knowledge-based computers for patient monitoring during labour.