This is the last in a series of three papers documenting two large-scale human reliability analysis (HRA) empirical studies – the International HRA Empirical Study and the US HRA Empirical Study. The goal of the two studies was to develop an empirically-based understanding of the performance, strengths, and weaknesses of HRA methods by comparing HRA method predictions against actual operator performance in simulated accident scenarios on nuclear power plant (NPP) simulators. This paper first addresses areas where there is convergence between the two studies and where differences lie. Then it summarizes the combined insights and conclusions, including key findings on HRA in general through lessons learned about the HRA methods assessed in the studies and specific recommendations for improving guidance, practice and methods. Then it discusses the relevance and usefulness of simulator data for HRA in general. Finally, it presents the key achievements and overall conclusions of the two studies taken together.

This is the first in a series of three papers documenting two large-scale human reliability analysis (HRA) empirical studies – the International HRA Empirical Study and the US HRA Empirical Study. The two studies are the first major efforts in recent years to benchmark HRA methods by comparing HRA method predictions against actual operator performance in responding to accidents simulated on nuclear power plant (NPP) full-scale simulators. The studies aimed to gain knowledge and insights concerning the strengths and weaknesses of the studied HRA methods and the factors contributing to inter-analyst (or intra-method) variability. In addition, the studies also compared the results of the same HRA method applied by different analysis teams. This paper provides the background and motivation of the studies, the overall study design, the simulation scenarios and human failure events to be analyzed, and concluding remarks concerning lessons learned on benchmarking HRA methods with crew performance of scenarios on NPP simulators.

This is the second in a series of three papers documenting two large-scale human reliability analysis (HRA) empirical studies – the International HRA Empirical Study and the US HRA Empirical Study. The goal of the two studies was to develop an empirically-based understanding of the performance, strengths, and weaknesses of HRA methods by comparing HRA method predictions against actual operator performance in simulated accident scenarios on full-scale nuclear power plant (NPP) simulators. The first paper (Paper 1) provides background information for the studies, an overview of their design and methodology, and a description of the simulation scenarios and associated human failure events (HFEs) addressed in the HRA analyses. This paper first discusses the overall simulator data followed by quantitative comparisons of the HRA methods’ predictions with the simulator data. Then, it presents a summary of the results of and knowledge and insights gained from the comparisons between method predictions obtained with the same method.

There is a diversity of human reliability analysis (HRA) methods available for use in assessing human performance within probabilistic risk assessments (PRA). Due to the significant differences in the methods, including the scope, approach, and underlying models, there is a need for an empirical comparison investigating the validity and reliability of the methods. To accomplish this empirical comparison, a benchmarking study comparing and evaluating HRA methods in assessing operator performance in simulator experiments is currently underway. In order to account for as many effects as possible in the construction of this benchmarking study, a literature review was conducted, reviewing past benchmarking studies in the areas of psychology and risk assessment. A number of lessons learned through these studies is presented in order to aid in the design of future HRA benchmarking endeavors.

Since the Reactor Safety Study in the early 1970's, human reliability analysis (HRA) has been evolving towards a better ability to account for the factors and conditions that can lead humans to take unsafe actions and thereby provide better estimates of the likelihood of human error for probabilistic risk assessments (PRAs). The purpose of this paper is to provide an overview of recent reviews of operational events and advances in the behavioral sciences that have impacted the evolution of HRA methods and contributed to improvements. The paper discusses the importance of human errors in complex human-technical systems, examines why humans contribute to accidents and unsafe conditions, and discusses how lessons learned over the years have changed the perspective and approach for modeling human behavior in PRAs of complicated domains such as nuclear power plants. It is argued that it has become increasingly more important to understand and model the more cognitive aspects of human performance and to address the broader range of factors that have been shown to influence human performance in complex domains. The paper concludes by addressing the current ability of HRA to adequately predict human failure events and their likelihood.

A diversity of Human Reliability Analysis (HRA) methods are currently available to treat human performance in Probabilistic Risk Assessments (PRAs). This range of methods reflects traditional concerns with human-machine interfaces and with the basic feasibility of actions in PRA scenarios as well as the more recent attention paid to Errors of Commission and decision- making performance. Given the differences in the scope of the methods and their underlying models, there is a substantial interest in assessing HRA methods and ultimately in validating the approaches and models underlying these methods. A significant step in this direction is an international evaluation study of HRA methods, based on comparing the observed performance in simulator experiments with the outcomes predicted in HRA analyses. Its aim is to develop an empirically- based understanding of the performance, strengths, and weaknesses of the methods. This paper presents the overall methodology for this initial assessment study.

An expert elicitation approach has been developed to estimate probabilities for unsafe human actions (UAs) based on error-forcing contexts (EFCs) identified through the ATHEANA (A Technique for Human Event Analysis) search process. The expert elicitation approach integrates the knowledge of informed analysts to quantify UAs and treat uncertainty (‘quantification-including-uncertainty’). The analysis focuses on (a) the probabilistic risk assessment (PRA) sequence EFCs for which the UAs are being assessed, (b) the knowledge and experience of analysts (who should include trainers, operations staff, and PRA/human reliability analysis experts), and (c) facilitated translation of information into probabilities useful for PRA purposes. Rather than simply asking the analysts their opinion about failure probabilities, the approach emphasizes asking the analysts what experience and information they have that is relevant to the probability of failure. The facilitator then leads the group in combining the different kinds of information into a consensus probability distribution. This paper describes the expert elicitation process, presents its technical basis, and discusses the controls that are exercised to use it appropriately. The paper also points out the strengths and weaknesses of the approach and how it can be improved. Specifically, it describes how generalized contextually anchored probabilities (GCAPs) can be developed to serve as reference points for estimates of the likelihood of UAs and their distributions.

A quantification process has been developed that explicitly uses the conceptual basis of the ATHEANA Human Reliability Analysis (HRA) method and the results of its search process for identifying potential unsafe human actions and their error-forcing contexts, The quantification approach integrates the knowledge of informed analysts to quantify unsafe acts and treats uncertainty explicitly. The approach emphasizes obtaining expert evidence, rather than asking for expert judgment. Initial applications of the method appear to work well and generate reasonable results. This process includes protection against bias and overconfidence.

In May of 1998, a technical basis and implementation guidelines document for A Technique for Human Event Analysis (ATHEANA) was issued as a draft report for public comment (NUREG-1624). In conjunction with the release of the draft NUREG, a paper review of the method, its documentation, and the results of an initial test of the method was held over a two-day period in Seattle, Washington, in June of 1998. Four internationally-known and respected experts in human reliability analysis (HRA) were selected to serve as the peer reviewers and were paid for their services. In addition, approximately 20 other individuals with an interest in HRA and ATHEANA also attended the peer review meeting and were invited to provide comments. The peer review team was asked to comment on any aspect of the method or the report in which improvements could be made and to discuss its strengths and weaknesses. All of the reviewers thought the ATEANA method had made significant contributions to the field of PRA/HRA, in particular by addressing the most important open questions and issues in HRA, by attempting to develop an integrated approach, and by developing a framework capable of identifying types of unsafe actions that generally have not been considered using existing methods. The reviewers had many concerns about specific aspects of the methodology and made many recommendations for ways to improve and extend the method, and to make its application more cost effective and useful to PRA in general. Details of the reviewers` comments and the ATHEANA team`s responses to specific criticisms will be discussed.

Hypotheses dealing with human sequential processing of information are derived to test an anchoring-and-adjustment mechanism of information processing and contrast-inertia models of H.J. Enhorn and R.M. Hogarth (1987). A computer-based research paradigm loosely modeled after a missile warning officer's activity was developed for the experimental effort. Results show that the order in which pieces of evidence are submitted to the decision-makers has a critical effect on their belief, often resulting in contradictory opinions as to the presence or absence of an enemy attack. This effect, predicted by the contrast-inertia model, is especially important when mixed evidence (confirming/disconfirming) is presented to the subjects. In addition it is hypothesized that a possible framing effect causes a bias in the subjects' beliefs, revealing an asymmetric preference for the attack hypothesis. The simple contrast-inertia model predicts the empirical data reasonably well, although its sensitivity coefficients may be complex functions of the initial anchor strength and direction, and of the number of stages in the belief-updating sequence.<>