This study examined the effects of tuning the parameters of the incremental function of MYCIN, the independent function of PROSPECTOR, a probability model that assumes independence, and a simple additive linear equation. me parameters of each of these models were optimized to provide solutions which most nearly approximated those from a full probability model for a large set of simple networks. Surprisingly, MYCIN, PROSPECTOR, and the linear equation performed equivalently; the independence model was clearly more accurate on the networks studied.

Expert systems applications that involve uncertain inference can be represented by a multidimensional contingency table. These tables offer a general approach to inferring with uncertain evidence, because they can embody any form of association between any number of pieces of evidence and conclusions. (Simpler models may be required, however, if the number of pieces of evidence bearing on a conclusion is large.) This paper presents a method of using these tables to make uncertain inferences without assumptions of conditional independence among pieces of evidence or heuristic combining rules. As evidence is accumulated, new joint probabilities are calculated so as to maintain any dependencies among the pieces of evidence that are found in the contingency table. The new conditional probability of the conclusion is then calculated directly from these new joint probabilities and the conditional probabilities in the contingency table.

This study compares the inherent intuitiveness or usability of the most prominent methods for managing uncertainty in expert systems, including those of EMYCIN, PROSPECTOR, Dempster-Shafer theory, fuzzy set theory, simplified probability theory (assuming marginal independence), and linear regression using probability estimates. Participants in the study gained experience in a simple, hypothetical problem domain through a series of learning trials. They were then randomly assigned to develop an expert system using one of the six Uncertain Inference Systems (UISs) listed above. Performance of the resulting systems was then compared. The results indicate that the systems based on the PROSPECTOR and EMYCIN models were significantly less accurate for certain types of problems compared to systems based on the other UISs. Possible reasons for these differences are discussed.

This paper examines the accuracy of the PROSPECTOR model for uncertain reasoning. PROSPECTOR's solutions for a large number of computer-generated inference networks were compared to those obtained from probability theory and minimum cross-entropy calculations. PROSPECTOR's answers were generally accurate for a restricted subset of problems that are consistent with its assumptions. However, even within this subset, we identified conditions under which PROSPECTOR's performance deteriorates.

Findings that decision makers can come to different conclusions depending on the order in which they receive information have been termed the "information order bias." When trained, experienced individuals exhibit similar behaviors; however, it has been argued that this result is not a bias, but rather, a patternmatching process. This study provides a critical examination of this claim. It also assesses both experts' susceptibility to an outcome framing bias and the effects of varying task loads on judgment. Using a simulation of state-of-the-art ship defensive systems operated by experienced, active-duty U.S. Navy officers, we found no evidence of a framing bias, while task load had a minor, but systematic effect. The order in which information was received had a significant impact, with the effect being consistent with a judgment bias. Nonetheless, we note that patternmatching processes, similar to those that produce inferential and reconstructive effects on memory, could also explain our results. Actual or potential applications of this research include decision support system interfaces or training programs that might be developed to reduce judgment bias.

Methods for grouping occupational tasks are required to support a broad range of personnel actions and organizational planning activities. Having subject matter experts sort tasks into groups is the only methodology generally recognized for these purposes. For many applications, however, and training in particular, analyses that cover large areas of an organization may be required. For such uses, manual sorting is costly and may be infeasible. A new method, based on statistical clustering using task coperformance, is described. Results indicate that this method can replicate much of the structure of the experts’ groups and so can be used to facilitate task grouping. Implications of this new approach are discussed.

: Estimates of the cost of providing training, in forms ranging from classroom instruction to on-the-job training, are needed to support decisions about who gets trained, when, where, and on what skills. To counter the myriad of uncontrollable factors that may obscure the relationship between manpower, personnel, and training policy changes and organizational outcomes, an organizational simulation of Air Force occupations called the Training Impact Decisions System (TIDES) was developed. An important first step in obtaining these occupation-level outcome estimates in TIDES is to identify groups of tasks with similar knowledge and skill requirements, because economies will be realized when these tasks are trained at the same time. This report compares results from using two different methods to identify groups of Air Force Occupational Survey tasks where these training economies would occur, including methods based on subject matter experts' judgments and statistical clustering using task coperformance. The results from two field applications indicated that the statistical methods could replicate much of the structure of the experts' clusters, and so, could be used to facilitate the process of identifying these task groups. Use of these methods to form task clusters which could be used to support a broad range of training and personnel decisions is also discussed.

Human reasoning under uncertainty has been shown to consist of a series of "local computations" in which a complex problem is broken into a series of simpler decisions. For practical and epistemological reasons, expert systems which must reason under uncertainty have taken a similar approach. Thus, choice of an appropriate Uncertainty Representation Scheme (URS) is dependent upon the robustness of the scheme to variations in the local computations that are expected in the application area. Researchers have based their claims for robustness of a particular URS on the apparent match between the scheme's stated assumptions and the expected "statistical" characteristics of the local computations. The limited empirical data that are available suggest that these claims may be ill-founded. Rather, this paper argues that a scheme's robustness should be measured relative to humans' local computations which are used to solve the problem, and we present a methodology for doing so. Application of this methodology suggests that representations of local computations are strongly influenced by the choice of a URS and that the accuracy of the resulting solutions are substantially influenced. Implications for full scale applications and directions for further research are also discussed.

Most of the approaches to uncertain reasoning developed for expert systems require user judgements about the degree to which evidence is present or absent. These user inputs are incorporated directly into the uncertainty calculations and influence the advice a completed system offers. Although all the other system parameters are set by the developer when the system is built and “tuned,” user inputs are likely to vary according to individual opinions. This empirical study examines the effects of user inputs on system accuracy. Subjects used one of two uncertain reasoning models to build and tune a system that captured their knowledge of a hypothetical, inherently uncertain domain. In the course of doing so, each subject also provided personal judgments of how evidence values for a uniform set of test cases mapped onto the uncertainty parameter values input by users. Our analysis examines the error introduced by using each subject's inputs for the test cases in conjunction with each of the other subject's systems. We also discuss the practical implications of our findings for system builders.

This study compares the inherent intuitiveness or usability of the most prominent methods for managing uncertainty in expert systems, including those of EMYCIN, PROSPECTOR, Dempster-Shafer theory, fuzzy set theory, simplified probability theory (assuming marginal independence), and linear regression using probability estimates. Participants in the study gained experience in a simple, hypothetical problem domain through a series of learning trials. They were then randomly assigned to develop an expert system using one of the six Uncertain Inference Systems (UISs) listed above. Performance of the resulting systems was then compared. The results indicate that the systems based on the PROSPECTOR and EMYCIN models were significantly less accurate for certain types of problems compared to systems based on the other UISs. Possible reasons for these differences are discussed.

Abstract : This document summarizes the research and development activities undertaken to develop the Training Decisions System (TDS). The TDS is a computer-based decision aid to be used in planning the what (training content), the where (technical school, Field Training Detachment (FTD), on-the-job training (OJT)), and the when (at what point in an airman's career). Further, the TDS incorporates optimization strategies to allow training managers to ask what if questions related to current and possible future policy changes within the Air Force training environment. In addition, this report contains a brief conceptual overview of the three major data-based subsystems and the fourth integrating/optimization subsystem which compose the present TDS. The first subsystem of the TDS is the Task Characteristics Subsystem (TCS). The TCS identifies what tasks are required to be trained and where to allocate those tasks for the most efficient training. The second subsystem of the TDS is the Field Utilization Subsystem (FUS). Alternative utilization, Training, Career path simulation, Integration optimization, Management information system, Modeling, Task training module.

: This document summarizes the research undertaken to develop the Field Utilization Subsystem (FUS), one of four basic subsystems of the prototype Training Decisions System (TDS). The TDS is a computer assisted decision system that will be developed to aid in planning the what, where, and when of training for Air Force career ladders. The FUS addresses the where and when to train in a career ladder. This three-component subsystem has first the task of describing the current Utilization and Training (U&T) pattern for an Air Force speciality (AFS) and second, provides a means for collecting alternatives to the current U&T pattern. The third component is responsible for determining which U&T pattern alternatives are preferred by various Air Force manpower, personnel, and training managers. The U&T patterns show job structures and personnel flow. Job analysis, or job-typing, using Comprehensive Occupational Data Analysis Programs (CODAP) routines and current Air Force job-typing methodologies helped to establish job structures for the patterns. Dynamic cross-KPATHing analysis, as a starting point, and Subject-Matter Expert (SME) judgments were instrumental in identifying training states and transition probabilities in relation to the job in the current U&T patterns. SME judgments were the primary source for alternative U&T pattern generation. In turn, the alternative U&T patterns in narrative and diagram form were the source for determining managers' preferences through the administration of surveys. The research produced a coherent and internally consistent model. Keywords: Air force training, Enlisted personnel, Job training. (SDW)

Uncertainty is a pervasive feature of the domains in which expert systems are designed to function. Research design to test uncertain inference methods for accuracy and robustness, in accordance with standard engineering practice is reviewed. Several studies were conducted to assess how well various methods perform on problems constructed so that correct answers are known, and to find out what underlying features of a problem cause strong or weak performance. For each method studied, situations were identified in which performance deteriorates dramatically. Over a broad range of problems, some well known methods do only about as well as a simple linear regression model, and often much worse than a simple independence probability model. The results indicate that some commercially available expert system shells should be used with caution, because the uncertain inference models that they implement can yield rather inaccurate results.