The concepts-of-operation proposed for the Next Generation Air Transportation System (NextGen) implicitly require a significant improvement over existing standards for flightdeck human-computer interaction. Whereas in today’s airspace operations there is no routing penalty for delayed response to a required ATC maneuver, flights in high density NextGen airspace that are unable to respond to off-nominal situations in a timely manner, will lose their slot and be shifted to a downgraded level of airspace resulting in flight delays and/or increased route distance. Current design and certification processes for avionics, aircraft, and pilots prove the reliability of the “deterministic” automated functions in a comprehensive manner. The design and certification requirements for ensuring and testing the reliability of the inherently “non-deterministic” operator interaction with the automation are not rigorous and are the source of operational inefficiencies and reduced safety margins. Unless the design and certification process are radically modified and refocused, pilots will find themselves with the same types of issues that researchers documented with the introduction of the “glass cockpit” in the 1980s and 1990’s. This paper provides a quantifiable definition of Human Computer Interaction performance and explicit measures of individual and crew proficiency. A method for estimating revenue-service cost savings generated by improved proficiency is described along with an example of the cost savings benefits accrued by a hypothetical large U.S. domestic carrier experiencing improved proficiency in response to FMS error messages ($45M per year). A discussion of the implications and limitations of the definition of proficiency and the cost savings model is provided.

We present our experience using CogTool, a tool originally designed for ease of use and learning by non-psychologist design practitioners, as a means for rapid theory exploration. We created seven iterations of a “model prototype” of an aviation task where each iteration produced errors that pointed to additional theory or device knowledge that should be incorporated to prevent those errors. This theory and knowledge was put into the next iteration by modifying the mock-up of the device, not by changing the implementation of the underlying cognitive model. This trick allowed us to rapidly change theory and knowledge and understand what must eventually migrate to the underlying cognitive model to provide general support for predictions of novice behavior in multi-step procedures with complex devices.

This research has two important sets of purposes. The first set of purposes was to evaluate working memory load and persistent behavior of computer users navigating complex websites, design and test the effectiveness of feedback intended to reduce revisits to previously-visited (dead-end) states, and determine the extent to which reducing revisits improves overall web navigation performance on difficult search tasks. The second purpose was to contribute to the development of the Cognitive Walkthrough for the Web (CWW), verifying and refining the psychological validity of CWW predictions of search difficulty. Participants in the primary experiment completed 36 tasks that varied in difficulty on an experimental website simulating an actual online encyclopedia. The between-subjects independent variables were link color change (no link color change, cumulative link color change, task-specific link color change), heading feedback (heading feedback, no heading feedback), and order (original, revised). The within-subject independent variables were search difficulty (difficult, easy), and task set (first-half, second-half), the latter being a measure of learning effects. Revisits to subordinate links — a clear, direct measure of working memory errors and perseverance — was the main dependent variable.Results showed that task-specific link color change dramatically reduced revisits to subordinate links, and there was a significant interaction of task set with link color change. In the no-link-color condition, individuals varied dramatically in both mean revisits and maximum revisits, and certain individuals, possibly with low working memory capacity or tendency to perseverate, derived the most benefits from task-specific link color change. However, even task-specific feedback reduced clicks on links by only about a half click, and tasks that CWW predicts to be difficult are not significantly reduced in difficulty. The three independent variables that CWW uses to predict search difficulty and usability problems and predicted clicks on links turn out to equally valid for experimental websites with no link color change and for real-world websites that do provide link color change feedback.

CoLiDeS and SNIF-ACT are empirically-validated, complementary models. SNIF-ACT applies rational analyses to information foraging anywhere on the Web. CoLiDeS describes how people attend to and comprehend information patches on individual webpages. Integrating CoLiDeS and SNIF-ACT would better predict how people forage the Web for information to solve everyday ill-structured problems. CoLiDeS and SNIF-ACT 2 CoLiDeS and SNIF-ACT: Complementary Models for Searching and Sensemaking on the Web Information foraging theory depicts the human species as hungry for information, and as Pirolli (2005) has perceptively pointed out, navigating the Web has become a common way to find information needed to solve such ill-structured everyday problems as selecting treatment for a medical condition. Information foraging theorists have used ACT-R spreading activation models of information scent to generate reliable predications of how people navigate the Web by following an information scent trail. They have used mathematical models from rational analyses to calculate and compare utility values and accurately describe how people decide which particular information patch to graze in, when to select links to move to another webpage, when to back up to a previously visited information patch, and when to abandon a website and search for a new and hopefully better information patch (Pirolli & Card, 1999; Pirolli, 2005). Although everyone has ill-structured everyday problems to solve, due to differences in background knowledge people vary enormously in their ability to comprehend the information available on the Web. Information foraging theory depicts people as hungry for information, but people in reality consume only information that they can comprehend. Information is useless to a person unless the person can comprehend the information. Due to differences in background knowledge people also vary in search strategies and attention management, ability to predict what links might be nested under superordinate categorical headings, and consequent ability to scan headings to identify what is in different patches. As a result of these differences in comprehension ability and attention management, people vary in both their ability to comprehend the information they find on the Web and their ability to find information by navigating the Web. Differences in background knowledge result from differences in culture, general reading knowledge, and amount of experience using Web browsers and computers, and researchers developing the CoLiDeS cognitive model and the Cognitive Walkthrough for the Web (CWW) have used the semantic spaces in Latent Semantic Analysis to generate reliable predictions for users that differ in general reading knowledge and culture (Blackmon, Mandalia, Kitajima, & Polson, 2007). The SNIF-ACT model (Pirolli, 2005; Pirolli & Fu, 2003; Fu & Pirolli, in press), exemplifies the information foraging and rational analysis approach to predicting information search behavior anywhere on the Web. In contrast, the CoLiDeS (Kitajima, Blackmon, & Polson, 2000, 2005) model exemplifies the comprehension-based approach to predicting information foraging behavior at the microcosmic level of individual webpages, building bottom-up from the perspective of actions taken on an individual webpage. Whereas SNIF-ACT is founded on the ACT-R computational model, CoLiDeS is founded on Kintsch's (1998) Construction-Integration model of text comprehension, action planning, and problem solving – useful for understanding how people solve such ill-structured problems as comprehending and selecting treatment for a medical condition. The core argument of this paper will be that integrating these two complementary, empirically well-validated models of Web navigation – CoLiDeS and SNIF-ACT – would improve our ability to predict information search and sensemaking on the Web for the full gamut of human users of varying abilities. Information foraging anywhere on the Web CoLiDeS and SNIF-ACT are complementary models, both starting with a user's goal to search for information. SNIF-ACT focuses on decisions to forage in a particular information patch, usually defined CoLiDeS and SNIF-ACT 3 as a complex website, or to leave the patch in search of patches of information with higher levels of information scent for the user's goal. As Figure 1 illustrates, an information patch can be defined at many different levels, from a particular website in the huge universe of websites on the Internet down to a collection of patches that compose a single webpage. SNIFACT computes the utility of staying within the current information patch compared to going back a page, clicking a link to go forward to a new page, or leaving the website. To date SNIF-ACT treats a webpage as a single information patch (Fu & Pirolli, in press), but there is no known barrier to extending SNIF-ACT to deal with a webpage as a collection of patches. In contrast, CoLiDeS considers the current webpage as a collection of patches – called subregions in earlier publications (e.g., Kitajima, Blackmon, & Polson, 2005) – and uses information scent to select which particular patch to forage. Figure 2 illustrates a collection of patches on a single webpage. When either CoLiDeS or a human user is drawn to a patch with high information scent for the goal, the consequences are good if the patch actually contains a link that is on the solution path. In many cases, however, a human user is drawn to a patch with high information scent, where there are multiple highscent links, none of which are on the solution path. This situation usually results in the user clicking many high-scent links that are not on the solution path (Blackmon et al, 2005, 2007). In these cases, information scent actively misleads the user, and the situation commonly occurs where items can be cross-classified but the Web designer makes the item accessible only by a link within one of categories. Two other closely related problems can shackle persons who follow an information scent trail. One is the problem posed when a "correct patch" has relatively high scent but the "correct link" within that patch has very weak scent. Based on the mathematical models of rational utility of when to abandon a patch – and confirmed by empirical evidence (Blackmon, Kitajima, & Polson, 2005) – weak-scent correct links pose serious difficulties because people tend to abandon the patch without clicking the weak-scent correct link. The second closely related problem – discussed in the next section – is that clusters of links are often highly general categories that have some information scent for most goals but relatively low scent for any one particular goal. This dilemma calls attention to even deeper problems of how people select patches anywhere on the Web, because (a) website-level patches will have very low scent, except for webpages deep in the hierarchy listed in the results webpages of a search engine like Google, and (b) information found by search engines like Google is liable to be very unreliable despite high scent. Figure 1. Information patches at all levels: individual websites within the universe of all websites on the Web, subsites, webpages, or patches within webpages CoLiDeS and SNIF-ACT 4 Figure 2. Patches within an individual webpage CoLiDeS and SNIF-ACT 5 Sensemaking for information foraging on the Web Information foraging theory draws from earlier models of foraging for food, and such models always take into account the nutritional content of the food – for example, calorie count, protein content, salt, minerals, and vitamins – and the tendency of organisms to avoid harmful constituents in the food source – for example, bacteria or toxins in the food or water that would cause the animal to become ill after eating the food. Sensemaking is a crucial element of information foraging, the analog of nutritional content of food and avoidance of constituents that would be harmful to the animal's health. CoLiDeS is founded in the construction-integration architecture for text comprehension, action planning, and problem solving. Based on a theory of comprehension we can make three claims about sensemaking in Web navigation: (a) information discovered through information foraging is worthless to a person unless the person has the background knowledge required to comprehend it, (b) unreliable, untrue information is harmful and should be avoided, and (c) inability to adequately comprehend links, headings, and/or page layout conventions can seriously lower a person's success in finding the information needed or desired for solving an ill-structured everyday problem (see evidence on unfamiliar links reported in Blackmon, Kitajima & Polson, 2005). Figure 2 shows an example of an unfamiliar link, "Oceania," that is unfamiliar even for college-level readers. The Oceania link is liable to cause problems for a user searching for trails in New Zealand, because even college-level readers are unlikely to know that New Zealand can be considered part of Oceania. Comprehension of the information found. In an extensive body of research, Kintsch has demonstrated the necessary role of background knowledge in constructing a situation model of the text. The situation model is required for text comprehension, for learning from text, for action planning and for problem solving (see review of this research in Kintsch, 1998). For example, in regard to finding information to solve the everyday ill-structured problem of finding information to select medical treatment, Patel and colleagues (e.g., Patel, Arocha, & Kushniruk, 2002) have documented patients' problems comprehending medical information about their condition, especially patients who have a narrative model of their disease and not a biomedical model like physicians and other medical professionals have. Reliability of the information found. As Bhavnani et al. (2003) have argued, background knowledge is also crucial for det

Researchers have identified low proficiency in pilot response to flight management system error messages and have documented pilot perceptions that the messages contribute to the overall difficulty in learning and using the flight management system. It is well known that sharp reductions in pilot proficiency occur when pilots are asked to perform tasks that are time-critical, occur very infrequently, and are not guided by salient visual cues on the user-interface. This paper describes the results of an analysis of the pilot human-computer interaction required to respond to 67 flight management system error messages from a representative modem flight management system. Thirty-six percent of the messages require prompt pilot response, occur very infrequently, and are not guided by visual cues. These results explain, in part, issues with pilot proficiency, and demonstrate the need for deliberate design of the messages to account for the properties of human-computer interaction. Guidelines for improved training and design of the error messages are discussed.

The Flight Management System (FMS) has been identified by researchers, airline pilots, and airline instructors as hard to learn and difficult to use. Using the FMS to execute airline missiontasksrequiresthedevelopmentandmaintenanceofapilot’scognitiveskillstointeract with the FMS user-interface. This cognition is guided by visual cues (e.g. labels, prompts), user-interfaceconventions,andmemorizedactionsequences.Pilotactionspromptedbyvisual cues on the user-interface take less time to learn and reduce the likelihood of errors while performing infrequent tasks in revenue-service operations. The analysis described in this paper identified the presence of visual cues to completely guide all pilot actions for twenty five percent (25%) of 102 airline mission tasks performed using a modern FMS. Further, forty-five percent (45%) of the tasks were identified as occurring infrequently and were not completely supported by salient visual cues. The low percentage of tasks supported entirely by visual cues contributes to pilots perceptions about the difficulty in learning and using the FMS. Implications for training the FMS and the design of improved user-interfaces are discussed.

The Cognitive Walkthrough for the Web (CWW) is a partially automated usability evaluation method for identifying and repairing website navigation problems. Building on five earlier experiments [3,4], we first conducted two new experiments to create a sufficiently large dataset for multiple regression analysis. Then we devised automatable problem-identification rules and used multiple regression analysis on that large dataset to develop a new CWW formula for accurately predicting problem severity. We then conducted a third experiment to test the prediction formula and refined CWW against an independent dataset, resulting in full cross-validation of the formula. We conclude that CWW has high psychological validity, because CWW gives us (a) accurate measures of problem severity, (b) high success rates for repairs of identified problems (c) high hit rates and low false alarms for identifying problems, and (d) high rates of correct rejections and low rates of misses for identifying non-problems.

We have developed design rules for diagrammatic instructions for initial setup, basic maintenance and troubleshooting (i.e., clearing paper jams) tasks. The study reported here evaluated these rules. Four groups of novice users cleared paper jams in a laser printer using one of four different diagrammatic instructions. Diagrams presented to the first two groups all followed the rules but differed in the number of actions per diagram (one verses 3-4). The remaining groups' instructions contained diagrams that violated one or more rules. Instructions that followed the diagram design rules resulted in no errors. Diagrams that prevented users from correctly identifying the location of a subtask resulted in the most severe errors. Other rule violations resulted in fewer errors of lesser severity so if users were shown the general location of the problem they could perform a subtask. Times to complete each subtask were similar unless the location rule was violated.

This cooperative agreement supported Dr. Peter Polson's participation in two interrelated research programs. The first was the development of the Situation-Goal-Behavior (SGB) Model that is both a formal description of an avionics system's logic and behavior and a representation of a system that can be understood by avionics designers, pilots, and training developers. The second was the development of a usability inspection method based on an approximate model, RAFIV, of pilot interactions with the Flight Management System (FMS). The main purpose of this report is to integrate the two models and provide a context in order to better characterize the accomplishments of this research program. A major focus of both the previous and this Cooperative Agreement was the development of usability evaluation methods that can be effectively utilized during all phases of the design, development, and certification process of modern avionics systems. The current efforts to validate these methods have involved showing that they generate useful analyses of known operational and training problems with the current generation of avionics systems in modern commercial airliners. This report is organized into seven sections. Following the overview, the second section describes the Goal-Situation-Behavior model and its applications. The next section summarizes the foundations of the RAFIV model and describes the model in some detail. The contents of both these sections are derived from previous reports referenced in footnotes. The fourth section integrates these two models into a complete design evaluation and training development framework. The fifth section contains conclusions and possible future directions for research. References are in Section 6. Section 7 contains the titles and abstracts of the papers paper describing in more detail the results of this research program.

A completed cognitive task analysis (CTA) describes theskills and conceptual knowledge necessary to competently/expertly perform complex jobs/tasks, such as piloting a commercial aircraft. The primary motivation for doing a CTA is to improve the performance of human-machine systems by developing better training programs, developing tests to certify job competence, improving teamwork, and/or designing computer systems to support human workers.

Methods for identifying usability problems in web page designs should ideally also provide practical methods for repairing the problems found. Blackmon et al. [2] proved the usefulness of the Cognitive Walkthrough for the Web (CWW) for identifying three types of problems that interfere with users' navigation and information search tasks. Extending that work, this paper reports a series of two experiments that develop and prove the effectiveness of both full-scale and quick-fix CWW repair methods. CWW repairs, like CWW problem identification, use Latent Semantic Analysis (LSA) to objectively estimate the degree of semantic similarity (information scent) between representative user goal statements (100-200 words) and heading/link texts on each web page. In addition to proving the effectiveness of CWW repairs, the experiments reported here replicate CWW predictions that users will face serious difficulties if web developers fail to repair the usability problems that CWW identifies in web page designs [2].