Static analysis (SA) tools examine code for flaws without executing the code, and produce warnings ("alerts") about possible flaws. A human auditor then evaluates the validity of the purported code flaws. The effort required to manually audit all alerts and repair all confirmed code flaws is often too much for a project's budget and schedule. An alert triaging tool enables strategically prioritizing alerts for examination, and could use classifier confidence. We developed and tested classification models that predict if static analysis alerts are true or false positives, using a novel combination of multiple static analysis tools, features from the alerts, alert fusion, code base metrics, and archived audit determinations. We developed classifiers using a partition of the data, then evaluated the performance of the classifier using standard measurements, including specificity, sensitivity, and accuracy. Test results and overall data analysis show accurate classifiers were developed, and specifically using multiple SA tools increased classifier accuracy, but labeled data for many types of flaws were inadequately represented (if at all) in the archive data, resulting in poor predictive accuracy for many of those flaws.

: Coding errors cause the majority of software vulnerabilities. For example, 64% of the nearly 2,500 vulnerabilities in the National Vulnerability Database in 2004 were caused by programming errors. The CERT Division's Source Code Analysis Laboratory (SCALe) offers conformance testing of C language software systems against the CERT C Secure Coding Standard and the CERT Oracle Secure Coding Standard for Java, using various analysis tools available from commercial software vendors. Unfortunately, the current SCALe analysis process and tools do not collect any statistics about the accuracy of the code analysis tools or about the coding violations they flag, such as frequency of occurrence. This paper describes the approach used to add the ability to collect and statistically analyze data regarding coding violations and tool characteristics along with the initial results. The collected data will be used over time to improve the effectiveness of the SCALe analysis.

Software Quality Professional sporadically publishes discussions among some of the active members of the Software Division on controversial topics. Following is one of those discussions.

: Extensive cost overruns in major defense programs are common, and studies have identified poor cost estimation as a main contributor. Research and experience have identified several factors associated with poor cost estimates. These include the following: (1) optimistic expectations about the program's scope and technology such that it can be delivered on schedule and within budget; (2) the enormous amount of unknowns and uncertainty that exist when these estimates are made about large-scale, unprecedented systems that take years to develop and deploy; and (3) the heavy reliance, of necessity, on expert judgment. In this paper, we describe a new, integrative approach for pre-Milestone A cost estimation called quantifying uncertainty in early life cycle cost estimation (QUELCE). QUELCE synthesizes scenario building, Bayesian belief network modeling, and Monte Carlo simulation into an estimation method that quantifies uncertainties, allows subjective inputs, visually depicts influential relationships among change drivers and outputs, and assists with explicit description and documentation underlying an estimate. We use scenario analysis and dependency structure matrix techniques to limit the combinatorial effects of multiple interacting program change drivers to make modeling and analysis more tractable. Finally, we describe results and insights gained from applying the method retrospectively to a major defense program.

This paper describes an analysis that justifies applying the pattern "Increased Review for Intellectual Property (IP) Theft by Departing Insiders." The pattern helps organizations plan, prepare, and implement a strategy to mitigate the risk of insider theft of IP. The analysis shows that organizations can reduce their risk of insider theft of IP through increased review of departing insiders' actions during a relatively small window of time prior to their departure. Preliminary research results show that approximately 70 percent of insider IP thieves can be caught by following the pattern's recommendation of reviewing insiders' actions for theft events during only the last two months of their employment. These results provide practical guidance for practitioners wishing to fine tune the application of the pattern for their organizations. "Increased Review for IP Theft by Departing Insiders" is part of the CERT® Insider Threat Center's evolving library of enterprise architectural patterns for mitigating the insider threat, based on the Center's collected data. The Center's larger goal is to foster greater organizational resilience to insider threat, using repeated application of patterns from the library.

: Difficulties with estimating the costs of developing new systems have been well documented, and are compounded by the fact that estimates are now prepared much earlier in the acquisition lifecycle, before there is concrete technical information available on the particular program to be developed. This report describes an innovative synthesis of analytical techniques into a cost estimation method that models and quantifies the uncertainties associated with early lifecycle cost estimation. The method described in this report synthesizes scenario building, Bayesian Belief Network (BBN) modeling and Monte Carlo simulation into an estimation method that quantifies uncertainties, allows subjective inputs, visually depicts influential relationships among program change drivers and outputs, and assists with the explicit description and documentation underlying an estimate. It uses scenario analysis and design structure matrix (DSM) techniques to limit the combinatorial effects of multiple interacting program change drivers to make modeling and analysis more tractable. Representing scenarios as BBNs enables sensitivity analysis, exploration of scenarios, and quantification of uncertainty. The methods link to existing cost estimation methods and tools to leverage their cost estimation relationships and calibration. As a result, cost estimates are embedded within clearly defined confidence intervals and explicitly associated with specific program scenarios or alternate futures.

The Department of Defense (DoD) is becoming increasingly aware of the importance of data quality to its operations, leading to an interest in methods and techniques that can be used to determine and improve data quality.The Office of the Secretary of Defense for Acquisition, Technology, and Logistics (OSD [AT&L]), Director, Defense Research & Engineering (DDR&E) sponsored a workshop to bring together leading researchers and practitioners to identify opportun-

Organizations rely on valid data to make informed decisions. When data integrity is compromised, the veracity of the decision-making process is likewise threatened. Detecting data anomalies and defects is an important step in understanding and improving data quality. The study described in this report investigated statistical anomaly detection techniques for identifying potential errors associated with the accuracy of quantitative earned value management (EVM) data values reported by government contractors to the Department of Defense. This research demonstrated the effectiveness of various statistical techniques for discovering quantitative data anomalies. The following tests were found to be effective when used for EVM variables that represent cumulative values: Grubbs' test, Rosner test, box plot, autoregressive integrated moving average (ARIMA), and the control chart for individuals. For variables related to contract values, the moving range control chart, moving range technique, ARIMA, and Tukey box plot were equally effective for identifying anomalies in the data. One or more of these techniques could be used to evaluate data at the point of entry to prevent data errors from being embedded and then propagated in downstream analyses. A number of recommendations regarding future work in this area are proposed in this report.

Just as oil fueled the industrial revolution, data are fueling progress in the digital revolution. As the dependence on data grows, data quality research efforts are growing as well.

Abstract : An organization's measurement and analysis infrastructure directly Impacts the quality of the decisions made by people at all organizational levels Ensuring information quality is a challenge for most organizations-partly because they might not be fully aware of their own data quality levels Without this information they cannot know the full business impact of poor or unknown data quality or determine how to begin improving their data This report describes common errors in measurement and analysis and the need for a criterion-based assessment method that will allow organizations to evaluate key characteristics of their measurement programs.

The output of 18 software architecture evaluations is analyzed. The goal of the analysis is to find patterns in the important quality attributes and risk themes identified in the evaluations. The major results are*A categorization of risk themes. *The observation that twice as many risk themes are risks of ''omission'' as are risks of ''commission''. *A failure to find a relationship between the business and mission goals of a system and the risk themes from an evaluation of that system. *A failure to find a correlation between the domain of a system being evaluated and the important quality attributes for that system. *A wide diversity of names used for various quality attributes. The results of this investigation have application to practitioners by suggesting activities on which developers should put greater focus. They also have application to researchers by suggesting further areas of investigation.

The output of 18 software architecture evaluations are analyzed to find patterns in the risk themes identified in the evaluations. The major results are: i) A categorization of risk themes ii) The observation that twice as many risk themes are risks of "omission " as are risks of "commission ". iii) A failure to find a relationship between the business and mission goals of a system and the risk themes from an evaluation of that system. iv) A failure to find a relationship between the domain of a system being evaluated and the risk themes associated with the development of that system. The results of this investigation have application to practitioners by suggesting activities on which developers should put greater focus. They also have application to researchers by suggesting further areas of investigation.