Background: Using design metrics to predict fault-prone elements of a software design can help to focus attention on classes that need redesign and more extensive testing. However, some design metrics have been pointed out to be theoretically invalid, and the usefulness of some metrics is questioned. Aim: To identify a set of object-oriented metrics that are theoretically valid, and useful for identifying fault-prone classes in a design. Method: Drawing on four well-known sets of design metrics (CK, LK, MOOD and QMOOD), we propose a consolidated set of metrics that covers many aspects of object-oriented software design. We conduct two experiments, first using a single large system and then considering successive releases of that system, to compare the usefulness of the consolidated set with the other four sets for within-project prediction of fault-prone classes. Results: Both experiments suggest the consolidated set is effective at identifying fault-prone classes, outperforming the other metric sets (though at a cost of more false alarms). Conclusion: This paper adds to knowledge about the usefulness of existing sets of design metrics for within-project defect prediction, and identifies a consolidated set of metrics that is more effective than the existing sets at identifying fault-prone classes.

To date most research in software effort estimation has not taken into account any form of chronological split when selecting projects for training and testing sets. A chronological split represents the use of a project's starting and completion dates, such that any model that estimates effort for a new project p only uses as its training set projects that were completed prior to p's starting date. Three recent studies investigated the use of chronological splits, using a type of chronological split called a moving window, which represented a subset of the most recent projects completed prior to a project p's starting date. They found some evidence in favour of using windows whenever projects were recent. These studies all defined window sizes as being fixed numbers of recent projects. In practice, we suggest that estimators are more likely to think in terms of elapsed time than the size of the data set, when deciding which projects to include in a training set. Therefore, this paper investigates the effect on accuracy when using moving windows of various durations to form training sets on which to base effort estimates. Our results show that the use of windows based on duration can affect the accuracy of estimates (in this data set, a window of about three years duration appears best), but to a lesser extent than windows based on a fixed number of projects.

Context The research literature on software development projects usually assumes that effort is a good proxy for cost. Practice, however, suggests that there are circumstances in which costs and effort should be distinguished. Objectives: We determine similarities and differences between size, effort, cost, duration, and number of defects of software projects. Method: We compare two established repositories (ISBSG and EBSPM) comprising almost 700 projects from industry. Results: We demonstrate a (log)-linear relation between cost on the one hand, and size, duration and number of defects on the other. This justifies conducting linear regression for cost. We establish that ISBSG is substantially different from EBSPM, in terms of cost (cheaper) and duration (faster), and the relation between cost and effort. We show that while in ISBSG effort is the most important cost factor, this is not the case in other repositories, such as EBSPM in which size is the dominant factor. Conclusion: Practitioners and researchers alike should be cautious when drawing conclusions from a single repository.

CONTEXT: Several studies in effort estimation havefound that it can be effective to use only recent project data for building an effort estimation model. The generality of this timeaware approach has been explored across a variety of effort estimation model approaches, organizations and definitions of recency. However, other studies have shown that it is not alwayshelpful. A question arises: how can one tell whether the approachwould be effective for a given target project? OBJECTIVE: Toinvestigate a potential method to decide between selecting recentor all project data. METHOD: Using a single-company ISBSGdata set1 studied previously in similar research, we propose andevaluate a selection method. The method utilizes a variant ofcross-validation based on recent projects to make the decision.RESULTS: There are significant differences in the estimation accuracybetween using the proposed method and using the growingportfolio (always using all available data). The method could alsoselect the better approach on average. However, the differencein estimation accuracy between using the proposed method andalways using moving windows was not statistically significant.CONCLUSIONS: The selection method could select the betterapproach on average. The results contribute to developing amethod for suggesting a better approach for practitioners.

Autonomous systems are making their way to the market. The transition from tasks performed by humans to tasks performed by machines begs for an answer to one of the most challenging questions in this area of research: Will humans understand and trust what a machine does? Analyzing human and machine behaviors offers the foundational steps toward finding answers to this question. This paper contributes a novel methodology for transforming low-level actions by each agent into high-level categorization of strategies to contrast the behaviors of humans and machines using a computational red teaming environment with a red (evader) and a blue (pursuer) agent. Two orthogonal sources of uncertainty were examined: the uncertainty in the blue agent's situation awareness about the red, and the red agent's uncertainty resulting from deceptive actions by the blue. For each uncertainty source, two different experiments were conducted by varying the controller of the red agent. In one experiment, the red agent was controlled by one of the 34 human subjects; and in the second, by an evolved neural network. The blue agent was controlled by a scripted rule-based system. In this time-critical task, the results revealed that humans tend to follow systemic and consistent strategies, sometimes ignoring the information available to them. On the other hand, machines tend to evolve more complex and diverse strategies. This finding calls for new computational intelligence techniques to enable the fusion of these different strategies into forms that each party can understand and use effectively.

CONTEXT: Studies have shown contradictory results on the effectiveness of using a moving window of only the most recent projects for effort estimation, compared to using the full history of past data. Moving windows improved the accuracy of effort estimates for a single-company subset of the ISBSG dataset ( www.isbsg.org ), but not for three single-company subsets of the Finnish dataset ( www.4sumpartners.com ). The contradiction may be caused by different characteristics of the data sets: in particular, they differ noticeably in heterogeneity of industry sector. GOAL: To investigate the effect on estimation accuracy of differences in the characteristics of the data sets. METHOD: Conduct an experiment with a virtual data set, composed from the three subsets of the Finnish dataset. The composite data set is similar to the ISBSG subset in that it includes data from multiple industry sectors; the largest group of projects in both data sets comes from the same industry sector; and in both data sets the projects are concentrated in a similar number of years. RESULTS: The conclusions is the same as in the past study using the individual Finnish subsets: in the composite data set, moving windows are of no help. CONCLUSIONS: In this instance, increased heterogeneity of projects does not explain the contradiction. It is still not clear when windows may be helpful. Practitioners and researchers should not assume automatically that only the most recent data is best for effort estimation.

Ensemble methods aim at combining multiple learning machines to improve the efficacy in a learning task in terms of prediction accuracy, scalability, and other measures. These methods have been applied to evolutionary machine learning techniques including learning classifier systems (LCSs). In this article, we first propose a conceptual framework that allows us to appropriately categorize ensemble‐based methods for fair comparison and highlights the gaps in the corresponding literature. The framework is generic and consists of three sequential stages: a pre‐gate stage concerned with data preparation; the member stage to account for the types of learning machines used to build the ensemble; and a post‐gate stage concerned with the methods to combine ensemble output. A taxonomy of LCSs‐based ensembles is then presented using this framework. The article then focuses on comparing LCS ensembles that use feature selection in the pre‐gate stage. An evaluation methodology is proposed to systematically analyze the performance of these methods. Specifically, random feature sampling and rough set feature selection‐based LCS ensemble methods are compared. Experimental results show that the rough set‐based approach performs significantly better than the random subspace method in terms of classification accuracy in problems with high numbers of irrelevant features. The performance of the two approaches are comparable in problems with high numbers of redundant features.

BACKGROUND: Several studies in software effort estimation have found that it can be effective to use a window of recent projects as training data for building an effort estimation model. The generality of the windowing approach still remains uncertain across the variety of effort estimation approaches that are based on different theory. Recent studies have focused on the use of windows with effort estimation models based on a machine learning approach, which could make better estimates than conventional linear regression. OBJECTIVE: To investigate the effect of using a window on estimation accuracy with a machine learning-based method, Artificial Neural Networks (ANN). ANN was recently found as a popular and good performance method, and is based on a different theory from other Machine Learning-based methods used in past studies. METHOD: Using a single-company ISBSG dataset studied previously in similar research, we examine the effect of using a fixed-size windowing policy on the accuracy of estimates using ANN. RESULTS: There is a difference in the estimation accuracy between using a window and not using a window. Using windows of 50 to 120 projects reduced mean absolute errors by 5-7%. The effective range of window sizes was different from previous studies. CONCLUSIONS: Windowing significantly improves estimation accuracy with ANN. The results support past studies, in that the effective window sizes were different among estimation models. The results contribute to understanding characteristics of the windowing approach.

CONTEXT: Recent studies have shown that estimation accuracy can be affected by only using a window of recent projects (instead of all past projects) as training data for building an effort estimation model. The effect and its extent can be affected by the effort estimation methods used, and the windowing policy used (fixed size or fixed duration). The generality of the windowing approach remains uncertain, because only a few effort estimation methods have been examined with each policy. OBJECTIVE: To investigate the effect on estimation accuracy of using the fixed-duration window policy, particularly in comparison to the fixed-size window policy, when using Classification and Regression Trees (CART) as the estimation method. METHOD: Using a single-company ISBSG data set studied previously in similar research, we examine the effects of using a fixed-duration windowing policy on the accuracy of estimates using CART. RESULTS: Fixed-duration windows rarely improve the accuracy of estimates with CART, compared to using all past projects as training data. Few window sizes lead to statistically significant differences. The effect is smaller than when fixed-size windows are used. CONCLUSIONS: Fixed-duration windows are not helpful with this data set when using CART as the estimation method. The results support the preference for the fixed-size window policy that was found in previous research. This contributes further to understanding the effect of using windows.

Context: Recent studies have shown that estimation accuracy can be affected by only using a window of recent projects as training data for building an effort estimation model. The idea has been extended for regression-based estimation by weighting projects differently according to their order within the window. This significantly improved the accuracy of estimation in a single-company dataset from the ISBSG repository.Objective: To investigate the effects on estimation accuracy of using weighted moving windows with a new dataset, and compare results across datasets.Method: Using a dataset drawn from the Finnish dataset (studied previously with regard to windows but not with weighting), and using a fixed-size window policy, we examine the effect on estimation accuracy of using weighted moving windows.Results: The use of weighting functions could improve the estimation accuracy significantly, compared to using unweighted windows, with larger window sizes. The steepness of the weighting functions affects their effectiveness. However, in this dataset it is better to use a growing portfolio (retaining all past projects as training data) than to use windows.Conclusions: The results reinforce previous studies: the use of weighting functions can significantly improve the accuracy of regression-based estimation, compared to not using weighting, but in this dataset the use of moving windows reduces estimation accuracy.

BACKGROUND: Several studies in software effort estimation have shown that it can be effective to use a window of recent projects, rather than the growing portfolio of all past projects, as training data. Practitioners need a method for choosing between those approaches when estimating effort for a target project. OBJECTIVE: This study examined the usefulness of the fitted values for choosing between moving windows and the growing portfolio. METHOD: An empirical experiment was conducted with a single-company ISBSG dataset. RESULTS: The fitted values could be useful for the selection on average. CONCLUSIONS: This positive result contributes to understanding when windows may be appropriate.

The International Software Benchmarking Standards Group (ISBSG) maintains a repository of data about completed software projects. A common use of the ISBSG dataset is to investigate models to estimate a software project's size, effort, duration, and cost. The aim of this paper is to determine which and to what extent variables in the ISBSG dataset have been used in software engineering to build effort estimation models. For that purpose a systematic mapping study was applied to 107 research papers, obtained after a filtering process, that were published from 2000 until the end of 2013, and which listed the independent variables used in the effort estimation models. The usage of ISBSG variables for filtering, as dependent variables, and as independent variables is described. The 20 variables (out of 71) mostly used as independent variables for effort estimation are identified and analysed in detail, with reference to the papers and types of estimation methods that used them. We propose guidelines that can help researchers make informed decisions about which ISBSG variables to select for their effort estimation models. (C) 2015 Elsevier Inc. All rights reserved.

Context: Recent studies have shown that estimation accuracy can be affected by only using a window of recent projects as training data for building an effort estimation model. The effect and its extent can be affected by effort estimation methods (e.g. linear regression (LR) or estimation by analogy (EbA)), windowing policies (fixed-size or fixed-duration), and between organizations. However, different effects between organizations have only been explored with LR as the estimation method, and different effects between estimation methods and windowing policies have mainly been explored with data from only one organization. Objective: To further investigate the effect on estimation accuracy of using windows, with different windowing policies, when using EbA as the estimation method. Also, to compare the effect of LR with EbA as an estimation method, when using windows. Method: Using a data set studied with LR in previous research, we examine the effects of using windows on the accuracy of effort estimates, using EbA with both fixed-size and fixed-duration windowing policies. Results: With this data set, fixed-size windows, no matter their size, do not improve the accuracy of estimates obtained using EbA. This reinforces previous research with this data set, which used LR as the estimation approach. However, fixed-duration windows can improve the accuracy of estimates obtained with EbA. This contradicts previous research with this data set, which used LR as the estimation approach. Variations in the settings for EbA can change the sizes at which windows are helpful. Conclusions: This study reinforces that the effect of using windows can be affected by the effort estimation approach, and by the windowing policy. Contrary to previous research, fixed-duration windows are found to be more helpful than fixed-size windows, and significant improvements are found with EbA that were not found with LR. Further research is needed to understand these differences.

In construction of an effort estimation model, it seems effective to use a window of training data so that the model is trained with only recent projects. Considering the chronological order of projects within the window, and weighting projects according to their order within the window, may also affect estimation accuracy. In this study, we examined the effects of weighted moving windows on effort estimation accuracy. We compared weighted and non‐weighted moving windows under the same experimental settings. We confirmed that weighting methods significantly improved estimation accuracy in larger windows, although the methods also significantly worsened accuracy in smaller windows. This result contributes to understanding properties of moving windows. Copyright © 2014 John Wiley & Sons, Ltd.

Context: Evolutionary algorithms have proved to be successful for generating test data for path coverage testing. However in this approach, the set of target paths to be covered may include some that are infeasible. It is impossible to find test data to cover those paths. Rather than searching indefinitely, or until a fixed limit of generations is reached, it would be desirable to stop searching as soon it seems likely that feasible paths have been covered and all remaining un-covered target paths are infeasible.Objective: The objective is to develop criteria to halt the evolutionary test data generation process as soon as it seems not worth continuing, without compromising testing confidence level.Method: Drawing on software reliability growth models as an analogy, this paper proposes and evaluates a method for determining when it is no longer worthwhile to continue searching for test data to cover uncovered target paths. We outline the method, its key parameters, and how it can be used as the basis for different decision rules for early termination of a search. Twenty-one test programs from the SBSE path testing literature are used to evaluate the method.Results: Compared to searching for a standard number of generations, an average of 30-75% of total computation was avoided in test programs with infeasible paths, and no feasible paths were missed due to early termination. The extra computation in programs with no infeasible paths was negligible.Conclusions: The method is effective and efficient. It avoids the need to specify a limit on the number of generations for searching. It can help to overcome problems caused by infeasible paths in search-based test data generation for path testing. (C) 2014 Elsevier B.V. All rights reserved.

Several studies in software effort estimation have found that it can be effective to use a window of recent projects as training data for building an effort estimation model. Windows can be defined as having a fixed size (containing a fixed number of projects), or as having a fixed duration. A recent study extended the idea of windows, by weighting projects differently according to their order within the window, and found that weighted moving windows could significantly improve estimation accuracy. That study used fixed-size windows. This study examines the effect on effort estimation accuracy of weighted moving windows that are based on fixed duration. We compare weighted and unweighted moving windows under the same experimental settings. Weighting methods are found to improve estimation accuracy significantly in larger windows, and the methods also significantly improved accuracy in smaller windows in terms of MRE. This result contributes further to understanding properties of moving windows.

Classification rule induction involves two main processes: finding the optimal conjuncts (attribute intervals or attribute-value pairs) and their combination (disjuncts or rules) to classify different concepts in the data. The evolutionary rule learning approaches employ an evolutionary algorithm, such as a genetic algorithm, to perform both these search operations simultaneously. This approach often leads to significant problems including population bloating and stalled evolutionary search in real-valued attribute problems, especially with higher dimensions. In this paper, we present an online evolutionary rule learning approach referred to as ERL-AID that decouples the above search processes and employs a discretization algorithm that works on the attribute space and a genetic algorithm to combine the discretized attributes into appropriate classification rules. ERL-AID applies a sliding window approach to process inputs in an online fashion. The proposed system is able to produce compact rule sets with competitive performance and could scale to higher dimensions. The experimental results show the competitiveness of our algorithm.

BACKGROUND: Several studies in software effort estimation have found that it can be effective to use a window of recent projects as training data for building an effort estimation model. The previous studies evaluated the use of a window with popular estimation models: linear regression (LR) and estimation by analogy (EbA). Many effort estimation models have been proposed, and the generality of windowing approach still remains uncertain for other effort estimation models, especially for those based on different theory. OBJECTIVE: This study investigates the effect of using a window on estimation accuracy with Classification and regression trees (CART). CART was recently found as a good performance method, and is based on a different theory from LR and EbA. METHOD: We compared the estimation accuracy of a windowing approach and growing approach with the same data set and procedure as the past studies. RESULTS: There is a difference in the estimation accuracy between using a window and not using a window. However, the effctive range of using windows on CART is narrower than that on LR. CONCLUSIONS: Windowing is also effective with CART. However, the range of effectiveness is narrower. The results contribute to the generality of the effectiveness of windowing approach.

Effort estimation for software customisation projects is a challenging task. Similar to the COCOMO approach, an initial estimate can be based on basic project characteristics, and then modified according to various productivity drivers. The aim of this paper is to build on an initial estimation model, generated from a set of contributing factors for software customisation projects, and to improve the accuracy of the model through calibration of effort multipliers derived from customisation project characteristics. Projects from Release 12 of the ISBSG Data Repository were analysed to see which of the productivity drivers were usefully related to variations in software productivity, after other project characteristics had been taken into account. The effect on estimation accuracy of multiplying an initial estimate by each of these productivity drivers was evaluated, in terms of changes in mean and median absolute error, and mean and median magnitude of relative error. Finally, a set of useful productivity drivers is identified for customisation projects, and their effect is quantified. They are found to improve effort estimation accuracy significantly.

Given the unprecedented growth in air transport, air traffic controllers and safety managers are exploring new approaches for collision risk assessment specifically in real time or near future term. In this paper a real time predictive approach for identifying worst case sector collision risk is proposed. An evolutionary framework is introduced to evolve flight maneuvers that may lead a traffic scenario to high collision risk. The proposed methodology discretizes the execution time of a baseline air traffic scenario into discrete time intervals based traffic scenarios. These traffic scenarios are then initialized at their given time intervals in a sector which are then perturbed with flight maneuvers to maximize collision risk. Flight maneuvers are evolved using an evolutionary framework with the objective of maximizing the collision risk for a given traffic scenario. Results indicate the effectiveness of the proposed methodology to successfully identify flight maneuvers which may increases the collision risk for a look ahead time. Results also indicate that when most of the flights are in the first half of their flight path, DESCEND and TURN RIGHT maneuvers may increase the collision risk. Also maneuvers, when flights are exiting the sector, do not significantly effect the collision risk. It was also found that traffic flow should be managed when flights enter the sector or when flights are exiting the sector (to facilitate coordination with neighboring sector controller) as this may reduce the collision risk.