Reproducibility is important for having confidence in evolutionary machine learning algorithms. Although the focus of reproducibility is usually to recreate an aggregate prediction error score using fixed random seeds, this is not sufficient. Firstly, multiple runs of an algorithm, without a fixed random seed, should ideally return statistically equivalent results. Secondly, it should be confirmed whether the expected behaviour of an algorithm matches its actual behaviour, in terms of how an algorithm targets a reduction in prediction error. Confirming the behaviour of an algorithm is not possible when using a total error aggregate score. Using an error decomposition framework as a methodology for improving the reproducibility of results in evolutionary computation addresses both of these factors. By estimating decomposed error using multiple runs of an algorithm and multiple training sets, the framework provides a greater degree of certainty about the prediction error. Also, decomposing error into bias, variance due to the algorithm (internal variance), and variance due to the training data (external variance) more fully characterises evolutionary algorithms. This allows the behaviour of an algorithm to be confirmed. Applying the framework to a number of evolutionary algorithms shows that their expected behaviour can be different to their actual behaviour. Identifying a behaviour mismatch is important in terms of understanding how to further refine an algorithm as well as how to effectively apply an algorithm to a problem.

Bagging is a commonly used ensemble method, which involves the bootstrap sampling of training observations. Bootstraps are typically needed to decouple the models. However, this is based on the well-understood properties of deterministic algorithms. To perform bagging for stochastic algorithms, there is an implicit assumption that both variance due to the training data and variance due to the algorithm need to be reduced. This assumption may not be correct and only a reduction in variance due to the algorithm may need to be targeted. Bootstrapping may cause an unnecessary increase in error due to bias because it reduces the number of unique training observations. Bootstrapped and non-bootstrapped ensembles are compared across multiple machine learning algorithms and data sets using an extended error decomposition. The results show that bootstrap sampling is often not required as it increases error due to bias and internal variance. The prediction error associated with an algorithm and data set needs to be more fully decomposed in order to distinguish between the different sources of error. This allows the most appropriate ensemble method to be chosen.

Movement analysis is distinguished by an emphasis on understanding via observation and association. However, an important component of movement from the human and computer modeling perspective is the processes that bring about movement behavior in the first place. This article contextualizes the graphical causal modeling framework (for association, intervention, and counterfactual causal analysis) in GIScience, and more specifically within movement analysis studies. This is done by modeling the movement behavior of football players, applied to spatiotemporal data generated by an agent‐based simulation. The movement dataset is thoroughly analyzed to infer the statistical associations among its variables, to estimate the effect of an intervention on some of those variables, and to answer a few counterfactual questions from the observations. We conclude that causal graphs (i.e., directed acyclic graphs), if implemented correctly, can assist analysts in infering causal relations from movement data. This research suggests the integration of causal graphs and agent‐based paradigms as one solution for computational movement analysis.

A common method for quantifying the interdisciplinarity of a publication is to measure the diversity of the publication’s cited references based on their disciplines. Here we examine the criteria that must be satisfied to develop a meaningful interdisciplinary measure based on citations and discuss the stages where uncertainty or bias may be introduced. In addition, using the Rao-Stirling diversity measure as an exemplar for such citation-based measures, we show how bootstrapping can be used to estimate a confidence interval for interdisciplinarity. Using an academic publication database, this approach is used to develop and assess a reliability measure for interdisciplinarity that extends current methods. Our results highlight issues with citation analysis for measuring interdisciplinarity and offer an approach to improve the confidence in assessing this concept. Specific guidelines for assessing the confidence in the Rao-Stirling diversity measure and subsequently other similar diversity measures are presented, hopefully reducing the likelihood of drawing false inferences about interdisciplinarity in the future.

Grammars provide a convenient and powerful mechanism to define the space of possible solutions for a range of problems. While recent work has shed light on the matters of initialisation and grammar design with respect to grammatical evolution (GE), their impact on other methods, such as random search and context-free grammar genetic programming (CFG-GP), is largely unknown. This paper examines GE, random search and CFG-GP on benchmark problems using different initialisation routines and grammar designs. Results suggest that CFG-GP is less sensitive to initialisation and grammar design than both GE and random search: we also demonstrate that observed cases of poor performance by CFG-GP are managed through simple adjustment of tuning parameters. We conclude that CFG-GP is a strong base from which to conduct grammar-guided evolutionary search, and that future work should focus on understanding the parameter space of CFG-GP for better application.

Genetic programming (GP) is a common method for performing symbolic regression that relies on the use of ephemeral random constants in order to adequately scale predictions. Suitable values for these constants must be drawn from appropriate, but typically unknown, distributions for the problem being modeled. While rarely used with GP, $Z$ -score standardization of feature and response spaces often significantly improves the predictive performance of GP by removing scale issues and reducing error due to bias. However, in some cases it is also associated with erratic error due to variance. This article demonstrates that this variance component increases in the presence of gaps at the boundaries of the training data explanatory variable intervals. An initial solution to this problem is proposed that augments training data with pseudo instances located at the boundaries of the intervals. When applied to benchmark problems, particularly with small training samples, this solution reduces error due to variance and, therefore, total error. Augmentation is shown to also stabilize error in larger problems; however, results suggest that standardized GP works well on such problems with little need for training data augmentation.

Modelling a complex system of autonomous individuals moving through space and time essentially entails understanding the (heterogeneous) spatiotemporal context, interactions with other individuals, their internal states and making any underlying causal interrelationships explicit, a task for which agents (including vector-agents) are specifically well-suited. Building on a conceptual model of agent space-time and reasoning behaviour, a design guideline for an implemented vector-agent model is presented. The movement of football players was chosen as it is appropriately constrained in space, time and individual actions. Sensitivity-variability analysis was applied to measure the performance of different configurations of system components on the emergent movement patterns. The model output varied more when the condition of the contextual actors (players' role-areas) was manipulated. The current study shows how agent-based modelling can contribute to our understanding of movement and how causally relevant evidence can be produced, illustrated through a spatiotemporally constrained football case-study.

While interdisciplinary research (IDR) has attracted much attention, this has not yet resulted in a coherent body of knowledge of interdisciplinarity. One of the impediments is a lack of consensus on its conceptualization and measurement. Some of the proposed measures have shown to misalign empirically, meaning that conclusions about IDR can differ across measures. To clarify this disagreement conceptually, and to stimulate better coherence in measurement, this paper starts with a review of the IDR definitions. From a synthesis of these definitions, we provide a conceptual definition and a logical structure of the construct, and derive evaluation criteria for its measures. We use these to evaluate 21 measures of IDR. The results show that measures vary widely in meeting the criteria, which can explain some of the observed inconsistencies in earlier studies. We discuss the most common limitations and present empirical analyses to gauge their severity. We present several suggestions for future measurement of the interdisciplinarity of research. We hope that with these suggestions, researchers can draw more consistent conclusions, aiding in the development of a coherent body of knowledge of this ever-important phenomenon.

The important process of choosing between algorithms and their many module choices is difficult, even for experts. Automated machine learning allows users at all skill levels to perform this process. It is currently performed using aggregated total error, which does not indicate whether a stochastic algorithm or module is stable enough to consistently perform better than other candidates. It also does not provide an understanding of how the modules contribute to total error. This paper explores the decomposition of error for the refinement of genetic programming. Automated algorithm refinement is examined through choosing a pool of candidate modules and swapping pairs of modules to reduce the largest component of decomposed error. It is shown that a pool of candidates that are not examined for diversity in targeting different components of error can provide inconsistent module preferences. Manual algorithm refinement is also examined by choosing refinements based on their well-understood behaviour in reducing a particular error component. The results show that an effective process should exploit both the advantages of targeted improvements identified using a manual process and the simplicity of an automated process by choosing a hierarchy of the most important modules for reducing error components.

Current spatiotemporal data has facilitated movement studies to shift objectives from descriptive models to explanations of the underlying causes of movement. From both a practical and theoretical standpoint, progress in developing approaches for these explanations should be founded on a conceptual model. This paper presents such a model in which three conceptual levels of abstraction are proposed to frame an agent-based representation of movement decision-making processes: ‘attribute,’ ‘actor,’ and ‘autonomous agent’. These in combination with three temporal, spatial, and spatiotemporal general forms of observations distinguish nine (3 × 3) representation typologies of movement data within the agent framework. Thirdly, there are three levels of cognitive reasoning: ‘association,’ ‘intervention,’ and ‘counterfactual’. This makes for 27 possible types of operation embedded in a conceptual cube with the level of abstraction, type of observation, and degree of cognitive reasoning forming the three axes. The conceptual model is an arena where movement queries and the statement of relevant objectives takes place. An example implementation of a tightly constrained spatiotemporal scenario to ground the agent-structure was summarised. The platform has been well-defined so as to accommodate different tools and techniques to drive causal inference in computational movement analysis as an immediate future step.

While interdisciplinary research has attracted much attention, this has not yet resulted in a coherent body of knowledge of interdisciplinarity. One of the impediments is a lack of consensus on its conceptualization and measurement. Some of the proposed measures have shown to misalign empirically, meaning that conclusions about IDR can differ across measures. To clarify this disagreement conceptually, and to stimulate better coherence in measurement, this paper starts with a review of the IDR definitions. From a synthesis of these definitions, we provide a conceptual definition of the construct along with evaluation criteria for its measures. We use these to evaluate current measures of IDR. The results show that most measures meet the usability criteria, but measures vary widely in meeting other criteria, which can explain some of the observed inconsistencies in earlier studies. The "Integration score" from Rafols and Meyer performed best. By using the results in selecting measures, researchers can, in our view, draw more consistent conclusions, aiding in the development of a coherent body of knowledge of this ever-important phenomenon.

The Hill–Robertson effect describes how, in a finite panmictic diploid population, selection at one diallelic locus reduces the fixation probability of a selectively favoured allele at a second, linked diallelic locus. Here we investigate the influence of population structure on the Hill–Robertson effect in a population of sizeN. We model population structure as a network by assuming that individuals occupy nodes on a graph connected by edges that link members who can reproduce with each other. Three regular networks (fully connected, ring and torus), two forms of scale-free network and a star are examined. We find that (i) the effect of population structure on the probability of fixation of the favourable allele is invariant for regular structures, but on some scale-free networks and a star, this probability is greatly reduced; (ii) compared to a panmictic population, the mean time to fixation of the favoured allele is much greater on a ring, torus and linear scale-free network, but much less on power-2 scale-free and star networks; (iii) the likelihood with which each of the four possible haplotypes eventually fix is similar across regular networks, but scale-free populations and the star are consistently less likely and much faster to fix the optimal haplotype; (iv) increasing recombination increases the likelihood of fixing the favoured haplotype across all structures, whereas the time to fixation of that haplotype usually increased, and (v) star-like structures were overwhelmingly likely to fix the least fit haplotype and did so significantly more rapidly than other populations. Last, we find that small (N< 64) panmictic populations do not exhibit the scaling property expected from Hill & Robertson (1966Genet. Res.8, 269–294. (doi:10.1017/S0016672300010156)).

Software effort estimation (SEE) models are typically developed based on an underlying assumption that all data points are equally relevant to the prediction of effort for future projects. The dynamic nature of several aspects of the software engineering process could mean that this assumption does not hold in at least some cases. This study employs three kernel estimator functions to test the stationarity assumption in three software engineering datasets that have been used in the construction of software effort estimation models. The kernel estimators are used in the generation of non-uniform weights which are subsequently employed in weighted linear regression modeling. Prediction errors are compared to those obtained from uniform models. Our results indicate that, for datasets that exhibit underlying non-stationary processes, uniform models are more accurate than non-uniform models. In contrast, the accuracy of uniform and non-uniform models for datasets that exhibited stationary processes was essentially equivalent. The results of our study also confirm prior findings that the accuracy of effort estimation models is independent of the type of kernel estimator function used in model development.

Translocated populations often share demographic, environmental and genetic risks associated with relict populations. Models that predict translocation impacts on source and founder populations are therefore necessary to ensure that harvesting for release does not jeopardize either population. However, current models generally focus on maximizing the long‐term demographic viability and retention of allelic diversity in the founder population only. We therefore risk estimating a release number that is considered ‘appropriate' for maintaining viability and allelic diversity in the founder population but is in fact detrimental to the source. Our objectives were to determine a sustainable number of individuals for release that maximized the viability and allelic diversity of source and founder populations. We developed a spatially explicit individual‐based model that simulated hypothetical translocation scenarios for the threatened New Zealand frog, Leiopelma pakeka , which was restricted to Te Hoiere/Maud Island and has been heavily targeted for translocations. Source and translocated populations were simulated over 200 generations, representing 200 years across various initial population sizes from 20 to 300 frogs. To account for the potential loss of allelic diversity from the source population via harvesting, along with genetic drift and founder events typically associated with translocations, an additional set of simulations determined the probability of retaining a single diploid locus within populations with rare and common alleles. Our results showed that at least 120 frogs were required for populations to remain viable and maximize allelic retention in the long‐term, regardless of allelic diversity. However, harvesting more than 150 frogs led to ongoing or the onset of declines in the viability and allelic retention of source populations. We recommend that future translocations of L. pakeka harvest no more than 140 frogs from each of the sub‐populations on Te Hoiere/Maud Island. We also recommend the described approach to simulate hypothetical translocation scenarios for other managed species.

It seems logical to assert that the dynamic nature of software engineering practice would mean that software effort estimation (SEE) modelling should take into account project start and completion dates. That is, we should build models for future projects based only on data from completed projects; and we should prefer data from recent similar projects over data from older similar projects. Research in SEE modelling generally ignores these recommendations. In this study two different model development approaches that take project timing into account are applied to two publicly available datasets and the outcomes are compared to those drawn from three baseline (non-time-aware) models. Our results indicate: that it is feasible to build accurate effort estimation models using project timing information; that the models differ from those built without considering time, in terms of the parameters included and their weightings; and that there is no statistical significance difference as to which of the two model building approaches is superior in terms of accuracy.

Symbolic regression is a common application of genetic programming where model structure and corresponding parameters are evolved in unison. In the majority of work exploring symbolic regression, features are used directly without acknowledgement of their relative scale or unit. This paper extends recent work on the importance of standardisation of features when conducting symbolic regression. Specifically, z-score standardisation of input features is applied to both inputs and response to ensure that evolution explores a model space with zero mean and unit variance. This paper demonstrates that standardisation allows a simpler function set to be used without increasing bias. Additionally, it is demonstrated that standardisation can significantly improve the performance of coefficient optimisation through gradient descent to produce accurate models. Through analysis of several benchmark data sets, we demonstrate that feature standardisation enables simple but effective approaches that are comparable in performance to the state-of-the-art in symbolic regression.

An error function can be used to select between candidate models but it does not provide a thorough understanding of the behavior of a model. A greater understanding of an algorithm can be obtained by performing a bias-variance decomposition. Splitting the error into bias and variance is effective for understanding a deterministic algorithm such as k-nearest neighbor, which provides the same predictions when performed multiple times using the same data. However, simply splitting the error into bias and variance is not sufficient for nondeterministic algorithms, such as genetic programming (GP), which potentially produces a different model each time it is run, even when using the same data. This article presents an extended bias-variance decomposition that decomposes error into bias, external variance (error attributable to limited sampling of the problem), and internal variance (error due to random actions performed in the algorithm itself). This decomposition is applied to GP to expose the three components of error, providing a unique insight into the role of maximum tree depth, number of generations, size/complexity of function set, and data standardization in influencing predictive performance. The proposed tool can be used to inform targeted improvements for reducing specific components of model error.

Software effort estimation models are typically developed based on an underlying assumption that all data points are equally relevant to the prediction of effort for future projects. The dynamic nature of several aspects of the software engineering process could mean that this assumption does not hold in at least some cases. This study employs three kernel estimator functions to test the stationarity assumption in five software engineering datasets that have been used in the construction of software effort estimation models. The kernel estimators are used in the generation of nonuniform weights which are subsequently employed in weighted linear regression modeling. In each model, older projects are assigned smaller weights while the more recently completed projects are assigned larger weights, to reflect their potentially greater relevance to present or future projects that need to be estimated. Prediction errors are compared to those obtained from uniform models. Our results indicate that, for the datasets that exhibit underlying nonstationary processes, uniform models are more accurate than the nonuniform models; that is, models based on kernel estimator functions are worse than the models where no weighting was applied. In contrast, the accuracies of uniform and nonuniform models for datasets that exhibited stationary processes were essentially equivalent. Our analysis indicates that as the heterogeneity of a dataset increases, the effect of stationarity is overridden. The results of our study also confirm prior findings that the accuracy of effort estimation models is independent of the type of kernel estimator function used in model development.

O interesse das organizações em desenvolver estratégias de Big Data está aumentando significativamente. No entanto, a expectativa do valor desses benefícios e dos custos envolvidos na aquisição ou desenvolvimento dessas soluções não é homogênea para todas as empresas, gerando imperfeições competitivas no mercado de recursos estratégicos. A capacidade de gestão da informação (CGI) tem como premissa fornecer as informações necessárias para que as estratégias de Big Data sejam bem-sucedidas. Este artigo se propõe a analisar a CGI como um agente de imperfeição no mercado de fatores estratégicos de Big Data. As hipóteses foram testadas a partir de uma Survey com 101 respondentes e analisadas com a utilização de SEM-PLS. Os resultados indicam uma influência CGI positiva na expectativa de valor e uma negativa na expectativa de custo. A expectativa de custo afeta inversamente a intenção de comprar ou desenvolver os recursos para implantar estratégias de Big Data. A expectativa de valor tem um efeito positivo em ambas as intenções.

The ability of organizations to produce, collect, manage, analyze, and transform data has increased rapidly over the past decade (Delen & Zolbanin, 2018). This has resulted in significant new challenges regarding how data can be leveraged for improving business decisions and how this new scenario changes business processes and operations (Vidgen, Shaw, & Grant, 2017). The widespread adoption of advanced analytical methods (e.g., machine learning) has attracted significant interest (Gupta, Deokar, Iyer, Sharda, & Schrader, 2018; Vassakis, Petrakis, & Kopanakis, 2018) particularly because the required data storage and methods can be accessed remotely through web-based interfaces such as cloud services. This has resulted in an increased belief that businesses must actively engage with this technology to remain competitive. However, this Red Queen scenario comes at a cost as collecting, curating, and managing large datasets requires expertise and dedicated staff, often consuming resources that do not contribute to core business activities. Consider the fact that there is an increasing role for data scientists and data engineers, among others, within organizations (Davenport & Patil, 2012). Roles such as Chief Data Officer (CDO) and Chief Analytics Officer (CAO) are now commonplace within most organizations.