The ChaLearn AutoML Challenge (The authors are in alphabetical order of last name, except the first author who did most of the writing and the second author who produced most of the numerical analyses and plots.) (NIPS 2015 - ICML 2016) consisted of six rounds of a machine learning competition of progressive difficulty, subject to limited computational resources. It was followed by a one-round AutoML challenge (PAKDD 2018). The AutoML setting differs from former model selection/hyper-parameter selection challenges, such as the one we previously organized for NIPS 2006: the participants aim to develop fully automated and computationally efficient systems, capable of being trained and tested without human intervention, with code submission. This chapter analyzes the results of these competitions and provides details about the datasets, which were not revealed to the participants. The solutions of the winners are systematically benchmarked over all datasets of all rounds and compared with canonical machine learning algorithms available in scikit-learn. All materials discussed in this chapter (data and code) have been made publicly available at http://automl.chalearn.org/.

This book presents ground-breaking advances in the domain of causal structure learning. casting the cause-effect problem (Does altitude cause a change in atmospheric pressure, or vice versa?) as a binary classification problem, to be tackled by machine learning algorithms.

We organized a challenge in causal discovery from observational data with the aim of devising a “causation coefficient” to score pairs of variables. The participants were provided with a large database of thousands of pairs of variables {X, Y } (80% semi-artificial data and 20% real data) from which samples were drawn independently (i.e. ignoring possible time dependencies). The goal was to discover whether the data supports the hypothesis that Y = f(X, noise), which for the purpose of this challenge was our definition of causality (X causes Y). The participants adopted a machine learning approach, which contrasts with previously published model-based methods. They extracted numerous features of the joint empirical distribution of X and Y and built a classifier to separate pairs belonging to the class “X causes Y” from other cases (“Y causes X”, “X and Y are related” but not in a causal way, a third variable may be causing both X and Y, “X and Y are independent”). The classifier was trained from examples provided by the organizers and tested on independent test data for which the truth values of causal relationships was known only to the organizers. The participants achieved an Area under the ROC Curve (AUC) over 0.8 in the first phase deployed on the Kaggle challenge, which ran from March through September 2013 (round 1). The participants were then invited to improve upon the code efficiency by submitting fast causation coefficients on the Codalab platform (round 2). The causation coefficients developed by the winners have been made available under open source licenses. We have made all data and code publicly available at http://www.causality.inf.ethz.ch/CEdata/ .

With the development of high throughput technology in the past twenty years, it has become easier and cheaper to simultaneously measure tens of thousands of molecules in biological systems. One of the major challenges is how to extract knowledge from these high dimensional datasets and infer the underlying mechanisms of the system. In this review, we discuss several topics related to causal discovery from biomedical data, including causal structural learning from observational and experimental data, estimation of causal effects, and using causal information for predictive modeling.

Detecting anomalies and novel events is vital to the financial services industry. These events may often be indicative of illegal activities such as fraud, risk, identity theft, network intrusion, account takeover and money laundering which may result in undesired outcomes such as disruption in service and other breakdowns. As financial environments change, digital adoption grows and the data moves at increasing speed and volume, the problem of detecting anomalies in real time at large scale becomes increasingly challenging. This is further compounded by the fact that more and more anomaly detection applications require operational decision making in real time. Several new ideas are emerging to tackle this challenge, including semi-supervised learning methods, deep learning based approaches and network/graph based solutions. These approaches must often be able to work in real time by consuming and processing large volumes of data produced in real time. The 2017 KDD Workshop on Anomaly Detection in Finance held at Halifax, Nova Scotia on Aug 14, 2017, brought together researchers and practitioners to discuss these new approaches and solutions. This half-day workshop consisted of two keynote speeches, two industry perspective talks, five full papers and eight spotlight talks. The talks and papers covered many ideas that are of general interest to the entire anomaly detection community in addition to discussing challenges specific to the financial services industry. We are happy to publish a select set of papers from the workshop in this issue of the Proceedings of Machine Learning Research. We open this Volume with a brief introduction to the applications in the financial services industry and how the papers in this Volume address the challenges for these applications.

To date, studies of biological risk factors have revealed inconsistent relationships with subsequent post-traumatic stress disorder (PTSD). The inconsistent signal may reflect the use of data analytic tools that are ill equipped for modeling the complex interactions between biological and environmental factors that underlay post-traumatic psychopathology. Further, using symptom-based diagnostic status as the group outcome overlooks the inherent heterogeneity of PTSD, potentially contributing to failures to replicate. To examine the potential yield of novel analytic tools, we reanalyzed data from a large longitudinal study of individuals identified following trauma in the general emergency room (ER) that failed to find a linear association between cortisol response to traumatic events and subsequent PTSD. First, latent growth mixture modeling empirically identified trajectories of post-traumatic symptoms, which then were used as the study outcome. Next, support vector machines with feature selection identified sets of features with stable predictive accuracy and built robust classifiers of trajectory membership (area under the receiver operator characteristic curve (AUC) = 0.82 (95% confidence interval (CI) = 0.80-0.85)) that combined clinical, neuroendocrine, psychophysiological and demographic information. Finally, graph induction algorithms revealed a unique path from childhood trauma via lower cortisol during ER admission, to non-remitting PTSD. Traditional general linear modeling methods then confirmed the newly revealed association, thereby delineating a specific target population for early endocrine interventions. Advanced computational approaches offer innovative ways for uncovering clinically significant, non-shared biological signals in heterogeneous samples.

We introduce DataQC, an automated system for data attribute anomaly detection for the purpose of improving data quality. Large organizations can have non-standardized or inconsistent data quality checking practices being followed across different departments. The key motivation behind the development of such a system is to 1) achieve a standard for anomaly detection 2) facilitate quick identification of obvious anomalies 3) reduce human judgment in data anomaly detection 4) facilitate prompt corrective action by data scientists. Most of the methods and techniques used during the development of this system are well known and have been widely used by finance professionals who deal with data. Our contribution is to provide a system that improves overall efficiency, interpretability, and objectivity for detecting data attribute anomalies.

Brain science is an evolving research area inviting great enthusiasm with its potential for providing insights and thereby, preventing, and treating multiple neuronal disorders affecting millions of patients. Discovery of relationships, such as brain connectivity, is a major goal in basic, translational, and clinical science. Algorithms for causal discovery are used in diverse fields for tackling problems similar to the task of reconstruction of neuronal brain connectivity. Our aim is to understand the strengths and limitations of these methods, measure performance and its determinants, and provide insights to enhance their performance and applicability. We performed extensive empirical testing and benchmarking of reconstruction performance of several state-of-the-art algorithms along with several ensemble techniques used to combine them. Our experiments used a clear and broadly relevant gold standard based on calcium fluorescence time series recordings of thousands of neurons sampled from a previously validated realistic, neuronal model. Correlation, entropy-based measures, Cross-Correlation for short time lags, and Generalized Transfer Entropy had the best performances with area under ROC curve (AUC) in the range of 0.7-0.8 even for smaller sample sizes of n = 100 to 1,000 and converged quickly (at less than n = 1,000). Ensembles of best-performing methods using random forests and neural networks generated AUC of ~0.9 with n = 10,000. Several important insights regarding parameter choice and sample size were gained for guiding the experimental design of studies. Our data are also supportive of the feasibility of reliably reconstructing complex neuronal connectivity using existing techniques.

The ChaLearn AutoML Challenge team conducted a large scale evaluation of fully auto- matic, black-box learning machines for feature-based classi cation and regression problems. The test bed was composed of 30 data sets from a wide variety of application domains and ranged across di erent types of complexity. Over six rounds, participants succeeded in delivering AutoML software capable of being trained and tested without human intervention. Although improvements can still be made to close the gap between human-tweaked and AutoML models, this competition contributes to the development of fully automated environments by challenging practitioners to solve problems under speci c constraints and sharing their approaches; the platform will remain available for post-challenge submissions at http://codalab.org/AutoML.

Reverse-engineering of causal pathways that implicate diseases and vital cellular functions is a fundamental problem in biomedicine. Discovery of the local causal pathway of a target variable (that consists of its direct causes and direct effects) is essential for effective intervention and can facilitate accurate diagnosis and prognosis. Recent research has provided several active learning methods that can leverage passively observed high-throughput data to draft causal pathways and then refine the inferred relations with a limited number of experiments. The current study provides a comprehensive evaluation of the performance of active learning methods for local causal pathway discovery in real biological data. Specifically, 54 active learning methods/variants from 3 families of algorithms were applied for local causal pathways reconstruction of gene regulation for 5 transcription factors in S. cerevisiae . Four aspects of the methods’ performance were assessed, including adjacency discovery quality, edge orientation accuracy, complete pathway discovery quality, and experimental cost. The results of this study show that some methods provide significant performance benefits over others and therefore should be routinely used for local causal pathway discovery tasks. This study also demonstrates the feasibility of local causal pathway reconstruction in real biological systems with significant quality and low experimental cost.

Conventional research methodologies and data analytic approaches in psychiatric research are unable to reliably infer causal relations without experimental designs, or to make inferences about the functional properties of the complex systems in which psychiatric disorders are embedded. This article describes a series of studies to validate a novel hybrid computational approach--the Complex Systems-Causal Network (CS-CN) method-designed to integrate causal discovery within a complex systems framework for psychiatric research. The CS-CN method was first applied to an existing dataset on psychopathology in 163 children hospitalized with injuries (validation study). Next, it was applied to a much larger dataset of traumatized children (replication study). Finally, the CS-CN method was applied in a controlled experiment using a 'gold standard' dataset for causal discovery and compared with other methods for accurately detecting causal variables (resimulation controlled experiment). The CS-CN method successfully detected a causal network of 111 variables and 167 bivariate relations in the initial validation study. This causal network had well-defined adaptive properties and a set of variables was found that disproportionally contributed to these properties. Modeling the removal of these variables resulted in significant loss of adaptive properties. The CS-CN method was successfully applied in the replication study and performed better than traditional statistical methods, and similarly to state-of-the-art causal discovery algorithms in the causal detection experiment. The CS-CN method was validated, replicated, and yielded both novel and previously validated findings related to risk factors and potential treatments of psychiatric disorders. The novel approach yields both fine-grain (micro) and high-level (macro) insights and thus represents a promising approach for complex systems-oriented research in psychiatry.

BACKGROUND:Predicting Posttraumatic Stress Disorder (PTSD) is a pre-requisite for targeted prevention. Current research has identified group-level risk-indicators, many of which (e.g., head trauma, receiving opiates) concern but a subset of survivors. Identifying interchangeable sets of risk indicators may increase the efficiency of early risk assessment. The study goal is to use supervised machine learning (ML) to uncover interchangeable, maximally predictive combinations of early risk indicators.METHODS:Data variables (features) reflecting event characteristics, emergency department (ED) records and early symptoms were collected in 957 trauma survivors within ten days of ED admission, and used to predict PTSD symptom trajectories during the following fifteen months. A Target Information Equivalence Algorithm (TIE*) identified all minimal sets of features (Markov Boundaries; MBs) that maximized the prediction of a non-remitting PTSD symptom trajectory when integrated in a support vector machine (SVM). The predictive accuracy of each set of predictors was evaluated in a repeated 10-fold cross-validation and expressed as average area under the Receiver Operating Characteristics curve (AUC) for all validation trials.RESULTS:The average number of MBs per cross validation was 800. MBs' mean AUC was 0.75 (95% range: 0.67-0.80). The average number of features per MB was 18 (range: 12-32) with 13 features present in over 75% of the sets.CONCLUSIONS:Our findings support the hypothesized existence of multiple and interchangeable sets of risk indicators that equally and exhaustively predict non-remitting PTSD. ML's ability to increase prediction versatility is a promising step towards developing algorithmic, knowledge-based, personalized prediction of post-traumatic psychopathology.

ChaLearn is organizing the Automatic Machine Learning (AutoML) contest for IJCNN 2015, which challenges participants to solve classification and regression problems without any human intervention. Participants' code is automatically run on the contest servers to train and test learning machines. However, there is no obligation to submit code; half of the prizes can be won by submitting prediction results only. Datasets of progressively increasing difficulty are introduced throughout the six rounds of the challenge. (Participants can enter the competition in any round.) The rounds alternate phases in which learners are tested on datasets participants have not seen, and phases in which participants have limited time to tweak their algorithms on those datasets to improve performance. This challenge will push the state of the art in fully automatic machine learning on a wide range of real-world problems. The platform will remain available beyond the termination of the challenge.

Field of cancerization in the airway epithelium has been increasingly examined to understand early pathogenesis of non-small cell lung cancer. However, the extent of field of cancerization throughout the lung airways is unclear. Here we sought to determine the differential gene and microRNA expressions associated with field of cancerization in the peripheral airway epithelial cells of patients with lung adenocarcinoma. We obtained peripheral airway brushings from smoker controls (n=13) and from the lung contralateral to the tumor in cancer patients (n=17). We performed gene and microRNA expression profiling on these peripheral airway epithelial cells using Affymetrix GeneChip and TaqMan Array. Integrated gene and microRNA analysis was performed to identify significant molecular pathways. We identified 26 mRNAs and 5 miRNAs that were significantly (FDR <0.1) up-regulated and 38 mRNAs and 12 miRNAs that were significantly down-regulated in the cancer patients when compared to smoker controls. Functional analysis identified differential transcriptomic expressions related to tumorigenesis. Integration of miRNA-mRNA data into interaction network analysis showed modulation of the extracellular signal-regulated kinase/mitogen-activated protein kinase (ERK/MAPK) pathway in the contralateral lung field of cancerization. In conclusion, patients with lung adenocarcinoma have tumor related molecules and pathways in histologically normal appearing peripheral airway epithelial cells, a substantial distance from the tumor itself. This finding can potentially provide new biomarkers for early detection of lung cancer and novel therapeutic targets.

ObjectiveInflammatory mediators, such as prostaglandin E2 (PGE2) and interleukin‐1β (IL‐1β), are produced by osteoarthritic (OA) joint tissue, where they may contribute to disease pathogenesis. We undertook the present study to examine whether inflammation, evidenced in plasma and peripheral blood leukocytes (PBLs), reflects the presence, progression, or specific symptoms of symptomatic knee OA.MethodsPatients with symptomatic knee OA were enrolled in a 24‐month prospective study of radiographic progression. Standardized knee radiographs were obtained at baseline and 24 months. At baseline, levels of the plasma lipids PGE2 and 15‐hydroxyeicosatetraenoic acid (15‐HETE) were measured, and transcriptome analysis of PBLs was performed by microarray and quantitative polymerase chain reaction.ResultsBaseline PGE2 synthase (PGES) levels determined by PBL microarray gene expression and plasma PGE2 levels distinguished patients with symptomatic knee OA from non‐OA controls (area under the receiver operating characteristic curve [AUC] 0.87 and 0.89, respectively, P < 0.0001). Baseline plasma 15‐HETE levels were significantly elevated in patients with symptomatic knee OA versus non‐OA controls (P < 0.0195). In the 146 patients who completed the 24‐month study, elevated baseline expression of IL‐1β, tumor necrosis factor α, and cyclooxygenase 2 (COX‐2) messenger RNA in PBLs predicted higher risk of radiographic progression as evidenced by joint space narrowing (JSN). In a multivariate model, AUC point estimates of models containing COX‐2 in combination with demographic traits overlapped the confidence interval of the base model in 2 of the 3 JSN outcome measures (JSN >0.0 mm, JSN >0.2 mm, and JSN >0.5 mm; AUC 0.62–0.67).ConclusionThe inflammatory plasma lipid biomarkers PGE2 and 15‐HETE identify patients with symptomatic knee OA, and the PBL inflammatory transcriptome identifies a subset of patients with symptomatic knee OA who are at increased risk of radiographic progression. These findings may reflect low‐grade inflammation in OA and may be useful as diagnostic and prognostic biomarkers in clinical development of disease‐modifying OA drugs.

Background: Pro deployment identification of soldiers at risk for long-term posttraumatic stress psychopathology after home coming is important to guide decisions about deployment. Early post deployment identification can direct early interventions to those in need and thereby prevents the development of chronic psychopathology. Both hold significant public health benefits given large numbers of deployed soldiers, but has so far not been achieved. Here, we aim to assess the potential for pre- and early post deployment prediction of resilience or posttraumatic stress development in soldiers by application of machine learning (ML) methods.Methods: ML feature selection and prediction algorithms were applied to a prospective cohort of 561 Danish soldiers deployed to Afghanistan in 2009 to identify unique risk indicators and forecast long-term posttraumatic stress responses.Results: Robust pre- and early postdeployment risk indicators were identified, and included individual PTSD symptoms as well as total level of PTSD symptoms, previous trauma and treatment, negative emotions, and thought suppression. The predictive performance of these risk indicators combined was assessed by cross-validation. Together, these indicators forecasted long term posttraumatic stress responses with high accuracy (pre-deployment: AUC=0.84 (95% CI=0.81-0.87), post-deployment: AUC=0.88 (95% CI=0.85-0.91)).Limitations: This study utilized a previously collected data set and was therefore not designed to exhaust the potential of ML methods. Further, the study relied solely on self-reported measures.Conclusions: Pre-deployment and early post-deployment identification of risk for long-term posttraumatic psychopathology are feasible and could greatly reduce the public health costs of war. (C) 2015 Elsevier B.V. All rights reserved

Brain science is a frontier research area with great promise for understanding, preventing, and treating multiple diseases affecting millions of patients. Its key task of reconstructing neuronal brain connectivity poses unique Big Data Analysis challenges distinct from those in clinical or "-omics" domains. Our goal is to understand the strengths and limitations of reconstruction algorithms, measure performance and its determinants, and ultimately enhance performance and applicability. We devised a set of experiments in a well-controlled setting using an established gold-standard based on calcium fluorescence time series recordings of thousands of neurons sampled from a previously validated neuronal model of complex time-varying causal neuronal connections. Following empirical testing of several state-of-the-art reconstruction algorithms, and using the best-performing algorithms, we constructed features of a classifier and predicted the presence or absence of connections using meta-learning. This approach combines information-theoretic, feature construction, and pattern recognition meta-learning methods to considerably improve the Area under ROC curve performance. Our data are very promising toward the feasibility of reliably reconstructing complex neuronal connectivity.

Objective: Pro-and anti-inflammatory mediators, such as IL-1 beta and IL1Ra, are produced by joint tissues in osteoarthritis (OA), where they may contribute to pathogenesis. We examined whether inflammatory events occurring within joints are reflected in plasma of patients with symptomatic knee osteoarthritis (SKOA).Design: 111 SKOA subjects with medial disease completed a 24-month prospective study of clinical and radiographic progression, with clinical assessment and specimen collection at 6-month intervals. The plasma biochemical marker IL1Ra was assessed at baseline and 18 months; other plasma biochemical markers were assessed only at 18 months, including IL-1 beta, TNF alpha, VEGF, IL-6, IL-6R alpha, IL-17A, IL-17A/F, IL17F, CRP, sTNF-RII, and MMP-2.Results: In cross-sectional studies, WOMAC (total, pain, function) and plasma IL1Ra were modestly associated with radiographic severity after adjustment for age, gender and body mass index (BMI). In addition, elevation of plasma IL1Ra predicted joint space narrowing (JSN) at 24 months. BMI did associate with progression in some but not all analyses. Causal graph analysis indicated a positive association of IL1Ra with JSN; an interaction between IL1Ra and BMI suggested either that BMI influences IL1Ra or that a hidden confounder influences both BMI and IL1Ra. Other protein biomarkers examined in this study did not associate with radiographic progression or severity.Conclusions: Plasma levels of IL1Ra were modestly associated with the severity and progression of SKOA in a causal fashion, independent of other risk factors. The findings may be useful in the search for prognostic biomarkers and development of disease-modifying OA drugs. (C) 2015 Osteoarthritis Research Society International. Published by Elsevier Ltd. All rights reserved.