The ChaLearn AutoML Challenge (The authors are in alphabetical order of last name, except the first author who did most of the writing and the second author who produced most of the numerical analyses and plots.) (NIPS 2015 - ICML 2016) consisted of six rounds of a machine learning competition of progressive difficulty, subject to limited computational resources. It was followed by a one-round AutoML challenge (PAKDD 2018). The AutoML setting differs from former model selection/hyper-parameter selection challenges, such as the one we previously organized for NIPS 2006: the participants aim to develop fully automated and computationally efficient systems, capable of being trained and tested without human intervention, with code submission. This chapter analyzes the results of these competitions and provides details about the datasets, which were not revealed to the participants. The solutions of the winners are systematically benchmarked over all datasets of all rounds and compared with canonical machine learning algorithms available in scikit-learn. All materials discussed in this chapter (data and code) have been made publicly available at http://automl.chalearn.org/.

This book illustrates the thrust of the scientific community to use machine learning concepts for tackling a complex problem: given time series of neuronal spontaneous activity, which is the underlying connectivity between the neurons in the network? The contributing authors also develop tools for the advancement of neuroscience through machine learning techniques, with a focus on the major open problems in neuroscience. While the techniques have been developed for a specific application, they address the more general problem of network reconstruction from observational time series, a problem of interest in a wide variety of domains, including econometrics, epidemiology, and climatology, to cite only a few. The book is designed for the mathematics, physics and computer science communities that carry out research in neuroscience problems. The content is also suitable for the machine learning community because it exemplifies how to approach the same problem from different perspectives.

ObjectiveTo evaluate prediction of laboratory diagnosis of acute respiratoryinfection (ARI) from participatory data using machine learningmodels.IntroductionARIs have epidemic and pandemic potential. Prediction of presenceof ARIs from individual signs and symptoms in existing studieshave been based on clinically-sourced data1. Clinical data generallyrepresents the most severe cases, and those from locations with accessto healthcare institutions. Thus, the viral information that comes fromclinical sampling is insufficient to either capture disease incidence ingeneral populations or its predictability from symptoms. Participatorydata — information that individuals today can produce on their own— enabled by the ubiquity of digital tools, can help fill this gap byproviding self-reported data from the community. Internet-basedparticipatory efforts such as Flu Near You2have augmented existingARI surveillance through early and widespread detection of outbreaksand public health trends.MethodsThe GoViral platform3was established to obtain self-reportedsymptoms and diagnostic specimens from the community (Table 1summarizes participation detail). Participants from states with themost data, MA, NY, CT, NH, and CA were included. Age, gender,zip code, and vaccination status were requested from each participant.Participants submitted saliva and nasal swab specimens and reportedsymptoms from: fever, cough, sore throat, shortness of breath, chills,fatigue, body aches, headache, nausea, and diarrhea. Pathogenswere confirmed via RT-PCR on a GenMark respiratory panel assay(full virus list reported previously3).Observations with missing, invalid or equivocal lab tests wereremoved. Table 2 summarizes the binary features. Age categorieswere:≤20, > 20 and < 40, and≥40 to represent young, middle-aged, and old. Missing age and gender values were imputed based onoverall distributions.Three machine learning algorithms—Support Vector Machines(SVMs)4, Random Forests (RFs)5, and Logistic Regression (LR) wereconsidered. Both individual features and their combinations wereassessed. Outcome was the presence (1) or absence (0) of laboratorydiagnosis of ARI.ResultsTen-fold cross validation was repeated ten times. Evaluationsmetrics used were: positive predictive value (PPV), negativepredictive value (NPV), sensitivity, and specificity6. LR and SVMsyielded the best PPV of 0.64 (standard deviation:±0.08) with coughand fever as predictors. The best sensitivity of 0.59 (±0.14) was fromLR using cough, fever, and sore throat. RFs had the best NPV andspecificity of 0.62 (±0.15) and 0.83 (±0.10) respectively with theCDC ILI symptom profile of fever and (cough or sore throat). Addingdemographics and vaccination status did not improve performanceof the classifiers. Results are consistent with studies using clinically-sourced data: cough and fever together were found to be the bestpredictors of flu-like illness1. Because our data include mildlyinfectious and asymptomatic cases, the classifier sensitivity and PPVare low compared to results from clinical data.ConclusionsEvidence of fever and cough together are good predictors of ARIin the community, but clinical data may overestimate this due tosampling bias. Integration of participatory data can not only improvepopulation health by actively engaging the general public2but alsoimprove the scope of studies solely based on clinically-sourcedsurveillance data.Table 1. Details of included participants.Table 2. Coding of binary features

The amounts and types of available multimodal tumor data are rapidly increasing, and their integration is critical for fully understanding the underlying cancer biology and personalizing treatment. However, the development of methods for effectively integrating multimodal data in a principled manner is lagging behind our ability to generate the data. In this article, we introduce an extension to a multiview nonnegative matrix factorization algorithm (NNMF) for dimensionality reduction and integration of heterogeneous data types and compare the predictive modeling performance of the method on unimodal and multimodal data. We also present a comparative evaluation of our novel multiview approach and current data integration methods. Our work provides an efficient method to extend an existing dimensionality reduction method. We report rigorous evaluation of the method on large-scale quantitative protein and phosphoprotein tumor data from the Clinical Proteomic Tumor Analysis Consortium (CPTAC) acquired using state-of-the-art liquid chromatography mass spectrometry. Exome sequencing and RNA-Seq data were also available from The Cancer Genome Atlas for the same tumors. For unimodal data, in case of breast cancer, transcript levels were most predictive of estrogen and progesterone receptor status and copy number variation of human epidermal growth factor receptor 2 status. For ovarian and colon cancers, phosphoprotein and protein levels were most predictive of tumor grade and stage and residual tumor, respectively. When multiview NNMF was applied to multimodal data to predict outcomes, the improvement in performance is not overall statistically significant beyond unimodal data, suggesting that proteomics data may contain more predictive information regarding tumor phenotypes than transcript levels, probably due to the fact that proteins are the functional gene products and therefore a more direct measurement of the functional state of the tumor. Here, we have applied our proposed approach to multimodal molecular data for tumors, but it is generally applicable to dimensionality reduction and joint analysis of any type of multimodal data.

Display Omitted Using multiple linked data sources for network inference is common in many domains.Infectious disease transmission is a key area for multimodal network models.We identify challenges in existing multimodal network inference methods.Bayesian methods are the focus; they can concurrently infer networks & parameters.This review identifies approaches for integrating new biomedical data types. Networks inference problems are commonly found in multiple biomedical subfields such as genomics, metagenomics, neuroscience, and epidemiology. Networks are useful for representing a wide range of complex interactions ranging from those between molecular biomarkers, neurons, and microbial communities, to those found in human or animal populations. Recent technological advances have resulted in an increasing amount of healthcare data in multiple modalities, increasing the preponderance of network inference problems. Multi-domain data can now be used to improve the robustness and reliability of recovered networks from unimodal data. For infectious diseases in particular, there is a body of knowledge that has been focused on combining multiple pieces of linked information. Combining or analyzing disparate modalities in concert has demonstrated greater insight into disease transmission than could be obtained from any single modality in isolation. This has been particularly helpful in understanding incidence and transmission at early stages of infections that have pandemic potential. Novel pieces of linked information in the form of spatial, temporal, and other covariates including high-throughput sequence data, clinical visits, social network information, pharmaceutical prescriptions, and clinical symptoms (reported as free-text data) also encourage further investigation of these methods. The purpose of this review is to provide an in-depth analysis of multimodal infectious disease transmission network inference methods with a specific focus on Bayesian inference. We focus on analytical Bayesian inference-based methods as this enables recovering multiple parameters simultaneously, for example, not just the disease transmission network, but also parameters of epidemic dynamics. Our review studies their assumptions, key inference parameters and limitations, and ultimately provides insights about improving future network inference methods in multiple applications.

The recent announcement of the Precision Medicine Initiative by President Obama has brought precision medicine (PM) to the forefront for healthcare providers, researchers, regulators, innovators, and funders alike. As technologies continue to evolve and datasets grow in magnitude, a strong computational infrastructure will be essential to realize PM's vision of improved healthcare derived from personal data. In addition, informatics research and innovation affords a tremendous opportunity to drive the science underlying PM. The informatics community must lead the development of technologies and methodologies that will increase the discovery and application of biomedical knowledge through close collaboration between researchers, clinicians, and patients. This perspective highlights seven key areas that are in need of further informatics research and innovation to support the realization of PM.

Brain science is an evolving research area inviting great enthusiasm with its potential for providing insights and thereby, preventing, and treating multiple neuronal disorders affecting millions of patients. Discovery of relationships, such as brain connectivity, is a major goal in basic, translational, and clinical science. Algorithms for causal discovery are used in diverse fields for tackling problems similar to the task of reconstruction of neuronal brain connectivity. Our aim is to understand the strengths and limitations of these methods, measure performance and its determinants, and provide insights to enhance their performance and applicability. We performed extensive empirical testing and benchmarking of reconstruction performance of several state-of-the-art algorithms along with several ensemble techniques used to combine them. Our experiments used a clear and broadly relevant gold standard based on calcium fluorescence time series recordings of thousands of neurons sampled from a previously validated realistic, neuronal model. Correlation, entropy-based measures, Cross-Correlation for short time lags, and Generalized Transfer Entropy had the best performances with area under ROC curve (AUC) in the range of 0.7-0.8 even for smaller sample sizes of n = 100 to 1,000 and converged quickly (at less than n = 1,000). Ensembles of best-performing methods using random forests and neural networks generated AUC of ~0.9 with n = 10,000. Several important insights regarding parameter choice and sample size were gained for guiding the experimental design of studies. Our data are also supportive of the feasibility of reliably reconstructing complex neuronal connectivity using existing techniques.

The ChaLearn AutoML Challenge team conducted a large scale evaluation of fully auto- matic, black-box learning machines for feature-based classi cation and regression problems. The test bed was composed of 30 data sets from a wide variety of application domains and ranged across di erent types of complexity. Over six rounds, participants succeeded in delivering AutoML software capable of being trained and tested without human intervention. Although improvements can still be made to close the gap between human-tweaked and AutoML models, this competition contributes to the development of fully automated environments by challenging practitioners to solve problems under speci c constraints and sharing their approaches; the platform will remain available for post-challenge submissions at http://codalab.org/AutoML.

Networks inference problems are commonly found in multiple biomedical subfields such as genomics, metagenomics, neuroscience, and epidemiology. Networks are useful for representing a wide range of complex interactions ranging from those between molecular biomarkers, neurons, and microbial communities, to those found in human or animal populations. Recent technological advances have resulted in an increasing amount of healthcare data in multiple modalities, increasing the preponderance of network inference problems. Multi-domain data can now be used to improve the robustness and reliability of recovered networks from unimodal data. For infectious diseases in particular, there is a body of knowledge that has been focused on combining multiple pieces of linked information. Combining or analyzing disparate modalities in concert has demonstrated greater insight into disease transmission than could be obtained from any single modality in isolation. This has been particularly helpful in understanding incidence and transmission at early stages of infections that have pandemic potential. Novel pieces of linked information in the form of spatial, temporal, and other covariates including high-throughput sequence data, clinical visits, social network information, pharmaceutical prescriptions, and clinical symptoms (reported as free-text data) also encourage further investigation of these methods. The purpose of this review is to provide an in-depth analysis of multimodal infectious disease transmission network inference methods with a specific focus on Bayesian inference. We focus on analytical Bayesian inference-based methods as this enables recovering multiple parameters simultaneously, for example, not just the disease transmission network, but also parameters of epidemic dynamics. Our review studies their assumptions, key inference parameters and limitations, and ultimately provides insights about improving future network inference methods in multiple applications.

There are several publicly accessible patient forums where patients can post questions related to their health conditions. The objective of this study was to develop a query-retrieval system that can mine such forums and identify existing questions most similar to the provided question. This pilot study based on a bag-of-words model with latent semantic analysis and cosine similarity suggests that text similarity-based mining holds promise for identification of diabetes-related questions from patient forums and informing self-care management. Further studies involving advance natural language processing tools can be used to reduce false positives and uncover semantically related questions.

ChaLearn is organizing the Automatic Machine Learning (AutoML) contest for IJCNN 2015, which challenges participants to solve classification and regression problems without any human intervention. Participants' code is automatically run on the contest servers to train and test learning machines. However, there is no obligation to submit code; half of the prizes can be won by submitting prediction results only. Datasets of progressively increasing difficulty are introduced throughout the six rounds of the challenge. (Participants can enter the competition in any round.) The rounds alternate phases in which learners are tested on datasets participants have not seen, and phases in which participants have limited time to tweak their algorithms on those datasets to improve performance. This challenge will push the state of the art in fully automatic machine learning on a wide range of real-world problems. The platform will remain available beyond the termination of the challenge.

A lack of recruitment of appropriate subjects plagues most clinical research trials. One barrier is an efficient way to identify eligible subjects. Researchers worked to harness computing power to improve automated identification of potential subjects for clinical trials with modest success. We use text classification to automatically identify patients for a hypothetical Acute Coronary Syndrome clinical research study from intensive care unit discharge summaries. We apply several state of the art classification methods including Bayesian Logistic Regression, AdaBoost, Support Vector Machines, and Random Forests to build models from administrative manually assigned ICD-9 codes. We then apply these models to discharge summaries labeled by a board certified cardiologist for patients eligible for the hypothetical research study. The best models perform with 0.95 area under the ROC curve for identifying eligible patients. This pilot study suggests that text-based classification holds promise for identification of potential clinical trial subjects. Our methods require further validation in studies involving multiple inclusion and exclusion criteria.

Brain science is a frontier research area with great promise for understanding, preventing, and treating multiple diseases affecting millions of patients. Its key task of reconstructing neuronal brain connectivity poses unique Big Data Analysis challenges distinct from those in clinical or "-omics" domains. Our goal is to understand the strengths and limitations of reconstruction algorithms, measure performance and its determinants, and ultimately enhance performance and applicability. We devised a set of experiments in a well-controlled setting using an established gold-standard based on calcium fluorescence time series recordings of thousands of neurons sampled from a previously validated neuronal model of complex time-varying causal neuronal connections. Following empirical testing of several state-of-the-art reconstruction algorithms, and using the best-performing algorithms, we constructed features of a classifier and predicted the presence or absence of connections using meta-learning. This approach combines information-theoretic, feature construction, and pattern recognition meta-learning methods to considerably improve the Area under ROC curve performance. Our data are very promising toward the feasibility of reliably reconstructing complex neuronal connectivity.

We organized a Challenge to unravel the connectivity of simulated neuronal networks. The provided data was solely based on fluorescence time series of spontaneous activity in a network constituted by 1000 neurons. The task of the participants was to compute the effective connectivity between neurons, with the goal to reconstruct as accurately as possible the ground truth topology of the network. The procured dataset is similar to the one measured in in vivo and in vitro recordings of calcium fluorescence imaging, and therefore the algorithms developed by the participants may largely contribute in the future to unravel major topological features of living neuronal networks from just the analysis of recorded data, and without the need of slow, painstaking experimental connectivity labeling methods. Among 143 entrants, 16 teams participated in the final round of the challenge to compete for prizes. The winners significantly outperformed the baseline method provided by the organizers. To measure influences between neurons the participants used an array of diverse methods, including transfer entropy, regression algorithms, correlation, deep learning, and network deconvolution. The development of "connectivity reconstruction" techniques is a major step in brain science, with many ramifications in the comprehension of neuronal computation, as well as the understanding of network dysfunctions in neuropathologies.

The spectrum of modern molecular high-throughput assaying includes diverse technologies such as microarray gene expression, miRNA expression, proteomics, DNA methylation, among many others. Now that these technologies have matured and become increasingly accessible, the next frontier is to collect "multi-modal'' data for the same set of subjects and conduct integrative, multi-level analyses. While multi-modal data does contain distinct biological information that can be useful for answering complex biology questions, its value for predicting clinical phenotypes and contributions of each type of input remain unknown. We obtained 47 datasets/predictive tasks that in total span over 9 data modalities and executed analytic experiments for predicting various clinical phenotypes and outcomes. First, we analyzed each modality separately using uni-modal approaches based on several state-of-the-art supervised classification and feature selection methods. Then, we applied integrative multi-modal classification techniques. We have found that gene expression is the most predictively informative modality. Other modalities such as protein expression, miRNA expression, and DNA methylation also provide highly predictive results, which are often statistically comparable but not superior to gene expression data. Integrative multi-modal analyses generally do not increase predictive signal compared to gene expression data.

We present a feasibility study using text classification to classify tweets about alcohol use. Alcohol use is the most widely used substance in the US and is the leading risk factor for premature morbidity and mortality globally. Understanding use patterns and locations is an important step toward prevention, moderation, and control of alcohol outlets. Social media may provide an alternate way to measure alcohol use in real time. This feasibility study explores text classification methodologies for identifying alcohol use tweets. We labeled 34,563 geo-located New York City tweets collected in a 24 hour period over New Year's Day 2012. We preprocessed the tweets into stem/ not stemmed and unigram/ bigram representations. We then applied multinomial naïve Bayes, a linear SVM, Bayesian logistic regression, and random forests to the classification task. Using 10 fold cross-validation, the algorithms performed with area under the receiver operating curve of 0.66, 0.91, 0.93, and 0.94 respectively. We also compare to a human constructed Boolean search for the same tweets and the text classification method is competitive with this hand crafted search. In conclusion, we show that the task of automatically identifying alcohol related tweets is highly feasible and paves the way for future research to improve these classifiers.

We are organizing a challenge to reverse engineer the structure of neuronal networks from patterns of activity recorded with calcium fluorescence imaging. Unraveling the brain structure at the neuronal level at a large scale is an important step in brain science, with many ramifications in the comprehension of animal and human intelligence and learning capabilities, as well as understanding and curing neuronal diseases and injuries. However, uncovering the anatomy of the brain by disentangling the neural wiring with its very fine and intertwined dendrites and axons, making both local and far reaching synapses, is a very arduous task: traditional methods of axonal tracing are tedious, difficult, and time consuming. This challenge proposes to approach the problem from a different angle, by reconstructing the effective connectivity of a neuronal network from observations of neuronal activity of thousands of neurons, which can be obtained with state-of-the-art fluorescence calcium imaging. To evaluate the effectiveness of proposed algorithms, we will use data obtained with a realistic simulator of real neurons for which we have ground truth of the neuronal connections. We produced simulated calcium imaging data, taking into account a model of fluorescence and light scattering. The task of the participants is to reconstruct a network of 1000 neurons from time series of neuronal activities obtained with this model. This challenge is part of the official selection of the WCCI 2014 competition program.