With the growing access to administrative health databases, retrospective studies have become crucial evidence for medical treatments. Yet, non-randomized studies frequently face selection biases, requiring mitigation strategies. Propensity score matching (PSM) addresses these biases by selecting comparable populations, allowing for analysis without further methodological constraints. However, PSM has several drawbacks. Different matching methods can produce significantly different Average Treatment Effects (ATE) for the same task, even when meeting all validation criteria. To prevent cherry-picking the best method, public authorities must involve field experts and engage in extensive discussions with researchers. To address this issue, we introduce a novel metric, A2A, to reduce the number of valid matches. A2A constructs artificial matching tasks that mirror the original ones but with known outcomes, assessing each matching method’s performance comprehensively from propensity estimation to ATE estimation. When combined with Standardized Mean Difference, A2A enhances the precision of model selection, resulting in a reduction of up to 50

Despite the vast body of literature on Active Learning (AL), there is no comprehensive and open benchmark allowing for efficient and simple comparison of proposed samplers. Additionally, the variability in experimental settings across the literature makes it difficult to choose a sampling strategy, which is critical due to the one-off nature of AL experiments. To address those limitations, we introduce OpenAL, a flexible and open-source framework to easily run and compare sampling AL strategies on a collection of realistic tasks. The proposed benchmark is augmented with interpretability metrics and statistical analysis methods to understand when and why some samplers outperform others. Last but not least, practitioners can easily extend the benchmark by submitting their own AL samplers.

Drug repurposing provides an opportunity to redeploy drugs, which ideally are already approved for use in humans, for the treatment of other diseases. For example, the repurposing of dexamethasone and baricitinib has played a crucial role in saving patient lives during the ongoing SARS-CoV-2 pandemic. There remains a need to expand therapeutic approaches to prevent life-threatening complications in patients with COVID-19. Using an in silico approach based on structural similarity to drugs already in clinical trials for COVID-19, potential drugs were predicted for repurposing. For a subset of identified drugs with different targets to their corresponding COVID-19 clinical trial drug, a mechanism of action analysis was applied to establish whether they might have a role in inhibiting the replication of SARS-CoV-2. Of sixty drugs predicted in this study, two with the potential to inhibit SARS-CoV-2 replication were identified using mechanism of action analysis. Triamcinolone is a corticosteroid that is structurally similar to dexamethasone; gallopamil is a calcium channel blocker that is structurally similar to verapamil. In silico approaches indicate possible mechanisms of action for both drugs in inhibiting SARS-CoV-2 replication. The identification of these drugs as potentially useful for patients with COVID-19 who are at a higher risk of developing severe disease supports the use of in silico approaches to facilitate quick and cost-effective drug repurposing. Such drugs could expand the number of treatments available to patients who are not protected by vaccination.

If Uncertainty Quantification (UQ) is crucial to achieve trustworthy Machine Learning (ML), most UQ methods suffer from disparate and inconsistent evaluation protocols. We claim this inconsistency results from the unclear requirements the community expects from UQ. This opinion paper offers a new perspective by specifying those requirements through five downstream tasks where we expect uncertainty scores to have substantial predictive power. We design these downstream tasks carefully to reflect real-life usage of ML models. On an example benchmark of 7 classification datasets, we did not observe statistical superiority of state-of-the-art intrinsic UQ methods against simple baselines. We believe that our findings question the very rationale of why we quantify uncertainty and call for a standardized protocol for UQ evaluation based on metrics proven to be relevant for the ML practitioner.

In Active learning, a trained model is used to select samples to label to maximize its performance. Choosing the best sample selection strategy for a one-shot experiment is hard, but metrics have been proven to help by detecting strategies performing worse than random or detecting and avoiding noisy samples. Cardinal is a python framework that assists the practitioner in selecting a strategy using metrics and the researcher in developing those metrics. Cardinal caches experiments to compute insights costlessly, keeps track of logged metrics, and proposes extensive documentation. It also interfaces with other packages to use state-of-the-art strategies.

SARS-CoV-2, the causative virus of COVID-19 continues to cause an ongoing global pandemic. Therapeutics are still needed to treat mild and severe COVID-19. Drug repurposing provides an opportunity to deploy drugs for COVID-19 more rapidly than developing novel therapeutics. Some existing drugs have shown promise for treating COVID-19 in clinical trials. This in silico study uses structural similarity to clinical trial drugs to identify two drugs with potential applications to treat early COVID-19. We apply in silico validation to suggest a possible mechanism of action for both. Triamcinolone is a corticosteroid structurally similar to Dexamethasone. Gallopamil is a calcium channel blocker structurally similar to Verapamil. We propose that both these drugs could be useful to treat early COVID-19 infection due to the proximity of their targets within a SARS-CoV-2-induced protein-protein interaction network to kinases active in early infection, and the APOA1 protein which is linked to the spread of COVID-19.

This work explores the effect of noisy sample selection in active learning strategies. We show on both synthetic problems and real-life use-cases that knowledge of the sample noise can significantly improve the performance of active learning strategies. Building on prior work, we propose a robust sampler, Incremental Weighted K-Means that brings significant improvement on the synthetic tasks but only a marginal uplift on real-life ones. We hope that the questions raised in this paper are of interest to the community and could open new paths for active learning research.

Active Learning (AL) is an active domain of research, but is seldom used in the industry despite the pressing needs. This is in part due to a misalignment of objectives, while research strives at getting the best results on selected datasets, the industry wants guarantees that Active Learning will perform consistently and at least better than random labeling. The very one-off nature of Active Learning makes it crucial to understand how strategy selection can be carried out and what drives poor performance (lack of exploration, selection of samples that are too hard to classify, ...). To help rebuild trust of industrial practitioners in Active Learning, we present various actionable metrics. Through extensive experiments on reference datasets such as CIFAR100, Fashion-MNIST, and 20Newsgroups, we show that those metrics brings interpretability to AL strategies that can be leveraged by the practitioner.

Functional connectomes reveal biomarkers of individual psychological or clinical traits. However, there is great variability in the analytic pipelines typically used to derive them from rest-fMRI cohorts. Here, we consider a specific type of studies, using predictive models on the edge weights of functional connectomes, for which we highlight the best modeling choices. We systematically study the prediction performances of models in 6 different cohorts and a total of 2000 individuals, encompassing neuro-degenerative (Alzheimer's, Post-traumatic stress disorder), neuro-psychiatric (Schizophrenia, Autism), drug impact (Cannabis use) clinical settings and psychological trait (fluid intelligence). The typical prediction procedure from rest-fMRI consists of three main steps: defining brain regions, representing the interactions, and supervised learning. For each step we benchmark typical choices: 8 different ways of defining regions -either pre-defined or generated from the rest-fMRI data- 3 measures to build functional connectomes from the extracted time-series, and 10 classification models to compare functional interactions across subjects. Our benchmarks summarize more than 240 different pipelines and outline modeling choices that show consistent prediction performances in spite of variations in the populations and sites. We find that regions defined from functional data work best; that it is beneficial to capture between-region interactions with tangent-based parametrization of covariances, a midway between correlations and partial correlation; and that simple linear predictors such as a logistic regression give the best predictions. Our work is a step forward to establishing reproducible imaging-based biomarkers for clinical settings.

Online A/B testing evaluates the impact of a new technology by running it in a real production environment and testing its performance on a subset of the users of the platform. It is a well-known practice to run a preliminary offline evaluation on historical data to iterate faster on new ideas, and to detect poor policies in order to avoid losing money or breaking the system. For such offline evaluations, we are interested in methods that can compute offline an estimate of the potential uplift of performance generated by a new technology. Offline performance can be measured using estimators known as counterfactual or off-policy estimators. Traditional counterfactual estimators, such as capped importance sampling or normalised importance sampling, exhibit unsatisfying bias-variance compromises when experimenting on personalized product recommendation systems. To overcome this issue, we model the bias incurred by these estimators rather than bound it in the worst case, which leads us to propose a new counterfactual estimator. We provide a benchmark of the different estimators showing their correlation with business metrics observed by running online A/B tests on a large-scale commercial recommender system.

Processing neuroimaging data on the cortical surface traditionally requires dedicated heavy-weight software suites. Here, we present an initial support of cortical surfaces in Python within the neuroimaging data processing toolbox Nilearn. We provide loading and plotting functions for different surface data formats with minimal dependencies, along with examples of their application. Limitations of the current implementation and potential next steps are discussed.

Resting-state functional Magnetic Resonance Imaging (R-fMRI) holds the promise to reveal functional biomarkers of neuropsychiatric disorders. However, extracting such biomarkers is challenging for complex multi-faceted neuropathologies, such as autism spectrum disorders. Large multi-site datasets increase sample sizes to compensate for this complexity, at the cost of uncontrolled heterogeneity. This heterogeneity raises new challenges, akin to those face in realistic diagnostic applications. Here, we demonstrate the feasibility of inter-site classification of neuropsychiatric status, with an application to the Autism Brain Imaging Data Exchange (ABIDE) database, a large (N=871) multi-site autism dataset. For this purpose, we investigate pipelines that extract the most predictive biomarkers from the data. These R-fMRI pipelines build participant-specific connectomes from functionally-defined brain areas. Connectomes are then compared across participants to learn patterns of connectivity that differentiate typical controls from individuals with autism. We predict this neuropsychiatric status for participants from the same acquisition sites or different, unseen, ones. Good choices of methods for the various steps of the pipeline lead to 67% prediction accuracy on the full ABIDE data, which is significantly better than previously reported results. We perform extensive validation on multiple subsets of the data defined by different inclusion criteria. These enables detailed analysis of the factors contributing to successful connectome-based prediction. First, prediction accuracy improves as we include more subjects, up to the maximum amount of subjects available. Second, the definition of functional brain areas is of paramount importance for biomarker discovery: brain areas extracted from large R-fMRI datasets outperform reference atlases in the classification tasks.

The disparity between the chronological age of an individual and their brain-age measured based on biological information has the potential to offer clinically-relevant biomarkers of neurological syndromes that emerge late in the lifespan. While prior brain-age prediction studies have relied exclusively on either structural or functional brain data, here we investigate how multimodal brain-imaging data improves age prediction. Using cortical anatomy and whole-brain functional connectivity on a large adult lifespan sample (N = 2354, age 19-82), we found that multimodal data improves brain-based age prediction, resulting in a mean absolute prediction error of 4.29 years. Furthermore, we found that the discrepancy between predicted age and chronological age captures cognitive impairment. Importantly, the brain-age measure was robust to confounding effects: head motion did not drive brain-based age prediction and our models generalized reasonably to an independent dataset acquired at a different site (N = 475). Generalization performance was increased by training models on a larger and more heterogeneous dataset. The robustness of multimodal brain-age prediction to confounds, generalizability across sites, and sensitivity to clinically-relevant impairments, suggests promising future application to the early prediction of neurocognitive disorders.

Resting-state functional Magnetic Resonance Imaging (rs-fMRI) holds the promise of easy-to-acquire and widespectrum biomarkers. However, there are few predictivemodeling studies on resting state, and processing pipelines all vary. Here, we systematically study resting state functionalconnectivity (FC)-based prediction across three different cohorts. Analysis pipelines consist of four steps: Delineation of brain regions of interest (ROIs), ROI-level rs-fMRI time series signal extraction, FC estimation and linear model classification analysis of FC features. For each step, we explore various methodological choices: ROI set selection, FC metrics, and linear classifiers to compare and evaluate the dominant strategies for the sake of prediction accuracy. We achieve good prediction results on the three different targets. With regard to pipeline selection, we obtain consistent results in two pipeline steps -FC metrics and linear classifiers- that are vital in the diagnosis of rs-fMRI based disease biomarkers. Regarding brain ROIs selection, we observe that the effects of different diseases are best characterized by different strategies: Schizophrenia discrimination is best performed in dataset-specific ROIs, which is not clearly the case for other pathologies. Overall, we outline some dominant strategies, in spite of the specificity of each brain disease in term of FC pattern disruption.

Functional brain imaging provides key information to characterize neurodegenerative diseases, such as Alzheimer's disease AD. Specifically, the metabolic activity measured through fluorodeoxyglucose positron emission tomography FDG-PET and the connectivity extracted from resting-state functional magnetic resonance imaging fMRI, are promising biomarkers that can be used for early assessment and prognosis of the disease and to understand its mechanisms. FDG-PET is the best suited functional marker so far, as it gives a reliable quantitative measure, but is invasive. On the other hand, non-invasive fMRI acquisitions do not provide a straightforward quantification of brain functional activity. To analyze populations solely based on resting-state fMRI, we propose an approach that leverages a metabolic prior learned from FDG-PET. More formally, our classification framework embeds population priors learned from another modality at the voxel-level, which can be seen as a regularization term in the analysis. Experimental results show that our PET-informed approach increases classification accuracy compared to pure fMRI approaches and highlights regions known to be impacted by the disease.

Recent studies have shown that resting-state spontaneous brain activity unveils intrinsic cerebral functioning and complete information brought by prototype task study. From these signals, we will set up a functional atlas of the brain, along with an across-subject variability model. The novelty of our approach lies in the integration of neuroscientific priors and inter-individual variability in a probabilistic description of the rest activity. These models will be applied to large datasets. This variability, ignored until now, may lead to learning of fuzzy atlases, thus limited in term of resolution. This program yields both numerical and algorithmic challenges because of the data volume but also because of the complexity of modelisation.

Statistical machine learning methods are increasingly used for neuroimaging data analysis. Their main virtue is their ability to model high-dimensional datasets, e.g. multivariate analysis of activation images or resting-state time series. Supervised learning is typically used in decoding or encoding settings to relate brain images to behavioral or clinical observations, while unsupervised learning can uncover hidden structures in sets of images (e.g. resting state functional MRI) or find sub-populations in large cohorts. By considering different functional neuroimaging applications, we illustrate how scikit-learn, a Python machine learning library, can be used to perform some key analysis steps. Scikit-learn contains a very large set of statistical learning algorithms, both supervised and unsupervised, and its application to neuroimaging data provides a versatile tool to study the brain.

Functional Magnetic Resonance Images acquired during resting-state provide information about the functional organization of the brain through measuring correlations between brain areas. Independent components analysis is the reference approach to estimate spatial components from weakly structured data such as brain signal time courses; each of these components may be referred to as a brain network and the whole set of components can be conceptualized as a brain functional atlas. Recently, new methods using a sparsity prior have emerged to deal with low signal-to-noise ratio data. However, even when using sophisticated priors, the results may not be very sparse and most often do not separate the spatial components into brain regions. This work presents post-processing techniques that automatically sparsify brain maps and separate regions properly using geometric operations, and compares these techniques according to faithfulness to data and stability metrics. In particular, among threshold-based approaches, hysteresis thresholding and random walker segmentation, the latter improves significantly the stability of both dense and sparse models.