We study conformal prediction for regression using calibration data with outcomes that are doubly censored (clipped) at known fixed thresholds. We show that existing methods are unsatisfactory in this setting, as they yield intervals that may have higher marginal coverage than desired and yet lose conditional coverage precisely for the easier-to-predict cases whose outcomes are typically fully observed. This reveals that marginal coverage, the usual target of conformal prediction, may not be the ideal goal under clipping. We address this challenge by introducing a new nonconformity score and calibration methods at both ends of this trade-off: one for tight marginal coverage, and a two-step method that prioritizes conditional coverage. We characterize their finite-sample coverage and oracle-like asymptotic behavior under suitable consistency of the underlying model, and we compare them to more direct adaptations of existing approaches.

We propose a method to quantify uncertainty around individual survival distribution estimates using right-censored data, compatible with any survival model. Unlike classical confidence intervals, the survival bands produced by this method offer predictive rather than population-level inference, making them useful for personalized risk screening. For example, in a low-risk screening scenario, they can be applied to flag patients whose survival band at 12 months lies entirely above 50%, while ensuring that at least half of flagged individuals will survive past that time on average. Our approach builds on recent advances in conformal inference and integrates ideas from inverse probability of censoring weighting and multiple testing with false discovery rate control. We provide asymptotic guarantees and show promising performance in finite samples with both simulated and real data.

BackgroundCurrent methods of measuring disease progression of neurodegenerative disorders, including Parkinson's disease (PD), largely rely on composite clinical rating scales, which are prone to subjective biases and lack the sensitivity to detect progression signals in a timely manner. Digital health technology (DHT)-derived measures offer potential solutions to provide objective, precise, and sensitive measures that address these limitations. However, the complexity of DHT datasets and the potential to derive numerous digital features that were not previously possible to measure pose challenges, including in selection of the most important digital features and construction of composite digital biomarkers.MethodsWe present a comprehensive machine learning based framework to construct composite digital biomarkers for progression tracking. This framework consists of a marginal (univariate) digital feature screening, a univariate association test, digital feature selection, and subsequent construction of composite (multivariate) digital disease progression biomarkers using Penalized Generalized Estimating Equations (PGEE). As an illustrative example, we applied this framework to data collected from a PD longitudinal observational study. The data consisted of Opal™ sensor-based movement measurements and MDS-UPDRS Part III scores collected at 3-month intervals for 2 years in 30 PD and 10 healthy control participants.ResultsIn our illustrative example, 77 out of 235 digital features from the study passed univariate feature screening, with 11 features selected by PGEE to include in construction of the composite digital measure. Compared to MDS-UPDRS Part III, the composite digital measure exhibited a smoother and more significant increasing trend over time in PD groups with less variability, indicating improved ability for tracking disease progression. This composite digital measure also demonstrated the ability to classify between de novo PD and healthy control groups.ConclusionMeasures from DHTs show promise in tracking neurodegenerative disease progression with increased sensitivity and reduced variability as compared to traditional clinical scores. Herein, we present a novel framework and methodology to construct composite digital measure of disease progression from high-dimensional DHT datasets, which may have utility in accelerating the development and application of composite digital biomarkers in drug development.

PURPOSE:Eastern Cooperative Oncology Group performance status (ECOG PS) is a key confounder in comparative effectiveness research, predicting treatment and survival, but is often incomplete in electronic health records (EHRs). Imputation on the basis of classification metrics alone may introduce differences in survival between patients with known and imputed ECOG PS, complicating comparative effectiveness research. We developed an approach to impute ECOG PS so that those with known and imputed ECOG PS are indistinguishable in their survival, reducing potential biases introduced by the imputation. METHODS:We analyzed deidentified data from an EHR-derived database for patients with advanced non-small cell lung cancer (aNSCLC) at their first line of treatment. Our novel imputation method involved (1) sample-splitting patients with known ECOG PS into modeling and thresholding data sets, (2) developing a predictive model of ECOG PS, (3) determining an optimal threshold aligning clinical outcomes, where a choice of outcome metric may depend on the use case, and (4) applying the model and threshold to impute missing ECOG PS. We evaluated the approach using binary classification metrics and alignment of survival metrics between observed and imputed ECOG PS. RESULTS:Of 62,101 patients, 13,297 (21%) had missing ECOG PS at the start of their first treatment. Our method achieved similar or better performance in accuracy (73.3%), sensitivity (42.4%), and specificity (81%) compared with other techniques, with smaller survival metric differences between observed and imputed ECOG PS, with differences of 0.07 in hazard ratio, -0.36 months in median survival for good ECOG PS (<2), and -0.39 months for poor ECOG PS (≥2). CONCLUSION:Our imputed ECOG PS aligning clinical outcomes enhanced the use of real-world EHR data of patients with aNSCLC for comparative effectiveness research.

Reliable compound screening is fundamental to drug discovery, yet the process remains undermined by lack of robust risk controls of false compound selection or omission in current methods. To address these challenges, we introduced conformal selection as an enhanced approach to optimize the compound screening process with balanced risks and benefits. Leveraging conformal inference, our approach constructs p-values for each candidate molecule to quantify statistical evidence for selection. The final selection of molecules is determined by comparing these p-values against thresholds derived from multiple testing principles. Our approach offers rigorous control over the false discovery/omission rate, ensuring validity independent of data set size and requiring minimal assumptions. By avoiding the estimation of prediction errors required in previous approaches, our method achieves higher power, thereby improving the ability to identify promising candidates. We validate these advantages through numerical simulations on real-world data sets.

We present a conformal inference method for constructing lower prediction bounds for survival times from right-censored data, extending recent approaches designed for more restrictive type-I censoring scenarios. The proposed method imputes unobserved censoring times using a machine learning model, and then analyzes the imputed data using a survival model calibrated via weighted conformal inference. This approach is theoretically supported by an asymptotic double robustness property. Empirical studies on simulated and real data demonstrate that our method leads to relatively informative predictive inferences and is especially robust in challenging settings where the survival model may be inaccurate.

With the growing commonality of multi‐omics datasets, there is now increasing evidence that integrated omics profiles lead to more efficient discovery of clinically actionable biomarkers that enable better disease outcome prediction and patient stratification. Several methods exist to perform host phenotype prediction from cross‐sectional, single‐omics data modalities but decentralized frameworks that jointly analyze multiple time‐dependent omics data to highlight the integrative and dynamic impact of repeatedly measured biomarkers are currently limited. In this article, we propose a novel Bayesian ensemble method to consolidate prediction by combining information across several longitudinal and cross‐sectional omics data layers. Unlike existing frequentist paradigms, our approach enables uncertainty quantification in prediction as well as interval estimation for a variety of quantities of interest based on posterior summaries. We apply our method to four published multi‐omics datasets and demonstrate that it recapitulates known biology in addition to providing novel insights while also outperforming existing methods in estimation, prediction, and uncertainty quantification. Our open‐source software is publicly available at https://github.com/himelmallick/IntegratedLearner.

The quantitative structure-activity relationship (QSAR) regression model is a commonly used technique for predicting the biological activities of compounds using their molecular descriptors. Besides accurate activity estimation, obtaining a prediction uncertainty metric like a prediction interval is highly desirable. Quantifying prediction uncertainty is an active research area in statistical and machine learning (ML), but the implementation for QSAR remains challenging. However, most ML algorithms with high predictive performance require add-on companions for estimating the uncertainty of their prediction. Conformal prediction (CP) is a promising approach as its main components are agnostic to the prediction modes, and it produces valid prediction intervals under weak assumptions on the data distribution. We proposed computationally efficient CP algorithms tailored to the most widely used ML models, including random forests, deep neural networks, and gradient boosting. The algorithms use a novel approach to the derivation of nonconformity scores from the estimates of prediction uncertainty generated by the ensembles of point predictions. The validity and efficiency of proposed algorithms are demonstrated on a diverse collection of QSAR data sets as well as simulation studies. The provided software implementing our algorithms can be used as stand-alone or easily incorporated into other ML software packages for QSAR modeling.

Variable importance plays a pivotal role in interpretable machine learning as it helps measure the impact of factors on the output of the prediction model. Model agnostic methods based on the generation of "null" features via permutation (or related approaches) can be applied. Such analysis is often utilized in pharmaceutical applications due to its ability to interpret black-box models, including tree-based ensembles. A major challenge and significant confounder in variable importance estimation however is the presence of between-feature correlation. Recently, several adjustments to marginal permutation utilizing feature knockoffs were proposed to address this issue, such as the variable importance measure known as conditional predictive impact (CPI). Assessment and evaluation of such approaches is the focus of our work. We first present a comprehensive simulation study investigating the impact of feature correlation on the assessment of variable importance. We then theoretically prove the limitation that highly correlated features pose for the CPI through the knockoff construction. While we expect that there is always no correlation between knockoff variables and its corresponding predictor variables, we prove that the correlation increases linearly beyond a certain correlation threshold between the predictor variables. Our findings emphasize the absence of free lunch when dealing with high feature correlation, as well as the necessity of understanding the utility and limitations behind methods in variable importance estimation.

Journal Article Phase advance of bedtimes in Alzheimer's disease Get access Donald L Bliwise, Donald L Bliwise Sleep Center, Emory University School of Medicine, Atlanta, GA, USA Corresponding author: Donald L. Bliwise, PhD, Sleep Center, Emory University School of Medicine, 12 Executive Park Drive, Room 435, Atlanta, GA, USA. Email: dbliwis@emory.edu. https://orcid.org/0000-0002-5079-035X Search for other works by this author on: Oxford Academic Google Scholar Ting-Chuan Wang, Ting-Chuan Wang Merck Research Laboratories, Merck & Co., Inc., Rahway, NJ, USA Search for other works by this author on: Oxford Academic Google Scholar Vladimir Svetnik, Vladimir Svetnik Merck Research Laboratories, Merck & Co., Inc., Rahway, NJ, USA Search for other works by this author on: Oxford Academic Google Scholar Gary Zammit, Gary Zammit Clinilabs Drug Development Corporation, New York, NY, USAIcahn School of Medicine at Mount Sinai, New York, NY, USA Search for other works by this author on: Oxford Academic Google Scholar Peining Tao, Peining Tao Merck Research Laboratories, Merck & Co., Inc., Rahway, NJ, USA Search for other works by this author on: Oxford Academic Google Scholar Christopher Lines, Christopher Lines Merck Research Laboratories, Merck & Co., Inc., Rahway, NJ, USA Search for other works by this author on: Oxford Academic Google Scholar W Joseph Herring W Joseph Herring Merck Research Laboratories, Merck & Co., Inc., Rahway, NJ, USA Search for other works by this author on: Oxford Academic Google Scholar Sleep, Volume 46, Issue 11, November 2023, zsad191, https://doi.org/10.1093/sleep/zsad191 Published: 04 August 2023

Background: We used baseline polysomnography (PSG) data obtained during the clinical program development for suvorexant to compare the PSG profiles of people with Alzheimer's disease and insomnia (ADI) versus age-matched elderly individuals with insomnia (EI).Methods: Sleep laboratory baseline PSG data from participants age 55-80 years from 2 trials in people with insomnia and a trial in people with ADI were included. ADI participants had dementia of mild-to-moderate severity. Diagnostic criteria for insomnia, exclusion for other sleep problems, PSG recording procedures, and endpoint derivations were similar across the trials. All participants underwent a night of in-laboratory PSG prior to the baseline night to allow for screening/adaptation. Participants in the EI and ADI groups were compared with regard to sleep architecture, sleep micro-structure, and quantitative EEG power spectral endpoints. The analysis was performed on a post hoc basis using propensity score matching to compare sleep parameters separately in women and men while accounting for age group and total sleep time.Results: A total of 837 EI and 239 ADI participants were included, with the majority in each population (similar to 65%) being women. Compared to EI, those with ADI had a lower percentage of time spent in slow wave sleep (and a corresponding higher percentage of time spent in the lighter N1 sleep), a lower number of spindles per minute of N2 sleep, and lower absolute EEG power during NREM sleep, particularly in the lower-frequency bands. Trends for lower REM sleep percentage in ADI did not reach statistical significance.Conclusions: Our findings in this large data set, in which the influence of sleep problems was effectively subtracted out (since both groups had insomnia), provide strong confirmatory support of results from previous smaller studies in indicating that AD of mild-to-moderate severity is associated with less slow wave sleep, spindles, and lower-frequency EEG power. Trial registration: ClinicalTrials.gov, numbers NCT01097616, NCT01097629, NCT02750306 (c) 2022 Merck Sharp & Dohme LLC., a subsidiary Merck & Co., Inc., Rahway, NJ, USA and The Author(s). Published by Elsevier B.V. This is an open access article under the CC BY-NC-ND license (http:// creativecommons.org/licenses/by-nc-nd/4.0/).

Chronic diseases often require continuing care, and early response to treatment can be an important predictor of long-term efficacy. Often, an apparent lack of early efficacy may lead to discontinuation of treatment, with the decision made either by clinicians or by the patients themselves. Thus, it is important to determine whether or not a desired early outcome corresponds to a beneficial long-term effect of continuing treatment, and conversely, whether or not the absence of such an outcome corresponds to a lack of long-term benefit. However, primary clinical trials of such treatments are not commonly designed to answer such questions, for example by randomizing subjects to continue or discontinue treatment after observing early outcomes. We propose an approach to estimating the effect of continuing treatment after observing early outcomes using data from randomized controlled trials in which treatment discontinuationwas not part of the design. Our approach estimates average causal effects of continuing treatment on long-term outcomes in principal strata defined by the potential early outcomes under treatment. For illustration, we estimate the effects of continuing to take gaboxadol to treat insomnia conditional on early improvement in subjective sleep quality after two nights, based on a standard parallel-arm randomized controlled trial.

The orexin receptor antagonist suvorexant was previously reported to significantly improve total sleep time (TST), by 28 min per night versus placebo after 4 weeks, in a sleep laboratory polysomnography (PSG) study of patients with Alzheimer's disease and insomnia. The study included an exploratory evaluation of a consumer-grade wearable "watch" device for assessing sleep that we report on here. Participants who met diagnostic criteria for both probable Alzheimer's disease dementia and insomnia were randomized to suvorexant 10-20 mg (N = 142) or placebo (N = 143) in a double-blind, 4-week trial. Patients were provided with a consumer-grade wearable watch device (Garmin vivosmart(R) HR) to be worn continuously. Overnight sleep laboratory PSG was performed on three nights: screening, baseline and Night 29 (last dose). Watch treatment effects were assessed by change-from-baseline in watch TST at Week 4 (average TST per night). We also analysed Night 29 data only, with watch data restricted to the PSG recording time. In the 193 participants included in the Week 4 watch analysis (suvorexant = 97, placebo = 96), the suvorexant-placebo difference in watch TST was 4 min (p = .622). In patients with usable data for both assessments at the baseline and Night 29 PSG (suvorexant = 57, placebo = 50), the watch overestimated TST compared to PSG (e.g., placebo baseline = 412 min for watch and 265 min for PSG) and underestimated change-from-baseline treatment effects: the suvorexant-placebo difference was 20 min for watch TST (p = .405) and 35 min for PSG TST (p = .057). These findings show that the watch was less sensitive than PSG for evaluating treatment effects on TST.

A reciprocal LASSO (rLASSO) regularization employs a decreasing penalty function as opposed to conventional penalization approaches that use increasing penalties on the coefficients, leading to stronger parsimony and superior model selection relative to traditional shrinkage methods. Here we consider a fully Bayesian formulation of the rLASSO problem, which is based on the observation that the rLASSO estimate for linear regression parameters can be interpreted as a Bayesian posterior mode estimate when the regression parameters are assigned independent inverse Laplace priors. Bayesian inference from this posterior is possible using an expanded hierarchy motivated by a scale mixture of double Pareto or truncated normal distributions. On simulated and real datasets, we show that the Bayesian formulation outperforms its classical cousin in estimation, prediction, and variable selection across a wide range of scenarios while offering the advantage of posterior inference. Finally, we discuss other variants of this new approach and provide a unified framework for variable selection using flexible reciprocal penalties. All methods described in this article are publicly available as an R package at: https://github.com/himelmallick/BayesRecipe.

Protein redesign and engineering has become an important task in pharmaceutical research and development. Recent advances in technology have enabled efficient protein redesign by mimicking natural evolutionary mutation, selection, and amplification steps in the laboratory environment. For any given protein, the number of possible mutations is astronomical. It is impractical to synthesize all sequences or even to investigate all functionally interesting variants. Recently, there has been an increased interest in using machine learning to assist protein redesign, since prediction models can be used to virtually screen a large number of novel sequences. However, many state-of-the-art machine learning models, especially deep learning models, have not been extensively explored. Moreover, only a small selection of protein sequence descriptors has been considered. In this work, the performance of prediction models built using an array of machine learning methods and protein descriptor types, including two novel, single amino acid descriptors and one structure-based three-dimensional descriptor, is benchmarked. The predictions were evaluated on a diverse collection of public and proprietary data sets, using a variety of evaluation metrics. The results of this comparison suggest that Convolution Neural Network models built with amino acid property descriptors are the most widely applicable to the types of protein redesign problems faced in the pharmaceutical industry.

Abstract Introduction Suvorexant, an orexin receptor antagonist, improved total sleep time (TST) in a sleep laboratory polysomnography (PSG) study of patients with Alzheimer’s disease (AD) and insomnia. The study included a pilot evaluation of an actigraphy watch for continuously recording patient’s sleep and daytime activity. We report on the utility of the watch for assessing sleep in relation to gold-standard PSG. Methods This was a randomized, double-blind, 4-week trial (ClinicalTrials.gov NCT02750306). Participants who met diagnostic criteria for both probable AD dementia and insomnia were randomized to suvorexant 10-20mg or placebo. Overnight sleep laboratory PSG was performed on 3 nights: screening, baseline, and Night-29 (last dose). An actigraphy watch (Garmin vívosmart® HR) was worn continuously by the patient. Separate analyses were performed for PSG and watch. We compared treatment effects on change-from-baseline in PSG-TST at Night-29 and WATCH-TST at Week-4 (average TST per night over Week-4). We also analyzed Night-29 data only with watch data restricted to the PSG recording time. Results A total of 274 participants were included in the Night-29 PSG analysis (suvorexant=135, placebo=139) and 223 in the Week-4 watch analysis (suvorexant=113, placebo=110). Suvorexant improved Night-29 PSG-TST by 28 minutes versus placebo (p=0.001) and Week-4 WATCH-TST by 17 minutes versus placebo (p=0.144). In the subgroup who had usable data for both assessments at Night-29 (suvorexant=57, placebo=50), the watch overestimated TST compared to PSG (e.g. placebo baseline scores = 412 minutes for WATCH-TST and 265 minutes for PSG-TST) and underestimated change-from-baseline treatment effects: the suvorexant versus placebo difference was 35 minutes for PSG-TST (p=0.057) and 20 minutes for WATCH-TST (p=0.405). Conclusion The watch was less sensitive than PSG for evaluating treatment effects on TST. However, results obtained with the watch were directionally similar to PSG in indicating a benefit of suvorexant versus placebo for improving TST in AD patients with insomnia. Support Merck Sharp & Dohme Corp., a subsidiary of Merck & Co., Inc., Kenilworth, NJ, USA

BACKGROUND:Experimental investigation of sleep-wake dynamics in animals is an important part of pharmaceutical development. Typically, it involves recording of electroencephalogram, electromyogram, locomotor activity, and electrooculogram. Visual identification, or scoring, of the sleep-wake states from these recordings is time-consuming. We sought to develop software for automated sleep-wake scoring capable of processing large databases of multi-channel signal recordings in a range of species.NEW METHOD:We used a large historical database of signal recordings and scores in non-human primates, dogs, mice, and rats, to develop a deep Convolutional Neural Network (CNN) classification algorithm for automatically scoring sleep-wake states. We compared the performance of the CNN algorithm with that of a widely used Machine Learning algorithm, Random Forest (RF).RESULTS:CNN accuracy in sleep-wake scoring of data in non-human primates and dogs was significantly higher than RF accuracy (0.75 vs. 0.66 for non-human primates and 0.73 vs. 0.64 for dogs). In rodents, the difference between CNN and RF was smaller: 0.83 vs. 0.81 for mice and 0.78 vs. 0.77 for rats. The variability of CNN accuracy was lower than that of RF for non-human primates, dogs and mice but similar for rats.COMPARISON WITH EXISTING METHODS:Deep Learning algorithms have not been previously evaluated across a range of species for animal sleep-wake scoring.CONCLUSIONS:We recommend use of CNN for sleep-wake scoring in non-human primates and dogs, and RF for sleep-wake scoring in rodents.

Abstract Introduction Suvorexant, an orexin receptor antagonist that enables sleep to occur via competitive antagonism of wake-promoting orexins, improved total sleep time (TST) in a sleep laboratory polysomnography (PSG) study of patients with AD and insomnia. Here we report on the effects of suvorexant on sleep architecture in the study. Methods This was a randomized, double-blind, 4-week trial (ClinicalTrials.gov NCT02750306). Participants who met diagnostic criteria for both probable AD dementia (of mild to moderate severity) and insomnia were randomized to suvorexant 10mg (could be increased to 20mg based on clinical response) or matching placebo. Overnight sleep laboratory PSG was performed on 3 nights: screening, baseline, and Night-29 (last night of dosing). Suvorexant differences from placebo in changes-from-baseline at Night-29 for sleep architecture were analyzed as exploratory endpoints. Results A total of 274 participants were included in the analysis (suvorexant N=135, placebo N=139). At Night-29, suvorexant improved TST by 28 minutes versus placebo (p=0.001). There were no significant differences between suvorexant and placebo in the % of TST spent in REM (1.3%, 95% CI: -0.5, 3.0), N1 (0.6%, 95% CI: -1.2, 2.5), N2 (-1.0%, 95% CI: -3.2, 1.2), or N3 (-0.6%, 95% CI: -1.8, 0.6). There was no significant difference between suvorexant and placebo in latency to REM (-5.4 minutes, 95% CI: -23.4, 12.7). Conclusion Suvorexant improves TST without altering the underlying sleep architecture in AD patients with insomnia. Support Merck Sharp & Dohme Corp., a subsidiary of Merck & Co., Inc., Kenilworth, NJ, USA

We evaluated a single-item Patient Global Impression-Severity (PGI-S) scale for assessing insomnia severity during the clinical development programme for suvorexant. The analyses used data from two randomised, double-blind, placebo-controlled, 3-month, Phase III clinical trials of suvorexant in patients with Diagnostic and Statistical Manual of Mental Disorders IV criteria insomnia. Patients assessed insomnia severity during the previous week using the PGI-S, a one-item questionnaire containing six response options ranging from 0 (none) to 5 (very severe), at baseline and at Week 2, and Months 1, 2, and 3 after randomisation. The seven-item Insomnia Severity Index (ISI) and other subjective and objective assessments were also completed by patients. PGI-S responses were compared primarily with the ISI using descriptive statistics and correlations. The PGI-S demonstrated favourable measurement characteristics (validity, reliability, responsiveness and sensitivity). PGI-S scores decreased from baseline to Month 3 in a similar pattern to the ISI total score, and the Spearman correlation coefficient between PGI-S and the ISI was .73. An improvement of >= 2 points on the PGI-S defined a treatment responder, based on comparison to the ISI definition of a responder (improvement of >= 6 points). Our present findings suggest that the PGI-S is a simple but valid, reliable, responsive, sensitive, and meaningful patient-reported assessment of insomnia severity. The PGI-S may be particularly useful as a companion outcome to sleep monitoring using wearable sleep devices or smartphones in at-home settings.

The need for assessment of agreement of biomarkers is ubiquitous in drug development. In this study, we focus on scaled agreement indices including within-subject coefficient of variation, intraclass correlation coefficient, and concordance correlation coefficient. We illustrate, by both simulated and real life datasets, the usage and added value of Bayesian estimation of agreement of biomarkers in early drug development. We discuss the solutions to small sample size, outliers, and nonnormally distributed data problems. Furthermore, we present how to address, coherently within the Bayesian framework, other practically relevant issues such as accommodation of covariates and model diagnostics.