Artificial intelligence (AI) holds great promise for supporting clinical trials, from patient recruitment and endpoint assessment to treatment response prediction. However, deploying AI without safeguards poses significant risks, particularly when evaluating patient endpoints that directly impact trial conclusions. We compared two AI frameworks against human-only assessment for medical image-based disease evaluation, measuring cost, accuracy, robustness, and generalization ability. To stress-test these frameworks, we injected bad models, ranging from random guesses to naive predictions, to ensure that observed treatment effects remain valid even under severe model degradation. We evaluated the frameworks using two randomized controlled trials with endpoints derived from spinal X-ray images. Our findings indicate that using AI as a supporting reader (AI-SR) is the most suitable approach for clinical trials, as it meets all criteria across various model types, even with bad models. This method consistently provides reliable disease estimation, preserves clinical trial treatment effect estimates and conclusions, and retains these advantages when applied to different populations.

The identification and estimation of heterogeneous treatment effects in biomedical clinical trials are challenging, because trials are typically planned to assess the treatment effect in the overall trial population. Nevertheless, the identification of how the treatment effect may vary across subgroups is of major importance for drug development. In this work, we review some existing simulation work and perform a simulation study to evaluate recent methods for identifying and estimating the heterogeneous treatments effects using various metrics and scenarios relevant for drug development. Our focus is not only on a comparison of the methods in general, but on how well these methods perform in simulation scenarios that reflect real clinical trials. We provide the R package benchtm that can be used to simulate synthetic biomarker distributions based on real clinical trial data and to create interpretable scenarios to benchmark methods for identification and estimation of treatment effect heterogeneity.

This article proposes a Workflow for Assessing Treatment effeCt Heterogeneity (WATCH) in clinical drug development targeted at clinical trial sponsors. WATCH is designed to address the challenges of investigating treatment effect heterogeneity (TEH) in randomized clinical trials, where sample size and multiplicity limit the reliability of findings. The proposed workflow includes four steps: analysis planning, initial data analysis and analysis dataset creation, TEH exploration, and multidisciplinary assessment. The workflow offers a general overview of how treatment effects vary by baseline covariates in the observed data and guides the interpretation of the observed findings based on external evidence and the best scientific understanding. The workflow is exploratory and not inferential/confirmatory in nature but should be preplanned before database lock and analysis start. It is focused on providing a general overview rather than a single specific finding or subgroup with a differential effect.

TorchSurv is a Python package that serves as a companion tool to perform deep survival modeling within the PyTorch environment. Unlike existing libraries that impose specific parametric forms, TorchSurv enables the use of custom PyTorch-based deep survival models. With its lightweight design, minimal input requirements, full PyTorch backend, and freedom from restrictive survival model parameterizations, TorchSurv facilitates efficient deep survival model implementation and is particularly beneficial for high-dimensional and complex input data scenarios.

Objective:Clinical trials involve the collection of a wealth of data, comprising multiple diverse measurements performed at baseline and follow-up visits over the course of a trial. The most common primary analysis is restricted to a single, potentially composite endpoint at one time point. While such an analytical focus promotes simple and replicable conclusions, it does not necessarily fully capture the multi-faceted effects of a drug in a complex disease setting. Therefore, to complement existing approaches, we set out here to design a longitudinal multivariate analytical framework that accepts as input an entire clinical trial database, comprising all measurements, patients, and time points across multiple trials.Methods:Our framework composes probabilistic principal component analysis with a longitudinal linear mixed effects model, thereby enabling clinical interpretation of multivariate results, while handling data missing at random, and incorporating covariates and covariance structure in a computationally efficient and principled way.Results:We illustrate our approach by applying it to four phase III clinical trials of secukinumab in Psoriatic Arthritis (PsA) and Rheumatoid Arthritis (RA). We identify three clinically plausible latent factors that collectively explain 74.5% of empirical variation in the longitudinal patient database. We estimate longitudinal trajectories of these factors, thereby enabling joint characterisation of disease progression and drug effect. We perform benchmarking experiments demonstrating our method’s competitive performance at estimating average treatment effects compared to existing statistical and machine learning methods, and showing that our modular approach leads to relatively computationally efficient model fitting.Conclusion:Our multivariate longitudinal framework has the potential to illuminate the properties of existing composite endpoint methods, and to enable the development of novel clinical endpoints that provide enhanced and complementary perspectives on treatment response.

Clinical trials are primarily conducted to estimate causal effects, but the data collected can also be invaluable for additional research, such as identifying prognostic measures of disease or biomarkers that predict treatment efficacy. However, these exploratory settings are prone to false discoveries (type-I errors) due to the multiple comparisons they entail. Unfortunately, many methods fail to address this issue, in part because the algorithms used are generally designed to optimize predictions and often only provide the measures used for variable selection, such as machine learning model importance scores, as a byproduct. To address the resulting unclear uncertainty in the selection sets, the knockoff framework offers a model-agnostic, robust approach to variable selection with guaranteed type-I error control. Here, we review the knockoff framework in the setting of clinical data, highlighting main considerations using simulation studies. We also extend the framework by introducing a novel knockoff generation method that addresses two main limitations of previously suggested methods relevant for clinical development settings. With this new method, we empirically obtain tighter bounds on type-I error control and gain an order of magnitude in computational efficiency in mixed data settings. We demonstrate comparable selections to those of the competing method for identifying prognostic biomarkers for C-reactive protein levels in patients with psoriatic arthritis in four clinical trials. Our work increases access to the knockoff framework for variable selection from clinical trial data. Hereby, this paper helps to address the current replicability crisis which can result in unnecessary research efforts, increased patient burden, and avoidable costs.

In 2020, Novartis Pharmaceuticals Corporation and the U.S. Food and Drug Administration (FDA) started a 4-year scientific collaboration to approach complex new data modalities and advanced analytics. The scientific question was to find novel radio-genomics-based prognostic and predictive factors for HR+/HER- metastatic breast cancer under a Research Collaboration Agreement. This collaboration has been providing valuable insights to help successfully implement future scientific projects, particularly using artificial intelligence and machine learning. This tutorial aims to provide tangible guidelines for a multi-omics project that includes multidisciplinary expert teams, spanning across different institutions. We cover key ideas, such as "maintaining effective communication" and "following good data science practices," followed by the four steps of exploratory projects, namely (1) plan, (2) design, (3) develop, and (4) disseminate. We break each step into smaller concepts with strategies for implementation and provide illustrations from our collaboration to further give the readers actionable guidance.

We present the motivation, experience, and learnings from a data challenge conducted at a large pharmaceutical corporation on the topic of subgroup identification. The data challenge aimed at exploring approaches to subgroup identification for future clinical trials. To mimic a realistic setting, participants had access to 4 Phase III clinical trials to derive a subgroup and predict its treatment effect on a future study not accessible to challenge participants. A total of 30 teams registered for the challenge with around 100 participants, primarily from Biostatistics organization. We outline the motivation for running the challenge, the challenge rules, and logistics. Finally, we present the results of the challenge, the participant feedback as well as the learnings. We also present our view on the implications of the results on exploratory analyses related to treatment effect heterogeneity.

There is growing interest in data science and the challenges that scientists can solve through its application. The growing interest is in part due to the promise of "extracting value from data." The pharmaceutical industry is no different in this regard reflected by the advancement and excitement surrounding data science. Data science brings new perspectives, new methods, new skill sets and the wider use of new data modalities. For example, there is a belief that extracting value from data integrated from multiple sources and modalities using advances in statistics, machine learning, informatics and computation can answer fundamental questions. These questions span a variety of themes including disease understanding, drug and target discovery, and trial design. By answering fundamental questions, we cannot only increase knowledge and understanding but more importantly inform decision making; accelerating drug development through data-driven prioritization, increasingly precise and accurate measurements, optimized trial designs and operational excellence. However, with the promise of data science, there are obstacles to overcome, especially if data science is to live up to this promise and deliver a positive impact. These obstacles include consensus on the definition of data science, the relationship between data science and existing fields such as statistics and computing science, what should be involved in the day-to-day practices of data science, and what is "good" practice. In this article, we cover these themes, highlighting issues with scientific practice from five perspectives and argue how advances in data science will not be immune, especially exploratory, investigative, and innovative activities. We propose a definition of data science as a coming together but also a refocusing of established disciplines leading to a framework for good practice. In doing so, we aim to begin a dialogue on good data science practice in the context of drug development, where there is no industry view or consensus.

In Oncology, trials evaluating drug combinations are becoming more common. While combination therapies bring the potential for greater efficacy, they also create unique challenges for ensuring drug safety. In Phase-I dose escalation trials of drug combinations, model-based approaches enable efficient use of information gathered, but the models need to account for trial complexities: appropriate modeling of interactions becomes increasingly important with growing numbers of drugs being tested simultaneously in a given trial. In principle, we can use data from multiple arms testing varying combinations to jointly estimate toxicity of the drug combinations. However, such efforts have highlighted limitations when modelling drug-drug interactions in the Bayesian Logistic Regression Model (BLRM) framework used to ensure patient safety. Previous models either do not account for non-monotonicity due to antagonistic toxicity, or exhibit the fundamental flaw of exponentially overpowering the contributions of the individual drugs in the dose-response. This specifically leads to issues when drug combinations exhibit antagonistic toxicity, in which case the toxicity probability gets vanishingly small as doses get very large. We put forward additional constraints inspired by Paracelsus' intuition of "the dose makes the poison" which avoid this flaw and present an improved interaction model which is compatible with these constraints. We create instructive data scenarios that showcase the improved behavior of this more constrained drug-drug interaction model in terms of preventing further dosing at overly toxic dose combinations and more sensible dose-finding under antagonistic drug toxicity. This model is now available in the open-source OncoBayes2 R package that implements the BLRM framework for an arbitrary number of drugs and trial arms.

Clinical data sharing can facilitate data-driven scientific research, allowing a broader range of questions to be addressed and thereby leading to greater understanding and innovation. However, sharing biomedical data can put sensitive personal information at risk. This is usually addressed by data anonymization, which is a slow and expensive process. An alternative to anonymization is construction of a synthetic dataset that behaves similar to the real clinical data but preserves patient privacy. As part of a collaboration between Novartis and the Oxford Big Data Institute, a synthetic dataset was generated based on images from COSENTYX® (secukinumab) ankylosing spondylitis (AS) clinical studies. An auxiliary classifier Generative Adversarial Network (ac-GAN) was trained to generate synthetic magnetic resonance images (MRIs) of vertebral units (VUs), conditioned on the VU location (cervical, thoracic and lumbar). Here, we present a method for generating a synthetic dataset and conduct an in-depth analysis on its properties along three key metrics: image fidelity, sample diversity and dataset privacy.

A bstract In 2020, Novartis Pharmaceuticals Corporation and the U.S. Food and Drug Administration (FDA) started a 4-year scientific collaboration to find novel radiogenomics-based prognostic and predictive factors for HR+/HER2-metastatic breast cancer under a Research Collaboration Agreement. This manuscript aims to detail the guiding principles and methodology for this study. We include a discussion of internal and external clinical, genomics, imaging datasets, data processing workflows, and machine learning model development strategies. We also prospectively define our success criteria to ensure robust scientific outputs. Disclosure This publication reflects the views of the authors and should not be construed to represent FDA’s views or policies.

One of the key challenges of personalized medicine is to identify which patients will respond positively to a given treatment. The area of subgroup identification focuses on this challenge, that is, identifying groups of patients that experience desirable characteristics, such as an enhanced treatment effect. A crucial first step towards the subgroup identification is to identify the baseline variables (eg, biomarkers) that influence the treatment effect, which are known as predictive variables. Many subgroup discovery algorithms return importance scores that capture the variables' predictive strength. However, a major limitation of these scores is that they do not answer the core question: "Which variables are actually predictive?" With our work we answer this question by using the knockoff framework, which is a general framework for controlling the false discovery rate when performing prognostic variable selection. In contrast, our work is the first that uses knockoffs for predictive variable selection. We introduce two novel knockoff filters: one parametric, building on variable importance scores derived from a penalized linear regression model, and one non-parametric, building on causal forest variable importance scores. We conduct extensive simulations to validate performance of the proposed methodology and we also apply the proposed methods to data from a randomized clinical trial.

Sharing data from clinical studies can facilitate innovative data-driven research and ultimately lead to better public health. However, sharing biomedical data can put sensitive personal information at risk. This is usually solved by anonymization, which is a slow and expensive process. An alternative to anonymization is sharing a synthetic dataset that bears a behaviour similar to the real data but preserves privacy. As part of the collaboration between Novartis and the Oxford Big Data Institute, we generate a synthetic dataset based on COSENTYX (secukinumab) Ankylosing Spondylitis clinical study. We apply an Auxiliary Classifier GAN to generate synthetic MRIs of vertebral units. The images are conditioned on the VU location (cervical, thoracic and lumbar). In this paper, we present a method for generating a synthetic dataset and conduct an in-depth analysis on its properties along three key metrics: image fidelity, sample diversity and dataset privacy.

Knockoffs provide a general framework for controlling the false discovery rate when performing variable selection. Much of the Knockoffs literature focuses on theoretical challenges and we recognize a need for bringing some of the current ideas into practice. In this paper we propose a sequential algorithm for generating knockoffs when underlying data consists of both continuous and categorical (factor) variables. Further, we present a heuristic multiple knockoffs approach that offers a practical assessment of how robust the knockoff selection process is for a given dataset. We conduct extensive simulations to validate performance of the proposed methodology. Finally, we demonstrate the utility of the methods on a large clinical data pool of more than 2000 patients with psoriatic arthritis evaluated in four clinical trials with an IL-17A inhibitor, secukinumab (Cosentyx), where we determine prognostic factors of a well established clinical outcome. The analyses presented in this paper could provide a wide range of applications to commonly encountered datasets in medical practice and other fields where variable selection is of particular interest.

OBJECTIVE:This study constructs a risk score for patients' progression to end-stage knee osteoarthritis (OA) within 4 years. DESIGN:The Osteoarthritis Initiative (OAI) was a longitudinal study of the onset and progression of knee OA. Using a recent definition of end-stage knee OA, we implement interval-censored survival forests to select predictors of this endpoint. We fit an interval-censored Cox model for time to end-stage knee OA, using the selected predictors. The risk score is the Cox model's fitted linear combination of the nine selected baseline structural and symptomatic knee OA variables. RESULTS:We fit our models on a training set of 2,701 patients, and we evaluate on an independent test set of 1,436 patients. On the test sample, we observe a concordance index of 0.86 between risk score and time to end-stage, AUC of 0.87 for predicting end-stage within 24, 36, and 48 months, and positive predictive values that increase with the risk score. This risk stratification algorithm could enrich clinical trial patient enrollment. By enrolling test sample patients with scores above a threshold, a trial could have included 91% of test set patients who reach end-stage within 4 years while only enrolling 45% of the test sample. CONCLUSION:Using statistical methods, we construct and validate an interpretable risk score for time to end-stage knee OA. This score can help disease-modifying OA treatment developers to select candidates with the highest risk of fast-progressing knee OA.

Functional data analysis, which analyzes data that can be represented by curves or images, has many potential applications in clinical trials. Motivated by a real example, this study constructs a functional mixed effects model for analyzing a clinical outcome that is observed continuously over a long period of time. A penalized spline (P-spline)-based method is applied to obtain the estimators of the mean function and the time-varying coefficients. Simulation studies are conducted to investigate the consistency, efficiency, and robustness of the method. To illustrate the use of the method, a real data analysis is performed and produces interpretable results.

Evidence synthesis models that combine multiple datasets of varying design, to estimate quantities that cannot be directly observed, require the formulation of complex probabilistic models that can be expressed as graphical models. An assessment of whether the different datasets synthesised contribute information that is consistent with each other, and in a Bayesian context, with the prior distribution, is a crucial component of the model criticism process. However, a systematic assessment of conflict suffers from the multiple testing problem, through testing for conflict at multiple locations in a model. We demonstrate the systematic use of conflict diagnostics, while accounting for the multiple hypothesis tests of no conflict at each location in the graphical model. The method is illustrated by a network meta-analysis to estimate treatment effects in smoking cessation programs and an evidence synthesis to estimate HIV prevalence in Poland.

Futility analyses provide a mechanism to stop a trial early because of low likelihood to achieve its efficacy objective. They are usually motivated by ethical and economic purposes, so that stopping a trial with poor efficacy could save patients and resources for other promising trials. There are various methods to address futility analyses in the literature but most focus on equally spaced interim looks. We consider a constrained optimization framework where the timing and the futility boundary are decided jointly to balance the risks between stopping trials which should continue, and continuing trials which should stop. The average sample size is used as a key parameter, which is evaluated under different degrees of power loss. Alternative objective functions and constraints are compared to assess the operating characteristics of the optimal futility scheme. Numerical results for single and multiple futility looks are provided. Supplementary materials for this article are available online.