ABSTRACT Molecular foundation models hold promise to provide accurate predictions for a large and diverse set of downstream tasks in bio‐medical research. Quality molecular representations are key and foundation model development has typically focused on a single representation or molecular view, which may have strengths or weaknesses on a given task. We develop Multi‐view Molecular Embedding with Late Fusion (MMELON), an approach that integrates pre‐trained graph, image and text foundation models and may be readily extended to additional views and models. The multi‐view model performs robustly and is validated on over 120 tasks, including molecular solubility, ADME properties, and activity against G Protein‐Coupled receptors (GPCRs). The GPCR model array is leveraged to perform a virtual screen in search of ligands binding to Alzheimer's disease related GPCRs. We identify a number of such targets and employ the multi‐view model to select strong binders from a compound screen. Predictions are validated through structure‐based modeling and identification of key binding motifs.

The growing adoption of diagnostic and prognostic algorithms in healthcare has led to concerns about the perpetuation of algorithmic bias against disadvantaged groups of individuals. Deep learning methods to detect and mitigate bias have revolved around modifying models, optimization strategies, and threshold calibration with varying levels of success and tradeoffs. However, there have been limited substantive efforts to address bias at the level of the data used to generate algorithms in healthcare datasets. We create a simple metric (AEquity) that utilizes a learning curve approximation to distinguish and mitigate bias via guided dataset collection or relabeling. We demonstrate this metric in two well-known examples: chest X-rays and healthcare cost utilization, and detect novel biases in the National Health and Nutrition Examination Survey. We demonstrate that utilizing AEquity to guide data-centric collection for each diagnostic finding in the chest radiograph dataset decreased bias by between 29% and 96.5% when measured by differences in area-under-the-curve. When we examined Black patients on Medicaid, at the intersection of race and socioeconomic status, we found that AEquity-based interventions reduced bias across a number of different fairness metrics including overall false negative rate by 33.3% (Bias Reduction Absolute = 1.88 x 10-1; 95% CI (1.4x10-1, 2.5x10-1); Bias Reduction (%) 33.3% (95% CI, 26.6-40.0)), Precision Bias by 7.50x10-2; 95% CI (7.48x10-2, 7.51x10-2); Bias Reduction (%) 94.6% (95% CI, 94.5-94.7%); False Discovery Rate by 94.5% (Absolute Bias Reduction = 3.50x10-2; 95% CI: (3.49x10-2, 3.50x10-2). Similarly, AEquity-guided data collection demonstrates bias reduction of up to 80% on mortality prediction with the National Health and Nutrition Examination Survey (Bias Reduction Absolute = 0.08; 95% CI (0.07, 0.09)). Additionally, we benchmark against balanced empirical risk minimization and calibration and we show that AEquity-guided data collection outperforms both standard approaches. Moreover, we demonstrate that AEquity works on fully connected networks, convolutional neural networks such as ResNet-50, transformer architectures such as on VIT-B-16, an 86 million parameter Vision Transformer, and nonparametric methods such as LightGBM In short, we demonstrate AEquity is a robust tool by applying it to different datasets and algorithms, intersectional analyses and measuring its effectiveness with respect to a range of traditional fairness metrics.

Microglia have been implicated as a key aspect of the pathology of Alzheimer’s disease (AD). However, high microglial heterogeneities, including disease-associated microglia (DAM), tau microglia (tau-pathology related), and neuroinflammation-like microglia (NIM), hinder the development of microglia-targeted treatment. In this study, we integrated ∼0.7 million single-nuclei RNA (snRNA)-seq transcriptomes derived from AD patient frozen brain samples using a variational autoencoder. We used trajectory analysis to identify microglial subtypes across AD progression, including DAM, tau microglia, and NIM. We conducted transition network analysis to identify putative molecular drivers of microglial subtypes across varying severities of AD and disease progression under the human protein-protein interactome network. We prioritized candidate drugs by specifically targeting transition modules using drug-gene signature enrichment analysis and we further validated drugs using two independent real-world patient databases (MarketScan [172 million insured individuals] and INSIGHT Clinical Research Network [15 million patients]). We showed that tau microglia were significantly associated with synaptic processes. Compared to DAM, upregulated genes of NIM were significantly enriched with key immune pathways (e.g., toll-like receptor). We identified potential AD pathobiological regulators (e.g., SYK , CTSB, PRKCA , INPP5D , and ADAM10 ) in transition networks between DAM and NIM. Via network-based drug repurposing prediction by specifically targeting NIM subpopulations and real-world patient data-based validation, we identified that usage of ketorolac (anti-inflammatory medicine) is significantly associated with reduced AD incidence in both MarketScan (hazard ratio [HR] = 0.81, 95% confidence interval [CI] 0.69-0.91, p-value = 0.002 after adjusting > 400 covariates) and INSIGHT (HR = 0.83, 95% CI 0.77-0.92, p-value = 0.004 after adjusting 267 covariates) patient databases. This study offers insights into pathobiology of AD-relevant microglial subtypes and identifies ketorolac as a potential anti-inflammatory treatment for AD.

Background:The growing adoption of diagnostic and prognostic algorithms in health care has led to concerns about the perpetuation of algorithmic bias against disadvantaged groups of individuals. Deep learning methods to detect and mitigate bias have revolved around modifying models, optimization strategies, and threshold calibration with varying levels of success and tradeoffs. However, there have been limited substantive efforts to address bias at the level of the data used to generate algorithms in health care datasets. Objective:The aim of this study is to create a simple metric (AEquity) that uses a learning curve approximation to distinguish and mitigate bias via guided dataset collection or relabeling. Methods:We demonstrate this metric in 2 well-known examples, chest X-rays and health care cost utilization, and detect novel biases in the National Health and Nutrition Examination Survey. Results:We demonstrated that using AEquity to guide data-centric collection for each diagnostic finding in the chest radiograph dataset decreased bias by between 29% and 96.5% when measured by differences in area under the curve. Next, we wanted to examine (1) whether AEquity worked on intersectional populations and (2) if AEquity is invariant to different types of fairness metrics, not just area under the curve. Subsequently, we examined the effect of AEquity on mitigating bias when measured by false negative rate, precision, and false discovery rate for Black patients on Medicaid. When we examined Black patients on Medicaid, at the intersection of race and socioeconomic status, we found that AEquity-based interventions reduced bias across a number of different fairness metrics including overall false negative rate by 33.3% (bias reduction absolute=1.88×10-1, 95% CI 1.4×10-1 to 2.5×10-1; bias reduction of 33.3%, 95% CI 26.6%-40%; precision bias by 7.50×10-2, 95% CI 7.48×10-2 to 7.51×10-2; bias reduction of 94.6%, 95% CI 94.5%-94.7%; false discovery rate by 94.5%; absolute bias reduction=3.50×10-2, 95% CI 3.49×10-2 to 3.50×10-2). Similarly, AEquity-guided data collection demonstrated bias reduction of up to 80% on mortality prediction with the National Health and Nutrition Examination Survey (bias reduction absolute=0.08, 95% CI 0.07-0.09). Then, we wanted to compare AEquity to state-of-the-art data-guided debiasing measures such as balanced empirical risk minimization and calibration. Consequently, we benchmarked against balanced empirical risk minimization and calibration and showed that AEquity-guided data collection outperforms both standard approaches. Moreover, we demonstrated that AEquity works on fully connected networks; convolutional neural networks such as ResNet-50; transformer architectures such as VIT-B-16, a vision transformer with 86 million parameters; and nonparametric methods such as Light Gradient-Boosting Machine. Conclusions:In short, we demonstrated that AEquity is a robust tool by applying it to different datasets, algorithms, and intersectional analyses and measuring its effectiveness with respect to a range of traditional fairness metrics.

Transcriptomic foundation models pretrained with masked language modeling can achieve low pretraining loss yet produce poor cell representations for downstream tasks. We introduce whole-cell expression decoding (WCED), where models reconstruct the entire gene vocabulary from a single CLS token embedding, even with limited inputs, creating a maximally informative bottleneck. WCED consistently outperforms MLM on all downstream metrics despite higher reconstruction error during training. Gene-level error tracking reveals that both methods preferentially learn genes whose expression co-varies with stable transcriptional programs rather than those driven by transient factors. We further add hierarchical cross-entropy loss that exploits Cell Ontology structure for zero-shot annotation at multiple granularity levels. Models trained with these objectives achieve best overall performance across CZI benchmarks, on zero-shot batch integration and linear probing cell-type annotation. Methods are implemented in biomed-multi-omic ( https://github.com/BiomedSciAI/biomed-multi-omic ), an open-source framework for transcriptomic foundation model development.

Large language models (LLMs) trained on text demonstrated remarkable results on natural language processing (NLP) tasks. These models have been adapted to decipher the language of DNA, where sequences of nucleotides act as "words" that encode genomic functions. However, the genome differs fundamentally from natural language, as it lacks clearly defined words or a consistent grammar. Although DNA language models (DNALMs) such as DNABERT, GENA-LM have achieved high level of performance on genome-related biological tasks, these models do not encode biological functions in the presence of sequence variations. To address this problem, we pre-train foundation models that effectively integrate sequence variations, in particular Single Nucleotide Polymorphisms (SNPs), as they underlie important biological functions. Specifically, we use ModernBERT to pre-train two different Biomedical Foundation Models (BMFM), namely, BMFM-DNA-REF in which the model is trained with sequences of varying lengths along with their reverse complements derived from the reference genome and BMFM-DNA-SNP in which the model is trained with sequences created using a novel representation scheme that encodes sequence variations. Our findings indicate that integrating sequence variations into DNALMs helps capture the biological functions as seen in improvements on all fine-tuning tasks. To explore the model's practical utility, we experimented with various strategies for SNP imputation on promoter detection task introduced in DNABERT-2. However, we acknowledge that the current benchmarks are limited in their ability to fully evaluate these models. To enable more comprehensive assessment in the future and encourage community contributions, we release our models through HuggingFace and the code to reproduce the results at https://github.com/BiomedSciAI/biomed-multi-omic

BackgroundUnderstanding the sensitivity and utility of clinical assessments across different HD stages is important for study/trial endpoint selection and clinical assessment development. The Integrated HD Progression Model (IHDPM) characterizes the complex symptom progression of HD and separates the disease into nine ordered disease states.ObjectiveTo generate a temporal map of discriminatory clinical measures across the IHDPM states.MethodsWe applied the IHDPM to all HD individuals in an integrated longitudinal HD dataset derived from four observational studies, obtaining disease state assignment for each study visit. Using large-scale screening, we estimated Cohen’s effect sizes to rank the discriminative power of 2,472 clinical measures for separating observations in disease state pairs. Individual trajectories through IHDPM states were examined. Discriminative analyses were limited to individuals with observations in both states of the pairs compared (N = 3,790).ResultsDiscriminative clinical measures were heterogeneous across the HD life course. UHDRS items were frequently identified as the best state pair discriminators, with UHDRS Motor items – most notably TMS – showing the highest discriminatory power between the early-disease states and early post-transition period states. UHDRS functional items emerged as strong discriminators from the transition period and on. Cognitive assessments showed good discriminative power between all state pairs examined, excepting state 1 vs. 2. Several non-UHDRS assessments were also flagged as excellent state discriminators for specific disease phases (e.g., SF-12). For certain state pairs, single assessment items other than total/summary scores were highlighted as having excellent discriminative power.ConclusionBy providing ranked quantitative scores indicating discriminatory ability of thousands of clinical measures between specific pairs of IHDPM states, our results will aid clinical trial designers select the most effective outcome measures tailored to their study cohort. Our observations may also assist in the development of end points targeting specific phases in the disease life course, through providing specific conceptual foci.

Although corticosteroids are an important treatment for inflammatory bowel disease (IBD) patients, many subjects develop dependence, leading to serious long-term side effects. We applied causal inference analyses to investigate the length of steroid use on reoperations in IBD patients. We identified subjects in the UK Biobank general practice dataset with at least one major GI surgery and followed them for at least 5 years to evaluate subsequent operations. We defined steroid dependence as at least 12 weeks of use (vs. acute steroid use) prior to baseline surgery. Of the 363 subjects included in our analyses, 163 (45%) were prescribed steroids on or before baseline surgery, and of these (N = 125 of 163, 77%) were dependent. Additional analyses for time-dependent data on prescriptions found a link between prescription length and reoperation. Among UC subjects with acute use, the odds of reoperation were significantly lower (OR: 0.32, 95% CI: 0.0-0.73). Steroid dependence resulted in a delay of reoperation (median 1.2 vs. 2.3 years, P = 0.01). Our findings indicate that long-term steroid use tends to increase the need for reoperation, whereas short-term use may reduce it.

Artificial intelligence (AI) and deep learning technologies hold promise for identifying effective drugs for human diseases, including pain. Here, we present an interpretable deep-learning-based ligand image- and receptor's three-dimensional (3D)-structure-aware framework to predict compound-protein interactions (LISA-CPI). LISA-CPI integrates an unsupervised deep-learning-based molecular image representation (ImageMol) of ligands and an advanced AlphaFold2-based algorithm (Evoformer). We demonstrated that LISA-CPI achieved ∼20% improvement in the average mean absolute error (MAE) compared to state-of-the-art models on experimental CPIs connecting 104,969 ligands and 33 G-protein-coupled receptors (GPCRs). Using LISA-CPI, we prioritized potential repurposable drugs (e.g., methylergometrine) and identified candidate gut-microbiota-derived metabolites (e.g., citicoline) for potential treatment of pain via specifically targeting human GPCRs. In summary, we presented that the integration of molecular image and protein 3D structural representations using a deep learning framework offers a powerful computational drug discovery tool for treating pain and other complex diseases if broadly applied.

INTRODUCTION:High microglial heterogeneities hinder the development of microglia-targeted treatment for Alzheimer's disease (AD). METHODS:We integrated 0.7 million single-nuclei RNA-sequencing transcriptomes from human brains using a variational autoencoder. We predicted AD-relevant microglial subtype-specific transition networks for disease-associated microglia (DAM), tau microglia, and neuroinflammation-like microglia (NIM). We prioritized drugs by specifically targeting microglia-specific transition networks and validated drugs using two independent real-world patient databases. RESULTS:We identified putative AD molecular drivers (e.g., SYK, CTSB, and INPP5D) in transition networks of DAM and NIM. Via specifically targeting NIM, we identified that usage of ketorolac was associated with reduced AD incidence in both MarketScan (hazard ratio [HR] = 0.89) and INSIGHT (HR = 0.83) Clinical Research Network databases, mechanistically supported by ketorolac-treated transcriptomic data from AD patient induced pluripotent stem cell-derived microglia. DISCUSSION:This study offers insights into the pathobiology of AD-relevant microglial subtypes and identifies ketorolac as a potential anti-inflammatory treatment for AD. HIGHLIGHTS:An integrative analysis of ≈ 0.7 million single-nuclei RNA-sequencing transcriptomes from human brains identified Alzheimer's disease (AD)-relevant microglia subtypes. Network-based analysis identified putative molecular drivers (e.g., SYK, CTSB, INPP5D) of transition networks between disease-associated microglia (DAM) and neuroinflammation-like microglia (NIM). Via network-based prediction and population-based validation, we identified that usage of ketorolac (a US Food and Drug Administration-approved anti-inflammatory medicine) was associated with reduced AD incidence in two independent patient databases. Mechanistic observation showed that ketorolac treatment downregulated the Type-I interferon signaling in patient induced pluripotent stem cell-derived microglia, mechanistically supporting its protective effects in real-world patient databases.

Accurate molecular representation of compounds is a fundamental challenge for prediction of drug targets and molecular properties. In this study, we present a molecular video-based foundation model, named VideoMol, pretrained on 120 million frames of 2 million unlabeled drug-like and bioactive molecules. VideoMol renders each molecule as a video with 60-frame and designs three self-supervised learning strategies on molecular videos to capture molecular representation. We show high performance of VideoMol in predicting molecular targets and properties across 43 drug discovery benchmark datasets. VideoMol achieves high accuracy in identifying antiviral molecules against common diverse disease-specific drug targets (i.e., BACE1 and EP4). Drugs screened by VideoMol show better binding affinity than molecular docking, revealing the effectiveness in understanding the three-dimensional structure of molecules. We further illustrate interpretability of VideoMol using key chemical substructures. Accurate molecular representation of compounds is a fundamental challenge for drug discovery. Here, the authors present a molecular video-based foundation model pretrained on 120 million frames of 2 million molecular videos, and apply it to molecular targets and properties prediction.

The high failure rate of clinical trials in Alzheimer's disease (AD) and AD-related dementia (ADRD) is due to a lack of understanding of the pathophysiology of disease, and this deficit may be addressed by applying artificial intelligence (AI) to "big data" to rapidly and effectively expand therapeutic development efforts. Recent accelerations in computing power and availability of big data, including electronic health records and multi-omics profiles, have converged to provide opportunities for scientific discovery and treatment development. Here, we review the potential utility of applying AI approaches to big data for discovery of disease-modifying medicines for AD/ADRD. We illustrate how AI tools can be applied to the AD/ADRD drug development pipeline through collaborative efforts among neurologists, gerontologists, geneticists, pharmacologists, medicinal chemists, and computational scientists. AI and open data science expedite drug discovery and development of disease-modifying therapeutics for AD/ADRD and other neurodegenerative diseases.

Quality molecular representations are key to foundation model development in bio-medical research. Previous efforts have typically focused on a single representation or molecular view, which may have strengths or weaknesses on a given task. We develop Multi-view Molecular Embedding with Late Fusion (MMELON), an approach that integrates graph, image and text views in a foundation model setting and may be readily extended to additional representations. Single-view foundation models are each pre-trained on a dataset of up to 200M molecules. The multi-view model performs robustly, matching the performance of the highest-ranked single-view. It is validated on over 120 tasks, including molecular solubility, ADME properties, and activity against G Protein-Coupled receptors (GPCRs). We identify 33 GPCRs that are related to Alzheimer's disease and employ the multi-view model to select strong binders from a compound screen. Predictions are validated through structure-based modeling and identification of key binding motifs.

Chronic gastrointestinal (GI) conditions, such as inflammatory bowel diseases (IBD), offer a promising opportunity to create classification systems that can enhance the accuracy of predicting the most effective therapies and prognosis for each patient. Here, we present a novel methodology to explore disease subtypes using our open-sourced BiomedSciAI toolkit. Applying methods available in this toolkit on the UK Biobank, including subpopulation-based feature selection and multi-dimensional subset scanning, we aimed to discover unique subgroups from GI surgery cohorts. Of a 12,073-patient cohort, a subgroup of 440 IBD patients was discovered with an increased risk of a subsequent GI surgery (OR: 2.21, 95% CI [1.81-2.69]). We iteratively demonstrate the discovery process using an additional cohort (with a narrower definition of GI surgery). Our results show that the iterative process can refine the subgroup discovery process and generate novel hypotheses to investigate determinants of treatment response.

Artificial intelligence (AI) and deep learning technologies hold promise for identifying drug-target interactions and prediction of molecular properties. Here, we present an interpretable deep learning-based Ligand Image- and receptor's three-dimensional (3D) Structures-Aware framework to predict Compound-Protein Interactions (LISA-CPI). LISA-CPI integrates an unsupervised deep learning-based molecular image representation (ImageMol, ~10 million druglike molecular images) of ligands, and an advanced AlphaFold2-based algorithm (Evoformer). We demonstrated that LISA-CPI achieved ~20% improvement on average in Mean Absolute Error (MAE) compared to state-of-the-art models on the experimental CPIs connecting 104,969 ligands and 30 G protein-coupled receptors (GPCRs). In particular, LISA-CPI showed the high accuracy of CPI predictions on 13 chronic pain-related GPCRs, with an average improvement of 20.8% in MAE (32.2% at best), and Pearson correlation coefficient (R) ranging from 0.57 to 0.81 with 0.68 on average. LISA-CPI also displayed high performance of CPI predictions on 10 benchmark kinase-ligand datasets, with Area Under Receiver Operating Characteristic (AUROC) ranging from 0.60 to 0.90 with 0.77 on average. Using LISA-CPI, we prioritized potential repurposable drugs (e.g., Methylergometrine) and identified potential gut microbiota-derived metabolites (e.g., Citicoline derived from Clostridium) for potential treatment of chronic pain by specifically targeting human GPCRs. In summary, we presented that a unique integration of ligand image and protein 3D structural representations using a deep learning framework offers a powerful computational drug discovery tool for treating chronic pain and other complex diseases if broadly applied.

Permissions Request permissions Contents list Environ. Sci.: Processes Impacts, 2023, 25, 1011 DOI: 10.1039/D3EM90020G This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. You can use material from this article in other publications without requesting further permissions from the RSC, provided that the correct acknowledgement is given. Read more about how to correctly acknowledge RSC content.

AbstractBackgroundBroad adoption of artificial intelligence (AI) algorithms in healthcare has led to perpetuation of bias found in datasets used for algorithm training. Methods to mitigate bias involve approaches after training leading to tradeoffs between sensitivity and specificity. There have been limited efforts to address bias at the level of the data for algorithm generation.MethodsWe generate a data-centric, but algorithm-agnostic approach to evaluate dataset bias by investigating how the relationships between different groups are learned at different sample sizes. We name this method AEquity and define a metric AEq. We then apply a systematic analysis of AEq values across subpopulations to identify and mitigate manifestations of racial bias.FindingsWe demonstrate that AEquity helps mitigate different biases in three different chest radiograph datasets, a healthcare costs dataset, and when using tabularized electronic health record data for mortality prediction. In the healthcare costs dataset, we show that AEquity is a more sensitive metric of label bias than model performance. AEquity can be utilized for label selection when standard fairness metrics fail. In the chest radiographs dataset, we show that AEquity can help optimize dataset selection to mitigate bias, as measured by nine different fairness metrics across nine of the most frequent diagnoses and four different protected categories (race, sex, insurance status, age) and the intersections of race and sex. We benchmark against approaches currently used after algorithm training including recalibration and balanced empirical risk minimization. Finally, we utilize AEquity to characterize and mitigate a previously unreported bias in mortality prediction with the widely used National Health and Nutrition Examination Survey (NHANES) dataset, showing that AEquity outperforms currently used approaches, and is effective at both small and large sample sizes.InterpretationAEquity can identify and mitigate bias in known biased datasets through different strategies and an unreported bias in a widely used dataset.SummaryAEquity, a machine learning approach can identify and mitigate bias the level of datasets used to train algorithms. We demonstrate it can mitigate known cases of bias better than existing methods, and detect and mitigate bias that was previously unreported.EVIDENCE IN CONTEXTEvidence before this studyMethods to mitigate algorithmic bias typically involve adjustments made after training, leading to a tradeoff between sensitivity and specificity. There have been limited efforts to mitigate bias at the level of the data.Added value of this studyThis study introduces a machine learning based method, AEquity, which analyzes the learnability of data from subpopulations at different sample sizes, which can then be used to intervene on the larger dataset to mitigate bias. The study demonstrates the detection and mitigation of bias in two scenarios where bias had been previously reported. It also demonstrates the detection and mitigation of bias the widely used National Health and Nutrition Examination Survey (NHANES) dataset, which was previously unknown.Implications of all available evidenceAEquity is a complementary approach that can be used early in the algorithm lifecycle to characterize and mitigate bias and thus prevent perpetuation of algorithmic disparities.

Recent advances in machine learning and artificial intelligence (AI) brought unprecedented promises across the fields of medicine, including nephrology. The clinical complexity and challenges in patient management highlight the potential benefit of data-driven, algorithmic approaches in nephrology.1 For example, neural networks and other deep learning methods have been applied, from analyzing kidney biopsy specimens to predicting kidney failures.2 Along with hopes and hype comes the increasing concern that data- and model-based decision making can in fact exacerbate bias and inequity in health care. Researchers have shown that a model-driven prediction of eGFR that has been used for decades could be racially biased by assigning higher eGFR estimates to patients identifying as Black, although uncertainty remains in the biological explanation underlying the race correction.3 Another model implemented in practice, the Kidney Donor Risk Index, assigns unwarrantedly higher predicted risk of kidney graft failure in patients identifying as Black, which can potentially exacerbate inequality in access to organs for transplantation.3 When a single race variable has significant potential to create bias, the likelihood of the presence of bias is much greater in black box AI models that often blindly take in a large number of variables. It is imperative, therefore, that both the developers and end users of AI-based clinical applications understand the ways in which biases arise in data and model outputs. Through this article, we aim to help the readers recognize biases in AI applications and get familiarized with methods to mitigate biases. Types of Bias in Clinical AI Applications Figure 1 illustrates the types of biases that can arise throughout different stages of AI development. At a high level, there is the algorithmic side and human side of biases, as described below.Figure 1: Stages of artificial intelligence application development and associated biases. Text in red indicates the type of bias, text in blue indicates nonalgorithmic bias mitigation, and text in tan indicates algorithmic debiasing method application.Bias in Data-Generating Process AI development begins with collecting patient data, which almost always comes from a selected sample among the underlying target population. Skewness in patient sampling can lead to disparate model performance in over- or under-represented subgroups. Differences in outcome ascertainment, such as higher sensitivity or specificity of an event of interest in electronic health data, can be another source of bias. A previous study has shown how the accurate capture of health care cost as the outcome resulted in a model preferentially recommending White patients for additional treatment resources because less money is spent on Black patients compared with White patients with similar levels of morbidity.4 Clinician bias and complex evaluation process have unfavored Black patients, leading to disparity in receiving kidney transplant; data accurately capturing this practice can generate a model that treats Black race as a risk factor for transplant failure, reinforcing the underlying inequity.5 Bias in Model Training, Testing, and Validation Often unknown to consumers of AI, numerous modeling decisions take place during the course of development. Unlike traditional medicine where publication of study protocols has become a standard practice, convoluted process of model selection, training and testing, and validation is seldom prespecified or communicated, although it has substantial effect on the outcomes. Data quality can be a function of sociodemographic factors if access to care is associated with reliable capture of data. If a group of people have a lot of missing data because of several barriers to health care, AI models will likely underperform for this group and can lead to more harm than benefits if missing data are simply excluded in model training. Similarly, various model updates that take place upon observing subpar performance, missing data treatment methods, and the decision threshold for hemodialysis can affect the performance of models predicting AKI.6 Bias in Interpretation and Application of AI Models The human side of bias plays a significant role in translation of AI to clinical benefits. Clinician trust and acceptance in AI can be a deciding factor over the actual model performance for the extent of real-world application of AI. Variability in the levels of health literacy and cultural acceptance among patients can lead to missed opportunities of improving patient outcomes through novel technologies. Importantly, the patient-provider relationship that is a product of history, culture, and mutual trust can be modified through deployment of AI in clinical nephrology, which can have unintended consequences such as loss of trust and authority or reduced adherence to medical advice.7 Ways to Mitigate and Prevent Biases in Clinical AI Application Algorithmic Debiasing Methods Biases that are algorithmic in nature, that is related to data sampling, model training, and obtaining outputs, can be addressed at least in part through the debiasing methods. In this sense, bias often refers to unwarranted statistical associations between patient attributes of interest and the outcome. Existing algorithmic debiasing methods can be categorized into preprocessing, in-processing, and postprocessing methods.8 Preprocessing methods treat the training data before model fitting to address imbalances in data. An intuitive example is the reweighing method that transforms the training data to achieve balance in groups defined by sensitive attributes of interest such as race or sex. In-processing methods modify how a model learns from data in a way that reduces the influence of a variable in the learning process. As the name implies, postprocessing takes place after a model is fitted and adjusts the outputs in a post hoc manner to address biases. Tools exist publicly to enable people to readily apply these methods in practice.8 Nonalgorithmic Bias Mitigation Completely unbiased sampling of data is usually unfeasible, so it is the responsibility of both developers and users of AI to evaluate patient representation bias. Comparing the distribution of patient demographics between training data and target population is a good starting point. Implicit bias in patient care, such as partial recommendation of novel treatment options, can be identified through examining electronic health data that reflect the practice patterns. Efforts to increase diversity in data collection and to provide equitable treatment options should accompany the aforementioned activities. In addition, detailed and transparent documentation of the modeling process, including publication of datasets and code, should become a norm in the field. It can also incentivise researchers to perform replication studies and sensitivity analyses that are critical in ascertaining clinical benefits of AI. Finally, patient and provider education is paramount to ensuring unbiased interpretation and utilization of AI. In conclusion, big data and AI utilization is an inevitable wave in medicine, and nephrology is no exception. Rigorous bias evaluation and mitigation throughout the development and application process can prevent biased AI from adversely affecting patients and health systems, especially those who are underserved. Recent efforts in providing the public with a guideline or playbook for navigating this process is important progress toward achieving fair and equitable utilization of AI.9 The epitome of AI is its ability to stay "live" and continuously learn over time, calling out the need for continuous monitoring and retraining of models to ensure unbiasedness of data and model outputs.10

Permissions Request permissions Contents list Environ. Sci.: Processes Impacts, 2023, 25, 889 DOI: 10.1039/D3EM90015K This article is licensed under a Creative Commons Attribution 3.0 Unported Licence. You can use material from this article in other publications without requesting further permissions from the RSC, provided that the correct acknowledgement is given. Read more about how to correctly acknowledge RSC content.