Precision medicine in ophthalmology requires accurate longitudinal predictions, but the fragmented nature of multimodal clinical data remains a barrier to forecasting. We introduce OphthaDT, an LLM-based digital twin for ophthalmology that serializes longitudinal patient histories from 3,220 patients across four Phase III clinical trials into structured narratives to forecast best corrected visual acuity (BCVA). In benchmarks spanning up to 100 weeks, OphthaDT demonstrated the lowest prediction error in neovascular age-related macular degeneration (nAMD), achieving an average mean absolute error (MAE) reduction of 6.0

Distinguishing cell types in a peripheral blood smear is critical for diagnosing blood diseases, such as leukemia subtypes. Artificial intelligence can assist in automating cell classification. For training robust machine learning algorithms, however, large and well-annotated single-cell datasets are pivotal. Here, we introduce a large, publicly available, annotated peripheral blood dataset comprising >40,000 single-cell images classified into 18 classes by cytomorphology experts from the Munich Leukemia Laboratory, the largest European laboratory for blood disease diagnostics. By making our dataset publicly available, we provide a valuable resource for medical and machine learning researchers and support the development of reliable and clinically relevant diagnostic tools for diagnosing hematological diseases.

To unravel the complexity of biological processes, it is necessary to resolve the underlying protein organization down to single proteins. Here, we present a protocol for secondary label-based unlimited multiplexed DNA-PAINT (SUM-PAINT), a DNA-PAINT-based super-resolution microscopy technique that is capable of resolving virtually unlimited protein species with single-protein resolution. We describe the steps to prepare neuronal cultures, troubleshoot and conduct SUM-PAINT experiments, and analyze the resulting feature-rich neuronal cell atlases using unsupervised machine learning approaches. For complete details on the use and execution of this protocol, please refer to Unterauer et al.1.

The demand for large quantities of high-quality annotated images poses a significant bottleneck for developing effective deep learning-based classifiers in the biomedical domain. We present a simple yet powerful solution to the cold start problem, i.e., selecting the most informative data for annotation within an unlabeled dataset. Our framework consists of three key components: (i) A self-supervised encoder to construct meaningful representations of unlabeled data, (ii) a sampling method selecting the most representative data points for annotation, and (iii) a classifier head using model ensembling to overcome the lack of validation data. We test our approach on four challenging public biomedical datasets. Our strategy outperforms the state-of-the-art approach in detecting the representative data points in all datasets and achieves a 7% improvement on a leukemia blood cell classification task. Our work offers a practical and efficient solution to the challenges associated with tedious and costly, high-quality data annotations in the biomedical field. We make our framework’s code publicly available on https://github.com/marrlab/initial-data-point-selection.

To understand biological processes, it is necessary to reveal the molecular heterogeneity of cells by gaining access to the location and interaction of all biomolecules. Significant advances were achieved by super -resolution microscopy, but such methods are still far from reaching the multiplexing capacity of proteomics. Here, we introduce secondary label -based unlimited multiplexed DNA -PAINT (SUM -PAINT), a high -throughput imaging method that is capable of achieving virtually unlimited multiplexing at better than 15 nm resolution. Using SUM -PAINT, we generated 30-plex single -molecule resolved datasets in neurons and adapted omics-inspired analysis for data exploration. This allowed us to reveal the complexity of synaptic heterogeneity, leading to the discovery of a distinct synapse type. We not only provide a resource for researchers, but also an integrated acquisition and analysis workflow for comprehensive spatial proteomics at single -protein resolution.

Imaging flow cytometry (IFC) allows rapid acquisition of numerous single-cell images per second, capturing information from multiple fluorescent channels. However, the traditional process of staining cells with fluorescently labeled conjugated antibodies for IFC analysis is time consuming, expensive, and potentially harmful to cell viability. To streamline experimental workflows and reduce costs, it is crucial to identify the most relevant channels for downstream analysis. In this study, we introduce PXPermute, a user-friendly and powerful method for assessing the significance of IFC channels, particularly for cell profiling. Our approach evaluates channel importance by permuting pixel values within each channel and analyzing the resulting impact on machine learning or deep learning models. Through rigorous evaluation of three multichannel IFC image datasets, we demonstrate PXPermute’s potential in accurately identifying the most informative channels, aligning with established biological knowledge. PXPermute can assist biologists with systematic channel analysis, experimental design optimization, and biomarker identification.

Poor generalization performance caused by distribution shifts in unseen domains often hinders the trustworthy deployment of deep neural networks. Many domain generalization techniques address this problem by adding a domain invariant regularization loss terms during training. However, there is a lack of modular software that allows users to combine the advantages of different methods with minimal effort for reproducibility. DomainLab is a modular Python package for training user specified neural networks with composable regularization loss terms. Its decoupled design allows the separation of neural networks from regularization loss construction. Hierarchical combinations of neural networks, different domain generalization methods, and associated hyperparameters, can all be specified together with other experimental setup in a single configuration file. Hierarchical combinations of neural networks, different domain generalization methods, and associated hyperparameters, can all be specified together with other experimental setup in a single configuration file. In addition, DomainLab offers powerful benchmarking functionality to evaluate the generalization performance of neural networks in out-of-distribution data. The package supports running the specified benchmark on an HPC cluster or on a standalone machine. The package is well tested with over 95 percent coverage and well documented. From the user perspective, it is closed to modification but open to extension. The package is under the MIT license, and its source code, tutorial and documentation can be found at https://github.com/marrlab/DomainLab.

Therapeutic antibodies are widely used to treat severe diseases. Most of them alter immune cells and act within the immunological synapse; an essential cell-to-cell interaction to direct the humoral immune response. Although many antibody designs are generated and evaluated, a high-throughput tool for systematic antibody characterization and prediction of function is lacking. Here, we introduce the first comprehensive open-source framework, scifAI (single-cell imaging flow cytometry AI), for preprocessing, feature engineering, and explainable, predictive machine learning on imaging flow cytometry (IFC) data. Additionally, we generate the largest publicly available IFC dataset of the human immunological synapse containing over 2.8 million images. Using scifAI, we analyze class frequency and morphological changes under different immune stimulation. T cell cytokine production across multiple donors and therapeutic antibodies is quantitatively predicted in vitro, linking morphological features with function and demonstrating the potential to significantly impact antibody design. scifAI is universally applicable to IFC data. Given its modular architecture, it is straightforward to incorporate into existing workflows and analysis pipelines, e.g., for rapid antibody screening and functional characterization.

Accurate morphological classification of white blood cells (WBCs) is an important step in the diagnosis of leukemia, a disease in which nonfunctional blast cells accumulate in the bone marrow. Recently, deep convolutional neural networks (CNNs) have been successfully used to classify leukocytes by training them on single-cell images from a specific domain. Most CNN models assume that the distributions of the training and test data are similar, i.e., the data are independently and identically distributed. Therefore, they are not robust to different staining procedures, magnifications, resolutions, scanners, or imaging protocols, as well as variations in clinical centers or patient cohorts. In addition, domain-specific data imbalances affect the generalization performance of classifiers. Here, we train a robust CNN for WBC classification by addressing cross-domain data imbalance and domain shifts. To this end, we use two loss functions and demonstrate their effectiveness in out-of-distribution (OOD) generalization. Our approach achieves the best F1 macro score compared to other existing methods and is able to consider rare cell types. This is the first demonstration of imbalanced domain generalization in hematological cytomorphology and paves the way for robust single cell classification methods for the application in laboratories and clinics.

Recent progress in computational pathology has been driven by deep learning. While code and data availability are essential to reproduce findings from preceding publications, ensuring a deep learning model's reusability is more challenging. For that, the codebase should be well-documented and easy to integrate into existing workflows and models should be robust toward noise and generalizable toward data from different sources. Strikingly, only a few computational pathology algorithms have been reused by other researchers so far, let alone employed in a clinical setting. To assess the current state of reproducibility and reusability of computational pathology algorithms, we evaluated peer-reviewed articles available in PubMed, published between January 2019 and March 2021, in 5 use cases: stain normalization; tissue type segmentation; evaluation of cell-level features; genetic alteration prediction; and inference of grading, staging, and prognostic information. We compiled criteria for data and code availability and statistical result analysis and assessed them in 160 publications. We found that only one-quarter (41 of 160 publications) made code publicly available. Among these 41 studies, three-quarters (30 of 41) analyzed their results statistically, half of them (20 of 41) released their trained model weights, and approximately a third (16 of 41) used an independent cohort for evaluation. Our review is intended for both pathologists interested in deep learning and researchers applying algorithms to computational pathology challenges. We provide a detailed overview of publications with published code in the field, list reusable data handling tools, and provide criteria for reproducibility and reusability.

Accurate classification of white blood cells in peripheral blood is essential for diagnosing hematological diseases. Due to constantly evolving clinical settings, data sources, and disease classifications, it is necessary to update machine learning classification models regularly for practical real-world use. Such models significantly benefit from sequentially learning from incoming data streams without forgetting previously acquired knowledge. However, models can suffer from catastrophic forgetting, causing a drop in performance on previous tasks when fine-tuned on new data. Here, we propose a rehearsal-based continual learning approach for class incremental and domain incremental scenarios in white blood cell classification. To choose representative samples from previous tasks, we employ exemplar set selection based on the model’s predictions. This involves selecting the most confident samples and the most challenging samples identified through uncertainty estimation of the model. We thoroughly evaluated our proposed approach on three white blood cell classification datasets that differ in color, resolution, and class composition, including scenarios where new domains or new classes are introduced to the model with every task. We also test a long class incremental experiment with both new domains and new classes. Our results demonstrate that our approach outperforms established baselines in continual learning, including existing iCaRL and EWC methods for classifying white blood cells in cross-domain environments.

the loss function in the latent feature space and output logits of the network. Our work shows how biological, epidemiological, and technical variabilities in hematologic single WBC classification can be addressed for training robust AI-based cell classifiers, paving the way for their safe and productive use in a clinical setting.

Imaging Flow Cytometry (IFC) enables rapid acquisition of thousands of single-cell images per second, capturing information from multiple fluorescent channels. However, the conventional process of staining cells with fluorescently labeled conjugated antibodies for IFC analysis is labor-intensive, costly, and potentially detrimental to cell viability. To streamline experimental workflows and reduce expenses, it is imperative to identify the most relevant channels for downstream analysis. In this study, we present PXPermute, a user-friendly and powerful method that assesses the significance of IFC channels for a given task, such as cell profiling. Our approach evaluates channel importance by permuting pixel values within each channel and analyzing the resulting impact on the performance of machine learning or deep learning models. Through rigorous evaluation on three multi-channel IFC image datasets, we demonstrate the superiority of PXPermute in accurately identifying the most informative channels, aligning with established biological knowledge. To facilitate systematic investigations of channel importance and aid biologists in optimizing their experimental designs, we have released PXPermute as an easy-to-use open-source Python package.

Massive, parallelized 3D stem cell cultures for engineering in vitro human cell types require imaging methods with high time and spatial resolution to fully exploit technological advances in cell culture technologies. Here, we introduce a large-scale integrated microfluidic chip platform for automated 3D stem cell differentiation. To fully enable dynamic high-content imaging on the chip platform, we developed a label-free deep learning method called Bright2Nuc to predict in silico nuclear staining in 3D from confocal microscopy bright-field images. Bright2Nuc was trained and applied to hundreds of 3D human induced pluripotent stem cell cultures differentiating toward definitive endoderm on a microfluidic platform. Combined with existing image analysis tools, Bright2Nuc segmented individual nuclei from bright-field images, quantified their morphological properties, predicted stem cell differentiation state, and tracked the cells over time. Our methods are available in an open-source pipeline, enabling researchers to upscale image acquisition and phenotyping of 3D cell culture.

Greater emphasis on reproducibility and reusability will advance computational pathology quickly and sustainably, ultimately optimizing clinical workflows and benefiting patient health.

Deep learning based classification of biomedical images requires expensive manual annotation by experts. Incomplete-supervision approaches including active learning, pre-training, and semi-supervised learning have thus been developed to increase classification performance with a limited number of annotated images. In practice, a combination of these approaches is often used to reach the desired performance for biomedical images. Most of these approaches are designed for natural images, which differ fundamentally from biomedical images in terms of color, contrast, image complexity, and class imbalance. In addition, it is not always clear which combination to use in practical cases. We, therefore, analyzed the performance of combining seven active learning, three pre-training, and two semi-supervised methods on four exemplary biomedical image datasets covering various imaging modalities and resolutions. The results showed that the ImageNet (pre-training) in combination with pseudo-labeling (semi-supervised learning) dominates the best performing combinations, while no particular active learning algorithm prevailed. For three out of four datasets, this combination reached over 90% of the fully supervised results by only adding 25% of labeled data. An ablation study also showed that pre-training and semi-supervised learning contributed up to 25% increase in F1-score in each cycle. In contrast, active learning contributed less than 5% increase in each cycle. Based on these results, we suggest employing the correct combination of pre-training and semi-supervised learning can be more efficient than active learning for biomedical image classification with limited annotated images. We believe that our study is an important step towards annotation-efficient model training for biomedical classification challenges.