Whole-slide multiplex brain tissue images for spatial proteomics are massive, information-dense, and challenging to analyze. We present mVISE, an interactive multiplex visual search engine that offers an alternative programming-free query-driven analysis method based on retrieving and profiling communities of similar cells, proximal cell pairs, and multicellular niches. The retrievals can be used for exploratory cell and tissue analysis, delineating brain regions and cortical layers, profiling and comparing brain regions/sub-regions/sub-layers, etc. mVISE is enabled by multiplex encoders that seamlessly integrate visual cues across imaging channels overcoming the limitations of current foundation models. We train separate encoders to learn each facet of tissue including cell morphologies, spatial protein expression (chemoarchitecture), cell arrangements (cytoarchitecture), and wiring patterns (myeloarchitecture) from a set of user-defined molecular marker panels, without the need for human annotations or intervention, and with visual confirmation of successful learning. Multiple encoders can be combined logically to drive specialized searches. We validated mViSE’s ability to retrieve single cells, proximal cell pairs, tissue patches, delineate cortical layers, brain regions, and sub-regions. mVISE is disseminated as an open-source QuPath plug-in tool.

Time-lapse imaging microscopy in nanowell grids (TIMING) is an integrated method for dynamic profiling of live immune-target cell interactions at single-cell resolution with broad applications and impact in immunology, immunotherapy and infectious diseases. Notwithstanding these applications, the current TIMING workflows necessitate fluorescent labeling of cells for automated image analysis operations including cell classification, segmentation, and tracking. Leveraging advances in computer vision methods for label-free phase contrast time-lapse microscopy and constraints specific to TIMING, especially spatial confinement of interacting cell cohorts in an array of nanoliter-capacity wells (nanowells); and temporal consistency, we show that TIMING analysis can now be performed in a fully label-free manner, with an accuracy comparable to the fluorescence-based TIMING. The proposed label-free TIMING (LF-TIMING) method offers reduced cellular phototoxicity and fluorescence photobleaching, reduced dye-induced artifacts that can interfere with physiological accuracy and enhanced live-cell imaging duration by eliminating reliance on fluorescent labels. Importantly, it expands the versatility of TIMING by enabling direct profiling of precious patient derived cells without the need for labeling while also freeing up fluorescence channels for investigating experimental structural or functional reporters, thus extending the molecular/subcellular features that can be profiled.

BACKGROUND:The adult mammalian cerebral cortex has a vertical laminar organization consisting of six neuronal layers, with each layer subserving a specific function. Accurate delineation and assignment of cortical neurons to their appropriate layer is important to understanding normal and diseased cerebral cortex function. NEW METHOD:We present a data-driven method for delineating cortical cell layers in coronal brain sections imaged using multiplexed immunofluorescence microscopy. Our method is based on spatial cluster analysis of neuronal features using an active machine learning-enhanced Dirichlet Process Mixture Model (Actively Informed Dirichlet Process Mixture Model; AIDPMM). It enables cytometric measurements to be parcellated based on cortical cell layer, facilitating unbiased, comprehensive, and quantitative profiling of the cell layers with respect to their thickness, cellular composition, cell-phenotypic status, and spatial arrangement. Cell profiles can be compared using conventional and spatial statistical methods across layers within the same brain, or across different experimental groups, allowing researchers to analyze the effects of manipulations with cell layer-specificity. RESULTS:The accuracy of the AIDPMM cortical layer delineation was validated by comparing layer-specific marker staining to AIDPMM delineated layers (intersection over union; IoU = 92.5%), and by measuring the concordance between computational and human-delineated cortical cell layer midlines (R2 = 93.5%). Applying our method to a mild traumatic brain injury model, we detected layer-specific microglia and astrocyte activation 14 days post injury that was modified by lithium+valproate treatment. COMPARISON WITH EXISTING METHODS:Delineating cortical layers has primarily been accomplished using either of two methods. The first method, manual delineation, is labor intensive and is generally performed for a limited number of regions as required by the study. Due to its subjective nature, this approach is prone to human error and bias. The second method involves the use of markers expressed in neurons belonging to specific cortical layer. Limitations of this approach include overlapping expression of the marker between layers, and a dearth of unique molecular markers. Neither of these methods provides any kind of validation or profiling for the layer delineation, and are subject to the experimenter's subjectivity. CONCLUSIONS:Our results indicate that AIDPMM is efficient, versatile, and readily amenable to visual inspection and proofreading, and provides an efficient, unbiased method for delineating cortical neuronal layers and profiling cytometric data by cell layer.

We present a weak to strong generalization methodology for fully automated training of a multi-head extension of the Mask-RCNN method with efficient channel attention for reliable segmentation of overlapping cell nuclei in multiplex cyclic immunofluorescent (IF) whole-slide images (WSIs), and present evidence for pseudo-label correction and coverage expansion, the key phenomena underlying weak to strong generalization. This method is designed to enable domain adaptation for multiplex spatial proteomics imaging data, eliminating the need for additional human annotations in the target domain. We also present metrics for automated self-diagnosis of segmentation quality in production environments, where human visual proofreading of massive WSI images is unaffordable. Our method was benchmarked against five current widely used methods and showed a significant improvement. The code, sample WSI images, and high-resolution segmentation results are provided in open form for community adoption and adaptation.

Whole-slide multiplex imaging of brain tissue generates massive information-dense images that are challenging to analyze and require custom software. We present an alternative query-driven programming-free strategy using a multiplex visual search engine (mViSE) that learns the multifaceted brain tissue chemoarchitecture, cytoarchitecture, and myeloarchitecture. Our divide-and-conquer strategy organizes the data into panels of related molecular markers and uses self-supervised learning to train a multiplex encoder for each panel with explicit visual confirmation of successful learning. Multiple panels can be combined to process visual queries for retrieving similar communities of individual cells or multicellular niches using information-theoretic methods. The retrievals can be used for diverse purposes including tissue exploration, delineating brain regions and cortical cell layers, profiling and comparing brain regions without computer programming. We validated mViSE's ability to retrieve single cells, proximal cell pairs, tissue patches, delineate cortical layers, brain regions and sub-regions. mViSE is provided as an open-source QuPath plug-in.

MOTIVATION:High-throughput time-lapse imaging is a fundamental tool for efficient living cell profiling at single-cell resolution. Label-free phase-contrast video microscopy enables noninvasive, nontoxic, and long-term imaging. The tradeoff between speed and throughput, however, implies that despite the state-of-the-art autofocusing algorithms, out-of-focus cells are unavoidable due to the migratory nature of immune cells (velocities >10 μm/min). Here, we propose PostFocus to (i) identify out-of-focus images within time-lapse sequences with a classifier, and (ii) deploy a de-noising diffusion probabilistic model to yield reliable in-focus images. RESULTS:De-noising diffusion probabilistic model outperformed deep discriminative models with a superior performance on the whole image and around cell boundaries. In addition, PostFocus improves the accuracy of image analysis (cell and contact detection) and the yield of usable videos. AVAILABILITY AND IMPLEMENTATION:Open-source code and sample data are available at: https://github.com/kwu14victor/PostFocus.

Deep learning approaches are state-of-the-art for semantic segmentation of medical images, but unlike many deep learning applications, medical segmentation is characterized by small amounts of annotated training data. Thus, while mainstream deep learning approaches focus on performance in domains with large training sets, researchers in the medical imaging field must apply new methods in creative ways to meet the more constrained requirements of medical datasets. We propose a framework for incrementally fine-tuning a multi-class segmentation of a high-resolution multiplex (multi-channel) immuno-flourescence image of a rat brain section, using a minimal amount of labelling from a human expert. Our framework begins with a modified Swin-UNet architecture that treats each biomarker in the multiplex image separately and learns an initial "global" segmentation (pre-training). This is followed by incremental learning and refinement of each class using a very limited amount of additional labeled data provided by a human expert for each region and its surroundings. This incremental learning utilizes the multi-class weights as an initialization and uses the additional labels to steer the network and optimize it for each region in the image. In this way, an expert can identify errors in the multi-class segmentation and rapidly correct them by supplying the model with additional annotations hand-picked from the region. In addition to increasing the speed of annotation and reducing the amount of labelling, we show that our proposed method outperforms a traditional multi-class segmentation by a large margin.

BigNeuron is an open community bench-testing platform with the goal of setting open standards for accurate and fast automatic neuron tracing. We gathered a diverse set of image volumes across several species that is representative of the data obtained in many neuroscience laboratories interested in neuron tracing. Here, we report generated gold standard manual annotations for a subset of the available imaging datasets and quantified tracing quality for 35 automatic tracing algorithms. The goal of generating such a hand-curated diverse dataset is to advance the development of tracing algorithms and enable generalizable benchmarking. Together with image quality features, we pooled the data in an interactive web application that enables users and developers to perform principal component analysis, t-distributed stochastic neighbor embedding, correlation and clustering, visualization of imaging and tracing data, and benchmarking of automatic tracing algorithms in user-defined data subsets. The image quality metrics explain most of the variance in the data, followed by neuromorphological features related to neuron size. We observed that diverse algorithms can provide complementary information to obtain accurate results and developed a method to iteratively combine methods and generate consensus reconstructions. The consensus trees obtained provide estimates of the neuron structure ground truth that typically outperform single algorithms in noisy datasets. However, specific algorithms may outperform the consensus tree strategy in specific imaging conditions. Finally, to aid users in predicting the most accurate automatic tracing results without manual annotations for comparison, we used support vector machine regression to predict reconstruction quality given an image volume and a set of automatic tracings. This resource describes a collection of neurons from a variety of light microscopy-based datasets, which can serve as a gold standard for testing automated tracing algorithms, as shown by comparison of the performance of 35 algorithms.

Movie M4. Representative example of a CAR8 cell that kills multiple tumor cells.

Abstract Motivation Reliable label-free methods are needed for detecting and profiling apoptotic events in time-lapse cell–cell interaction assays. Prior studies relied on fluorescent markers of apoptosis, e.g. Annexin-V, that provide an inconsistent and late indication of apoptotic onset for human melanoma cells. Our motivation is to improve the detection of apoptosis by directly detecting apoptotic bodies in a label-free manner. Results Our trained ResNet50 network identified nanowells containing apoptotic bodies with 92% accuracy and predicted the onset of apoptosis with an error of one frame (5 min/frame). Our apoptotic body segmentation yielded an IoU accuracy of 75%, allowing associative identification of apoptotic cells. Our method detected apoptosis events, 70% of which were not detected by Annexin-V staining. Availability and implementation Open-source code and sample data provided at https://github.com/kwu14victor/ApoBDproject.

Table ST1: Glossary of terms used in the paper Figure S1. Genetic modification and expansion of CAR+ T cells. Figure S2. In vitro expansion of CAR+ T cells. Figure S3. High-throughput cytotoxicity assay for monitoring T-cell target cell interactions in nanowell grids. Figure S4. CD19 expression on NALM-6 tumor cells or CD19+EL4 target cells as determined by immunofluoresecent staining. Figure S5. Composite micrographs illustrating representative examples of the interactions between single CAR+ T cells (E) and one or more NALM-6 tumor (T) cells. Figure S6. Increased probability of killing by individual CAR+ T cells at higher target cell densities. Figure S7. Donut plots summarizing the outcomes of the interaction between individual CAR8 cells and 1-3 CD19+-NALM-6 tumor cells. Figure S8. Timelapse Imaging Microscopy In Nanowell Grids (TIMING). Figure S9. At an E:T ratio of 1:1, identification of subgroups of killer CAR8 cells based on their motility and contact behavior with tumor cells. Figure S10. At an E:T of 1:1, the total duration of conjugation prior to NALM-6 tumor cell killing is no different for the CAR8 cells in the different subgroups. Figure S11. At an E:T ratio of 1:1, CAR8 cells in the S1 subgroup, demonstrate: (A) drop in motility, and (B) increased circularization upon conjugation to NALM-6 tumor cell. Figure S12. At an E:T ratio of 1:1, CAR8 cells in the different subgroups demonstrate different frequencies and kinetics of AICD subsequent to the interactions with NALM-6 cells. Figure S13. At an E:T ratio of 1:2-5, multi-killer CAR8 cells demonstrate no significant differences in their duration of conjugation prior to killing multiple NALM-6 tumor cells. Figure S14. Despite the increased crowding because of higher tumor cell density, multi-killer CAR8 cells displayed greater motility when conjugated to tumor cell in comparison to single-killer CAR8 cells that encountered only a single tumor cell. Figure S15. At an E:T ratio of 1:1, identification of subgroups of killer CAR4 cells based on their motility and contact behavior with tumor cells. Figure S16. At an E:T of 1:1, the total duration of conjugation prior to NALM-6 tumor cell killing is significantly longer for CAR4 cells in S2 subgroup in comparison to subgroups S1 and S3. Figure S17. At an E:T of 1:1, CAR4 cells in S2 subgroup induce apoptosis in tumor cells with delayed kinetics in comparison to CAR8 cells in the S2 subgroup. Figure S18. At an E:T ratio of 1:2-5, multi-killer CAR4 cells demonstrate increased circularization upon contact with one or more NALM-6 tumor cells. Figure S19. The ability of individual CAR4 cells to simultaneously conjugate to multiple NALM-6 tumor cells increases as the number of tumor cells within the nanowell increases. Figure S20. Comparison of the killing efficiency of individual single killer CAR+ T cells (E:T 1:1) with multi-killer CAR+ T cells (E:T 1:2-5) that killed multiple NALM-6 tumor cells.

Mapping biological processes in brain tissues requires piecing together numerous histological observations of multiple tissue samples. We present a direct method that generates readouts for a comprehensive panel of biomarkers from serial whole-brain slices, characterizing all major brain cell types, at scales ranging from subcellular compartments, individual cells, local multi-cellular niches, to whole-brain regions from each slice. We use iterative cycles of optimized 10-plex immunostaining with 10-color epifluorescence imaging to accumulate highly enriched image datasets from individual whole-brain slices, from which seamless signal-corrected mosaics are reconstructed. Specific fluorescent signals of interest are isolated computationally, rejecting autofluorescence, imaging noise, cross-channel bleed-through, and cross-labeling. Reliable large-scale cell detection and segmentation are achieved using deep neural networks. Cell phenotyping is performed by analyzing unique biomarker combinations over appropriate subcellular compartments. This approach can accelerate pre-clinical drug evaluation and system-level brain histology studies by simultaneously profiling multiple biological processes in their native anatomical context.

Arti ficial intelligence (AI) for the purpose of this review is an umbrella term for technologies emulating a nephropathologist ?s ability to extract information on diagnosis, prognosis, and therapy responsiveness from native or transplant kidney biopsies. Although AI can be used to analyze a wide variety of biopsy -related data, this review focuses on whole slide images traditionally used in nephropathology. AI applications in nephropathology have recently become available through several advancing technologies, including (i) widespread introduction of glass slide scanners, (ii) data servers in pathology departments worldwide, and (iii) through greatly improved computer hardware to enable AI training. In this review, we explain how AI can enhance the reproducibility of nephropathology results for certain parameters in the context of precision medicine using advanced architectures, such as convolutional neural networks, that are currently the state of the art in machine learning software for this task. Because AI applications in nephropathology are still in their infancy, we show the power and potential of AI applications mostly in the example of oncopathology. Moreover, we discuss the technological obstacles as well as the current stakeholder and regulatory concerns about developing AI applications in nephropathology from the perspective of nephropathologists and the wider nephrology community. We expect the gradual introduction of these technologies into routine diagnostics and research for selective tasks, suggesting that this technology will enhance the performance of nephropathologists rather than making them redundant.

Super Resolution (SR) microscopy leverages a variety of optical and computational techniques for overcoming the optical diffraction limit to acquire additional spatial details. However, added spatial details challenge existing segmentation tools. Confounding features include protein distributions that form membranes and boundaries, such as cellular and nuclear surfaces. We present a segmentation pipeline that retains the benefits provided by SR in surface separation while providing a tensor field to overcome these confounding features. The proposed technique leverages perceptual grouping to generate a tensor field that enables robust evolution of active contours despite ill-defined membrane boundaries.

The original version of this book was revised. The following corrections were implemented: The acronym was corrected to "MIL3ID" throughout the book. The equation on page 5 of Chapter 14 was modified to improve its accuracy and readability.

T cells engineered to express chimeric antigen receptor (CAR) targeting CD19 have shown promising clinical responses in patients with certain hematologic malignancies, however, it is desirable to be able to enrich cells with enhanced anti-tumor efficacy prior to infusion. We utilized a suite of high-throughput technologies with single-cell resolution, including Timelapse Imaging Microscopy In Nanowell Grids (TIMING) that integrates cytokine profiling to reveal that persistent motility of CD19- specific CAR T cells is correlated to desirable polyfunctionality (elimination of tumor cells and cytokine secretion), contributing to anti-tumor effects. We implemented a marker-free Boyden chamber-based method to enrich CAR+ T cells with persistent motility (motile cell). Integration of transcriptomic profiling, immune phenotyping and metabolism demonstrated that motile cells are more naïve-like with higher oxidative metabolism and spare respiratory capacity. Our result also revealed that the master metabolic regulator AMP kinase (AMPK) is required for CAR+ T cells with high motility. We used a xenograft leukemia mouse model (CD19+ NALM-6) and validated that the motile cells have enhanced persistence and superior anti-cancer effect in vivo compared to the parental un-sorted population. Collectively, our multi-dimensional results demonstrated that persistent motility is a selectable biomarker of expanded CAR+ T cell bioactivity.

Motivation: Automated profiling of cell–cell interactions from high-throughput time-lapse imaging microscopy data of cells in nanowell grids (TIMING) has led to fundamental insights into cell–cell interactions in immunotherapy. This application note aims to enable widespread adoption of TIMING by (i) enabling the computations to occur on a desktop computer with a graphical processing unit instead of a server; (ii) enabling image acquisition and analysis to occur in the laboratory avoiding network data transfers to/from a server and (iii) providing a comprehensive graphical user interface. Results: On a desktop computer, TIMING 2.0 takes 5 s/block/image frame, four times faster than our previous method on the same computer, and twice as fast as our previous method (TIMING) running on a Dell PowerEdge server. The cell segmentation accuracy (f-number = 0.993) is superior to our previous method (f-number1⁄4 0.821). A graphical user interface provides the ability to inspect the video analysis results, make corrective edits efficiently (one-click editing of an entire nanowell video sequence in 5–10 s) and display a summary of the cell killing efficacy measurements. Availability and implementation: Open source Python software (GPL v3 license), instruction manual, sample data and sample results are included with the Supplement (https://github.com/ RoysamLab/TIMING2). Contact: nvaradar@central.uh.edu or broysam@central.uh.edu Supplementary information: Supplementary data are available at Bioinformatics online.

T cell-based therapies have shown promising results, but adoptive T cell therapy (ACT) for solid tumors like melanoma treatment has met limited success. The goal of this study was to identify attributes of T cell fitness important for optimal anti-tumor efficacy. This was accomplished by integrated profiling of expanded tumor infiltrating lymphocytes, TILs (responders (CR) and progressive diseases (PD)) co-cultured overnight with their autologous primary tumor cells with the aid of a suite of single-cell, transcriptional, proteomic and functional assays to construct a genome-scale metabolic model of the metabolism of TILs and tumor cells in direct competition with each other. Sorted live TILs after co-culture were used to performed RNA-seq, and proteome-wide profiling by mass spectrometry. Gene enrichment of the CR TIL suggested an increased expression in genes involving processes such as glycolysis, hypoxia, adipogenesis, and mTORC1 signaling. PD tumors showed an increase for epithelial to mesenchymal transition (EMT) and glycolysis. Comparisons of transcripts and proteins in both CR and PD were tightly correlated (Spearman rank 0.765). Since patient-derived cell numbers were limited (<50,000), we utilized genome-scale metabolic models to infer relevant metabolic pathways by comparison to the human metabolic Atlas (HMR2). We observed high enrichment in fatty acid β-oxidation processes in different organelles related to CR TIL and tumor. Lastly, we examined if increasing fatty acid oxidation in TILs might enable their increased survival in metabolically replete environments. These results show that the fatty acid metabolism pathway is associated with the ACT efficacy, and it can contribute to improving treatment.

Automatic and accurate classification of apoptosis, or programmed cell death, will facilitate cell biology research. The state-of-the-art approaches in apoptosis classification use deep convolutional neural networks (CNNs). However, these networks are not efficient in encoding the part-whole relationships, thus requiring a large number of training samples to achieve robust generalization. This paper proposes an efficient variant of capsule networks (CapsNets) as an alternative to CNNs. Extensive experimental results demonstrate that the proposed CapsNets achieve competitive performances in target cell apoptosis classification, while significantly outperforming CNNs when the number of training samples is small. To utilize temporal information within microscopy videos, we propose a recurrent CapsNet constructed by stacking a CapsNet and a bi-directional long short-term recurrent structure. Our experiments show that when considering temporal constraints, the recurrent CapsNet achieves 93.8% accuracy and makes significantly more consistent prediction than NNs.