PURPOSE:To increase performance and generalization ability of artificial intelligence prostate cancer detection systems by simulating physiological size changes of the bladder and rectum and, thereby, associated deformations of the prostate and its lesions. MATERIALS AND METHODS:This retrospective study included 1028 bi-parametric MRI examinations of men (age range: 40-90 years) performed between 2014 and 2019, divided into training/test sets (771/257). We integrated an 'anatomy-informed' transformation into the training of nnU-Net, by simulating soft-tissue deformations of the prostate resulting from size changes of the rectum and bladder. The effects of these strategies were evaluated using free-response receiver operating characteristic (FROC) to assess lesion-level performance, along with a variant: weighted alternative FROC (wAFROC), which prioritizes patient-level effects with localization criteria. Change in sensitivity was tested using a clustered McNemar test. Patient-level performance was assessed with standard and localized receiver operating characteristics (ROC/LROC) analysis. RESULTS:On the independent test set, the anatomy-informed model simulating changes of both rectum and bladder significantly increased lesion-level detection of true positive lesions by 18.8% (from 48 to 57, p = 0.01) and demonstrated significantly higher performance in the wAFROC analysis (from 0.597 to 0.639, p < 0.01). Patient-level ROC increased slightly (from 0.779 to 0.782, p = 0.89), while LROC analysis demonstrated increased performance (from 0.471 to 0.546). CONCLUSION:Simulation of rectum and bladder size variations during model training led to significant improvement in lesion detection performance, which may be crucial for diagnostics and therapeutic measures depending on correct lesion localization, e.g. MRI-guided biopsies or focal therapy regimes.

Cardiovascular Magnetic Resonance Imaging (cardiac MRI) is the gold standard for quantifying ventricular function, from which several parameters are derived. Among these, long-axis strain (LAS) is valuable for diagnosis of cardiovascular diseases. Unlike global longitudinal strain (GLS), which needs multi-plane imaging, LAS can be effectively derived from a single-plane four-chamber long-axis (4CH) cardiac MRI. Conventional analysis focuses on end-diastolic (ED) to end-systolic (ES) LAS, overlooking intermediate dynamics that could help distinguish diseases. In this study, we present a novel framework for estimating left ventricular LAS across five cardiac phases. The proposed method combines a self-supervised deformable image registration model for key frame detection with a supervised segmentation for landmark identification. LAS is calculated between ED and intermediate phases K (ED2K) and between consecutive phases (K2K). The methodology was developed and validated on the publicly available M&M2 dataset. The evaluation demonstrated significant differences between healthy individuals and patients with four of the seven cardiac diseases investigated not only in ED2ES, but also mid-systole to ES and ES to peak-flow. This emphasizes the diagnostic potential of phase-specific LAS analysis. The method is fully automated and fast, underscoring its potential for clinical application. The code and reference annotations will be made publicly available. https://github.com/ Cardio- AI/cmr-las- phase2phase-analysis.

Purpose Machine learning approaches can only be reliably evaluated if training, validation, and test data splits are representative and not affected by the absence of classes. Surgical workflow and instrument recognition are two tasks that are complicated in this manner, because of heavy data imbalances resulting from different length of phases and their potential erratic occurrences. Furthermore, sub-properties like instrument (co-)occurrence are usually not particularly considered when defining the split. Methods We present a publicly available data visualization tool that enables interactive exploration of dataset partitions for surgical phase and instrument recognition. The application focuses on the visualization of the occurrence of phases, phase transitions, instruments, and instrument combinations across sets. Particularly, it facilitates assessment of dataset splits, especially regarding identification of sub-optimal dataset splits. Results We performed analysis of the datasets Cholec80, CATARACTS, CaDIS, M2CAI-workflow, and M2CAI-tool using the proposed application. We were able to uncover phase transitions, individual instruments, and combinations of surgical instruments that were not represented in one of the sets. Addressing these issues, we identify possible improvements in the splits using our tool. A user study with ten participants demonstrated that the participants were able to successfully solve a selection of data exploration tasks. Conclusion In highly unbalanced class distributions, special care should be taken with respect to the selection of an appropriate dataset split because it can greatly influence the assessments of machine learning approaches. Our interactive tool allows for determination of better splits to improve current practices in the field. The live application is available at https://cardio-ai.github.io/endovis-ml/ .

Federated learning is one popular paradigm to train a joint model in a distributed, privacy-preserving environment. But partial annotations pose an obstacle meaning that categories of labels are heterogeneous over clients. We propose to learn a joint backbone in a federated manner, while each site receives its own multi-label segmentation head. By using Bayesian techniques we observe that the different segmentation heads although only trained on the individual client's labels also learn information about the other labels not present at the respective site. This information is encoded in their predictive uncertainty. To obtain a final prediction we leverage this uncertainty and perform a weighted averaging of the ensemble of distributed segmentation heads, which allows us to segment "locally unknown" structures. With our method, which we refer to as FUNAvg, we are even on-par with the models trained and tested on the same dataset on average. The code is publicly available (https://github.com/Cardio-AI/FUNAvg).

Data augmentation (DA) is a key factor in medical image analysis, such as in prostate cancer (PCa) detection on magnetic resonance images. State-of-the-art computer-aided diagnosis systems still rely on simplistic spatial transformations to preserve the pathological label post transformation. However, such augmentations do not substantially increase the organ as well as tumor shape variability in the training set, limiting the model's ability to generalize to unseen cases with more diverse localized soft-tissue deformations. We propose a new anatomy-informed transformation that leverages information from adjacent organs to simulate typical physiological deformations of the prostate and generates unique lesion shapes without altering their label. Due to its lightweight computational requirements, it can be easily integrated into common DA frameworks. We demonstrate the effectiveness of our augmentation on a dataset of 774 biopsy-confirmed examinations, by evaluating a state-of-the-art method for PCa detection with different augmentation settings.

Summary Python is the most commonly used language for deep learning (DL). Existing Python packages for mass spectrometry imaging (MSI) data are not optimized for DL tasks. We, therefore, introduce pyM2aia, a Python package for MSI data analysis with a focus on memory-efficient handling, processing and convenient data-access for DL applications. pyM2aia provides interfaces to its parent application M2aia, which offers interactive capabilities for exploring and annotating MSI data in imzML format. pyM2aia utilizes the image input and output routines, data formats, and processing functions of M2aia, ensures data interchangeability, and enables the writing of readable and easy-to-maintain DL pipelines by providing batch generators for typical MSI data access strategies. We showcase the package in several examples, including imzML metadata parsing, signal processing, ion-image generation, and, in particular, DL model training and inference for spectrum-wise approaches, ion-image-based approaches, and approaches that use spectral and spatial information simultaneously. Availability and implementation Python package, code and examples are available at (https://m2aia.github.io/m2aia)

Prostate cancer (PCa) diagnosis on multi-parametric magnetic resonance images (MRI) requires radiologists with a high level of expertise. Misalignments between the MRI sequences can be caused by patient movement, elastic soft-tissue deformations, and imaging artifacts. They further increase the complexity of the task prompting radiologists to interpret the images. Recently, computer-aided diagnosis (CAD) tools have demonstrated potential for PCa diagnosis typically relying on complex co-registration of the input modalities. However, there is no consensus among research groups on whether CAD systems profit from using registration. Furthermore, alternative strategies to handle multi-modal misalignments have not been explored so far. Our study introduces and compares different strategies to cope with image misalignments and evaluates them regarding to their direct effect on diagnostic accuracy of PCa. In addition to established registration algorithms, we propose 'misalignment augmentation' as a concept to increase CAD robustness. As the results demonstrate, misalignment augmentations can not only compensate for a complete lack of registration, but if used in conjunction with registration, also improve the overall performance on an independent test set.

Minimally invasive surgeries have restricted surgical ports, demanding a high skill level from the surgeon. Surgical simulation potentially reduces this steep learning curve and additionally provides quantitative feedback. Markerless depth sensors show great promise for quantification, but most such sensors are not designed for accurate reconstruction of complex anatomical forms in close-range. This work compares three commercially available depth sensors, namely the Intel D405, D415, and the Stereolabs Zed-Mini in the range of 12–20 cm, for use in surgical simulation. Three environments are designed that closely mimic surgical simulation, comprising planar surfaces, rigid objects, and mitral valve models of silicone and realistic porcine tissue. The cameras are evaluated on Z-accuracy, temporal noise, fill rate, checker distance, point cloud comparisons, and visual inspection of surgical scenes, across several camera settings. The Intel cameras show sub-mm accuracy in most static environments. The D415 fails in reconstructing valve models, while the Zed-Mini provides lesser temporal noise and higher fill rate. The D405 could reconstruct anatomical structures like the mitral valve leaflet and a ring prosthesis, but performs poorly for reflective surfaces like surgical tools and thin structures like sutures. If a high temporal resolution is needed and lower spatial resolution is acceptable, the Zed-Mini is the best choice, whereas the Intel D405 is the most suited for close-range applications. The D405 shows potential for applications like deformable registration of surfaces, but is not yet suitable for applications like real-time tool tracking or surgical skill assessment.

Mass spectrometry imaging vows to enable simultaneous spatially resolved investigation of hundreds of metabolites in tissues, but it primarily relies on traditional ion images for non-data-driven metabolite visualization and analysis. The rendering and interpretation of ion images neither considers nonlinearities in the resolving power of mass spectrometers nor does it yet evaluate the statistical significance of differential spatial metabolite abundance. Here, we outline the computational framework moleculaR (https://github.com/CeMOS-Mannheim/moleculaR) that is expected to improve signal reliability by data-dependent Gaussian-weighting of ion intensities and that introduces probabilistic molecular mapping of statistically significant nonrandom patterns of relative spatial abundance of metabolites-of-interest in tissue. moleculaR also enables cross-tissue statistical comparisons and collective molecular projections of entire biomolecular ensembles followed by their spatial statistical significance evaluation on a single tissue plane. It thereby fosters the spatially resolved investigation of ion milieus, lipid remodeling pathways, or complex scores like the adenylate energy charge within the same image.

Mass spectrometry imaging (MSI) is a label-free analysis method for resolving biomolecules or pharmaceuticals in the spatial domain. It offers unique perspectives for the examination of entire organs or other tissue specimens. Owing to increasing capabilities of modern MSI devices, the use of 3D and multi-modal MSI becomes feasible in routine applications — resulting in hundreds of gigabytes of data.

Three-dimensional (3D) human cell culture models have emerged as a key technology for personalized medicine and for phenotypic compound screening in more disease-like in-vitro systems. Mass spectrometry imaging (MSI) is one of the most versatile label-free techniques that enables simultaneous generation of spatial maps for multiple relevant molecules in these 3D-models. Here, we present an integrated platform for 3D-MSI of 3D-cell cultures comprising 3D-printed metal casting molds for freezing and embedding, MS imaging of 100 serial cryosections and their computational elastic 3D-reconstruction. With this platform, we monitored multiple lipids that were selectively associated with different cell-types or cell-cell interactions within 300 μm-scale fibroblast and colon cancer biculture spheroids. Our findings suggest that 3D-printing-aided precise preparation of serial sections from small spheroids and visualization of marker molecules in 3D can provide a detailed overview of the cellular metabolic interplay in 3D cell culture models in cancer research and drug discovery.

To support minimally-invasive intraoperative mitral valve repair, quantitative measurements from the valve can be obtained using an infra-red tracked stylus. It is desirable to view such manually measured points together with the endoscopic image for further assistance. Therefore, hand-eye calibration is required that links both coordinate systems and is a prerequisite to project the points onto the image plane. A complementary approach to this is to use a vision-based endoscopic stereo-setup to detect and triangulate points of interest, to obtain the 3D coordinates. In this paper, we aim to compare both approaches on a rigid phantom and two patient-individual silicone replica which resemble the intraoperative scenario. The preliminary results indicate that 3D landmark estimation, either labeled manually or through partly automated detection with a deep learning approach, provides more accurate triangulated depth measurements when performed with a tailored image-based method than with stylus measurements.

Mitral valve repair is a surgery to restore the function of the mitral valve. To achieve this, a prosthetic ring is sewed onto the mitral annulus. Analyzing the sutures, which are punctured through the annulus for ring implantation, can be useful in surgical skill assessment, for quantitative surgery and for positioning a virtual prosthetic ring model in the scene via augmented reality. This work presents a neural network approach which detects the sutures in endoscopic images of mitral valve repair and therefore solves a landmark detection problem with varying amount of landmarks, as opposed to most other existing deep learning-based landmark detection approaches. The neural network is trained separately on two data collections from different domains with the same architecture and hyperparameter settings. The datasets consist of more than 1,300 stereo frame pairs each, with a total over 60,000 annotated landmarks. The proposed heatmap-based neural network achieves a mean positive predictive value (PPV) of 66.68$\pm$4.67% and a mean true positive rate (TPR) of 24.45$\pm$5.06% on the intraoperative test dataset and a mean PPV of 81.50\pm5.77\% and a mean TPR of 61.60$\pm$6.11% on a dataset recorded during surgical simulation. The best detection results are achieved when the camera is positioned above the mitral valve with good illumination. A detection from a sideward view is also possible if the mitral valve is well perceptible.

Planning the optimal time of intervention for pulmonary valve replacement surgery in patients with the congenital heart disease Tetralogy of Fallot (TOF) is mainly based on ventricular volume and function according to current guidelines. Both of these two biomarkers are most reliably assessed by segmentation of 3D cardiac magnetic resonance (CMR) images. In several grand challenges in the last years, U-Net architectures have shown impressive results on the provided data. However, in clinical practice, data sets are more diverse considering individual pathologies and image properties derived from different scanner properties. Additionally, specific training data for complex rare diseases like TOF is scarce. For this work, 1) we assessed the accuracy gap when using a publicly available labelled data set (the Automatic Cardiac Diagnosis Challenge (ACDC) data set) for training and subsequent applying it to CMR data of TOF patients and vice versa and 2) whether we can achieve similar results when applying the model to a more heterogeneous data base. Multiple deep learning models were trained with four-fold cross validation. Afterwards they were evaluated on the respective unseen CMR images from the other collection. Our results confirm that current deep learning models can achieve excellent results (left ventricle dice of $0.951\pm{0.003}$/$0.941\pm{0.007}$ train/validation) within a single data collection. But once they are applied to other pathologies, it becomes apparent how much they overfit to the training pathologies (dice score drops between $0.072\pm{0.001}$ for the left and $0.165\pm{0.001}$ for the right ventricle).

Background: Mass spectrometry imaging (MSI) is a label-free analysis method for resolving bio-molecules or pharmaceuticals in the spatial domain. It offers unique perspectives for the examination of entire organs or other tissue specimens. Owing to increasing capabilities of modern MSI devices, the use of 3D and multi-modal MSI becomes feasible in routine applications-resulting in hundreds of gigabytes of data. To fully leverage such MSI acquisitions, interactive tools for 3D image reconstruction, visualization, and analysis are required, which preferably should be open-source to allow scientists to develop custom extensions. Findings: We introduce M(2)aia (MSI applications for interactive analysis in MITK), a software tool providing interactive and memory-efficient data access and signal processing of multiple large MSI datasets stored in imzML format. M(2)aia extends MITK, a popular open-source tool in medical image processing. Besides the steps of a typical signal processing workflow, M(2)aia offers fast visual interaction, image segmentation, deformable 3D image reconstruction, and multi-modal registration. A unique feature is that fused data with individual mass axes can be visualized in a shared coordinate system. We demonstrate features of M(2)aia by reanalyzing an N-glycan mouse kidney dataset and 3D reconstruction and multi-modal image registration of a lipid and peptide dataset of a mouse brain, which we make publicly available. Conclusions: To our knowledge, M(2)aia is the first extensible open-source application that enables a fast, user-friendly, and interactive exploration of large datasets. M(2)aia is applicable to a wide range of MSI analysis tasks.

Anisotropic multi-slice Cardiac Magnetic Resonance (CMR) Images are conventionally acquired in patient-specific short-axis (SAX) orientation. In specific cardiovascular diseases that affect right ventricular (RV) morphology, acquisitions in standard axial (AX) orientation are preferred by some investigators, due to potential superiority in RV volume measurement for treatment planning. Unfortunately, due to the rare occurrence of these diseases, data in this domain is scarce. Recent research in deep learning-based methods mainly focused on SAX CMR images and they had proven to be very successful. In this work, we show that there is a considerable domain shift between AX and SAX images, and therefore, direct application of existing models yield sub-optimal results on AX samples. We propose a novel unsupervised domain adaptation approach, which uses task-related probabilities in an attention mechanism. Beyond that, cycle consistency is imposed on the learned patient-individual 3D rigid transformation to improve stability when automatically re-sampling the AX images to SAX orientations. The network was trained on 122 registered 3D AX-SAX CMR volume pairs from a multi-centric patient cohort. A mean 3D Dice of 0.86 ± 0.06 for the left ventricle, 0.65 ± 0.08 for the myocardium, and 0.77 ± 0.10 for the right ventricle could be achieved. This is an improvement of 25% in Dice for RV in comparison to direct application on axial slices. To conclude, our pre-trained task module has neither seen CMR images nor labels from the target domain, but is able to segment them after the domain gap is reduced. Code: https://github.com/Cardio-AI/3d-mri-domain-adaptation.

Phantoms for surgical training are able to mimic cutting and suturing properties and patient-individual shape of organs, but lack a realistic visual appearance that captures the heterogeneity of surgical scenes. In order to overcome this in endoscopic approaches, hyperrealistic concepts have been proposed to be used in an augmented reality-setting, which are based on deep image-to-image transformation methods. Such concepts are able to generate realistic representations of phantoms learned from real intraoperative endoscopic sequences.

MALDI mass spectrometry imaging (MSI) enables label-free, spatially resolved analysis of a wide range of analytes in tissue sections. Quantitative analysis of MSI datasets is typically performed on single pixels or manually assigned regions of interest (ROIs). However, many sparse, small objects such as Alzheimer's disease (AD) brain deposits of amyloid peptides called plaques are neither single pixels nor ROIs. Here, we propose a new approach to facilitate the comparative computational evaluation of amyloid plaque-like objects by MSI: a fast PLAQUE PICKER tool that enables a statistical evaluation of heterogeneous amyloid peptide composition. Comparing two AD mouse models, APP NL-G-F and APP PS1, we identified distinct heterogeneous plaque populations in the NL-G-F model but only one class of plaques in the PS1 model. We propose quantitative metrics for the comparison of technical and biological MSI replicates. Furthermore, we reconstructed a high-accuracy 3D-model of amyloid plaques in a fully automated fashion, employing rigid and elastic MSI image registration using structured and plaque-unrelated reference ion images. Statistical single-plaque analysis in reconstructed 3D-MSI objects revealed the Aβ1-42Arc peptide to be located either in the core of larger plaques or in small plaques without colocalization of other Aβ isoforms. In 3D, a substantially larger number of small plaques were observed than that indicated by the 2D-MSI data, suggesting that quantitative analysis of molecularly diverse sparsely-distributed features may benefit from 3D-reconstruction. Data are available via ProteomeXchange with identifier PXD020824.

In endoscopy, depth estimation is a task that potentially helps in quantifying visual information for better scene understanding. A plethora of depth estimation algorithms have been proposed in the computer vision community. The endoscopic domain however, differs from the typical depth estimation scenario due to differences in the setup andnature of the scene. Furthermore, it is unfeasible to obtain ground truth depth information owing to an unsuitable detection range of off-the-shelf depth sensors and difficulties in settingupadepth-sensor in a surgical environment. In this paper, an existing self-supervised approach, called Monodepth [1], from the field of autonomous driving is applied to a novel dataset of stereo-endoscopic images from reconstructivemitral valve surgery.While it is already known that endoscopic scenes are more challenging than outdoor driving scenes, the paper performs experiments to quantify the comparison, anddescribe thedomaingapandchallenges involved in the transfer of these methods.

In endoscopy, depth estimation is a task that potentially helps in quantifying visual information for better scene understanding. A plethora of depth estimation algorithms have been proposed in the computer vision community. The endoscopic domain however, differs from the typical depth estimation scenario due to differences in the setup and nature of the scene. Furthermore, it is unfeasible to obtain ground truth depth information owing to an unsuitable detection range of off-the-shelf depth sensors and difficulties in setting up a depth-sensor in a surgical environment. In this paper, an existing self-supervised approach, called Monodepth [1], from the field of autonomous driving is applied to a novel dataset of stereo-endoscopic images from reconstructive mitral valve surgery. While it is already known that endoscopic scenes are more challenging than outdoor driving scenes, the paper performs experiments to quantify the comparison, and describe the domain gap and challenges involved in the transfer of these methods.