Robust and objective quantification of ophthalmic and retroorbital structures from standard neuroimaging is an accessible but still untapped resource for clinical assessment. Current approaches mostly rely on subjective manual measurements, introducing variability and limiting clinical integration. We developed and validated an automated ocular morphometrics pipeline enabling diagnostic precision through standardised orbital measurements. The presented pipeline consists of: (1) supra-resolution orbital template generation from standard structural MRI, (2) expert-guided segmentation with anatomical landmark annotation, (3) registration-based propagation to individual scans, (4) ocular neuroimaging quality control, and (5) automated extraction of 38 bilateral volumetric and geometric parameters critical for comprehensive ophthalmic and ocular structural morphometry including entire optic nerve sheath volume. We also integrated posterior globe deformation mapping for structural analysis. High-resolution orbital space template generation and subsequent landmark-based segmentation succeeded for all tested T1- and T2-weighted datasets. Ocular morphometrics revealed a substantial and significant physiological asymmetry between the eyes with minimal-to-negligible inter-dataset variations for all orbital parameters and landmarks. The pipeline demonstrated excellent reproducibility with intraclass correlation coefficients exceeding 0.8 for 30/38 metrics. We then established normative ranges from adult datasets to facilitate clinical and research interpretation. Our posterior globe deformation mapping enables sensitive morphological assessment. This pipeline ( https://github.com/Eulenburg-LMU/MReye-Seg ) enables objective, standardised and comprehensive quantification of ophthalmic morphometrics, including total optic nerve sheath volume and posterior globe deformations as potential surrogates for raised intracranial pressure. By minimising measurement variability and processing time, MReye-Seg supports eye researchers and ophthalmologists in achieving diagnostic precision for ocular and retroorbital conditions.

Medical image registration is critical for clinical applications, and fair benchmarking of different methods is essential for monitoring ongoing progress in the field. To date, the Learn2Reg 2020-2023 challenges have released several complementary datasets and established metrics for evaluations. Building on this foundation, the 2024 edition expands the challenge’s scope to cover a wider range of registration scenarios, particularly in terms of modality diversity and task complexity, by introducing three new tasks, including large-scale multi-modal registration and unsupervised inter-subject brain registration, as well as the first microscopy-focused benchmark within Learn2Reg. The new datasets also inspired new method developments, including invertibility constraints, pyramid features, keypoints alignment and instance optimisation.

OBJECTIVES:The shape is commonly used to describe the objects. State-of-the-art algorithms in medical imaging are predominantly diverging from computer vision, where voxel grids, meshes, point clouds, and implicit surface models are used. This is seen from the growing popularity of ShapeNet (51,300 models) and Princeton ModelNet (127,915 models). However, a large collection of anatomical shapes (e.g., bones, organs, vessels) and 3D models of surgical instruments is missing. METHODS:We present MedShapeNet to translate data-driven vision algorithms to medical applications and to adapt state-of-the-art vision algorithms to medical problems. As a unique feature, we directly model the majority of shapes on the imaging data of real patients. We present use cases in classifying brain tumors, skull reconstructions, multi-class anatomy completion, education, and 3D printing. RESULTS:By now, MedShapeNet includes 23 datasets with more than 100,000 shapes that are paired with annotations (ground truth). Our data is freely accessible via a web interface and a Python application programming interface and can be used for discriminative, reconstructive, and variational benchmarks as well as various applications in virtual, augmented, or mixed reality, and 3D printing. CONCLUSIONS:MedShapeNet contains medical shapes from anatomy and surgical instruments and will continue to collect data for benchmarks and applications. The project page is: https://medshapenet.ikim.nrw/.

Although purely transformer-based architectures pretrained on large datasets are introduced as foundation models for general computer vision tasks, hybrid models that incorporate combinations of convolution and transformer blocks showed state-of-the-art performance in more specialized tasks. Nevertheless, despite the performance gain of both pure and hybrid transformer-based architectures compared to convolutional networks, their high training cost and complexity make it challenging to use them in real scenarios. In this work, we propose a novel and simple architecture based on only convolutional layers and show that by just taking advantage of the attention map visualizations obtained from a self-supervised pretrained vision transformer network, complex transformer-based networks, and even 3D architectures are outperformed with much fewer computation costs. The proposed architecture is composed of two encoder branches with the original image as input in one branch and the attention map visualizations of the same image from multiple self-attention heads from a pre-trained DINO model in the other branch. The results of our experiments on medical imaging datasets show that the extracted attention map visualizations from the attention heads of a pre-trained transformer architecture combined with the image provide strong prior knowledge for a pure CNN architecture to outperform CNN-based and transformer-based architectures. Project Page: dai-net.github.io

Although purely transformer-based architectures showed promising performance in many computer vision tasks, many hybrid models consisting of CNN and transformer blocks are introduced to fit more specialized tasks. Nevertheless, despite the performance gain of both pure and hybrid transformer-based architectures compared to CNNs in medical imaging segmentation, their high training cost and complexity make it challenging to use them in real scenarios. In this work, we propose simple architectures based on purely convolutional layers, and show that by just taking advantage of the attention map visualizations obtained from a self-supervised pretrained vision transformer network (e.g., DINO) one can outperform complex transformer-based networks with much less computation costs. The proposed architecture is composed of two encoder branches with the original image as input in one branch and the attention map visualizations of the same image from multiple self-attention heads from a pre-trained DINO model (as multiple channels) in the other branch. The results of our experiments on two publicly available medical imaging datasets show that the proposed pipeline outperforms U-Net and the state-of-the-art medical image segmentation models.

Graph deep learning has recently emerged as a powerful ML concept allowing to generalize successful deep neural architectures to non-euclidean structured data. Such methods have shown promising results on a broad spectrum of applications ranging from social science, biomedicine, and particle physics to computer vision, graphics, and chemistry. One of the limitations of the majority of current graph neural network architectures is that they are often restricted to the transductive setting and rely on the assumption that the underlying graph is known and fixed. Often, this assumption is not true since the graph may be noisy, or partially and even completely unknown. In such cases, it would be helpful to infer the graph directly from the data, especially in inductive settings where some nodes were not present in the graph at training time. Furthermore, learning a graph may become an end in itself, as the inferred structure may provide complementary insights next to the downstream task. In this paper, we introduce Differentiable Graph Module (DGM), a learnable function that predicts edge probabilities in the graph which are optimal for the downstream task. DGM can be combined with convolutional graph neural network layers and trained in an end-to-end fashion. We provide an extensive evaluation of applications from the domains of healthcare (disease prediction), brain imaging (age prediction), computer graphics (3D point cloud segmentation), and computer vision (zero-shot learning). We show that our model provides a significant improvement over baselines both in transductive and inductive settings and achieves state-of-the-art results.

Graphs are a powerful tool for representing and analyzing unstructured, non-Euclidean data ubiquitous in the healthcare domain. Two prominent examples are molecule property prediction and brain connectome analysis. Importantly, recent works have shown that considering relationships between input data samples has a positive regularizing effect on the downstream task in healthcare applications. These relationships are naturally modeled by a (possibly unknown) graph structure between input samples. In this work, we propose Graph-in-Graph (GiG), a neural network architecture for protein classification and brain imaging applications that exploits the graph representation of the input data samples and their latent relation. We assume an initially unknown latent-graph structure between graph-valued input data and propose to learn a parametric model for message passing within and across input graph samples, end-to-end along with the latent structure connecting the input graphs. Further, we introduce a Node Degree Distribution Loss (NDDL) that regularizes the predicted latent relationships structure. This regularization can significantly improve the downstream task. Moreover, the obtained latent graph can represent patient population models or networks of molecule clusters, providing a level of interpretability and knowledge discovery in the input domain, which is of particular value in healthcare.

The reconstruction loss and the Kullback-Leibler divergence (KLD) loss in a variational autoencoder (VAE) often play antagonistic roles, and tuning the weight of the KLD loss in $$\beta $$ -VAE to achieve a balance between the two losses is a tricky and dataset-specific task. As a result, current practices in VAE training often result in a trade-off between the reconstruction fidelity and the continuity/disentanglement of the latent space, if the weight $$\beta $$ is not carefully tuned. In this paper, we present intuitions and a careful analysis of the antagonistic mechanism of the two losses, and propose, based on the insights, a simple yet effective two-stage method for training a VAE. Specifically, the method aggregates a learned Gaussian posterior $$z \sim q_{\theta } (z|x)$$ with a decoder decoupled from the KLD loss, which is trained to learn a new conditional distribution $$p_{\phi } (x|z)$$ of the input data x. Experimentally, we show that the aggregated VAE maximally satisfies the Gaussian assumption about the latent space, while still achieves a reconstruction error comparable to when the latent space is only loosely regularized by $$\mathcal {N}(\textbf{0},I)$$ . The proposed approach does not require hyperparameter (i.e., the KLD weight $$\beta $$ ) tuning given a specific dataset as required in common VAE training practices. We evaluate the method using a medical dataset intended for 3D skull reconstruction and shape completion, and the results indicate promising generative capabilities of the VAE trained using the proposed method. Besides, through guided manipulation of the latent variables, we establish a connection between existing autoencoder (AE)-based approaches and generative approaches, such as VAE, for the shape completion problem. Codes and pre-trained weights are available at https://github.com/Jianningli/skullVAE .

We present a deep learning model based on an autoencoder for the reconstruction of cranial and facial defects using the Medical Open Network for Artificial Intelligence (MONAI) framework, which has been pre-trained on the MUG500+ and SkullFix dataset. The implementation follows the MONAI contribution guidelines, hence, it can be easily tried out and used, and extended by MONAI users. The primary goal of this paper lies in the investigation of open-sourcing codes and pre-trained deep learning models under the MONAI framework. The pre-trained models generated in this work deliver reasonable results on the cranial and facial reconstruction task and provide an ideal starting-point for other researchers interested in further investigating the topic. We released the codes and the pre-trained model at the official MONAI ‘research contributions’ GitHub repository: https://github.com/Project-MONAI/research-contributions/tree/master/SkullRec. This contribution has two novelties: 1. Pre-training an autoencoder on the MUG500+ and SkullFix dataset for cranial and facial reconstruction using MONAI, and open-sourcing the codes and weights for other MONAI users; 2. Demonstrating that existing MONAI tutorials can be easily adapted to new use cases, such as skull (cranial and facial) reconstruction.

We are on the road to further the role of high-resolution structural neuroimaging in the diagnostics of the most prevalent vestibular disorders. After creating an in-vivo template and atlas space for the inner ear, we have no investigated structural aging across all cochlear and vestibular anatomical regions in a representative cohort (n=87). We significant reduction along the aging process for cochlear width and length as well as some semicircular canal dimensions. Total intracranial volume was a highly relevant covariate in our analysis. Aging also seemed to affect most neuroimaging quality parameters as well and had to be controlled for.

Abstract Background The evaluation of automatic segmentation algorithms is commonly performed using geometric metrics. An analysis based on dosimetric parameters might be more relevant in clinical practice but is often lacking in the literature. The aim of this study was to investigate the impact of state-of-the-art 3D U-Net-generated organ delineations on dose optimization in radiation therapy (RT) for prostate cancer patients. Methods A database of 69 computed tomography images with prostate, bladder, and rectum delineations was used for single-label 3D U-Net training with dice similarity coefficient (DSC)-based loss. Volumetric modulated arc therapy (VMAT) plans have been generated for both manual and automatic segmentations with the same optimization settings. These were chosen to give consistent plans when applying perturbations to the manual segmentations. Contours were evaluated in terms of DSC, average and 95% Hausdorff distance (HD). Dose distributions were evaluated with the manual segmentation as reference using dose volume histogram (DVH) parameters and a 3%/3 mm gamma-criterion with 10% dose cut-off. A Pearson correlation coefficient between DSC and dosimetric metrics, i.e. gamma index and DVH parameters, has been calculated. Results 3D U-Net-based segmentation achieved a DSC of 0.87 (0.03) for prostate, 0.97 (0.01) for bladder and 0.89 (0.04) for rectum. The mean and 95% HD were below 1.6 (0.4) and below 5 (4) mm, respectively. The DVH parameters, V $$_{60/65/70\,{\mathrm{Gy}}}$$ 60 / 65 / 70 Gy for the bladder and V $$_{50/65/70\,{\mathrm{Gy}}}$$ 50 / 65 / 70 Gy for the rectum, showed agreement between dose distributions within $$\pm \, 5\%$$ ± 5 % and $$\pm \,2\%$$ ± 2 % , respectively. The D $$_{98/2\%}$$ 98 / 2 % and V $$_{95\%}$$ 95 % , for prostate and its 3 mm expansion (surrogate clinical target volume) showed agreement with the reference dose distribution within 2% and 3 Gy with the exception of one case. The average gamma pass-rate was 85%. The comparison between geometric and dosimetric metrics showed no strong statistically significant correlation. Conclusions The 3D U-Net developed for this work achieved state-of-the-art geometrical performance. Analysis based on clinically relevant DVH parameters of VMAT plans demonstrated neither excessive dose increase to OARs nor substantial under/over-dosage of the target in all but one case. Yet the gamma analysis indicated several cases with low pass rates. The study highlighted the importance of adding dosimetric analysis to the standard geometric evaluation.

BackgroundIn-vivo MR-based high-resolution volumetric quantification methods of the endolymphatic hydrops (ELH) are highly dependent on a reliable segmentation of the inner ear's total fluid space (TFS). This study aimed to develop a novel open-source inner ear TFS segmentation approach using a dedicated deep learning (DL) model.MethodsThe model was based on a V-Net architecture (IE-Vnet) and a multivariate (MR scans: T1, T2, FLAIR, SPACE) training dataset (D1, 179 consecutive patients with peripheral vestibulocochlear syndromes). Ground-truth TFS masks were generated in a semi-manual, atlas-assisted approach. IE-Vnet model segmentation performance, generalizability, and robustness to domain shift were evaluated on four heterogenous test datasets (D2-D5, n = 4 × 20 ears).ResultsThe IE-Vnet model predicted TFS masks with consistently high congruence to the ground-truth in all test datasets (Dice overlap coefficient: 0.9 ± 0.02, Hausdorff maximum surface distance: 0.93 ± 0.71 mm, mean surface distance: 0.022 ± 0.005 mm) without significant difference concerning side (two-sided Wilcoxon signed-rank test, p>0.05), or dataset (Kruskal-Wallis test, p>0.05; post-hoc Mann-Whitney U, FDR-corrected, all p>0.2). Prediction took 0.2 s, and was 2,000 times faster than a state-of-the-art atlas-based segmentation method.ConclusionIE-Vnet TFS segmentation demonstrated high accuracy, robustness toward domain shift, and rapid prediction times. Its output works seamlessly with a previously published open-source pipeline for automatic ELS segmentation. IE-Vnet could serve as a core tool for high-volume trans-institutional studies of the inner ear. Code and pre-trained models are available free and open-source under https://github.com/pydsgz/IEVNet.

Abstract Background Artificial intelligences (AIs) are emerging in the field of medical informatics in many areas. They are mostly used for diagnosis support in medical imaging but have potential uses in many other fields of medicine where large datasets are available. Aim To develop an artificial intelligence (AI) “ToxNet”, a machine-learning based computer-aided diagnosis (CADx) system, which aims to predict poisons based on patient’s symptoms and metadata from our Poison Control Center (PCC) data. To prove its accuracy and compare it against medical doctors (MDs). Methods The CADx system was developed and trained using data from 781,278 calls recorded in our PCC database from 2001 to 2019. All cases were mono-intoxications. Patient symptoms and meta-information (e.g., age group, sex, etiology, toxin point of entry, weekday, etc.) were provided. In the pilot phase, the AI was trained on 10 substances, the AI’s prediction was compared to naïve matching, literature matching, a multi-layer perceptron (MLP), and the graph attention network (GAT). The trained AI’s accuracy was then compared to 10 medical doctors in an individual and in an identical dataset. The dataset was then expanded to 28 substances and the predictions and comparisons repeated. Results In the pilot, the prediction performance in a set of 8995 patients with 10 substances was 0.66 ± 0.01 (F1 micro score). Our CADx system was significantly superior to naïve matching, literature matching, MLP, and GAT (p < 0.005). It outperformed our physicians experienced in clinical toxicology in the individual and identical dataset. In the extended dataset, our CADx system was able to predict the correct toxin in a set of 36,033 patients with 28 substances with an overall performance of 0.27 ± 0.01 (F1 micro score), also significantly superior to naïve matching, literature matching, MLP, and GAT. It also outperformed our MDs. Conclusion Our AI trained on a large PCC database works well for poison prediction in these experiments. With further research, it might become a valuable aid for physicians in predicting unknown substances and might be the first step into AI use in PCCs.

Objective Strong magnetic fields from magnetic resonance (MR) scanners induce a Lorentz force that contributes to vertigo and persistent nystagmus. Prior studies have reported a predominantly horizontal direction for healthy subjects in a 7 Tesla (T) MR scanner, with slow phase velocity (SPV) dependent on head orientation. Less is known about vestibular signal behavior for subjects in a weaker, 3T magnetic field, the standard strength used in the Human Connectome Project (HCP). The purpose of this study is to characterize the form and magnitude of nystagmus induced at 3T. Methods Forty-two subjects were studied after being introduced head-first, supine into a Siemens Prisma 3T scanner. Eye movements were recorded in four separate acquisitions over 20 minutes. A biometric eye model was fit to the recordings to derive rotational eye position and then SPV. An anatomical template of the semi-circular canals was fit to the T2 anatomical image from each subject, and used to derive the angle of the B 0 magnetic field with respect to the vestibular apparatus. Results Recordings from 37 subjects yielded valid measures of eye movements. The population-mean SPV ± SD for the horizontal component was −1.38 ± 1.27 deg/sec, and vertical component was −0.93 ± 1.44 deg/sec, corresponding to drift movement in the rightward and downward direction. Although there was substantial inter-subject variability, persistent nystagmus was present in half of subjects with no significant adaptation over the 20 minute scanning period. The amplitude of vertical drift was correlated with the roll angle of the vestibular system, with a non-zero vertical SPV present at a 0 degree roll. Interpretation Non-habituating vestibular signals of varying amplitude are present in resting state data collected at 3T.

Large-scale population-based studies in medicine are a key resource towards better diagnosis, monitoring, and treatment of diseases. They also serve as enablers of clinical decision support systems, in particular computer-aided diagnosis (CADx) using machine learning (ML). Numerous ML approaches for CADx have been proposed in literature. However, these approaches assume feature-complete data, which is often not the case in clinical data. To account for missing data, incomplete data samples are either removed or imputed, which could lead to data bias and may negatively affect classification performance. As a solution, we propose an end-to-end learning of imputation and disease prediction of incomplete medical datasets via Multi-graph Geometric Matrix Completion (MGMC). MGMC uses multiple recurrent graph convolutional networks, where each graph represents an independent population model based on a key clinical meta-feature like age, sex, or cognitive function. Graph signal aggregation from local patient neighborhoods, combined with multi-graph signal fusion via self-attention, has a regularizing effect on both matrix reconstruction and classification performance. Our proposed approach is able to impute class relevant features as well as perform accurate and robust classification on two publicly available medical datasets. We empirically show the superiority of our proposed approach in terms of classification and imputation performance when compared with state-of-the-art approaches. MGMC enables disease prediction in multimodal and incomplete medical datasets. These findings could serve as baseline for future CADx approaches which utilize incomplete datasets.

Background: Multivariable analyses (MVA) and machine learning (ML) applied on large datasets may have a high potential to provide clinical decision support in neuro-otology and reveal further avenues for vestibular research. To this end, we build base-ml, a comprehensive MVA/ML software tool, and applied it to three increasingly difficult clinical objectives in differentiation of common vestibular disorders, using data from a large prospective clinical patient registry (DizzyReg). Methods: Base-ml features a full MVA/ML pipeline for classification of multimodal patient data, comprising tools for data loading and pre-processing; a stringent scheme for nested and stratified cross-validation including hyper-parameter optimization; a set of 11 classifiers, ranging from commonly used algorithms like logistic regression and random forests, to artificial neural network models, including a graph-based deep learning model which we recently proposed; a multi-faceted evaluation of classification metrics; tools from the domain of “Explainable AI” that illustrate the input distribution and a statistical analysis of the most important features identified by multiple classifiers. Results: In the first clinical task, classification of the bilateral vestibular failure (N = 66) vs. functional dizziness (N = 346) was possible with a classification accuracy ranging up to 92.5% (Random Forest). In the second task, primary functional dizziness (N = 151) vs. secondary functional dizziness (following an organic vestibular syndrome) (N = 204), was classifiable with an accuracy ranging from 56.5 to 64.2% (k-nearest neighbors/logistic regression). The third task compared four episodic disorders, benign paroxysmal positional vertigo (N = 134), vestibular paroxysmia (N = 49), Menière disease (N = 142) and vestibular migraine (N = 215). Classification accuracy ranged between 25.9 and 50.4% (Naïve Bayes/Support Vector Machine). Recent (graph-) deep learning models classified well in all three tasks, but not significantly better than more traditional ML methods. Classifiers reliably identified clinically relevant features as most important toward classification. Conclusion: The three clinical tasks yielded classification results that correlate with the clinical intuition regarding the difficulty of diagnosis. It is favorable to apply an array of MVA/ML algorithms rather than a single one, to avoid under-estimation of classification accuracy. Base-ml provides a systematic benchmarking of classifiers, with a standardized output of MVA/ML performance on clinical tasks. To alleviate re-implementation efforts, we provide base-ml as an open-source tool for the community.

The human cataract, a developing opacification of the human eye lens, currently constitutes the world's most frequent cause for blindness. As a result, cataract surgery has become the most frequently performed ophthalmic surgery in the world. By removing the human lens and replacing it with an artificial intraocular lens (IOL), the optical system of the eye is restored. In order to receive a good refractive result, the IOL specifications, especially the refractive power, have to be determined precisely prior to surgery. In the last years, there has been a body of work to perform this prediction by using biometric information extracted from OCT imaging data, recently also by machine learning (ML) methods. Approaches so far consider only biometric information or physical modelling, but provide no effective combination, while often also neglecting IOL geometry. Additionally, ML on small data sets without sufficient domain coverage can be challenging. To solve these issues, we propose OpticNet, a novel optical refraction network based on an unsupervised, domain-specific loss function that explicitly incorporates physical information into the network. By providing a precise and differentiable light propagation eye model, physical gradients following the eye optics are backpropagated into the network. We further propose a new transfer learning procedure, which allows the unsupervised pre-training on the optical model and fine-tuning of the network on small amounts of surgical patient data. We show that our method outperforms the current state of the art on five OCT-image based data sets, provides better domain coverage within its predictions, and achieves better physical consistency.

Metabolic connectivity patterns on the basis of [18F]-FDG positron emission tomography (PET) are used to depict complex cerebral network alterations in different neurological disorders and therefore may have the potential to support diagnostic decisions. In this study, we established a novel statistical classification method taking advantage of differential time-dependent states of whole-brain metabolic connectivity following unilateral labyrinthectomy (UL) in the rat and explored its classification accuracy.The dataset consisted of repeated [18F]-FDG PET measurements at baseline and 1, 3, 7, and 15 days (= maximum of 5 classes) after UL with 17 rats per measurement day. Classification in different stages after UL was performed by determining connectivity patterns for the different classes by Pearson's correlation between uptake values in atlas-based segmented brain regions. Connections were fitted with a linear function, with which different thresholds on the correlation coefficient (r = [0.5, 0.85]) were investigated. Rats were classified by determining the congruence of their PET uptake pattern with the fitted connectivity patterns in the classes.Overall, the classification accuracy with this method was 84.3% for 3 classes, 75.0% for 4 classes, and 54.1% for 5 classes and outperformed random classification as well as machine learning classification on the same dataset. The optimal classification thresholds of the correlation coefficient and distance-to-fit were found to be |r| > 0.65 and d = 4 when using Siegel's slope estimator for fitting.This connectivity-based classification method can compete with machine learning classification and may have methodological advantages when applied to support PET-based diagnostic decisions in neurological network disorders (such as neurodegenerative syndromes).