Surface-based analysis of the cerebral cortex is ubiquitous in human neuroimaging with MRI. It is crucial for tasks like cortical registration, parcellation, and thickness estimation. Traditionally, such analyses require high-resolution, isotropic scans with good gray-white matter contrast, typically a T1-weighted scan with 1 mm resolution. This requirement precludes application of these techniques to most MRI scans acquired for clinical purposes, since they are often anisotropic and lack the required T1-weighted contrast. To overcome this limitation and enable large-scale neuroimaging studies using vast amounts of existing clinical data, we introduce recon-all-clinical, a novel methodology for cortical reconstruction, registration, parcellation, and thickness estimation for clinical brain MRI scans of any resolution and contrast. Our approach employs a hybrid analysis method that combines a convolutional neural network (CNN) trained with domain randomization to predict signed distance functions (SDFs), and classical geometry processing for accurate surface placement while maintaining topological and geometric constraints. The method does not require retraining for different acquisitions, thus simplifying the analysis of heterogeneous clinical datasets. We evaluated recon-all-clinical on multiple public datasets like ADNI, HCP, AIBL, OASIS and including a large clinical dataset of over 9,500 scans. The results indicate that our method produces geometrically precise cortical reconstructions across different MRI contrasts and resolutions, consistently achieving high accuracy in parcellation. Cortical thickness estimates are precise enough to capture aging effects, independently of MRI contrast, even though accuracy varies with slice thickness. Our method is publicly available at https://surfer.nmr.mgh.harvard.edu/fswiki/recon-all-clinical, enabling researchers to perform detailed cortical analysis on the huge amounts of already existing clinical MRI scans. This advancement may be particularly valuable for studying rare diseases and underrepresented populations where research-grade MRI data is scarce.

Over recent years, deep learning based image registration has achieved impressive accuracy in many domains, including medical imaging and, specifically, human neuroimaging with magnetic resonance imaging (MRI). However, the uncertainty estimation associated with these methods has been largely limited to the application of generic techniques (e.g., Monte Carlo dropout) that do not exploit the peculiarities of the problem domain, particularly spatial modeling. Here, we propose a principled way to propagate uncertainties (epistemic or aleatoric) estimated at the level of spatial location by these methods, to the level of global transformation models, and further to downstream tasks. Specifically, we justify the choice of a Gaussian distribution for the local uncertainty modeling, and then propose a framework where uncertainties spread across hierarchical levels, depending on the choice of transformation model. Experiments on publicly available data sets show that Monte Carlo dropout correlates very poorly with the reference registration error, whereas our uncertainty estimates correlate much better. Crucially, the results also show that uncertainty-aware fitting of transformations improves the registration accuracy of brain MRI scans. Finally, we illustrate how sampling from the posterior distribution of the transformations can be used to propagate uncertainties to downstream neuroimaging tasks. Code is available at: https://github.com/HuXiaoling/Regre4Regis.

The relationship between CSF measures of Alzheimer disease (AD) pathologies and their neurodegenerative signatures is not fully understood. This study seeks to employ machine learning approaches on clinical MRI data to identify patterns associated with amyloid and tau, aiming to guide diagnosis and therapeutic interventions. We selected brain volumes that differed significantly between AD pathology and control groups. Then we utilized logistic LASSO regression to identify and compare neurodegenerative signatures associated with amyloid and tau burdens using selected brain volumes. Our methods were applied to a dataset that included clinical MRI scans from patients who also had cerebrospinal fluid (CSF) biomarkers assessed within a year of the MRI scan (Figure 1). Model performance was measured using the cross-validation area under the receiver operating characteristic curve (AUROC), as well as the accuracy, sensitivity, and specificity in a 25% held-out test set. For amyloid status, among 30 amyloid-positive, tau-negative (A+T-) patients and 24 cognitively unimpaired A-T- controls, we identified 25 significant features associated with amyloid status, including subcortical areas (e.g., amygdala, thalamus, hippocampus), cortical areas (e.g., inferior temporal cortex, inferior parietal cortex, precuneus), ventricles (lateral ventricle, inferior lateral ventricle, and the 3 rd ventricle), cerebellum (cerebellum white matter and cerebellum cortex), and brain stem (Figure 2A). Our model with these features yielded an AUROC of 0.792 (95% CI: [0.783, 0.801]), and a sensitivity of 0.857 at specificity of 1 (Figure 2B). In the assessment of tau status, where the amyloid-negative, tau-positive group (A-T+) included 14 patients and the control group comprised 24. The volumes showed significant difference associated with tau status, including subcortical areas (amygdala, ventral diencephalon, hippocampus), the 4 th ventricle, and cerebellum white matter (Figure 3A). The model achieved an AUROC of 0.858 (95% C.I. [0.85, 0.866]), with a sensitivity of 1 at specificity of 1 (Figure 3B and C). The study underscores the potential of using widely available clinical MRI images to measure AD and associated pathologies in the brain, which hold significant promise for guiding treatment interventions.

Parcellation of mesh models for cortical analysis is a central problem in neuroimaging. Most classical and deep learning methods have requisites in terms of mesh topology, requiring inputs that are homeomorphic to a sphere (i.e., no holes or handles). Topology correction algorithms do exist, but their computational complexity is quadratic with the size of the topological defects - sometimes hours, effectively precluding segmentation of incorrect meshes, including those derived from imperfect segmentations or obtained from inherently noisy modalities like surface scanning. Furthermore, deep learning mesh segmentation also struggles surface scans of brains because they are relatively nondescript and require modeling of longer-range dependencies. Here we propose "pseudo-render-inverse-render" (PRIR), a novel perspective on cortical mesh parcellation that effectively reframes the problem as a 2D segmentation task using an direct-inverse rendering framework. Our approach: (i) renders the mesh from a number of perspectives, projecting the three components of the face normal vectors to a three-channel image; (ii) segments these images with U-Nets; (iii) maps the 2D segmentations back to vertices (inverse rendering); and (iv) aggregates the information from multiple views, postprocessing the output with a Markov Random Field to ensure smoothness and segmentation of occluded areas. PRIR is not affected by mesh topology and easily captures long-range dependencies with the U-Nets. Our results demonstrate: state-of-the-art accuracy on topologically correct white matter meshes; equally accurate performance on simulated surface scans; and robust segmentation of real surface scans.

In human neuroimaging studies, atlas registration enables mapping MRI scans to a common coordinate frame, which is necessary to aggregate data from multiple subjects. Machine learning registration methods have achieved excellent speed and accuracy but lack interpretability and flexibility at test time (since their deformation model is fixed). More recently, keypoint-based methods have been proposed to tackle these issues, but their accuracy is still subpar, particularly when fitting nonlinear transforms. Here we propose Registration by Regression (RbR), a novel atlas registration framework that: is highly robust and flexible; can be trained with cheaply obtained data; and operates on a single channel, such that it can also be used as pretraining for other tasks. RbR predicts the (x, y, z) atlas coordinates for every voxel of the input scan (i.e., every voxel is a keypoint), and then uses closed-form expressions to quickly fit transforms using a wide array of possible deformation models, including affine and nonlinear (e.g., Bspline, Demons, invertible diffeomorphic models, etc.). Robustness is provided by the large number of voxels informing the registration and can be further increased by robust estimators like RANSAC. Experiments on independent public datasets show that RbR yields more accurate registration than competing keypoint approaches, over a wide range of deformation models.

Synthetic data have emerged as an attractive option for developing machine-learning methods in human neuroimaging, particularly in magnetic resonance imaging (MRI)-a modality where image contrast depends enormously on acquisition hardware and parameters. This retrospective paper reviews a family of recently proposed methods, based on synthetic data, for generalizable machine learning in brain MRI analysis. Central to this framework is the concept of domain randomization, which involves training neural networks on a vastly diverse array of synthetically generated images with random contrast properties. This technique has enabled robust, adaptable models that are capable of handling diverse MRI contrasts, resolutions, and pathologies, while working out-of-the-box, without retraining. We have successfully applied this method to tasks such as whole-brain segmentation (SynthSeg), skull-stripping (SynthStrip), registration (SynthMorph, EasyReg), super-resolution, and MR contrast transfer (SynthSR). Beyond these applications, the paper discusses other possible use cases and future work in our methodology. Neural networks trained with synthetic data enable the analysis of clinical MRI, including large retrospective datasets, while greatly alleviating (and sometimes eliminating) the need for substantial labeled datasets, and offer enormous potential as robust tools to address various research goals.

Surface analysis of the cortex is ubiquitous in human neuroimaging with MRI, e.g., for cortical registration, parcellation, or thickness estimation. The convoluted cortical geometry requires isotropic scans (e.g., 1mm MPRAGEs) and good gray-white matter contrast for 3D reconstruction. This precludes the analysis of most brain MRI scans acquired for clinical purposes. Analyzing such scans would enable neuroimaging studies with sample sizes that cannot be achieved with current research datasets, particularly for underrepresented populations and rare diseases. Here we present the first method for cortical reconstruction, registration, parcellation, and thickness estimation for clinical brain MRI scans of any resolution and pulse sequence. The methods has a learning component and a classical optimization module. The former uses domain randomization to train a CNN that predicts an implicit representation of the white matter and pial surfaces (a signed distance function) at 1mm isotropic resolution, independently of the pulse sequence and resolution of the input. The latter uses geometry processing to place the surfaces while accurately satisfying topological and geometric constraints, thus enabling subsequent parcellation and thickness estimation with existing methods. We present results on 5mm axial FLAIR scans from ADNI and on a highly heterogeneous clinical dataset with 5,000 scans. Code and data are publicly available at https://surfer.nmr.mgh.harvard.edu/fswiki/recon-all-clinical

Brain surface analysis is essential to neuroscience, however, the complex geometry of the brain cortex hinders computational methods for this task. The difficulty arises from a discrepancy between 3D imaging data, which is represented in Euclidean space, and the non-Euclidean geometry of the highly-convoluted brain surface. Recent advances in machine learning have enabled the use of neural networks for non-Euclidean spaces. These facilitate the learning of surface data, yet pooling strategies often remain constrained to a single fixed-graph. This paper proposes a new learnable graph pooling method for processing multiple surface-valued data to output subject-based information. The proposed method innovates by learning an intrinsic aggregation of graph nodes based on graph spectral embedding. We illustrate the advantages of our approach with in-depth experiments on two large-scale benchmark datasets. The ablation study in the paper illustrates the impact of various factors affecting our learnable pooling method. The flexibility of the pooling strategy is evaluated on four different prediction tasks, namely, subject-sex classification, regression of cortical region sizes, classification of Alzheimer's disease stages, and brain age regression. Our experiments demonstrate the superiority of our learnable pooling approach compared to other pooling techniques for graph convolutional networks, with results improving the state-of-the-art in brain surface analysis.

The emerging field of geometric deep learning extends the application of convolutional neural networks to irregular domains such as graphs, meshes and surfaces. Several recent studies have explored the potential for using these techniques to analyse and segment the cortical surface. However, there has been no comprehensive comparison of these approaches to one another, nor to existing Euclidean methods, to date. This paper benchmarks a collection of geometric and traditional deep learning models on phenotype prediction and segmentation of sphericalised neonatal cortical surface data, from the publicly available Developing Human Connectome Project (dHCP). Tasks include prediction of postmenstrual age at scan, gestational age at birth and segmentation of the cortical surface into anatomical regions defined by the M-CRIB-S atlas. Performance was assessed not only in terms of model precision, but also in terms of network dependence on image registration, and model interpretation via occlusion. Networks were trained both on sphericalised and anatomical cortical meshes. Findings suggest that the utility of geometric deep learning over traditional deep learning is highly task-specific, which has implications for the design of future deep learning models on the cortical surface. The code, and instructions for data access, are available from https://github.com/Abdulah-Fawaz/Benchmarking-Surface-DL .

The varying cortical geometry of the brain creates numerous challenges for its analysis. Recent developments have enabled learning cortical data directly across multiple brain surfaces via graph convolutions. However, current graph learning algorithms fail when brain surface data are misaligned across subjects, thereby requiring to apply a costly alignment procedure in pre-processing. Adversarial training is widely used for unsupervised domain adaptation to improve segmentation performance on target data whose distribution differs from the training source data. In this paper, we exploit this technique to learn surface data across inconsistent graph alignments. This novel approach comprises a segmentator that uses graph convolution layers to enable parcellation across brain surfaces of varying geometry, and a discriminator that predicts the alignment-domain of surfaces from their segmentation. By trying to fool the discriminator, the adversarial training learns an alignment-invariant representation which yields consistent parcellations for differently-aligned surfaces. Using manually-labeled brain surface from MindBoggle, the largest publicly available dataset of this kind, we demonstrate a 2%–13% improvement in mean Dice over a non-adversarial training strategy, for test brain surfaces with no alignment or aligned on a different reference than source examples.

The analysis of the brain surface modeled as a graph mesh is a challenging task. Conventional deep learning approaches often rely on data lying in the Euclidean space. As an extension to irregular graphs, convolution operations are defined in the Fourier or spectral domain. This spectral domain is obtained by decomposing the graph Laplacian, which captures relevant shape information. However, the spectral decomposition across different brain graphs causes inconsistencies between the eigenvectors of individual spectral domains, causing the graph learning algorithm to fail. Current spectral graph convolution methods handle this variance by separately aligning the eigenvectors to a reference brain in a slow iterative step. This paper presents a novel approach for learning the transformation matrix required for aligning brain meshes using a direct data-driven approach. Our alignment and graph processing method provides a fast analysis of brain surfaces. The novel Spectral Graph Transformer (SGT) network proposed in this paper uses very few randomly sub-sampled nodes in the spectral domain to learn the alignment matrix for multiple brain surfaces. We validate the use of this SGT network along with a graph convolution network to perform cortical parcellation. Our method on 101 manually-labeled brain surfaces shows improved parcellation performance over a no-alignment strategy, gaining a significant speed (1400 fold) over traditional iterative alignment approaches.

Brain surface analysis is challenging due to the high variability of the cortical geometry. This paper presents a novel graph convolutional based approach for learning surface data directly across multiple surfaces. Current methods either rely on costly geometrical simplification processes or lack the ability to compare surface data across different domains. Our work leverages advances in spectral graph matching to align incompatible surface bases to a reference surface for direct learning of surface data. We illustrate with a cortical parcellation application the benefits of our method. We validate the algorithm over 101 manually labeled brain surfaces. The improvements in parcellation reveal a 29% increase in accuracy with drastic speed gains over conventional methods. The proposed method can be applied to other analysis of surface data, particularly relevant for studying neurological disorders.

Learning surface data is fundamental to neuroscience. Recent advances has enabled the use of graph convolution filters directly within neural network frameworks. These filters are, however, constrained to a single fixed-graph structure. A pooling strategy remains yet to be defined for learning graph-node data in non-predefined graph structures. This lack of flexibility in graph convolutional architectures currently limits applications on brain surfaces. Graph structures and number of mesh nodes, indeed, highly vary across brain geometries. This paper proposes a new general graph-based pooling method for processing full-sized surface-valued data, as input layers of graph neural networks, towards predicting subject-based variables, as output information. This novel method learns an intrinsic aggregation of input graph nodes based on the geometry of the input graph. This is leveraged using recent advances in spectral graph alignment where the surface parameterization becomes common across multiple brain geometries. These novel adaptive intrinsic pooling layers enable the exploration of entirely new architectures of graph neural networks, which were previously constrained to one single fixed structure in a dataset. We demonstrate the flexibility of the new pooling strategy in two proof-of-concept applications, namely, the classification of disease stages and regression of subject's ages using directly the surface data from varying mesh geometries.

In this work we present HyperDenseNet, a recently published 3D fully convolutional neural network that extends the definition of dense connectivity to multi-modal segmentation problems. In this architecture each modality is processed by a separate path, and dense connections occur not only between the pairs of layers within the same path, but also between those across different paths. Hence, the proposed network has total freedom to learn more complex combinations between the different modalities, within and in-between all the levels of abstraction, increasing significantly the learning representation. HyperDenseNet is evaluated on two challenging multi-modal brain tissue segmentation datasets, iSEG 2017 and MRBrainS'13, yielding significant improvements over many state-of-the-art segmentation networks on both benchmarks. Our code is publicly available at https://www.github.com/josedolz/HyperDenseNet.

Fully Convolutional Networks (FCNs) are widely used in medical image analysis for segmentation tasks. However, most FCNs fail to directly incorporate image geometry such as topology and boundary smoothness during segmentation. The sub-regions of the brain tumor in MRI images follow a particular topological order. However, the automatic segmentation of these brain tumors with conventional FCNs may violate the topological structure of brain tumors. FCNs could be constrained with a topological loss to enforce structured predictions. This paper presents the effect of such topological loss on brain tumor segmentation using the BraTS dataset.

Retinal swelling due to the accumulation of fluid is associated with the most vision-threatening retinal diseases. Optical coherence tomography (OCT) is the current standard of care in assessing the presence and quantity of retinal fluid and image-guided treatment management. Deep learning methods have made their impact across medical imaging, and many retinal OCT analysis methods have been proposed. However, it is currently not clear how successful they are in interpreting the retinal fluid on OCT, which is due to the lack of standardized benchmarks. To address this, we organized a challenge RETOUCH in conjunction with MICCAI 2017, with eight teams participating. The challenge consisted of two tasks: fluid detection and fluid segmentation. It featured for the first time: all three retinal fluid types, with annotated images provided by two clinical centers, which were acquired with the three most common OCT device vendors from patients with two different retinal diseases. The analysis revealed that in the detection task, the performance on the automated fluid detection was within the inter-grader variability. However, in the segmentation task, fusing the automated methods produced segmentations that were superior to all individual methods, indicating the need for further improvements in the segmentation performance.

Neuronal cell bodies mostly reside in the cerebral cortex. The study of this thin and highly convoluted surface is essential for understanding how the brain works. The analysis of surface data is, however, challenging due to the high variability of the cortical geometry. This paper presents a novel approach for learning and exploiting surface data directly across surface domains. Current approaches rely on geometrical simplifications, such as spherical inflations, a popular but costly process. For instance, the widely used FreeSurfer takes about 3 hours to parcellate brain surfaces on a standard machine. Direct learning of surface data via graph convolutions would provide a new family of fast algorithms for processing brain surfaces. However, the current limitation of existing state-of-the-art approaches is their inability to compare surface data across different surface domains. Surface bases are indeed incompatible between brain geometries. This paper leverages recent advances in spectral graph matching to transfer surface data across aligned spectral domains. This novel approach enables a direct learning of surface data across compatible surface bases. It exploits spectral filters over intrinsic representations of surface neighborhoods. We illustrate the benefits of this approach with an application to brain parcellation. We validate the algorithm over 101 manually labeled brain surfaces. The results show a significant improvement in labeling accuracy over recent Euclidean approaches, while gaining a drastic speed improvement over conventional methods.

Automated and accurate segmentation of cystoid structures in optical coherence tomography (OCT) is of interest in the early detection of retinal diseases. It is, however, a challenging task. We propose a novel method for localizing cysts in 3-D OCT volumes. The proposed work is biologically inspired and based on selective enhancement of the cysts, by inducing motion to a given OCT slice. A convolutional neural network is designed to learn a mapping function that combines the result of multiple such motions to produce a probability map for cyst locations in a given slice. The final segmentation of cysts is obtained via simple clustering of the detected cyst locations. The proposed method is evaluated on two public datasets and one private dataset. The public datasets include the one released for the OPTIMA cyst segmentation challenge (OCSC) in MICCAI 2015 and the DME dataset. After training on the OCSC train set, the method achieves a mean dice coefficient (DC) of 0.71 on the OCSC test set. The robustness of the algorithm was examined by cross validation on the DME and AEI (private) datasets and a mean DC values obtained were 0.69 and 0.79, respectively. Overall, the proposed system has the highest performance on all the benchmarks. These results underscore the strengths of the proposed method in handling variations in both data acquisition protocols and scanners.

Age related macula degeneration (AMD), Cystoid Macular Edema (CME) and glaucoma are retinal diseases that affects vision and often lead to irreversible blindness. Many imaging modalities have been used for screening of retinal diseases. Optical coherence tomography (OCT) is one such imaging modality that provides structural information of retina. Optical coherence tomography angiography (OCTA) also provides vascular information of retina in addition to structural information. With advancement in OCT and OCTA technology, the number of patients being scanned using these modalities is increasing exponentially. Manual analysis of vast data often makes human experts feel fatigue. Also, this extends the time to treat any patient and creates a demand to develop a fast and accurate automated OCT image analysis system. The system proposed in this thesis aims at analysing retinal anatomy and its diseases. We approach the problem of automatically analysing the retinal anatomy by segmenting its layers using a deep learning framework. In literature these algorithms usually requires preprocessing steps, such as denoising, image flatenning and edge detection, all of which involve separate parameter tunings. We propose a deep learning technique to automate all these steps and handle the presence/absence of pathologies. This model consists of a combination of Convolutional Neural Network (CNN) and Long Short Term Memory (LSTM). The CNN is used to extract layers of interest image and extract the edges, while the LSTM is used to trace the layer boundary. This model is trained on a mixture of normal and AMD cases using minimal data. Validation results on three public datasets show that the pixel-wise mean absolute error obtained with our system is 1.30 ± 0.48 which is lower than inter-marker error of 1.79 ± 0.76. The performance of the proposed module is also on par with the existing methods. We next propose three modules for disease analysis, first for diagnosing them at volume level, second at slice level and finally at pixel level by segmenting the abnormality. Firstly, we propose a novel method for glaucoma assessment at volume level using data from OCTA modality. The goal of this module is to gain insights about glaucoma indicators from both structural information (OCT volume) and vascular information (angioflow images). The proposed method achieves a mean sensitivity of 94% specificity of 91% and accuracy of 92% on 49 normal and 18 glaucomatous volumes. Secondly, for slice level disease(AMD and CME) classification from OCT volumes, we learn a decision boundary using a CNN in the new extremal representation space of an image. Evaluating on four publically available datasets with training set consisting of 3500 OCT slices and test set having 1995 OCT slices, this module achieves a mean sensitivity of 96%, specificity of 97% and accuracy of 96%. Finally, we propose an automated cyst segmentation algorithm from OCT volumes. We propose a biologically inspired method based on selective enhancement of the cysts, by inducing motion to a given OCT slice. A CNN is designed to learn a mapping function that combines the result of multiple such motions to produce a probability map for cyst locations in a given slice. The final segmentation of cysts is obtained via a simple clustering of the detected cyst locations. The proposed method is trained on OPTIMA Cyst segmentation challenge (OCSC) train test and achieves a mean Dice Coefficient (DC) of 0.71, 0.69 and 0.79 on the OCSC test set, DME dataset and AEI dataset respectively. Thus, the complete system can be employed in screening scenarios to aid retinal experts in accurate and faster analysis of retinal anatomy and disease.

Recently, dense connections have attracted substantial attention in computer vision because they facilitate gradient flow and implicit deep supervision during training. Particularly, DenseNet that connects each layer to every other layer in a feed-forward fashion and has shown impressive performances in natural image classification tasks. We propose HyperDenseNet, a 3-D fully convolutional neural network that extends the definition of dense connectivity to multi-modal segmentation problems. Each imaging modality has a path, and dense connections occur not only between the pairs of layers within the same path but also between those across different paths. This contrasts with the existing multi-modal CNN approaches, in which modeling several modalities relies entirely on a single joint layer (or level of abstraction) for fusion, typically either at the input or at the output of the network. Therefore, the proposed network has total freedom to learn more complex combinations between the modalities, within and in-between all the levels of abstraction, which increases significantly the learning representation. We report extensive evaluations over two different and highly competitive multi-modal brain tissue segmentation challenges, iSEG 2017 and MRBrainS 2013, with the former focusing on six month infant data and the latter on adult images. HyperDenseNet yielded significant improvements over many state-of-the-art segmentation networks, ranking at the top on both benchmarks. We further provide a comprehensive experimental analysis of features re-use, which confirms the importance of hyper-dense connections in multi-modal representation learning. Our code is publicly available.