An incomplete field-of-view (FOV) in diffusion magnetic resonance imaging (dMRI) can severely hinder the volumetric and bundle analyses of whole-brain white matter connectivity. Although existing works have investigated imputing the missing regions using deep generative models, it remains unclear how to specifically utilize additional information from paired multi-modality data and whether this can enhance the imputation quality and be useful for downstream tractography. To fill this gap, we propose a novel framework for imputing dMRI scans in the incomplete part of the FOV by integrating the learned diffusion features in the acquired part of the FOV to the complete brain anatomical structure. We hypothesize that by this design the proposed framework can enhance the imputation performance of the dMRI scans and therefore be useful for repairing whole-brain tractography in corrupted dMRI scans with incomplete FOV. We tested our framework on two cohorts from different sites with a total of 96 subjects and compared it with a baseline imputation method that treats the information from T1w and dMRI scans equally. The proposed framework achieved significant improvements in imputation performance, as demonstrated by angular correlation coefficient (p < 1E-5), and in downstream tractography accuracy, as demonstrated by Dice score (p < 0.01). Results suggest that the proposed framework improved imputation performance in dMRI scans by specifically utilizing additional information from paired multi-modality data, compared with the baseline method. The imputation achieved by the proposed framework enhances whole brain tractography, and therefore reduces the uncertainty when analyzing bundles associated with neurodegenerative.

Semi-Supervised Learning reduces the annotation burden for medical image segmentation but currently suffers from spatial context overfitting and confirmation bias due to noisy pseudo-labels. This study aims to propose a robust framework to overcome these limitations. We introduce a novel framework termed Spatially Decoupled Reliable Mutual Learning (SDRML). To address context overfitting, we propose a Spatial Decoupling strategy that utilizes translation consistency, compelling the model to focus on intrinsic anatomical features rather than fixed background contexts. To mitigate confirmation bias, we design a Reliable Mutual Learning mechanism incorporating a Confident Regional Cross-entropy loss. This loss dynamically filters low-confidence predictions, ensuring only reliable pseudo-labels guide the tri-model co-training process. Extensive experiments were conducted on the ACDC (2D MRI), Left Atrium (3D MRI), and Pancreas-CT datasets. SDRML significantly outperforms state-of-the-art methods across all benchmarks. Notably, it demonstrates superior robustness and segmentation accuracy in data-scarce scenarios, such as regimes with only 10

Multimodal magnetic resonance imaging (MRI) provides complementary information for brain tumor segmentation, and several methods leveraging full modalities have been proposed. However, capturing the full modality information is challenging due to commonplace data corruption, imperfect imaging protocols, and patient-related constraints. The unavailability of certain modalities can significantly undermine the performance of segmentation methods that rely on full-modality data. To address this issue, this paper proposes a deformation-aware and reconstruction-driven method for brain tumor segmentation in the presence of missing modalities. The proposed method introduces a local–global modeling module to enhance the intramodal feature representation ability of the modality-specific encoder. Considering the irregular shape of tumor regions, we develop a deformation-adaptive perceptual multimodal representation learning module that learns deformation information from an incomplete set of multimodal images, thereby guiding the network to accurately localize the tumor regions. Furthermore, we design a reconstruction-driven key-information mining module that recovers the original images from the features extracted by the encoder. This process further ensures that the encoder can extract the key tumor discriminative features. During the inference phase, the module is removed to mitigate additional computational burdens. Experimental results on two publicly available multimodal brain tumor benchmark datasets show that the proposed method outperforms existing brain tumor segmentation methods with missing modalities. The code is available at https://github.com/Linzy0227/SRMNet.

The rapid development of the Internet of Things has promoted the progress of human-computer interaction technology, in which gesture recognition, as a key component, provides diversified applications for smart homes, elderly care, and other fields. In recent years, with the wide application of Wi-Fi, gesture recognition based on channel state information (CSI) has attracted much attention due to its advantages of flexibility, convenience, and cost-effectiveness. In this paper, we propose a gesture recognition system that includes an image conversion module and a 3D convolutional network recognition module. In the image conversion module, we treat the CSI subcarrier information from multiple receivers as a one-dimensional time series and convert them into images separately. Then by aggregating all the image data on the new depth dimension, we finally input the processed data into the 3D Convolutional network recognition module for gesture recognition. This module focuses on capturing changes in CSI information over time and space. We evaluated our model on a Widar3.0 open data set, and the experimental results show that the proposed method has a good effect in realizing cross-domain gesture recognition.

Purpose:In brain diffusion magnetic resonance imaging (dMRI), the volumetric and bundle analyses of whole-brain tissue microstructure and connectivity can be severely impeded by an incomplete field of view (FOV). We aim to develop a method for imputing the missing slices directly from existing dMRI scans with an incomplete FOV. We hypothesize that the imputed image with a complete FOV can improve whole-brain tractography for corrupted data with an incomplete FOV. Therefore, our approach provides a desirable alternative to discarding the valuable brain dMRI data, enabling subsequent tractography analyses that would otherwise be challenging or unattainable with corrupted data. Approach:We propose a framework based on a deep generative model that estimates the absent brain regions in dMRI scans with an incomplete FOV. The model is capable of learning both the diffusion characteristics in diffusion-weighted images (DWIs) and the anatomical features evident in the corresponding structural images for efficiently imputing missing slices of DWIs in the incomplete part of the FOV. Results:For evaluating the imputed slices, on the Wisconsin Registry for Alzheimer's Prevention (WRAP) dataset, the proposed framework achieved PSNR b 0 = 22.397 , SSIM b 0 = 0.905 , PSNR b 1300 = 22.479 , and SSIM b 1300 = 0.893 ; on the National Alzheimer's Coordinating Center (NACC) dataset, it achieved PSNR b 0 = 21.304 , SSIM b 0 = 0.892 , PSNR b 1300 = 21.599 , and SSIM b 1300 = 0.877 . The proposed framework improved the tractography accuracy, as demonstrated by an increased average Dice score for 72 tracts ( p < 0.001 ) on both the WRAP and NACC datasets. Conclusions:Results suggest that the proposed framework achieved sufficient imputation performance in brain dMRI data with an incomplete FOV for improving whole-brain tractography, thereby repairing the corrupted data. Our approach achieved more accurate whole-brain tractography results with an extended and complete FOV and reduced the uncertainty when analyzing bundles associated with Alzheimer's disease.

Limited progress has been made in continual unsupervised learning of representations, especially in reusing, expanding, and continually disentangling learned semantic factors across data environments. We argue that this is because existing approaches treat continually-arrived data independently, without considering how they are related based on the underlying semantic factors. We address this by a new generative model describing a topologically-connected mixture of spike-and-slab distributions in the latent space, learned end-to-end in a continual fashion via principled variational inference. The learned mixture is able to automatically discover the active semantic factors underlying each data environment and to accumulate their relational structure based on that. This distilled knowledge of different data environments can further be used for generative replay and guiding continual disentangling of new semantic factors. We tested the presented method on a split version of 3DShapes to provide the first quantitative disentanglement evaluation of continually learned representations, and further demonstrated its ability to continually disentangle new representations in benchmark datasets.

Multimodal Magnetic Resonance Imaging (MRI) can provide valuable complementary information and substantially enhance the performance of brain tumor segmentation. However, it is common for certain modalities to be absent or missing during clinical diagnosis, which can significantly impair segmentation techniques that rely on complete modalities. Current advanced methods attempt to address this challenge by developing shared feature representations via modal fusion to handle different missing modality situations. Considering the importance of missing modality information in multimodal segmentation, this paper utilize a feature reconstruction method to recover the missing information, and proposes a joint learning-based feature reconstruction and enhancement method for incomplete modality brain tumor segmentation. The method leverages an information learning mechanism to transfer information from the complete modality to a single modality, enabling it to obtain complete brain tumor information, even without the support of other modalities. Additionally, the method incorporates a module for reconstructing missing modality features, which recovers fused features of the absent modality through utilizing the abundant potential information obtained from the available modalities. Furthermore, the feature enhancement mechanism improves shared feature representation by utilizing the information obtained from the missing modalities that have been reconstructed. These processes enable the method to obtain more comprehensive information regarding brain tumors in various missing modality circumstances, thereby enhancing the model's robustness. The performance of the proposed model was evaluated on BraTS datasets and compared with other deep learning algorithms using Dice similarity scores. On the BraTS2018 dataset, the proposed algorithm achieved a Dice similarity score of 86.28%, 77.02%, and 59.64% for whole tumors, tumor cores, and enhanced tumors, respectively. These results demonstrate the superiority of our framework over state-of-the-art methods in missing modalities situations.

Spurious correlation caused by subgroup underrepresentation has received increasing attention as a source of bias that can be perpetuated by deep neural networks (DNNs). Distributionally robust optimization has shown success in addressing this bias, although the underlying working mechanism mostly relies on upweighting under-performing samples as surrogates for those underrepresented in data. At the same time, while invariant representation learning has been a powerful choice for removing nuisance-sensitive features, it has been little considered in settings where spurious correlations are caused by significant underrepresentation of subgroups. In this paper, we take the first step to better understand and improve the mechanisms for debiasing spurious correlation due to subgroup underrepresentation in medical image classification. Through a comprehensive evaluation study, we first show that 1) generalized reweighting of under-performing samples can be problematic when bias is not the only cause for poor performance, while 2) naive invariant representation learning suffers from spurious correlations itself. We then present a novel approach that leverages robust optimization to facilitate the learning of invariant representations at the presence of spurious correlations. Finetuned classifiers utilizing such representation demonstrated improved abilities to reduce subgroup performance disparity, while maintaining high average and worst-group performance.

Modern applications increasingly require learning and forecasting latent dynamics from high-dimensional time-series. Compared to univariate time-series forecasting, this adds a new challenge of reasoning about the latent dynamics of an unobserved abstract state. Sequential latent variable models (SLVMs) present an attractive solution, although existing works either struggle with long-term forecasting or have difficulty learning across diverse dynamics. In this paper, we first present a conceptual framework of SLVMs to unify existing works, contrast their fundamental limitations, and identify an intuitive solution to long-term forecasting for diverse dynamics via meta-learning. We then present a few-shot forecasting framework for high-dimensional time-series: instead of learning a single dynamic function, we leverage data of diverse dynamics and learn to adapt latent dynamic functions to few-shot support series. This is realized via Bayesian meta-learning underpinned by: 1) a latent dynamic function conditioned on knowledge derived from few-shot support series, and 2) a meta-model that learns to extract such dynamic-specific knowledge via feed-forward embedding of support set. We compared the presented framework with a comprehensive set of baseline models 1) trained globally on the large meta-training set with diverse dynamics, 2) trained individually on single dynamics with and without fine-tuning to k -shot support series, and 3) extended to few-shot meta-formulations. We demonstrated that the presented framework is agnostic to the latent dynamic function of choice and, at meta-test time, is able to forecast for new dynamics given variable-shot of support series. 1

For multi-modal magnetic resonance (MR) brain tumor image segmentation, current methods usually directly extract the discriminative features from input images for tumor sub-region category determination and localization. However, the impact of information aliasing caused by the mutual inclusion of tumor sub-regions is often ignored. Moreover, existing methods usually do not take tailored efforts to highlight the single tumor sub-region features. To this end, a multi-modal MR brain tumor segmentation method with tumor prototype-driven and multi-expert integration is proposed. It could highlight the features of each tumor sub-region under the guidance of tumor prototypes. Specifically, to obtain the prototypes with complete information, we propose a mutual transmission mechanism to transfer different modal features to each other to address the issues raised by insufficient information on single-modal features. Furthermore, we devise a prototype-driven feature representation and fusion method with the learned prototypes, which implants the prototypes into tumor features and generates corresponding activation maps. With the activation maps, the sub-region features consistent with the prototype category can be highlighted. A key information enhancement and fusion strategy with multi-expert integration is designed to further improve the segmentation performance. The strategy can integrate the features from different layers of the extra feature extraction network and the features highlighted by the prototypes. Experimental results on three competition brain tumor segmentation datasets prove the superiority of the proposed method.

Clinical adoption of personalized virtual heart simulations faces challenges in model personalization and expensive computation. While an ideal solution is an efficient neural surrogate that at the same time is personalized to an individual subject, the state-of-the-art is either concerned with personalizing an expensive simulation model, or learning an efficient yet generic surrogate. This paper presents a completely new concept to achieve personalized neural surrogates in a single coherent framework of meta-learning (metaPNS). Instead of learning a single neural surrogate, we pursue the process of learning a personalized neural surrogate using a small amount of context data from a subject, in a novel formulation of few-shot generative modeling underpinned by: 1) a set-conditioned neural surrogate for cardiac simulation that, conditioned on subject-specific context data, learns to generate query simulations not included in the context set, and 2) a meta-model of amortized variational inference that learns to condition the neural surrogate via simple feed-forward embedding of context data. As test time, metaPNS delivers a personalized neural surrogate by fast feed-forward embedding of a small and flexible number of data available from an individual, achieving – for the first time – personalization and surrogate construction for expensive simulations in one end-to-end learning framework. Synthetic and real-data experiments demonstrated that metaPNS was able to improve personalization and predictive accuracy in comparison to conventionally-optimized cardiac simulation models, at a fraction of computation.

Prior knowledge about the imaging physics provides a mechanistic forward operator that plays an important role in image reconstruction, although myriad sources of possible errors in the operator could negatively impact the reconstruction solutions. In this work, we propose to embed the traditional mechanistic forward operator inside a neural function, and focus on modeling and correcting its unknown errors in an interpretable manner. This is achieved by a conditional generative model that transforms a given mechanistic operator with unknown errors, arising from a latent space of self-organizing clusters of potential sources of error generation. Once learned, the generative model can be used in place of a fixed forward operator in any traditional optimization-based reconstruction process where, together with the inverse solution, the error in prior mechanistic forward operator can be minimized and the potential source of error uncovered. We apply the presented method to the reconstruction of heart electrical potential from body surface potential. In controlled simulation experiments and in-vivo real data experiments, we demonstrate that the presented method allowed reduction of errors in the physics-based forward operator and thereby delivered inverse reconstruction of heart-surface potential with increased accuracy.

Traditional approaches to image reconstruction uses physics-based loss with data-efficient inference, although the difficulty to properly model the inverse solution precludes learning the reconstruction across a distribution of data. Modern deep learning approaches enable expressive modeling but rely on a large number of reconstructed images (labeled data) that are often not available in practice. To combine the best of the above two lines of works, we present a novel label-free image reconstruction network that is supervised by physics-based forward operators rather than labeled data. We further present an expressive yet disentangled spatial-temporal modeling of the inverse solution, where its latent dynamics is modeled by neural ordinary differential equations and its emission over non-Euclidean geometrical domains by graph convolutional neural networks. We applied the presented method to reconstruct electrical activity on the heart surface from body-surface potential. In simulation and real-data experiments in comparison to both traditional physics-based and modern data-driven reconstruction methods, we demonstrated the ability of the presented method to learn how to reconstruct using observational data without any corresponding labels.

Computer-aided diagnosis via deep learning relies on large-scale annotated data sets, which can be costly when involving expert knowledge. Semi-supervised learning (SSL) mitigates this challenge by leveraging unlabeled data. One effective SSL approach is to regularize the local smoothness of neural functions via perturbations around single data points. In this work, we argue that regularizing the global smoothness of neural functions by filling the void in between data points can further improve SSL. We present a novel SSL approach that trains the neural network on linear mixing of labeled and unlabeled data, at both the input and latent space in order to regularize different portions of the network. We evaluated the presented model on two distinct medical image data sets for semi-supervised classification of thoracic disease and skin lesion, demonstrating its improved performance over SSL with local perturbations and SSL with global mixing but at the input space only. Our code is available at https://github.com/Prasanna1991/LatentMixing.

Learning rich representation from data is an important task for deep generative models such as variational auto-encoder (VAE). However, by extracting high-level abstractions in the bottom-up inference process, the goal of preserving all factors of variations for top-down generation is compromised. Motivated by the concept of "starting small", we present a strategy to progressively learn independent hierarchical representations from high- to low-levels of abstractions. The model starts with learning the most abstract representation, and then progressively grow the network architecture to introduce new representations at different levels of abstraction. We quantitatively demonstrate the ability of the presented model to improve disentanglement in comparison to existing works on two benchmark data sets using three disentanglement metrics, including a new metric we proposed to complement the previously-presented metric of mutual information gap. We further present both qualitative and quantitative evidence on how the progression of learning improves disentangling of hierarchical representations. By drawing on the respective advantage of hierarchical representation learning and progressive learning, this is to our knowledge the first attempt to improve disentanglement by progressively growing the capacity of VAE to learn hierarchical representations.

Deep neural networks have shown great potential in image reconstruction problems in Euclidean space. However, many reconstruction problems involve imaging physics that are dependent on the underlying non-Euclidean geometry. In this paper, we present a new approach to learn inverse imaging that exploit the underlying geometry and physics. We first introduce a non-Euclidean encoding-decoding network that allows us to describe the unknown and measurement variables over their respective geometrical domains. We then learn the geometry-dependent physics in between the two domains by explicitly modeling it via a bipartite graph over the graphical embedding of the two geometry. We applied the presented network to reconstructing electrical activity on the heart surface from body-surface potential. In a series of generalization tasks with increasing difficulty, we demonstrated the improved ability of the presented network to generalize across geometrical changes underlying the data in comparison to its Euclidean alternatives.

Background: Machine learning models may help localize the site of origin of ventricular tachycardia (VT) using 12-lead electrocardiograms. However, population-based models suffer from inter-subject anatomical variations within ECG data, while patient-specific models face the open challenge of what pacing data to collect for training. Methods: This study presents and validates the first hybrid model that combines population and patient-specific machine learning for rapid "computer-guided pace-mapping". A population-based deep learning model was first trained offline to disentangle inter-subject variations and regionalize the site of VT origin. Given a new patient with a target VT, an on-line patient-specific model after being initialized by the population-based prediction was then built in real time by actively suggesting where to pace next and improving the prediction with each added pacing data, progressively guiding pace-mapping towards the site of VT origin. Results: The population model was trained on pace-mapping data from 38 patients and the patient-specific model was subsequently tuned on one patient. The resulting hybrid model was tested on a separate cohort of eight patients in localizing 1) 193 LV endocardial pacing sites, and 2) nine VTs with clinically determined exit sites. The hybrid model achieved a localization error of 5.3 +/- 2.6 mm using 5.4 +/- 2.5 pacing sites in localizing LV pacing sites, achieving a significantly higher accuracy with a significantly smaller amount of training sites in comparison to models without active guidance. Conclusion: The presented hybrid model has the potential to assist rapid pace-mapping of interventional targets in VT.

The success of deep learning in medical imaging is mostly achieved at the cost of a large labeled data set. Semi-supervised learning (SSL) provides a promising solution by leveraging the structure of unlabeled data to improve learning from a small set of labeled data. Self-ensembling is a simple approach used in SSL to encourage consensus among ensemble predictions of unknown labels, improving generalization of the model by making it more insensitive to the latent space. Currently, such an ensemble is obtained by randomization such as dropout regularization and random data augmentation. In this work, we hypothesize – from the generalization perspective – that self-ensembling can be improved by exploiting the stochasticity of a disentangled latent space. To this end, we present a stacked SSL model that utilizes unsupervised disentangled representation learning as the stochastic embedding for self-ensembling. We evaluate the presented model for multi-label classification using chest X-ray images, demonstrating its improved performance over related SSL models as well as the interpretability of its disentangled representations.

Digital manufacturing technologies, especially three-dimensional (3-D) printing, are rapidly transforming the way we experience retail. Consumer products can now be deeply customized to an individual both in terms of fit, as well as style. However, consumers typically lack the design tools and expertise to express their design intent. In this article, we explore a human-centered approach for leveraging artificial intelligence (AI) to scaffold 3-D design tasks to make them accessible to more consumers. Our sunglasses kiosk prototype allows users to iteratively express their design intent at a high-level, using voice. The AI generates new 3-D designs of sunglasses that reflect the user's refined design intent. A user study examines the attitudes and behaviors toward this AI-powered kiosk approach.