Proximal operators are fundamental across many applications in signal processing and machine learning, including solving ill-posed inverse problems. Recent work has introduced Learned Proximal Networks (LPNs), providing parametric functions that compute exact proximals for data-driven and potentially non-convex regularizers. However, in many settings it is important to include additional structure to these regularizers–and their corresponding proximals–such as shift and scale equivariance. In this work, we show how to obtain learned functions parametrized by neural networks that provably compute exact proximal operators while being equivariant to shifts and scaling, which we dub Affine-Equivariant Learned Proximal Networks (AE-LPNs). We demonstrate our results on synthetic, constructive examples, and then on real data via denoising in out-of-distribution settings. Our equivariant learned proximals enhance robustness to noise distributions and affine shifts far beyond training distributions, improving the practical utility of learned proximal operators

Diffusion models have emerged as a dominant paradigm for generative modeling across a wide range of domains, including prompt-conditional generation. The vast majority of samplers, however, rely on forward discretization of the reverse diffusion process and use score functions that are learned from data. Such forward and explicit discretizations can be slow and unstable, requiring a large number of sampling steps to produce good-quality samples. In this work we develop a text-to-image (T2I) diffusion model based on backward discretizations, dubbed ProxT2I, relying on learned and conditional proximal operators instead of score functions. We further leverage recent advances in reinforcement learning and policy optimization to optimize our samplers for task-specific rewards. Additionally, we develop a new large-scale and open-source dataset comprising 15 million high-quality human images with fine-grained captions, called LAION-Face-T2I-15M, for training and evaluation. Our approach consistently enhances sampling efficiency and human-preference alignment compared to score-based baselines, and achieves results on par with existing state-of-the-art and open-source text-to-image models while requiring lower compute and smaller model size, offering a lightweight yet performant solution for human text-to-image generation.

In recent years, a variety of learned regularization frameworks for solving inverse problems in imaging have emerged. These offer flexible modeling together with mathematical insights. The proposed methods differ in their architectural design and training strategies, making direct comparison challenging due to non-modular implementations. We address this gap by collecting and unifying the available code into a common framework. This unified view allows us to systematically compare the approaches and highlight their strengths and limitations, providing valuable insights into their future potential. We also provide concise descriptions of each method, complemented by practical guidelines.

Diffusion models have quickly become some of the most popular and powerful generative models for high-dimensional data. The key insight that enabled their development was the realization that access to the score---the gradient of the log-density at different noise levels---allows for sampling from data distributions by solving a reverse-time stochastic differential equation (SDE) via forward discretization, and that popular denoisers allow for unbiased estimators of this score. In this paper, we demonstrate that an alternative, backward discretization of these SDEs, using proximal maps in place of the score, leads to theoretical and practical benefits. We leverage recent results in _proximal matching_ to learn proximal operators of the log-density and, with them, develop Proximal Diffusion Models (`ProxDM`). Theoretically, we prove that $\widetilde{\mathcal O}(d/\sqrt{\varepsilon})$ steps suffice for the resulting discretization to generate an $\varepsilon$-accurate distribution w.r.t. the KL divergence. Empirically, we show that two variants of `ProxDM` achieve significantly faster convergence within just a few sampling steps compared to conventional score-matching methods.

The application of STI in human in vivo has been practically infeasible because of its time-consuming acquisition scheme. We propose a novel image reconstruction algorithm for STI that leverages data-driven priors to tackle this issue. Our method, called DeepSTI, learns the data prior implicitly via a deep neural network that resembles the proximal operator of a regularizer function. The dipole inversion problem is then solved iteratively using the learned proximal network. Experimental results demonstrate superior performance of DeepSTI over state-of-the-art methods. DeepSTI is the first reconstruction method to achieve high quality results for human STI with fewer than six orientations.

Recently, image deblurring task driven by the encoder-decoder network has made a tremendous amount of progress. However, these encoder-decoder-based networks still have two disadvantages: (1) due to the lack of feedback mechanism in the decoder design, the reconstruction results of existing networks are still sub-optimal; (2) these networks introduce multiple modules, such as the self-attention mechanism, to improve the performance, which also increases the computational burden. To overcome these issues, this paper proposes a novel feedback-mechanism-based encoder-decoder network (namely, FMNet) that is equipped with two key components: (1) the feedback-mechanism-based decoder and (2) the dual gated attention module. To improve reconstruction quality, the feedback-mechanism-based decoder is proposed to leverage the feedback information via the feedback attention module, which adaptively selects useful features in the feedback path. To decrease the computational cost, an efficient dual gated attention module is proposed to perform the attention mechanism in the frequency domain twice, which improves deblurring performance while reducing the computational cost by avoiding redundant convolutions and feature channels. The superiority of FMNet in terms of both deblurring performance and computational efficiency is demonstrated via comparisons with state-of-the-art methods on multiple public datasets.

Proximal operators are ubiquitous in inverse problems, commonly appearing as part of algorithmic strategies to regularize problems that are otherwise ill-posed. Modern deep learning models have been brought to bear for these tasks too, as in the framework of plug-and-play or deep unrolling, where they loosely resemble proximal operators. Yet, something essential is lost in employing these purely data-driven approaches: there is no guarantee that a general deep network represents the proximal operator of any function, nor is there any characterization of the function for which the network might provide some approximate proximal. This not only makes guaranteeing convergence of iterative schemes challenging but, more fundamentally, complicates the analysis of what has been learned by these networks about their training data. Herein we provide a framework to develop learned proximal networks (LPN), prove that they provide exact proximal operators for a data-driven nonconvex regularizer, and show how a new training strategy, dubbed proximal matching, provably promotes the recovery of the log-prior of the true data distribution. Such LPN provide general, unsupervised, expressive proximal operators that can be used for general inverse problems with convergence guarantees. We illustrate our results in a series of cases of increasing complexity, demonstrating that these models not only result in state-of-the-art performance, but provide a window into the resulting priors learned from data.

Quantitative Susceptibility Mapping (QSM) is a technique that derives tissue magnetic susceptibility distributions from phase measurements obtained through Magnetic Resonance (MR) imaging. This involves solving an ill-posed dipole inversion problem, however, and thus time-consuming and cumbersome data acquisition from several distinct head orientations becomes necessary to obtain an accurate solution. Most recent (supervised) deep learning methods for single-phase QSM require training data obtained via multiple orientations. In this work, we present an alternative unsupervised learning approach that can efficiently train on single-orientation measurement data alone, named ProxiMO (Proximal Multi-Operator), combining Learned Proximal Convolutional Neural Networks (LP-CNN) with multi-operator imaging (MOI). This integration enables LP-CNN training for QSM on single-phase data without ground truth reconstructions. We further introduce a semi-supervised variant, which further boosts the reconstruction performance, compared to the traditional supervised fashions. Extensive experiments on multicenter datasets illustrate the advantage of unsupervised training and the superiority of the proposed approach for QSM reconstruction. Code is available at https://github.com/shmuelor/ProxiMO.

Recent studies on deepfake detection have achieved promising results when training and testing faces are from the same dataset. However, their results severely degrade when confronted with forged samples that the model has not yet seen during training. In this paper, deepfake data to help detect deepfakes. this paper present we put a new insight into diffusion model-based data augmentation, and propose a Masked Conditional Diffusion Model (MCDM) for enhancing deepfake detection. It generates a variety of forged faces from a masked pristine one, encouraging the deepfake detection model to learn generic and robust representations without overfitting to special artifacts. Extensive experiments demonstrate that forgery images generated with our method are of high quality and helpful to improve the performance of deepfake detection models.

The separation of signal contributions from paramagnetic and diamagnetic susceptibility sources in MRI has important implications for understanding the biological functions and health conditions of the brain. However, general and flexible deep-learning-based tools that can provide this information in humans in vivo are limited. For instance, the state-of-the-art deep-learning-based source separation method in quantitative susceptibility mapping (QSM) demands high-quality paramagnetic and diamagnetic maps for training and only allows phase measurement from a single head orientation as input. Furthermore, no method currently exists to separate these contributions when considering the susceptibility anisotropy as in the more challenging framework of susceptibility tensor imaging (STI). In this paper, we present a unified and flexible algorithm for source separation for both QSM and STI, dubbed WaveSep. Our method allows for an arbitrary number of input measurements at any head orientations for better estimation accuracy given multiple input measurements, does not require ground-truth paramagnetic and diamagnetic data for training, and is able to estimate the anisotropic second-order susceptibility tensors without requiring significant additional measurements. Our method first solves the dipole inversion problem by using state-of-the-art, off-the-shelf data-driven models based on learned proximal operators, and then separates the paramagnetic and diamagnetic sources using a Wavelet-based separation approach, without the need for retraining. Experimental results on both simulation and in-vivo human brain data demonstrate the superior performance of WaveSep for susceptibility source separation in QSM, and unprecedented separation results in STI. Code is available at https://github.com/ZhenghanFang/WaveSep.

Retinal vessel segmentation is an essential part of ocular disease diagnosis. However, due to complex vascular structure, large‐scale variations of retinal vessels, as well as inefficiency of vessel segmentation speed, accurate and fast automatic vessel segmentation for retinal images is still technically challenging. To tackle these issues, we present a lightweight multi‐scale‐aware network (LMSA‐Net) for retinal vessel segmentation. The network leverages the encoder‐decoder structure that was used in U‐Net. In the encoder, we propose a ghosted sandglass residual (GSR) block, aiming at greatly reducing the parameters and computational cost while obtaining richer semantic information. After that, a multi‐scale feature‐aware aggregation (MFA) module is designed to perceive multi‐scale semantic information for effective information extraction. Then, a global adaptive upsampling (GAU) module is proposed to guide the effective fusion of high‐ and low‐level semantic information in the decoder. Experiments are conducted on three public datasets, including DRIVE, CHASE_DB1, and STARE. The experimental results indicate the effectiveness of the LMSA‐Net, which can achieve better segmentation performance than other state‐of‐the‐art methods.

Susceptibility tensor imaging (STI) is an emerging magnetic resonance imaging technique that characterizes the anisotropic tissue magnetic susceptibility with a second-order tensor model. STI has the potential to provide information for both the reconstruction of white matter fiber pathways and detection of myelin changes in the brain at mm resolution or less, which would be of great value for understanding brain structure and function in healthy and diseased brain. However, the application of STI in vivo has been hindered by its cumbersome and time-consuming acquisition requirement of measuring susceptibility induced MR phase changes at multiple head orientations. Usually, sampling at more than six orientations is required to obtain sufficient information for the ill-posed STI dipole inversion. This complexity is enhanced by the limitation in head rotation angles due to physical constraints of the head coil. As a result, STI has not yet been widely applied in human studies in vivo. In this work, we tackle these issues by proposing an image reconstruction algorithm for STI that leverages data-driven priors. Our method, called DeepSTI, learns the data prior implicitly via a deep neural network that approximates the proximal operator of a regularizer function for STI. The dipole inversion problem is then solved iteratively using the learned proximal network. Experimental results using both simulation and in vivo human data demonstrate great improvement over state-of-the-art algorithms in terms of the reconstructed tensor image, principal eigenvector maps and tractography results, while allowing for tensor reconstruction with MR phase measured at much less than six different orientations. Notably, promising reconstruction results are achieved by our method from only one orientation in human in vivo, and we demonstrate a potential application of this technique for estimating lesion susceptibility anisotropy in patients with multiple sclerosis.

PURPOSE Preclinical MR fingerprinting (MRF) suffers from long acquisition time for organ-level coverage due to demanding image resolution and limited undersampling capacity. This study aims to develop a deep learning-assisted fast MRF framework for sub-millimeter T1 and T2 mapping of entire macaque brain on a preclinical 9.4 T MR system. METHODS Three dimensional MRF images were reconstructed by singular value decomposition (SVD) compressed reconstruction. T1 and T2 mapping for each axial slice exploited a self-attention assisted residual U-Net to suppress aliasing-induced quantification errors, and the transmit-field (B1 + ) measurements for robustness against B1 + inhomogeneity. Supervised network training used MRF images simulated via virtual parametric maps and a desired undersampling scheme. This strategy bypassed the difficulties of acquiring fully sampled preclinical MRF data to guide network training. The proposed fast MRF framework was tested on experimental data acquired from ex vivo and in vivo macaque brains. RESULTS The trained network showed reasonable adaptability to experimental MRF images, enabling robust delineation of various T1 and T2 distributions in the brain tissues. Further, the proposed MRF framework outperformed several existing fast MRF methods in handling the aliasing artifacts and capturing detailed cerebral structures in the mapping results. Parametric mapping of entire macaque brain at nominal resolution of 0.35 × $$ \times $$ 0.35 × $$ \times $$ 1 mm3 can be realized via a 20-min 3D MRF scan, which was sixfold faster than the baseline protocol. CONCLUSION Introducing deep learning to MRF framework paves the way for efficient organ-level high-resolution quantitative MRI in preclinical applications.

As the coronavirusdisease 2019 (COVID-19) pandemic continues, fast and automatic COVID-19-related pneumonia lesion segmentation method is in an urgent need. The current state-of-the-art methods for segmentation generally require sufficient amounts of annotated data for training. However, human expert annotation of such lesion on chest computed tomography (CT) scans is time-consuming and labor-intensive due to its heterogeneous appearance, ambiguous boundary, and large number of slices in 3-D CT images. Therefore, the purpose of this study is to present a novel annotation-efficient learning method for COVID-19 pneumonia lesion segmentation on CT. To make the best use of limited human expert annotation resources, we propose an error-aware unified semisupervised and active learning method. A novel error estimation network is proposed to estimate a voxelwise segmentation loss map, which is used to guide learning from unlabeled data for semisupervised learning and choose the most informative images to annotate next for active learning. Validation is carried out on segmenting pneumonia lesions in 110 chest CT scans. The experimental result demonstrates that the proposed method significantly boosts the segmentation accuracy given limited amount of human annotation, compared with a conventional fully supervised baseline (60.9% Dice to 72.0% at 30% labeled data). The performance is also competitive compared with other state-of-the-art annotation-efficient segmentation methods. The proposed method can significantly reduce the annotation effort needed to achieve accurate COVID-19 pneumonia lesion segmentation.

In human MRI studies, magnetic resonance fingerprinting (MRF) allows simultaneous T1 and T2 mapping in 10 s using 48-fold undersampled data. However, when "reverse translated" to preclinical research involving small laboratory animals, the undersampling capacity of the MRF method decreases to 8 fold because of the low SNR associated with high spatial resolution. In this study, we aim to develop a deep-learning based method to reliably quantify T1 and T2 in the mouse brain from highly undersampled MRF data, and to demonstrate its efficacy in tracking T1 and T2 variations induced by MR tracers. The proposed method employs U-Net as the backbone for spatially constrained T1 and T2 mapping. Several strategies to improve the robustness of mapping results are evaluated, including feature extraction with sliding window averaging, implementing physics-guided training objectives, and implementing data-consistency constraint to iteratively refine the inferred maps by a cascade of U-Nets. The quantification network is trained using mouse-brain MRF datasets acquired before and after Manganese (Mn2+) enhancement. Experimental results show that robust T1 and T2 mapping can be achieved from MRF data acquired in 30 s (4-fold further acceleration), by using a simple combination of sliding window averaging for feature extraction and U-Net for parametric quantification. Meanwhile, the T1 variations induced by Mn2+ in mouse brain are faithfully detected. Code is available at https://github.com/guyn-idealab/Mouse-MRF-DL/ .

Accurate segmentation of white matter, gray matter and cerebrospinal fluid from neonatal brain MR images is of great importance in characterizing early brain development. Deep-learning-based methods have been successfully applied to neonatal brain MRIs with superior performance if testing subjects were acquired with the same imaging protocols/scanners as training subjects. However, for the testing subjects acquired with different imaging protocols/scanners, they cannot achieve accurate segmentation results due to large appearance/pattern differences between the testing and training subjects. Besides, imaging artifacts, like head motion, which are inevitable during the imaging acquisition process, also pose a challenge for the segmentation methods. To address these issues, in this paper, we propose a harmonized neonatal brain MR image segmentation model that harmonizes testing images acquired by different protocols/scanners into the domain of training images through a cycle-consistent generative adversarial network (CycleGAN). Meanwhile, the artifacts can be largely alleviated during the harmonization. Then, a densely-connected U-Net based segmentation model trained in the domain of training images can be applied robustly for segmenting the harmonized testing images. Comparisons with existing methods illustrate the better performance of the proposed method on neonatal brain MR images from cross-sites, a grand segmentation challenge, as well as images with artifacts.

Background Coronavirus disease 2019 (COVID-19) has widely spread all over the world since the beginning of 2020. It is desirable to develop automatic and accurate detection of COVID-19 using chest CT. Purpose To develop a fully automatic framework to detect COVID-19 using chest CT and evaluate its performance. Materials and Methods In this retrospective and multicenter study, a deep learning model, the COVID-19 detection neural network (COVNet), was developed to extract visual features from volumetric chest CT scans for the detection of COVID-19. CT scans of community-acquired pneumonia (CAP) and other non-pneumonia abnormalities were included to test the robustness of the model. The datasets were collected from six hospitals between August 2016 and February 2020. Diagnostic performance was assessed with the area under the receiver operating characteristic curve, sensitivity, and specificity. Results The collected dataset consisted of 4352 chest CT scans from 3322 patients. The average patient age (±standard deviation) was 49 years ± 15, and there were slightly more men than women (1838 vs 1484, respectively; P = .29). The per-scan sensitivity and specificity for detecting COVID-19 in the independent test set was 90% (95% confidence interval [CI]: 83%, 94%; 114 of 127 scans) and 96% (95% CI: 93%, 98%; 294 of 307 scans), respectively, with an area under the receiver operating characteristic curve of 0.96 (P < .001). The per-scan sensitivity and specificity for detecting CAP in the independent test set was 87% (152 of 175 scans) and 92% (239 of 259 scans), respectively, with an area under the receiver operating characteristic curve of 0.95 (95% CI: 0.93, 0.97). Conclusion A deep learning model can accurately detect coronavirus 2019 and differentiate it from community-acquired pneumonia and other lung conditions. © RSNA, 2020 Online supplemental material is available for this article.

Brain extraction is a fundamental prerequisite step in neuroimage analysis for fetus. Due to surrounding maternal tissues and unpredictable movement, brain extraction from fetal Magnetic Resonance (MR) images is a challenging task. In this paper, we propose a novel deep learning-based multi-step framework for brain extraction from 3D fetal MR images. In the first step, a global localization network is applied to estimate probability maps for brain candidates. Connected-component labeling algorithm is applied to eliminate small erroneous components and accurately locate the candidate brain area. In the second step, a local refinement network is implemented in the brain candidate area to obtain fine-grained probability maps. Final extraction results are derived by a fusion network with the two cascaded probability maps obtained from previous two steps. Experimental results demonstrate that our proposed method has superior performance compared with existing deep learning-based methods.

Purpose To develop a rapid 2D MR fingerprinting technique with a submillimeter in-plane resolution using a deep learning-based tissue quantification approach. Methods A rapid and high-resolution MR fingerprinting technique was developed for brain T-1 and T-2 quantification. The 2D acquisition was performed using a FISP-based MR fingerprinting sequence and a spiral trajectory with 0.8-mm in-plane resolution. A deep learning-based method was used to replace the standard template matching method for improved tissue characterization. A novel network architecture (i.e., residual channel attention U-Net) was proposed to improve high-resolution details in the estimated tissue maps. Quantitative brain imaging was performed with 5 adults and 2 pediatric subjects, and the performance of the proposed approach was compared with several existing methods in the literature. Results In vivo measurements with both adult and pediatric subjects show that high-quality T-1 and T-2 mapping with 0.8-mm in-plane resolution can be achieved in 7.5 seconds per slice. The proposed deep learning method outperformed existing algorithms in tissue quantification with improved accuracy. Compared with the standard U-Net, high-resolution details in brain tissues were better preserved by the proposed residual channel attention U-Net. Experiments on pediatric subjects further demonstrated the potential of the proposed technique for fast pediatric neuroimaging. Alongside reduced data acquisition time, a 5-fold acceleration in tissue property mapping was also achieved with the proposed method. Conclusion A rapid and high-resolution MR fingerprinting technique was developed, which enables high-quality T-1 and T-2 quantification with 0.8-mm in-plane resolution in 7.5 seconds per slice.

In the originally published version of this chapter, the Acknowledgements section was missing. This has been corrected and an Acknowledgements section has been added.