The 2019 fastMRI challenge was an open challenge designed to advance research in the field of machine learning for MR image reconstruction. The goal for the participants was to reconstruct undersampled MRI k-space data. The original challenge left an open question as to how well the reconstruction methods will perform in the setting where there is a systematic difference between training and test data. In this work we tested the generalization performance of the submissions with respect to various perturbations, and despite differences in model architecture and training, all of the methods perform very similarly.

In current seismic acquisition practice, there is an increasing drive for sparsely (in space) acquired data, often in irregular geometry. These surveys can trade off subsurface information for efficiency/cost - creating a problem of "missing seismic data" that can greatly hinder subsequent processing and interpretation. Reconstruction of regularly sampled dense data from highly sparse, irregular data can therefore aid in processing and interpretation of these far sparser, more efficient seismic surveys. Here, two methods are compared to solve the reconstruction problem in both space-time and wavenumber-frequency domain. This requires an operator that maps sparse to dense data: the operator is generally unknown, being the inverse of a known data sampling operator. As such, here the deterministic inversion is efficiently solved by least squares optimisation using a numerically efficient Python-based linear operator representation. An alternative approach is probabilistic and uses deep learning. Here, two deep learning architectures are benchmarked against each other and the deterministic approach; a Recurrent Inference Machine (RIM), which is designed specifically to solve inverse problems given known forward operators, and the well-known U-Net. The trained deep learning networks are capable of successfully mapping sparse to dense seismic data for a range of different datasets and decimation percentages, thereby significantly reducing spatial aliasing in the wavenumber-frequency domain. The deterministic inversion on the contrary, could not reconstruct the missing data and thus did not reduce the undesired spatial aliasing. Results show that the application of Deep Learning for seismic reconstruction is promising, but the treatment of large-volume, multi-component seismic datasets will require dedicated learning architectures not yet realisable with existing tools.

Summary Currently, missing seismic data because of too sparse sampling or irregularities in acquisition greatly hinders accurate seismic processing and interpretation. Reconstruction of highly sparse, irregular data to dense data can therefore aid in processing and interpretation of far sparser, more efficient seismic surveys. Here, two methods to solve the reconstruction problem, that requires an inverse operator mapping sparse to dense data, are compared in both space-time and wavenumber-frequency domain. The deterministic inversion is efficiently solved by least squares optimisation using an numerically-efficient Python-based linear operator representation. An alternative method is the probabilistic approach that uses deep learning. Here, the specific deep learning architecture is a Recurrent Inference Machine (RIM), which is designed specifically to solve inverse problems given known forward operators. Our trained RIM proved to be able to map sparse to dense seismic data by reducing spatial aliasing in the wavenumber-frequency domain caused by sub-Nyquist spatial sampling in space-time domain. The RIM is therefore an improvement over deterministic inversion that could not undo spatial aliasing, leaving the benchmarking of RIMs to other deep learning architectures as a subject of ongoing study.

Can we image and monitor the internal ocean structure at global scales? Can we monitor in vast expanses of the Earth’s cryosphere subsurface with meter-length resolution? Can we characterize the interior structures of asteroids and comets out in space efficiently and with high confidence? At the core of these questions lies the understanding and development of wave-based imaging systems, based on seismic or radar, that rely on highly-sparse, high-quality data, but whose output image quality is comparable to that of densely sampled, wide aperture array-based data. Traditionally, exploration seismology has long relied on wide aperture, dense data sets together with high-end imaging such as reverse-time migration and full-waveform inversion to produce high resolution subsurface models. Given the recent rise of drone-like, autonomous systems, in this talk, we present approaches that can take highly-sparse data as would be recorded by autonomous platforms, into accurate high-resolution images as if they had been acquired by densely-sampled, wide aperture source and receiver arrays. We demonstrate two approaches that could achieve this goal. The first is the use of sparse multicomponent sources and receivers capable of exciting/recording fields and their spatial gradients, together with a gradient-based wavefield reconstruction approach and subsequent imaging. The second approach relies on a new deep learning architecture, the so-called Recurrent Inference Machine, designed specifically for inverse problems – showing that it can surpass the capabilities of deterministic approaches to data reconstruction and imaging. We illustrate these approaches using a numerical model for oceanic turbulence, where we show the compressive sensing potential of these acquisition, reconstruction and imaging methods for acoustic imaging of the ocean's internal structure – overcoming current limitations in data acquisition and processing for seismic oceanography. Finally, we postulate that these approaches, though still in their early days, will pave the way in enabling breakthrough imaging systems at the frontiers of geo-imaging, e.g., for oceanography at global scales, in imaging the Earth’s cryosphere or for planetary exploration.

Iterative learning to infer approaches have become popular solvers for inverse problems. However, their memory requirements during training grow linearly with model depth, limiting in practice model expressiveness. In this work, we propose an iterative inverse model with constant memory that relies on invertible networks to avoid storing intermediate activations. As a result, the proposed approach allows us to train models with 400 layers on 3D volumes in an MRI image reconstruction task. In experiments on a public data set, we demonstrate that these deeper, and thus more expressive, networks perform state-of-the-art image reconstruction.

Deep learning allows for accelerated magnetic resonance image (MRI) reconstruction, thereby shortening measurement times. Rather than using sparsifying transforms, a prerequisite in Compressed Sensing (CS), suitable MRI prior distributions are learned from data. In clinical practice, both the underlying anatomy as well as image acquisition settings vary. For this reason, deep neural networks must be able to reapply what they learn across different measurement conditions. We propose to use Recurrent Inference Machines (RIM) as a framework for accelerated MRI reconstruction. RIMs solve inverse problems in an iterative and recurrent inference procedure by repeatedly reassessing the state of their reconstruction, and subsequently making incremental adjustments to it in accordance with the forward model of accelerated MRI. RIMs learn the inferential process of reconstructing a given signal, which, in combination with the use of internal states as part of their recurrent architecture, makes them less dependent on learning the features pertaining to the source of the signal itself. This gives RIMs a low tendency to overfit, and a high capacity to generalize to unseen types of data. We demonstrate this ability with respect to anatomy by reconstructing brain and knee scans, as well as other MRI acquisition settings, by reconstructing scans of different contrast and resolution, at different field strength, subjected to varying acceleration levels. We show that RIMs outperform CS not only with respect to quality metrics, but also according to a rating given by an experienced neuroradiologist in a double blinded experiment. Finally, we show with qualitative results that our model can be applied to prospectively under-sampled raw data, as acquired by pre-installed acquisition protocols.

We present a machine-learning method for the reconstruction of the undistorted images of background sources in strongly lensed systems. This method treats the source as a pixelated image and utilizes the recurrent inference machine to iteratively reconstruct the background source given a lens model. Our architecture learns to maximize the likelihood of the model parameters (source pixels) given the data using the physical forward model (ray-tracing simulations) while implicitly learning the prior of the source structure from the training data. This results in better performance compared to linear inversion methods, where the prior information is limited to the two-point covariance of the source pixels approximated with a Gaussian form, and often specified in a relatively arbitrary manner. We combine our source reconstruction network with a convolutional neural network that predicts the parameters of the mass distribution in the lensing galaxies directly from telescope images, allowing a fully automated reconstruction of the background source images and the foreground mass distribution.

We, team AImsterdam, summarize our submission to the fastMRI challenge (Zbontar et al., 2018). Our approach builds on recent advances in invertible learning to infer models as presented in Putzky and Welling (2019). Both, our single-coil and our multi-coil model share the same basic architecture.

We use convolutional neural networks (CNNs) and recurrent neural networks (RNNs) to estimate the parameters of strong gravitational lenses from interferometric observations. We explore multiple strategies and find that the best results are obtained when the effects of the dirty beam are first removed from the images with a deconvolution performed with an RNN-based structure before estimating the parameters. For this purpose, we use the recurrent inference machine (RIM) introduced in Putzky & Welling (2017). This provides a fast and automated alternative to the traditional CLEAN algorithm. We obtain the uncertainties of the estimated parameters using variational inference with Bernoulli distributions. We test the performance of the networks with a simulated test dataset as well as with five ALMA observations of strong lenses. For the observed ALMA data we compare our estimates with values obtained from a maximum-likelihood lens modeling method which operates in the visibility space and find consistent results. We show that we can estimate the lensing parameters with high accuracy using a combination of an RNN structure performing image deconvolution and a CNN performing lensing analysis, with uncertainties less than a factor of two higher than those achieved with maximum-likelihood methods. Including the deconvolution procedure performed by RIM, a single evaluation can be done in about a second on a single GPU, providing a more than six orders of magnitude increase in analysis speed while using about eight orders of magnitude less computational resources compared to maximum-likelihood lens modeling in the uv-plane. We conclude that this is a promising method for the analysis of mm and cm interferometric data from current facilities (e.g., ALMA, JVLA) and future large interferometric observatories (e.g., SKA), where an analysis in the uv-plane could be difficult or unfeasible.

Accelerated MRI reconstruction is important for making MRI faster and thus applicable in a broader range of problem domains. Computational tools allow for high-resolution imaging without the need to perform time-consuming measurements. Most recently, deep learning approaches have been applied to this problem. However, none of these methods have been shown to transfer well across different measurement settings. We propose to use Recurrent Inference Machines as a framework for accelerated MRI, which allows us to leverage the power of deep learning without explicit domain knowledge. We show in experiments that the model can generalize well across different setups, while at the same time it outperforms another deep learning method and a compressed sensing approach.

Inverse problems are typically solved by first defining a model and then choosing an inference procedure. With this separation of modeling from inference, inverse problems can be framed in a modular way. For example, variational inference can be applied to a broad class of models. The modularity, however, typically goes away after model parameters have been trained under a chosen inference procedure. During training, model and inference often interact in a way that the model parameters will ultimately be adapted to the chosen inference procedure, posing the two components inseparable after training. But if model and inference become inseperable after training, why separate them in the first place? We propose a novel learning framework which abandons the dichotomy between model and inference. Instead, we introduce Recurrent Inference Machines (RIM), a class of recurrent neural networks (RNN) that directly learn to solve inverse problems. We demonstrate the effectiveness of RIMs in experiments on various image reconstruction tasks. We show empirically that RIMs exhibit the desirable convergence behavior of classical inference procedures, and that they can outperform state-of- the-art methods when trained on specialized inference tasks. Our approach bridges the gap between inverse problems and deep learning, providing a framework for fast progression in the field of inverse problems.

Much of the recent research on solving iterative inference problems focuses on moving away from hand-chosen inference algorithms and towards learned inference. In the latter, the inference process is unrolled in time and interpreted as a recurrent neural network (RNN) which allows for joint learning of model and inference parameters with back-propagation through time. In this framework, the RNN architecture is directly derived from a hand-chosen inference algorithm, effectively limiting its capabilities. We propose a learning framework, called Recurrent Inference Machines (RIM), in which we turn algorithm construction the other way round: Given data and a task, train an RNN to learn an inference algorithm. Because RNNs are Turing complete [1, 2] they are capable to implement any inference algorithm. The framework allows for an abstraction which removes the need for domain knowledge. We demonstrate in several image restoration experiments that this abstraction is effective, allowing us to achieve state-of-the-art performance on image denoising and super-resolution tasks and superior across-task generalization.

Neural population activity in cortical circuits is not solely driven by external inputs, but is also modulated by endogenous states which vary on multiple time-scales. To understand information processing in cortical circuits, we need to understand the statistical structure of internal states and their interaction with sensory inputs. Here, we present a statistical model for extracting hierarchically organised neural population states from multi-channel recordings of neural spiking activity. Population states are modelled using a hidden Markov decision tree with state-dependent tuning parameters and a generalised linear observation model. We present a variational Bayesian inference algorithm for estimating the posterior distribution over parameters from neural population recordings. On simulated data, we show that we can identify the underlying sequence of population states and reconstruct the ground truth parameters. Using population recordings from visual cortex, we find that a model with two levels of population states outperforms both a one-state and a two-state generalised linear model. Finally, we find that modelling of state-dependence also improves the accuracy with which sensory stimuli can be decoded from the population response.