BACKGROUND:Cardiac real-time MRI (RT-MRI) in combination with MR-compatible spirometry (MRcS) offers unique opportunities to study heart-lung interactions. In contrast to other techniques that monitor respiration during MRI, MRcS provides quantitative respiratory data. Though MRcS is well tolerated, shortening of the scanning time with MRcS would be desirable, especially in young and sick patients. PURPOSE:The aim of the study was to predict airflow and lung volume based on RT-MR images after a short learning phase of combined RT-MRI and MRcS to provide respiratory data for subsequent short axis stack-based volumetries. METHODS:Cardiac RT-MRI (1.5 T; short axis; 30 frames/s) was acquired during free breathing in combination with MRcS in adult healthy subjects (n = 10). MR images with MRcS were recorded during a learning phase to collect training data. The iterative Lucas-Kanade method was applied to estimate optical flow from the captured MR images. A ridge regression model was fitted to predict airflow and thus also the lung volume from the estimated optical flow. Hyperparameters were estimated using leave-one-out cross validation and the performance was assessed on a held-out test dataset. Different durations and compositions of the learning phase were investigated to develop the most efficient measurement protocol. Coefficient of determination (R2), relative mean squared error (rMSE), Bland-Altman analysis on absolute tidal volume difference (aTVD), and absolute maximal airflow difference (aMFD) were used to validate the predictions on held-out test data. RESULTS:MRI combined with MRcS can train a machine learning algorithm to provide excellent predictive quantitative respiratory volume and flow for the remaining study. The optimal trade-off between predictive power and time necessary for training was reached with a shortened cardiac volumetry protocol covering only about two breaths per slice and every second slice (airflow: mean R2: 0.984, mean rMSE: 0.015, Bias aMFD: -0.01 L/s with +0.084/-0.1 95% CI and volume: mean R2: 0.990, mean rMSE: 0.003, Bias aTVD: 4.27 mL with +33/-24 95% CI) at a total duration of 100 s. Shorter protocols or application of the algorithm to subsequent studies in the same subject or even in different subjects still provided useful qualitative data. CONCLUSION:Machine-learning-based prediction of respiratory flow and lung volume from cardiac RT-MR images after a short training phase with MRcS is feasible and can help to shorten the time with MRcS while providing accurate respiratory data during RT-MRI.

Learning meaningful representations is at the heart of many tasks in the field of modern machine learning. Recently, a lot of methods were introduced that allow learning of image representations without supervision. These representations can then be used in downstream tasks like classification or object detection. The quality of these representations is close to supervised learning, while no labeled images are needed. This survey paper provides a comprehensive review of these methods in a unified notation, points out similarities and differences of these methods, and proposes a taxonomy which sets these methods in relation to each other. Furthermore, our survey summarizes the most recent experimental results reported in the literature in form of a meta-study. Our survey is intended as a starting point for researchers and practitioners who want to dive into the field of representation learning.

Phase retrieval is the problem of reconstructing images from magnitude-only measurements. In many real-world applications the problem is underdetermined. When training data is available, generative models allow optimization in a lower-dimensional latent space, hereby constraining the solution set to those images that can be synthesized by the generative model. However, not all possible solutions are within the range of the generator. Instead, they are represented with some error. To reduce this representation error in the context of phase retrieval, we first leverage a novel variation of intermediate layer optimization (ILO) to extend the range of the generator while still producing images consistent with the training data. Second, we introduce new initialization schemes that further improve the quality of the reconstruction. With extensive experiments on the Fourier phase retrieval problem and thorough ablation studies, we can show the benefits of our modified ILO and the new initialization schemes. Additionally, we analyze the performance of our approach on the Gaussian phase retrieval problem.

In this paper, we present our approach for the Helsinki Deblur Challenge (HDC2021). The task of this challenge is to deblur images of characters without knowing the point spread function (PSF). The organizers provided a dataset of pairs of sharp and blurred images. Our method consists of three steps: First, we estimate a warping transformation of the images to align the sharp images with the blurred ones. Next, we estimate the PSF using a quasi-Newton method. The estimated PSF allows to generate additional pairs of sharp and blurred images. Finally, we train a deep convolutional neural network to reconstruct the sharp images from the blurred images. Our method is able to successfully reconstruct images from the first 10 stages of the HDC 2021 data. Our code is available at https://github.com/hhu-machine-learning/hdc2021-psfnn.

Computed tomography (CT) has become an essential part of modern science and medicine. A CT scanner consists of an X-ray source that is spun around an object of interest. On the opposite end of the X-ray source, a detector captures X-rays that are not absorbed by the object. The reconstruction of an image is a linear inverse problem, which is usually solved by filtered back projection. However, when the number of measurements is small, the reconstruction problem is ill-posed. This is for example the case when the X-ray source is not spun completely around the object, but rather irradiates the object only from a limited angle. To tackle this problem, we present a deep neural network that is trained on a large amount of carefully-crafted synthetic data and can perform limited-angle tomography reconstruction even for only 30° or 40° sinograms. With our approach we won the first place in the Helsinki Tomography Challenge 2022.

Model-based reinforcement learning algorithms try to learn an agent by training a model that simulates the environment. However, the size of such models tends to be quite large which could be a burden as well. In this paper, we address the question, how we could design a model with fewer parameters than previous model-based approaches while achieving the same performance in the 100 K-interactions regime. For this purpose, we create a world model that combines a vector quantized-variational autoencoder to encode observations and a convolutional long short-term memory to model the dynamics. This is connected to a model-free proximal policy optimization agent to train purely on simulated experience from this world model. Detailed experiments on the Atari environments show that it is possible to reach comparable performance to the SimPLe method with a significantly smaller world model. A series of ablation studies justify our design choices and give additional insights.

Deep neural networks have been successful in many reinforcement learning settings. However, compared to human learners they are overly data hungry. To build a sample-efficient world model, we apply a transformer to real-world episodes in an autoregressive manner: not only the compact latent states and the taken actions but also the experienced or predicted rewards are fed into the transformer, so that it can attend flexibly to all three modalities at different time steps. The transformer allows our world model to access previous states directly, instead of viewing them through a compressed recurrent state. By utilizing the Transformer-XL architecture, it is able to learn long-term dependencies while staying computationally efficient. Our transformer-based world model (TWM) generates meaningful, new experience, which is used to train a policy that outperforms previous model-free and model-based reinforcement learning algorithms on the Atari 100k benchmark.

Learning meaningful representations is at the heart of many tasks in the field of modern machine learning. Recently, a lot of methods were introduced that allow learning of image representations without supervision. These representations can then be used in downstream tasks like classification or object detection. The quality of these representations is close to supervised learning, while no labeled images are needed. This survey paper provides a comprehensive review of these methods in a unified notation, points out similarities and differences of these methods, and proposes a taxonomy which sets these methods in relation to each other. Furthermore, our survey summarizes the most-recent experimental results reported in the literature in form of a meta-study. Our survey is intended as a starting point for researchers and practitioners who want to dive into the field of representation learning.

Reconstructing images from magnitude measurements is an important and difficult problem arising in many research areas, such as X-ray crystallography, astronomical imaging and more. While optimization-based approaches often struggle with the non-convexity and non-linearity of the problem, learning-based approaches are able to produce reconstructions of high quality for data similar to a given training dataset. In this work, we analyze a class of methods based on conditional generative adversarial networks (CGAN). We show how the benefits of optimization-based and learning-based methods can be combined to improve reconstruction quality. Furthermore, we show that these combined methods are able to generalize to out-of-distribution data and analyze their robustness to measurement noise. In addition to that, we compare how the methods are impacted by missing measurements. Extensive ablation studies demonstrate that all components of our approach are essential and justify the choice of network architecture.

Zielsetzung Die kardiale Echtzeit-MRT ermöglicht die Akquisition qualitativ hochwertiger Bilder unter physiologischen Bedingungen. Schwierigkeiten bei der quantitativen Datenanalyse erschweren den Einsatz in der klinischen Routine. Ziel dieser Studie war die Entwicklung einer Methode zur quantitativen Analyse der kardialen Echtzeit-MRT unter freier Atmung im klinischen Alltag und der Vergleich der Bildqualität und der Ergebnisse der Analyse der Herzfunktion bei pädiatrischen Patienten mit der konventionellen kardialen MRT mit Atemanhalten.

Zielsetzung Ziel der Studie war es, zu untersuchen, ob eine MR-kompatible Spirometrie in Kombination mit Echtzeit-MR-Bildgebung des Herzens während freier Atmung toleriert wird und hierdurch eine Stabilisierung der atemabhängigen Bewegung gelingt. Weiterhin sollte geklärt werden, ob so die atemabhängige Modulation der ventrikulären Volumina quantifiziert werden kann und welche zusätzlichen Informationen dadurch sowohl zur Herzfunktion als auch zur Herz-Lungen-Interaktion im Vergleich zur Standardvolumetrie in Atemanhaltetechnik ermöglicht werden.

Cardiac real-time magnetic resonance imaging (RT-MRI) provides high-quality images even during free-breathing. Difficulties in post-processing impede its use in clinical routine. To demonstrate the feasibility of quantitative analysis of cardiac free-breathing RT-MRI and to compare image quality and volumetry during free-breathing RT-MRI in pediatric patients to standard breath-hold cine MRI. Pediatric patients (n = 22) received cardiac RT-MRI volumetry during free breathing (1.5 T; short axis; 30 frames per s) in addition to standard breath-hold cine imaging in end-expiration. Real-time images were binned retrospectively based on electrocardiography and respiratory bellows. Image quality and volumetry were compared using the European Cardiovascular Magnetic Resonance registry score, structure visibility rating, linear regression and Bland–Altman analyses. Additional time for binning of real-time images was 2 min. For both techniques, image quality was rated good to excellent. RT-MRI was significantly more robust against artifacts (P < 0.01). Linear regression revealed good correlations for the ventricular volumes. Bland–Altman plots showed a good limit of agreement (LoA) for end-diastolic volume (left ventricle [LV]: LoA -0.1 ± 2.7 ml/m2, right ventricle [RV]: LoA -1.9 ± 3.4 ml/m2), end-systolic volume (LV: LoA 0.4 ± 1.9 ml/m2, RV: LoA 0.6 ± 2.0 ml/m2), stroke volume (LV: LoA -0.5 ± 2.3 ml/m2, RV: LoA -2.6 ± 3.3 ml/m2) and ejection fraction (LV: LoA -0.5 ± 1.6%, RV: LoA -2.1 ± 2.8%). Compared to standard cine MRI with breath hold, RT-MRI during free breathing with retrospective respiratory binning offers good image quality, reduced image artifacts enabling fast quantitative evaluations of ventricular volumes in clinical practice under physiological conditions.

Purpose To test the feasibility of cardiac real‐time MRI in combination with retrospective gating by MR‐compatible spirometry, to improve motion control, and to allow quantification of respiratory‐induced changes during free‐breathing. Methods Cross‐sectional real‐time MRI (1.5T; 30 frames/s) using steady‐state free precession contrast during free‐breathing was combined with MR‐compatible spirometry in healthy adult volunteers ( n = 4). Retrospective binning assigned images to classes that were defined by electrocardiogram and spirometry. Left ventricular eccentricity index as an indicator of septal position and ventricular volumes in different respiratory phases were calculated to assess heart–lung interactions. Results Real‐time MRI with MR‐compatible spirometry is feasible and well tolerated. Spirometry‐based binning improved motion control significantly. The end‐diastolic epicardial eccentricity index increased significantly during inspiration (1.04 ± 0.04 to 1.19 ± 0.05; P < .05). During inspiration, right ventricular end‐diastolic volume (79 ± 17 mL/m 2 to 98 ± 18 mL/m 2 ), stroke volume (41 ± 8 mL/m 2 to 59 ± 11 mL/m 2 ) and ejection fraction (53 ± 3% to 60 ± 1%) increased significantly, whereas the end‐systolic volume remained almost unchanged. Left ventricular end‐diastolic volume, left ventricular stroke volume, and left ventricular ejection fraction decreased during inspiration, whereas the left ventricular end‐systolic volume increased. The relationship between stroke volume and end‐diastolic volume (Frank‐Starling relationship) based on changes induced by respiration allowed for a slope estimate of the Frank‐Starling curve to be 0.9 to 1.1. Conclusion Real‐time MRI during free‐breathing combined with MR‐compatible spirometry and retrospective binning improves image stabilization, allows quantitative image analysis, and importantly, offers unique opportunities to judge heart–lung interactions.

Fourier phase retrieval is the problem of reconstructing a signal given only the magnitude of its Fourier transformation. Optimization-based approaches, like the well-established Gerchberg-Saxton or the hybrid input output algorithm, struggle at reconstructing images from magnitudes that are not oversampled. This motivates the application of learned methods, which allow reconstruction from non-oversampled magnitude measurements after a learning phase. In this paper, we want to push the limits of these learned methods by means of a deep neural network cascade that reconstructs the image successively on different resolutions from its non-oversampled Fourier magnitude. We evaluate our method on four different datasets (MNIST, EMNIST, Fashion-MNIST, and KMNIST) and demonstrate that it yields improved performance over other non-iterative methods and optimization-based methods.

Given the increasing threat of adversarial attacks on deep neural networks (DNNs), research on efficient detection methods is more important than ever. In this work, we take a closer look at adversarial attack detection based on the class scores of an already trained classification model. We propose to train a support vector machine (SVM) on the class scores to detect adversarial examples. Our method is able to detect adversarial examples generated by various attacks, and can be easily adopted to a plethora of deep classification models. We show that our approach yields an improved detection rate compared to an existing method, whilst being easy to implement. We perform an extensive empirical analysis on different deep classification models, investigating various state-of-the-art adversarial attacks. Moreover, we observe that our proposed method is better at detecting a combination of adversarial attacks. This work indicates the potential of detecting various adversarial attacks simply by using the class scores of an already trained classification model.

Einleitung und Ziel: Die Echtzeit-MRT ist eine neue Methode, die eine hohe Abtastrate ermöglicht, während gleichzeitig eine hohe räumliche Auflösung erhalten bleibt. Dadurch wird eine Darstellung von Herzzyklen bei freier Atmung ermöglicht, was sowohl unter dem Aspekt der physiologischen Herz-Lungen-Interaktion als auch für den Patientenkomfort weitreichende Vorteile erzielt. Ziel dieser Studie ist die Entwicklung und Validierung einer Methode zur nicht-invasiven Quantifizierung der physiologischen Herz-Lungen-Interaktion.

In this paper, we propose the application of conditional generative adversarial networks to solve various phase retrieval problems. We show that including knowledge of the measurement process at training time leads to an optimization at test time that is more robust to initialization than existing approaches involving generative models. In addition, conditioning the generator network on the measurements enables us to achieve much more detailed results. We empirically demonstrate that these advantages provide meaningful solutions to the Fourier and the compressive phase retrieval problem and that our method outperforms well-established projection-based methods as well as existing methods that are based on neural networks. Like other deep learning methods, our approach is robust to noise and can therefore be useful for real-world applications.

Reconstructing images from their Fourier magnitude measurements is a problem that often arises in different research areas. This process is also referred to as phase retrieval. In this work, we consider a modified version of the phase retrieval problem, which allows for a reference image to be added onto the image before the Fourier magnitudes are measured. We analyze an unrolled Gerchberg-Saxton (GS) algorithm that can be used to learn a good reference image from a dataset. Furthermore, we take a closer look at the learned reference images and propose a simple and efficient heuristic to construct reference images that, in some cases, yields reconstructions of comparable quality as approaches that learn references. Our code is available at https://github.com/tuelwer/reference-learning.

Alpha matting aims to estimate the translucency of an object in a given image. The resulting alpha matte describes pixel-wise to what amount foreground and background colors contribute to the color of the composite image. While most methods in literature focus on estimating the alpha matte, the process of estimating the foreground colors given the input image and its alpha matte is often neglected, although foreground estimation is an essential part of many image editing workflows. In this work, we propose a novel method for foreground estimation given the alpha matte. We demonstrate that our fast multi-level approach yields results that are comparable with the state-of-the-art while outperforming those methods in computational runtime and memory usage.