3D phase imaging recovers an object's volumetric refractive index from intensity and/or holographic measurements. Partially coherent methods, such as illumination-based differential phase contrast (DPC), are particularly simple to implement in a commercial brightfield microscope. 3D DPC acquires images at multiple focus positions and with different illumination source patterns in order to reconstruct 3D refractive index. Here, we present a practical extension of the 3D DPC method that does not require a precise motion stage for scanning the focus and uses optimized illumination patterns for improved performance. The user scans the focus by hand, using the microscope's focus knob, and the algorithm self-calibrates the axial position to solve for the 3D refractive index of the sample through a computational inverse problem. We further show that the illumination patterns can be optimized by an end-to-end learning procedure. Combining these two, we demonstrate improved 3D DPC with a commercial microscope whose only hardware modification is LED array illumination.

We couple a differentiable forward model of diffuser-based single-shot 3D microscopy (Fourier DiffuserScope) with an invertible and differentiable reconstruction algorithm to jointly optimize the imaging system and the reconstruction parameters in a memory-efficient manner.

3D differential phase contrast (3D DPC) microscopy uses asymmetric illumination patterns and axial scanning to recover volumetric maps of refractive index. To avoid the expense of automated axial scanning, we demonstrate 3D DPC without a z-stage by hand spinning the microscope’s defocus knob to scan the object axially while updating illumination patterns on the LED-array microscope. We utilize an inverse problem optimization to retrieve the sample’s volumetric information with measurements from unknown axial positions by jointly solving for each measurement’s axial position. Finally, we explore how to optimize the LED-array illumination patterns for varying axial sampling rates.

Deep learning (DL) based unrolled reconstructions have shown state-of-the-art performance for under-sampled magnetic resonance imaging (MRI). Similar to compressed sensing, DL can leverage high-dimensional data (e.g. 3D, 2D+time, 3D+time) to further improve performance. However, network size and depth are currently limited by the GPU memory required for backpropagation. Here we use a memory-efficient learning (MEL) framework which favorably trades off storage with a manageable increase in computation during training. Using MEL with multidimensional data, we demonstrate improved image reconstruction performance for in-vivo 3D MRI and 2D+time cardiac cine MRI. MEL uses far less GPU memory while marginally increasing the training time, which enables new applications of DL to high-dimensional MRI.

3D refractive index imaging methods usually rely on a weak-scattering approximation that does not allow for thick samples to be imaged accurately. Recent methods such as 3D Fourier ptychographic microscopy (FPM) instead use multiple-scattering models which allow for thicker objects to be imaged. In practice the illumination-side coding of 3D FPM requires redundant information and may produce inaccurate reconstructions for thick samples. Here, we propose augmenting 3D FPM with detection-side coding using a spatial light modulator (SLM) and optimize the SLM coding strategy with physics-based machine learned pupil coding designs that are optimized for 3D reconstructions. We compare our learned designs to random-, defocus-, Zernike aberrations-based pupil codes in simulated and experimental results.

Computational illumination microscopy has enabled imaging of a sample’s phase, spatial features beyond the diffraction limit (Fourier Ptychography), and 3D refractive index from intensity-based measurements captured on an LED array microscope. However, these methods require up to hundreds of images, limiting applications, particularly live sample imaging. Here, we demonstrate how the experimental design of a computational microscope can be optimized using data-driven methods to learn a compressed set of measurements, thereby improving the temporal resolution of the system. Specifically, we consider the image reconstruction as a physics-based network and learn the experimental design to optimize the system’s overall performance for a desired temporal resolution. Finally, we will discuss how the system’s experimental design can be learned on synthetic training data.

We demonstrate 3D phase imaging using asymmetric illumination patterns and defocused intensity measurements taken with continuous axial motion. The sample’s 3D refractive index is reconstructed with a motion-corrected transfer function.

Critical aspects of computational imaging systems, such as experimental design and image priors, can be optimized through deep networks formed by the unrolled iterations of classical physics-based reconstructions. Termed physics-based networks, they incorporate both the known physics of the system via its forward model, and the power of deep learning via data-driven training. However, for realistic large-scale physics-based networks, computing gradients via backpropagation is infeasible due to the memory limitations of graphics processing units. In this work, we propose a memory-efficient learning procedure that exploits the reversibility of the network's layers to enable physics-based learning for large-scale computational imaging systems. We demonstrate our method on a compressed sensing example, as well as two large-scale real-world systems: 3D multi-channel magnetic resonance imaging and super-resolution optical microscopy.

Deep neural networks have emerged as effective tools for computational imaging including quantitative phase microscopy of transparent samples. To reconstruct phase from intensity, current approaches rely on supervised learning with training examples; consequently, their performance is sensitive to a match of training and imaging settings. Here we propose a new approach to phase microscopy by using an untrained deep neural network for measurement formation, encapsulating the image prior and imaging physics. Our approach does not require any training data and simultaneously reconstructs the sought phase and pupil-plane aberrations by fitting the weights of the network to the captured images. To demonstrate experimentally, we reconstruct quantitative phase from through-focus images blindly (i.e. no explicit knowledge of the aberrations).

The goal of this tutorial is to explain step-by-step how to implement physics-based learning for the rapid prototyping of a computational imaging system. We provide a basic overview of physics-based learning, the construction of a physics-based network, and its reduction to practice. Specifically, we advocate exploiting the auto-differentiation functionality twice, once to build a physics-based network and again to perform physics-based learning. Thus, the user need only implement the forward model process for their system, speeding up prototyping time. We provide an open-source Pytorch implementation of a physics-based network and training procedure for a generic sparse recovery problem

The goal of this work is to incorporate Convolutional Neural Networks (CNNs) into the 3D deconvolution process without training. CNNs are well suited to the problem of 2D deconvolution, however training a CNN on 3D volumes requires excessive time and impractical amounts of training data. To circumvent these problems, we use a CNN architecture as if it were a handcrafted prior, similar to the work deep image prior. Using this method, we achieve high SSIM and PSNR metrics relative to other modern techniques for deconvolving through-focus fluorescence measurements to recover a 3D volume with no training data and minimal hyperparameter tuning.

Computational imaging systems jointly design computation and hardware to retrieve information which is not traditionally accessible with standard imaging systems. Recently, critical aspects such as experimental design and image priors are optimized through deep neural networks formed by the unrolled iterations of classical physics-based reconstructions (termed physics-based networks). However, for real-world large-scale systems, computing gradients via backpropagation restricts learning due to memory limitations of graphical processing units. In this work, we propose a memory-efficient learning procedure that exploits the reversibility of the network's layers to enable data-driven design for large-scale computational imaging. We demonstrate our methods practicality on two large-scale systems: super-resolution optical microscopy and multi-channel magnetic resonance imaging.

Coded-illumination can enable quantitative phase microscopy of transparent samples with minimal hardware requirements. Intensity images are captured with different source patterns and a non-linear phase retrieval optimization reconstructs the image. The non-linear nature of the processing makes optimizing the illumination pattern designs complicated. Traditional techniques for experimental design (e.g. condition number optimization, spectral analysis) consider only linear measurement formation models and linear reconstructions. Deep neural networks (DNNs) can efficiently represent the non-linear process and can be optimized over via training in an end-to-end framework. However, DNNs typically require a large amount of training examples and parameters to properly learn the phase retrieval process, without making use of the known physical models. Here, we aim to use both our knowledge of the physics and the power of machine learning together. We develop a new data-driven approach to optimizing coded-illumination patterns for a LED array microscope for a given phase reconstruction algorithm. Our method incorporates both the physics of the measurement scheme and the non-linearity of the reconstruction algorithm into the design problem. This enables efficient parameterization, which allows us to use only a small number of training examples to learn designs that generalize well in the experimental setting without retraining. We show experimental results for both a well-characterized phase target and mouse fibroblast cells using coded-illumination patterns optimized for a sparsity-based phase reconstruction algorithm. Our learned design results using 2 measurements demonstrate similar accuracy to Fourier Ptychography with 69 measurements.

Fourier Ptychographic Microscopy (FPM) is a computational imaging method that is able to super-resolve features beyond the diffraction-limit set by the objective lens of a traditional microscope. This is accomplished by using synthetic aperture and phase retrieval algorithms to combine many measurements captured by an LED array microscope with programmable source patterns. FPM provides simultaneous large field-of-view and high resolution imaging, but at the cost of reduced temporal resolution, thereby limiting live cell applications. In this work, we learn LED source pattern designs that compress the many required measurements into only a few, with negligible loss in reconstruction quality or resolution. This is accomplished by recasting the super-resolution reconstruction as a Physics-based Neural Network and learning the experimental design to optimize the network's overall performance. Specifically, we learn LED patterns for different applications (e.g. amplitude contrast and quantitative phase imaging) and show that the designs we learn through simulation generalize well in the experimental setting. Further, we discuss a context-specific loss function, practical memory limitations, and interpretability of our learned designs.

Fourier Ptychographic Microscopy (FPM) and Differential Phase Contrast (DPC) are quantitative phase imaging (QPI) methods that recover the complex transmittance function of a sample through coded illumination measurements and phase retrieval optimization. The successes of these methods rely upon acquiring several or possibly hundreds of illumination-encoded measurements. The multi-shot nature of such methods limits their temporal resolution. Similar to motion-induced blur during a long photographic exposure, motion occurring during these acquisitions causes spatial distortion and errors in the reconstructed phase, which inhibits these methods' ability to image fast moving live samples. Here we present a novel approach to correct for motion during QPI capture that relies on motion navigation to register measurements together prior to phase retrieval. The different illumination patterns required for QPI cause the measurements to have a different contrasts. This makes it difficult to use standard registration approaches to estimate complex sample motion directly from the measurements. Instead, we use a color-multiplexed navigator signal (red) that is comprised of a constant illumination pattern and leverage a color camera to separate it from the primary QPI information (green). The reliable motion estimate allows measurements to be shared across time points through image registration. This enables a full set of measurements for a phase retrieval problem to be solved at each time point. We demonstrate proof-of-concept experimental results in which blurring due to live sample motion (swimming Zebra fish, cell motion, and organelle movement) is reduced.

We present a novel method to perform individual particle (e.g. cells or viruses) coincidence correction through joint channel design and algorithmic methods. Inspired by multiple-user communication theory, we modulate the channel response, with Node-Pore Sensing, to give each particle a binary Barker code signature. When processed with our modified successive interference cancellation method, this signature enables both the separation of coincidence particles and a high sensitivity to small particles. We identify several sources of modeling error and mitigate most effects using a data-driven self-calibration step and robust regression. Additionally, we provide simulation analysis to highlight our robustness, as well as our limitations, to these sources of stochastic system model error. Finally, we conduct experimental validation of our techniques using several encoded devices to screen a heterogeneous sample of several size particles.

The temporal resolution of quantitative phase imaging with Differential Phase Contrast (DPC) is limited by the requirement for multiple illumination-encoded measurements. This inhibits imaging of fast-moving samples. We present a computational approach to model and correct for non-rigid sample motion during the DPC acquisition in order to improve temporal resolution to that of a single-shot method and enable imaging of motion dynamics at the framerate of the sensor. Our method relies on the addition of a simultaneously-acquired color-multiplexed reference signal to enable non-rigid registration of measurements prior to phase retrieval. We show experimental results where we reduce motion blur from fast-moving live biological samples.

Blind deconvolution is the process of deconvolving a known acquired signal with an unknown point spread function (PSF) or channel encoding, by jointly estimating the system’s input as well as the system’s PSF. Models for this problem often present as the convolution or product of the underlying signal and the system’s PSF. The acquired signal is bi-linear in these two unknown signals (i.e. linear in one if the other is constant), which results in an inverse problem that is non-convex. In addition, the number of observations to uniquely reconstruct the signal and the system’s PSF is often unattainable as the number of degrees of freedom grows as the product of the size of the signal with the size of the PSF, thus the problem is often underdetermined. While the task seems daunting, there has been a plethora of work over the past twenty years with the aim of performing this task with exact recovery, similar to that of sparse vector recovery in compressed sensing and of low rank matrix recovery in matrix completion. There are a variety of works in the area of exact recovery to pose convex and non-convex problem alike as well as theory regarding under what conditions this is possible. We shall see that advances in these fields compliment the blind deconvolution problem as it can be posed as a linear inverse problem with rank constraint and relaxed to a convex program. In this literature review the point of view as a practitioner in the areas of signal processing and computational imaging (or computational sensing) is taken. This review is laid out as follows: section 2 goes over into several applications and importance, section 3 reviews the technique of lifting as well as several convex and non-convex problem formulations, section 4 explains some of the theoretical guarantees that accompany convex and non-convex problem formulations, and finally section 5 discusses a broader picture regarding computation and memory for large scale problems.

A resistive pulse sensing device is able to extract quantities such as concentration and size distribution of particles, e.g. cells or microspheres, as they flow through the device's sensor region, i.e. channel, in an electrolyte solution. The dynamic range of detectable particle sizes is limited by the channel dimensions. In addition, signal interference from multiple particles transiting the channel simultaneously, i.e. coincidence event, further hinder the dynamic range. Coincidence data is often considered unusable and is discarded, reducing the throughput and introducing possible biases and errors into the distributions. Here, we propose a two-step solution. We code the channel such that the system response results in a Manchester encoded Barker-Code sequence, allowing us to take advantage of the code's pulse compression properties. We pose the parameter estimation problem as a sparse inverse problem, which enables estimation of particle sizes and velocities while resolving coincidences, and solve it with a successive interference cancellation algorithm. We introduce modifications to the algorithm to account for device fabrication variations and natural stochastic variations in flow. We demonstrate the ability to resolve coincidences and possible increases in the device's dynamic range by screening particles of different size through a Barker encoded device.