We propose an accurate and computationally efficient 3D scattering model, multi-layer Born (MLB), and use it to recover the 3D refractive index (RI) of thick biological samples. For inverse problems recovering the complex field of thick samples, weak scattering models (e.g., first Born) may fail or underestimate the RI, especially with a large index contrast. Multi-slice (MS) beam propagation methods model multiple scattering to provide more realistic reconstructions; however, MS does not properly account for highly oblique scattering, nor does it model backward scattering. Our proposed MLB model uses a first Born model at each of many slices, accurately capturing the oblique scattering effects and estimating the backward scattering process. When used in conjunction with an inverse solver, the model provides more accurate RI reconstructions for high-resolution phase tomography. Importantly, MLB retains a reasonable computation time that is critical for practical implementation with iterative inverse algorithms.

3D quantitative phase (refractive index) microscopy reveals volumetric structure of biological specimens. Optical diffraction tomography (ODT) is a common technique for 3D phase imaging. By angularly scanning a spatially coherent light source and measuring scattered fields on the imaging plane, 3D refractive index (RI) is recovered by solving an inverse problem. However, ODT often linearizes the process by using a weakly scattering model, e.g. the first Born approximation or Rytov approximation, which underestimate the RI and fail to reconstruct realistic shape of high RI contrast multiple scattering objects. On the other hand, non-linear models such as the multi-slice or beam propagation methods mitigate artifacts by modeling multiple scattering. However, they ignore back-scattering and intra-slice scattering and make a paraxial approximation by assuming each slice is infinitesimally thin. In this work, we propose a new 3D scattering model Multi-layer Born (MLB), which treats the object as thin 3D slabs with finite thickness and applies the first Born approximation on each slab as the field propagates through the object, increasing the accuracy significantly. In the meantime, a similar computation complexity is achieved comparing to the previously proposed multi-slice models. Therefore, MLB can achieve accuracy similar to that of FDTD or SEAGLE, a frequency domain solver, with orders of magnitude less computation time. In addition to forward scattering, multiple back-scattering effects are also captured by MLB unlike existing models. We apply MLB to recover the RI distribution of 3D phantoms and biological samples with intensity-only measurements from an LED array microscope and show that the results are superior to existing methods.

Linear operators used in iterative methods like conjugate gradient have typically been implemented either as ""matrix-driven"" subroutines backed by explicit sparse or dense matrices, or as ""matrix-free"" subroutines that implement specific linear operations directly (e.g. FFTs). The matrix-driven approach is generally more portable because it can target widely-available BLAS libraries, but it can be inefficient in terms of time and space complexity. In contrast, the matrix-free approach is more performant because it leverages structure in operations, but it requires each operator be re-implemented on each new platform. To increase performance and portability, we propose a hybrid approach that represents linear operators as expression trees. Leaf nodes in the tree are either matrix-free or matrix-driven operators, and interior nodes represent mathematical compositions (sums, products, transposes) or structural compositions (stacks, block diagonals, etc.) of the leaf operators. This representation enables expert-guided reordering and fusion transformations that can improve performance or reduce memory pressure. We implement our approach in a domain-specific language called Indigo. We assess Indigo on image reconstruction problems arising in four application areas: magnetic resonance imaging, ptychography, magnetic particle imaging, and fluorescent microscopy. We give performance results from vendor BLAS libraries, and we introduce specializations to Sparse BLAS routines that achieve near-Roofline performance on multi-core, many-core, and GPU systems.

Linear operators used in iterative methods like conjugate gradient have typically been implemented either as “matrix-driven” subroutines backed by explicit sparse or dense matrices, or as “matrix-free” subroutines that implement specific linear operations directly (e.g. FFTs). The matrix-driven approach is generally more portable because it can target widelyavailable BLAS libraries, but it can be inefficient in terms of time and space complexity. In contrast, the matrix-free approach is more performant because it leverages structure in operations, but it requires each operator be re-implemented on each new platform. To increase performance and portability, we propose a hybrid approach that represents linear operators as expression trees. Leaf nodes in the tree are either matrix-free or matrix-driven operators, and interior nodes represent mathematical compositions (sums, products, transposes) or structural compositions (stacks, block diagonals, etc.) of the leaf operators. This representation enables expert-guided reordering and fusion transformations that can improve performance or reduce memory pressure. We implement our approach in a domain-specific language called Indigo. We assess Indigo on image reconstruction problems arising in four application areas: magnetic resonance imaging, ptychography, magnetic particle imaging, and fluorescent microscopy. We give performance results from vendor BLAS libraries, and we introduce specializations to Sparse BLAS routines that achieve near-Roofline performance on multi-core, many-core, and GPU systems.

Coded aperture imaging for 3D reconstructions have been applied in both photography and microscopy to successfully recover depth information from images captured with a patterned pupil plane. Generally, single-shot coded aperture microscopy methods require the 3D sample to be extremely sparse and with no overlapping features. Here, we extend coded aperture microscopy methods to multi-shot 3D fluorescent imaging with wave-optical forward models, enabling thicker and denser 3D samples to be faithfully reconstructed with efficient data capture via compressed sensing and advanced inverse algorithms.. We demonstrate successful reconstruction of a 500um thick brine shrimp sample. We further study the minimum required number of codes to capture full fluorescent information under wide-field illumination The experiments are carried out with a 4f system and a spatial light modulator in the Fourier (aperture) domain to display the codes.

We demonstrate a 3D phase imaging model for scattering objects with intensity-only measurements taken with patterned illumination in an LED array microscope. 3D refractive index (RI) of polystyrene beads is recovered, and the preliminary results indicate that the proposed model outperforms existing methods in terms of quantitative accuracy of RI.

Multiple scattering of an electromagnetic wave as it passes through an object is a fundamental problem that limits the performance of current imaging systems. In this paper, we describe a new technique-called Series Expansion with Accelerated Gradient Descent on the Lippmann-Schwinger Equation-for robust imaging under multiple scattering based on a combination of an iterative forward model and a total variation regularizer. The proposed method can account for multiple scattering, which makes it advantageous in applications where single scattering approximations are inaccurate. Specifically, the method relies on a series expansion of the scattered wave with an accelerated-gradient method. This expansion guarantees the convergence of the forward model even for strongly scattering objects. One of our key insights is that it is possible to obtain an explicit formula for computing the gradient of an iterative forward model with respect to the unknown object, thus enabling fast image reconstruction with the state-of-theart fast iterative shrinkage/thresholding algorithm. The proposed method is validated on diffraction tomography, where complex electric field is captured at different illumination angles.

Phase-space refers to simultaneous space-frequency information (e.g. Wigner functions, light fields), which is directly related to spatial coherence properties (e.g. Mutual Intensity). We introduce a binary pupil masking technique that allows us to computationally reconstruct the phase space distribution of optical beams from a series of images. Previous work has shown phase space to be useful for 3D imaging and localization in a multiple scattering environment. Binary masks are easy to implement compared to gray masks or phase masks and the proposed scheme requires no interferometry. After designing the masks with nonredundant arrays, we measure an intensity image for each aperture mask and reconstruct the phase space through an auxiliary coherence function. We demonstrate experimentally the reconstruction of the phase space of a collection of 3D incoherent sources.

Optical phase-space functions describe spatial and angular information simultaneously; examples of optical phase-space functions include light fields in ray optics and Wigner functions in wave optics. Measurement of phase-space enables digital refocusing, aberration removal and 3D reconstruction. High-resolution capture of 4D phase-space datasets is, however, challenging. Previous scanning approaches are slow, light inefficient and do not achieve diffraction-limited resolution. Here, we propose a multiplexed method that solves these problems. We use a spatial light modulator (SLM) in the pupil plane of a microscope in order to sequentially pattern multiplexed coded apertures while capturing images in real space. Then, we reconstruct the 3D fluorescence distribution of our sample by solving an inverse problem via regularized least squares with a proximal accelerated gradient descent solver. We experimentally reconstruct a 101 Megavoxel 3D volume (1010×510×500µm with NA 0.4), demonstrating improved acquisition time, light throughput and resolution compared to scanning aperture methods. Our flexible patterning scheme further allows sparsity in the sample to be exploited for reduced data capture.

We propose a new compressive imaging method for reconstructing 2D or 3D objects from their scattered wave-field measurements. Our method relies on a novel, nonlinear measurement model that can account for the multiple scattering phenomenon, which makes the method preferable in applications where linear measurement models are inaccurate. We construct the measurement model by expanding the scattered wave-field with an accelerated-gradient method, which is guaranteed to converge and is suitable for large-scale problems. We provide explicit formulas for computing the gradient of our measurement model with respect to the unknown image, which enables image formation with a sparsity-driven numerical optimization algorithm. We validate the method both analytically and with numerical simulations.

We present a multi-shot coded aperture 3D fluorescent imaging method based on multiplexed phase-space sampling theory. Our reconstructed volumes achieve diffraction-limited resolution from only 100 raw images, by exploiting the sparse nature of the sample.

Understanding the mechanisms of perception, cognition, and behavior requires instruments that are capable of recording and controlling the electrical activity of many neurons simultaneously and at high speeds. All-optical approaches are particularly promising since they are minimally invasive and potentially scalable to experiments interrogating thousands or millions of neurons. Conventional light-field microscopy provides a single-shot 3D fluorescence capture method with good light efficiency and fast speed, but suffers from low spatial resolution and significant image degradation due to scattering in deep layers of brain tissue. Here, we propose a new compressive light-field microscopy method to address both problems, offering a path toward measurement of individual neuron activity across large volumes of tissue. The technique relies on spatial and temporal sparsity of fluorescence signals, allowing one to identify and localize each neuron in a 3D volume, with scattering and aberration effects naturally included and without ever reconstructing a volume image. Experimental results on live zebrafish track the activity of an estimated 800+ neural structures at 100 Hz sampling rate.

Phase-space measurements enable characterization of second-order spatial coherence properties and can be used for digital aberration removal or 3D position reconstruction. Previous methods use a scanning aperture to measure the phase space spectrogram, which is slow and light inefficient, while also attenuating information about higher-order correlations. We demonstrate a significant improvement of speed and light throughput by incorporating multiplexing techniques into our phase-space imaging system. The scheme implements 2D coded aperture patterning in the Fourier (pupil) plane of a microscope using a Spatial Light Modulator (SLM), while capturing multiple intensity images in real space. We compare various multiplexing schemes to scanning apertures and show that our phase-space reconstructions are accurate for experimental data with biological samples containing many 3D fluorophores.

Traditional light-field microscopy (LFM) relies on the intermediate reconstruction of visual renderings, followed by digital analysis, which applications in deeper layers of the brain as optical scattering destroys image quality. We have developed an LFM technique that eliminates the need to reconstruct a volume image, instead detecting and localizing neurons by leveraging the sparsity of brain activity in space and time for better performance in highly scattering environments.

We demonstrate the use of phase-space imaging for 3D localization of multiple point sources inside scattering material. The effect of scattering is to spread angular (spatial frequency) information, which can be measured by phase space imaging. We derive a multi-slice forward model for homogenous volumetric scattering, then develop a reconstruction algorithm that exploits sparsity in order to further constrain the problem. By using 4D measurements for 3D reconstruction, the dimensionality mismatch provides significant robustness to multiple scattering, with either static or dynamic diffusers. Experimentally, our high-resolution 4D phase-space data is collected by a spectrogram setup, with results successfully recovering the 3D positions of multiple LEDs embedded in turbid scattering media.

In freshman general physics, the professor briefly touched Einstein’s relativity and pointed out that if there were a fat person walking at the speed of light, he would look as thin as a piece of paper! This effect is called Lorenz contraction. The reason is roughly that the time and space is mixed at high speed and that we can “see” time as a negative spatial length. Hence we want to see some cool effect at the speed close to light. Let’s assume that we are sitting in a supercar, seeing the front. The above Lorenz contraction gives us a 3D scene in Euclidean coordinate, so there is still work to convert it into images we would see from the supercar. We think it would be cool if we can see a 360◦-by-180◦ all direction scene at any speed between static and speed of light. In addition, with full view panorama, we can choose wherever to travel as well as arbitrary field of view!

We demonstrate a compressive light field microscopy method for 3D neural activity measurement. Individual neurons are identified computationally, without the intermediate step of reconstructing volume images. Results on Zebrafish show volume activity recording at 100fps.

Measuring temperature in nanoscale spatial resolution either at or far from equilibrium is of importance in many scientific and technological applications. Although negatively charged nitrogen-vacancy (NV(-)) centers in diamond have recently emerged as a promising nanometric temperature sensor, the technique has been applied only under steady state conditions so far. Here, we present a three-point sampling method that allows real-time monitoring of the temperature changes over ±100 K and a pump-probe-type experiment that enables the study of nanoscale heat transfer with a temporal resolution of better than 10 μs. The utility of the time-resolved luminescence nanothermometry was demonstrated with 100 nm fluorescent nanodiamonds spin-coated on a glass substrate and submerged in gold nanorod solution heated by a near-infrared laser, and the validity of the measurements was verified with finite-element numerical simulations. The combined theoretical and experimental approaches will be useful to implement time-resolved temperature sensing in laser processing of materials and even for devices in operation at the nanometer scale.