Most computer vision systems and computational photography systems are visible light based which is a small fraction of the electromagnetic (EM) spectrum. In recent years radio frequency (RF) hardware has become more widely available, for example, many cars are equipped with a RADAR, and almost every home has a WiFi device. In the context of imaging, RF spectrum holds many advantages compared to visible light systems. In particular, in this regime, EM energy effectively interacts in different ways with matter. This property allows for many novel applications such as privacy preserving computer vision and imaging through absorbing and scattering materials in visible light such as walls. Here, we expand many of the concepts in computational photography in visible light to RF cameras. The main limitation of imaging with RF is the large wavelength that limits the imaging resolution when compared to visible light. However, the output of RF cameras is usually processed by computer vision and perception algorithms which would benefit from multi-modal sensing of the environment, and from sensing in situations in which visible light systems fail. To bridge the gap between computational photography and RF imaging, we expand the concept of light-field to RF. This work paves the way to novel computational sensing systems with RF.

Seeing around corners, also known as non-line-of-sight (NLOS) imaging is a computational method to resolve or recover objects hidden around corners. Recent advances in imaging around corners have gained significant interest. This paper reviews different types of existing NLOS imaging techniques and discusses the challenges that need to be addressed, especially for their applications outside of a constrained laboratory environment. Our goal is to introduce this topic to broader research communities as well as provide insights that would lead to further developments in this research area.

Imaging through scattering media in the visible part of the electromagnetic spectrum holds many applications in various industries. For example, seeing through fog would enable autonomous vehicles to navigate in degraded weather conditions, augment human drivers and allow airplanes to operate in dense fog conditions. Another domain is medical imaging, where the ability to see into the body in the visible spectrum would reduce ionizing radiation exposure and provide more clinical meaningful data.Recent advances in single photon avalanche diode (SPAD) counters, and specifically time-resolved single photon counters enabled various challenging imaging applications. The main advantages provided by SPAD devices for imaging are improved noise models and sensitivity, both are essential in low signal-to-noise (SNR) imaging modalities. Another interesting property of SPAD detectors is the ability to measure single photon events which exposes the statistical nature of light. Moreover, the ability to manufacture SPAD arrays naturally lead to a faster and simpler acquisition process as they alleviate the need for scanning.Here, we leverage a time-resolved SPAD camera and demonstrate its advantages for imaging through scattering media. Specifically we experimentally demonstrate seeing through fog and imaging through scattering layers. These examples directly leverage the single photon sensitivity in modeling and rejecting scattered light.

Imaging through fog has important applications in industries such as self-driving cars, augmented driving, airplanes, helicopters, drones and trains. Current solutions are based on radar that suffers from poor resolution (due to the long wavelength), or on time gating that suffers from low signal-to-noise ratio. Here we demonstrate a technique that recovers reflectance and depth of a scene obstructed by dense, dynamic, and heterogeneous fog. For practical use cases, the imaging system is designed in optical reflection mode with minimal footprint and is based on LIDAR hardware. Specifically, we use a single photon avalanche diode (SPAD) camera that time-tags individual detected photons. A probabilistic computational framework is developed to estimate the fog properties from the measurement itself, and distinguish between background photons reflected from the fog and signal photons reflected from the target. The method is experimentally evaluated on a wide range of fog densities created in a fog chamber. The suggested approach demonstrates recovering objects 57cm away from the camera when the visibility is 37cm. In that case it recovers depth with a resolution of 5cm and scene reflectance with an improvement of 4dB in PSNR and 3.4× reconstruction quality in SSIM over time gating techniques.

Non-line-of-right (NLOS) imaging has recently been demonstrated using traditional cameras. Here we present a data-driven technique for NLOS imaging. We experimentally analyze the role of scene geometry and clutter on reconstruction quality.

We experimentally demonstrate a single-pixel, time-resolved camera that, by using the temporal information, produces improved reconstruction quality and shorter acquisition times, compared to traditional, non-time-resolved, single-pixel approaches.

Vehicles, search and rescue personnel, and endoscopes use flash lights to locate, identify, and view objects in their surroundings. Here we show the first steps of how all these tasks can be done around corners with consumer cameras. Recent techniques for NLOS imaging using consumer cameras have not been able to both localize and identify the hidden object. We introduce a method that couples traditional geometric understanding and data-driven techniques. To avoid the limitation of large dataset gathering, we train the data-driven models on rendered samples to computationally recover the hidden scene on real data. The method has three independent operating modes: 1) a regression output to localize a hidden object in 2D, 2) an identification output to identify the object type or pose, and 3) a generative network to reconstruct the hidden scene from a new viewpoint. The method is able to localize 12cm wide hidden objects in 2D with 1.7cm accuracy. The method also identifies the hidden object class with 87.7% accuracy (compared to 33.3% random accuracy). This paper also provides an analysis on the distribution of information that encodes the occluded object in the accessible scene. We show that, unlike previously thought, the area that extends beyond the corner is essential for accurate object localization and identification.

Imaging through fog has important applications in industries such as self-driving cars, augmented driving, airplanes, helicopters, drones and trains. Here we show that time profiles of light reflected from fog have a distribution (Gamma) that is different from light reflected from objects occluded by fog (Gaussian). This helps to distinguish between background photons reflected from the fog and signal photons reflected from the occluded object. Based on this observation, we recover reflectance and depth of a scene obstructed by dense, dynamic, and heterogeneous fog. For practical use cases, the imaging system is designed in optical reflection mode with minimal footprint and is based on LIDAR hardware. Specifically, we use a single photon avalanche diode (SPAD) camera that time-tags individual detected photons. A probabilistic computational framework is developed to estimate the fog properties from the measurement itself, without prior knowledge. Other solutions are based on radar that suffers from poor resolution (due to the long wavelength), or on time gating that suffers from low signal-to-noise ratio. The suggested technique is experimentally evaluated in a wide range of fog densities created in a fog chamber. It demonstrates recovering objects 57cm away from the camera when the visibility is 37cm. In that case it recovers depth with a resolution of 5cm and scene reflectance with an improvement of 4dB in PSNR and 3.4x reconstruction quality in SSIM over time gating techniques.

We exploit the high time resolution of ultrafast imaging sensors to fold and modify the imaging optics of the camera. We demonstrate an order of magnitude compression in lens-sensor distance using a time-folded design.

Ultrafast sensors and depth cameras are key enablers for imaging through complex geometries, through scattering, and beyond the line of sight. However, despite accelerating advances in imaging electronics and imaging applications, the optics of such cameras have been inherited from conventional low-speed photography cameras. This has limited ultrafast cameras and their applications to the design constraints of conventional optics. Here, we exploit time as an extra dimension in the optical design and demonstrate that by folding large spaces in time using time-resolved cavities, one can enable new camera capabilities without losing the targeted information. We demonstrate lens tube compression by an order of magnitude, together with ultrafast multi-zoom imaging and ultrafast multispectral imaging by time-folding the optical path at different regions of the imaging optics. Considering the vast variety of designs that could emerge by time-folding conventional imaging optics, we expect this technique to have a broad impact on time-resolved imaging and depth-sensing optics.

Lensless imaging is an important and challenging problem. One notable solution to lensless imaging is a single-pixel camera that benefits from ideas central to compressive sampling. However, traditional single-pixel cameras require many illumination patterns that result in a long acquisition process. Here, we present a method for lensless imaging based on compressive ultrafast sensing. Each sensor acquisition is encoded with a different illumination pattern and produces a time series where time is a function of the photon's origin in the scene. Currently available hardware with picosecond time resolution enables time tagging photons as they arrive to an omnidirectional sensor. This allows lensless imaging with significantly fewer patterns compared to regular single-pixel imaging. To that end, we develop a framework for designing lensless imaging systems that use ultrafast detectors. We provide an algorithm for ideal sensor placement and an algorithm for optimized active illumination patterns. We show that efficient lensless imaging is possible with ultrafast measurement and compressive sensing. This paves the way for novel imaging architectures and remote sensing in extreme situations where imaging with a lens is not possible.

Get PDF Email Share Share with Facebook Tweet This Post on reddit Share with LinkedIn Add to CiteULike Add to Mendeley Add to BibSonomy Get Citation Copy Citation Text G. Satat, B. Heshmat, and R. Raskar, "All Photons Imaging Through Thick Layered Scattering Materials," in Imaging and Applied Optics 2017 (3D, AIO, COSI, IS, MATH, pcAOP), OSA Technical Digest (online) (Optica Publishing Group, 2017), paper JTu5A.8. Export Citation BibTex Endnote (RIS) HTML Plain Text Citation alert Save article

We demonstrate an imaging technique that allows identification and classification of objects hidden behind scattering media and is invariant to changes in calibration parameters within a training range. Traditional techniques to image through scattering solve an inverse problem and are limited by the need to tune a forward model with multiple calibration parameters (like camera field of view, illumination position etc.). Instead of tuning a forward model and directly inverting the optical scattering, we use a data driven approach and leverage convolutional neural networks (CNN) to learn a model that is invariant to calibration parameters variations within the training range and nearly invariant beyond that. This effectively allows robust imaging through scattering conditions that is not sensitive to calibration. The CNN is trained with a large synthetic dataset generated with a Monte Carlo (MC) model that contains random realizations of major calibration parameters. The method is evaluated with a time-resolved camera and multiple experimental results are provided including pose estimation of a mannequin hidden behind a paper sheet with 23 correct classifications out of 30 tests in three poses (76.6% accuracy on real-world measurements). This approach paves the way towards real-time practical non line of sight (NLOS) imaging applications.

Ultrafast imaging has been a key enabler to many novel imaging modalities, including looking behind corners and imaging behind scattering layers. With picosecond time resolution and unconventional sensing geometries, ultrafast imaging can fundamentally impact sensing capabilities in industrial and biomedical applications. This paper reviews the fundamentals, recent advances, and the future prospects of ultrafast imaging-based modalities.

Between 7-18 million Americans suffer from sleep disordered breathing (SDB), including those who suffer from obstructive sleep apnea (OSA). Despite this high prevalence and burden of OSA, existing diagnostic techniques remain impractical for widespread screening. In this study, we introduce a new model for OSA screening and describe an at-home wearable sleep mask (named ARAM) that can robustly track the wearers' sleep patterns. This monitoring is achieved using select sensors that enable screening and monitoring in a form-factor that can be easily self-instrumented. Based on feedback from sleep doctors and technicians, we incorporate the most valuable sensors for OSA diagnosis, while maintaining ease-of-use and comfort for the patient. We discuss the results of preliminary field trials, where both our sleep mask and a commercially available device were worn simultaneously to evaluate our device's robustness. Based on these results, we discuss next steps for the design of the screening system, including analyses techniques that would provide more efficient screening than existing systems.

Asthma is the most common chronic illness among children. The skills required to diagnose it make it an even greater concern. In this work, we present a child-friendly wearable device, which allows in-home diagnosis of asthma. The device acquires simultaneous measurements from multiple stethoscopes. The recordings are then sent to a specialist who uses assistive diagnosis algorithms that enable auscultation (listening to lung sounds with a stethoscope) at any location in the lungs volume by sound refocusing. The specialist is also presented with a sound “heat map” which shows the location of sound sources in the lungs. We present design considerations of our device, as well as the algorithms for assistive diagnosis and their analysis which demonstrate reduction of ambient and measurement noise by over 10dB.

Imaging through thick highly scattering media (sample thickness ≫ mean free path) can realize broad applications in biomedical and industrial imaging as well as remote sensing. Here we propose a computational “All Photons Imaging” (API) framework that utilizes time-resolved measurement for imaging through thick volumetric scattering by using both early arrived (non-scattered) and diffused photons. As opposed to other methods which aim to lock on specific photons (coherent, ballistic, acoustically modulated, etc.), this framework aims to use all of the optical signal. Compared to conventional early photon measurements for imaging through a 15 mm tissue phantom, our method shows a two fold improvement in spatial resolution (4 db increase in Peak SNR). This all optical, calibration-free framework enables widefield imaging through thick turbid media and opens new avenues in non-invasive testing, analysis and diagnosis.

We demonstrate an endoscopic imaging system for microscopic flow measurement (for example blood flow in tissue) using laser speckle contrast imaging. Computational imaging is used to reject sub-surface scattering and eliminates erroneous flow speed estimation.