We introduce MyStyle, a personalized deep generative prior trained with a few shots of an individual. MyStyle allows to reconstruct, enhance and edit images of a specific person, such that the output is faithful to the person's key facial characteristics. Given a small reference set of portrait images of a person (~ 100), we tune the weights of a pretrained StyleGAN face generator to form a local, low-dimensional, personalized manifold in the latent space. We show that this manifold constitutes a personalized region that spans latent codes associated with diverse portrait images of the individual. Moreover, we demonstrate that we obtain a personalized generative prior, and propose a unified approach to apply it to various ill-posed image enhancement problems, such as inpainting and super-resolution, as well as semantic editing. Using the personalized generative prior we obtain outputs that exhibit high-fidelity to the input images and are also faithful to the key facial characteristics of the individual in the reference set. We demonstrate our method with fair-use images of numerous widely recognizable individuals for whom we have the prior knowledge for a qualitative evaluation of the expected outcome. We evaluate our approach against few-shots baselines and show that our personalized prior, quantitatively and qualitatively, outperforms state-of-the-art alternatives.

Although deep learning has enabled a huge leap forward in image inpainting, current methods are often unable to synthesize realistic high-frequency details. In this paper, we propose applying super-resolution to coarsely reconstructed outputs, refining them at high resolution, and then downscaling the output to the original resolution. By introducing high-resolution images to the refinement network, our framework is able to reconstruct finer details that are usually smoothed out due to spectral bias – the tendency of neural networks to reconstruct low frequencies better than high frequencies. To assist training the refinement network on large upscaled holes, we propose a progressive learning technique in which the size of the missing regions increases as training progresses. Our zoom-in, refine and zoom-out strategy, combined with high-resolution supervision and progressive learning, constitutes a framework-agnostic approach for enhancing high-frequency details that can be applied to any CNN-based inpainting method. We provide qualitative and quantitative evaluations along with an ablation analysis to show the effectiveness of our approach. This seemingly simple, yet powerful approach, outperforms existing inpainting methods.

We introduce MyStyle, a personalized deep generative prior trained with a few shots of an individual. MyStyle allows to reconstruct, enhance and edit images of a specific person, such that the output is faithful to the person's key facial characteristics. Given a small reference set of portrait images of a person (~100), we tune the weights of a pretrained StyleGAN face generator to form a local, low-dimensional, personalized manifold in the latent space. We show that this manifold constitutes a personalized region that spans latent codes associated with diverse portrait images of the individual. Moreover, we demonstrate that we obtain a personalized generative prior, and propose a unified approach to apply it to various ill-posed image enhancement problems, such as inpainting and super-resolution, as well as semantic editing. Using the personalized generative prior we obtain outputs that exhibit high-fidelity to the input images and are also faithful to the key facial characteristics of the individual in the reference set. We demonstrate our method with fair-use images of numerous widely recognizable individuals for whom we have the prior knowledge for a qualitative evaluation of the expected outcome. We evaluate our approach against few-shots baselines and show that our personalized prior, quantitatively and qualitatively, outperforms state-of-the-art alternatives.

Figure 1: Using our personalized prior forMichelle Obama, we solve inpainting, super-resolution, and semantic editing (smile), while faithfully preservingher key facial characteristic. Each example shows the original input image,whichmaybe corrupted (top left), and the output based on our personalized (right) and generic (bottom left) face priors. The generic face prior is learned from a diverse set of images and produces results that do not preserve Obama’s key facial characteristics. Left and middle blocks ©U.S. Government, right block ©Tim Pierce.

Nanoscale materials are routinely developed, characterized, and evaluated as diagnostic and therapeutic agents for disease treatment in humans. However, the size and composition of many such agents result in poor clearance profiles within biological tissues, thereby posing profound challenges to translational clinical applications. Herein, we present a hyperspectral imaging technique capable of quantifying plasmonic nanoparticle biodistribution with single particle limit of detection and microanatomical detail. Using this method, we find that, although intravenous administration of plasmonic nanoparticles remains largely infeasible from a biodistribution perspective, alternate routes relevant to treatment of oral and gastrointestinal diseases are within translational reach.

The sky is a major component of the appearance of a photograph, and its color and tone can strongly influence the mood of a picture. In nighttime photography, the sky can also suffer from noise and color artifacts. For this reason, there is a strong desire to process the sky in isolation from the rest of the scene to achieve an optimal look. In this work, we propose an automated method, which can run as a part of a camera pipeline, for creating accurate sky alpha-masks and using them to improve the appearance of the sky. Our method performs end-to-end sky optimization in less than half a second per image on a mobile device. We introduce a method for creating an accurate sky-mask dataset that is based on partially annotated images that are inpainted and refined by our modified weighted guided filter. We use this dataset to train a neural network for semantic sky segmentation. Due to the compute and power constraints of mobile devices, sky segmentation is performed at a low image resolution. Our modified weighted guided filter is used for edge-aware upsampling to resize the alpha-mask to a higher resolution. With this detailed mask we automatically apply post-processing steps to the sky in isolation, such as automatic spatially varying white-balance, brightness adjustments, contrast enhancement, and noise reduction.

Taking photographs in low light using a mobile phone is challenging and rarely produces pleasing results. Aside from the physical limits imposed by read noise and photon shot noise, these cameras are typically handheld, have small apertures and sensors, use mass-produced analog electronics that cannot easily be cooled, and are commonly used to photograph subjects that move, like children and pets. In this paper we describe a system for capturing clean, sharp, colorful photographs in light as low as 0.3 lux, where human vision becomes monochromatic and indistinct. To permit handheld photography without flash illumination, we capture, align, and combine multiple frames. Our system employs "motion metering", which uses an estimate of motion magnitudes (whether due to handshake or moving objects) to identify the number of frames and the per-frame exposure times that together minimize both noise and motion blur in a captured burst. We combine these frames using robust alignment and merging techniques that are specialized for high-noise imagery. To ensure accurate colors in such low light, we employ a learning-based auto white balancing algorithm. To prevent the photographs from looking like they were shot in daylight, we use tone mapping techniques inspired by illusionistic painting: increasing contrast, crushing shadows to black, and surrounding the scene with darkness. All of these processes are performed using the limited computational resources of a mobile device. Our system can be used by novice photographers to produce shareable pictures in a few seconds based on a single shutter press, even in environments so dim that humans cannot see clearly.

By their nature, tumors pose a set of profound challenges to the immune system with respect to cellular recognition and response coordination. Recent research indicates that leukocyte subpopulations, especially tumor-associated macrophages (TAMs), can exert substantial influence on the efficacy of various cancer immunotherapy treatment strategies. To better study and understand the roles of TAMs in determining immunotherapeutic outcomes, significant technical challenges associated with dynamically monitoring single cells of interest in relevant live animal models of solid tumors must be overcome. However, imaging techniques with the requisite combination of spatiotemporal resolution, cell-specific contrast, and sufficient signal-to-noise at increasing depths in tissue are exceedingly limited. Here we describe a method to enable high-resolution, wide-field, longitudinal imaging of TAMs based on speckle-modulating optical coherence tomography (SM-OCT) and spectral scattering from an optimized contrast agent. The approach's improvements to OCT detection sensitivity and noise reduction enabled high-resolution OCT-based observation of individual cells of a specific host lineage in live animals. We found that large gold nanorods (LGNRs) that exhibit a narrow-band, enhanced scattering cross-section can selectively label TAMs and activate microglia in an in vivo orthotopic murine model of glioblastoma multiforme. We demonstrated near real-time tracking of the migration of cells within these myeloid subpopulations. The intrinsic spatiotemporal resolution, imaging depth, and contrast sensitivity reported herein may facilitate detailed studies of the fundamental behaviors of TAMs and other leukocytes at the single-cell level in vivo, including intratumoral distribution heterogeneity and roles in modulating cancer proliferation.

Optical coherence tomography (OCT) with significant speckle reduction can be used with highly-scattering contrast agents for noninvasive, contrast-enhanced imaging of living tissue at the cellular scale. The advantages of reduced speckle noise and improved targeted contrast can be harnessed to track objects as small as 2 μm in vivo, with the potential for cell tracking and counting in living subjects. Here we demonstrate the use of Large Gold Nanorods (LGNRs) as contrast agents for detecting individual micron-sized polystyrene beads and single myeloma cells in blood circulation using speckle modulating-OCT (SM-OCT). This is the first time that OCT has been used to image at the individual cell scale in vivo. This technical capability presents an exciting opportunity for the dynamic detection and quantification of tumor cells circulating in living subjects.

Current in vivo neuroimaging techniques provide limited field of view or spatial resolution and often require exogenous contrast. These limitations prohibit detailed structural imaging across wide fields of view and hinder intraoperative tumor margin detection. Here we present a novel neuroimaging technique, speckle-modulating optical coherence tomography (SM-OCT), which allows us to image the brains of live mice and ex vivo human samples with unprecedented resolution and wide field of view using only endogenous contrast. The increased visibility provided by speckle elimination reveals white matter fascicles and cortical layer architecture in brains of live mice. To our knowledge, the data reported herein represents the highest resolution imaging of murine white matter structure achieved in vivo across a wide field of view of several millimeters. When applied to an orthotopic murine glioblastoma xenograft model, SM-OCT readily identifies brain tumor margins with resolution of approximately 10 μm. SM-OCT of ex vivo human temporal lobe tissue reveals fine structures including cortical layers and myelinated axons. Finally, when applied to an ex vivo sample of a low-grade glioma resection margin, SM-OCT is able to resolve the brain tumor margin. Based on these findings, SM-OCT represents a novel approach for intraoperative tumor margin detection and in vivo neuroimaging.

Current in vivo neuroimaging techniques provide limited field of view or spatial resolution and often require exogenous contrast. These limitations prohibit detailed structural imaging across wide fields of view and hinder intraoperative tumor margin detection. Here we present a novel neuroimaging technique, speckle-modulating optical coherence tomography (SM-OCT), which allows us to image the brains of live mice and ex vivo human samples with unprecedented resolution and wide field of view using only endogenous contrast. The increased effective resolution provided by speckle elimination reveals white matter fascicles and cortical layer architecture in the brains of live mice. To our knowledge, the data reported herein represents the highest resolution imaging of murine white matter structure achieved in vivo across a wide field of view of several millimeters. When applied to an orthotopic murine glioblastoma xenograft model, SM-OCT readily identifies brain tumor margins with near single-cell resolution. SM-OCT of ex vivo human temporal lobe tissue reveals fine structures including cortical layers and myelinated axons. Finally, when applied to an ex vivo sample of a low-grade glioma resection margin, SM-OCT is able to resolve the brain tumor margin. Based on these findings, SM-OCT represents a novel approach for intraoperative tumor margin detection and in vivo neuroimaging.

Optical coherence tomography angiography (OCTA) is an important tool for investigating vascular networks and microcirculation in living tissue. Traditional OCTA detects blood vessels via intravascular dynamic scattering signals derived from the movements of red blood cells (RBCs). However, the low hematocrit and long latency between RBCs in capillaries make these OCTA signals discontinuous, leading to incomplete mapping of the vascular networks. OCTA imaging of microvascular circulation is particularly challenging in tumors due to the abnormally slow blood flow in angiogenic tumor vessels and strong attenuation of light by tumor tissue. Here, we demonstrate in vivo that gold nanoprisms (GNPRs) can be used as OCT contrast agents working in the second near-infrared window, significantly enhancing the dynamic scattering signals in microvessels and improving the sensitivity of OCTA in skin tissue and melanoma tumors in live mice. With GNPRs as contrast agents, the postinjection OCT angiograms showed 41 and 59% more microvasculature than preinjection angiograms in healthy mouse skin and melanoma tumors, respectively. By enabling better characterization of microvascular circulation in vivo, GNPR-enhanced OCTA could lead to better understanding of vascular functions during pathological conditions, more accurate measurements of therapeutic response, and improved patient prognoses.

Speckle-Modulating Optical Coherence Tomography (SM-OCT) is a method based on light manipulation for removing speckle noise without significant loss of resolution. By removing speckle noise, SM-OCT reveals small structures in the tissues of living animals.

We measured the reduction of speckle by frequency compounding using Gaussian pulses, which have the least time-bandwidth product. The experimental results obtained from a tissue mimicking phantom agree quantitatively with numerical simulations of randomly distributed point scatterers. For a fixed axial resolution, the amount of speckle reduction is found to approach a maximum as the number of bands increases while the total spectral range that they cover is kept constant. An analytical solution of the maximal speckle reduction is derived and shows that the maximum improves approximately as the inverse square root of the Gaussian pulse bandwidth. Since the axial resolution is proportional to the inverse of the pulse bandwidth, an optimized trade-off between speckle reduction and axial resolution is obtained. Considerations for the applications of the optimized trade-off are discussed.

5-ALA: 5-aminolevulinic acid iMRI: intraoperative magnetic resonance imaging OCT: optical coherence tomography SM-OCT: Speckle-Modulating OCT LGNRs: large gold nanorods Maximizing extent of resection has been correlated with improved outcomes in a variety of pediatric and adult brain tumors.1-11 While gross total resections are often achieved without the use of intraoperative adjuncts, difficulty in distinguishing tumor from normal brain can at times prevent the complete resection of brain tumors. Intraoperative imaging tools have increasingly been embraced with the goal of improving rates of gross total resection. In this review, we will discuss several currently used intraoperative imaging modalities, including intraoperative magnetic resonance imaging (iMRI), wide-field fluorescence, high-resolution fluorescence microscopy, and optical coherence tomography (OCT).12-17 We will also discuss recently reported advances in OCT for brain tumor margin detection that we have developed in our lab. iMRI is utilized in many neurosurgical centers and has been shown to increase extent of resection in glioma surgery.13-16 MRI technology is reliable and generates images that are easily interpreted by neurosurgeons. iMRI images the entire brain and is excellent for detecting macroscopic regions of residual tumor. The inherent spatial resolution of MRI does limit the ability of iMRI to detect small areas of residual tumor. Additionally, acquiring iMRI images cannot be done without completely halting the operation and a significant outlay of time. Optical imaging strategies can be used in conjunction with iMRI and can be used continuously throughout an operation, addressing some of the limitations of iMRI. Fluorescence guided glioblastoma surgery with 5-aminolevulinic acid (5-ALA) is practiced commonly in many neurosurgical centers throughout the world and has been shown to increase extent of resection.6 This type of wide-field fluorescence guided neurosurgery is relatively easy to incorporate into a normal surgical work flow and can be used continuously throughout resection. Unfortunately, the use of wide-field fluorescence-guided brain tumor surgery has been largely limited to glioblastoma as a disrupted blood-brain barrier and for 5-ALA, specific biochemical alterations in the tumor, seem to be required for the successful use of this imaging strategy. This has precluded the use of wide-field fluorescence in a variety of low-grade glial and pediatric tumors. Confocal microscopy, an optical imaging technique that allows for increased resolution and contrast by utilizing a spatial pinhole to block out-of-focus light, has been applied to 5-ALA fluorescence imaging and does allow for successful fluorescence imaging in some low-grade gliomas.18 Similar imaging strategies utilizing confocal and multiphoton microscopy and only endogenous contrast have demonstrated an ability to resolve individual axons and brain tumor margins in an ex-vivo setting.19-23 The practicality of multiphoton and confocal microscopy for surgical use may be limited by their very limited field of view and depth of penetration as well as the requirement for contact and the use of high-energy light sources that may have some risk of tissue damage.24 OCT, an imaging technique based on low-coherence interferometry, allows for fast imaging across a large area with several millimeters of tissue penetration and micron-scale resolution while utilizing only endogenous contrast.25-27 OCT systems have been successfully integrated into neurosurgical endoscopes and operating microscopes and preliminary studies have demonstrated that OCT is capable of distinguishing tumor from normal brain based on structural features and signal attenuation.28-32 Quantitative OCT signal attenuation thresholds have been also been investigated as a potential way of rapidly differentiating tumor from white matter.33,34 OCT systems in these studies have been limited by speckle artifact, a feature intrinsic to OCT imaging, which reduces the effective resolution of this modality and limits the signal to noise ratio for accurate tumor margin detection.FIGURE: A and B, OCT and SM-OCT B-scans of a mouse cornea. Scale bar, 100 μm. C and D, Close-up view on the regions marked in A and B. In C, due to the high density of scatterers in this tissue, speckle noise is masking the inner structure of the stroma. Scale bar, 50 μm. E, Microscope image of H&E-stained mouse cornea at 10 × magnification. Scale bar, 100 μm. F and G, OCT and SM-OCT B-scans of a mouse retina. Scale bar, 100 μm. H and I, Close-up view on the regions marked in F and G. IP, inner plexiform; IN, inner nuclear layer; OP, outer plexiform layer; ON, outer nuclear layer; ELM, external limiting membrane; RPE, retinal pigment epithelium; CH, choroid. Scale bar, 50 μm.35We have recently developed a novel OCT-based imaging technique called Speckle-Modulating OCT (SM-OCT) that allows for the near complete elimination of speckle noise.35 When applied to in vivo ocular imaging (Figure), SM-OCT allowed for the visualization of the lamellar structure of the corneal stroma and improved visualization of the individual layers of the retina.35 The enhanced signal-to-noise ratio afforded by the elimination of speckle has also allowed us to explore large gold nanorods (LGNRs) as a novel spectral contrast agent for in vivo imaging. The combination of SM-OCT and LGNR contrast has allowed us to visualize the movement of individual macrophages across a wide field of view in orthotopic glioblastoma tumors in live mice.36 We have also applied SM-OCT to tumor margin detection and in vivo neuroimaging in a label-free setting.37 Excitingly, SM-OCT allows for wide-field neuroimaging with unprecedented resolution. Structures ranging in size from individual myelinated axons to small white matter fascicles and cortical layers are clearly visualized. In an orthotopic murine glioblastoma model, SM-OCT was capable of identifying the tumor margin with micron-scale resolution. Finally, when applied to an ex vivo human low-grade glioma sample, SM-OCT was capable of resolving the tumor margin.37 We feel that SM-OCT has great potential as an intraoperative imaging tool for tumor margin detection and will focus future experiments on translating this technology for intraoperative use. Disclosures Dr Zerda is a Chan Zuckerberg Biohub investigator and a Pew-Stewart Scholar for Cancer Research supported by The Pew Charitable Trusts and The Alexander and Margaret Stewart Trust. Dr Liba is supported by a Stanford Bowes Bio-X Graduate Fellowship. Dr Yecies received funding support for this publication provided by the Tashia and John Morgridge Endowed Postdoctoral Fellowship from the Child Health Research Institute at Lucille Packard Children's Hospital as well as the National Institute of Neurological Disorders and Stroke of the National Institutes of Health under Award Number R25NS065741. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. The authors have no personal, financial, or institutional interest in any of the drugs, materials, or devices described in this article.

Optical Coherence Tomography (OCT) imaging of living subjects offers increased depth of penetration while maintaining high spatial resolution when compared to other optical microscopy techniques. However, since most protein biomarkers do not exhibit inherent contrast detectable by OCT, exogenous contrast agents must be employed for imaging specific cellular biomarkers of interest. While a number of OCT contrast agents have been previously studied, demonstrations of molecular targeting with such agents in live animals have been historically challenging and notably limited in success. Here we demonstrate for the first time that microbeads (µBs) can be used as contrast agents to target cellular biomarkers in lymphatic vessels and can be detected by OCT using a phase variance algorithm. This molecular OCT method enables in vivo imaging of the expression profiles of lymphatic vessel endothelial hyaluronan receptor 1 (LYVE-1), a biomarker that plays crucial roles in inflammation and tumor metastasis. In vivo OCT imaging of LVYE-1 showed that the biomarker was significantly down-regulated during inflammation induced by acute contact hypersensitivity (CHS). Our work demonstrated a powerful molecular imaging tool that can be used for high resolution studies of lymphatic function and dynamics in models of inflammation, tumor development, and other lymphatic diseases.

Nanoparticles have been explored extensively as potential biomedical imaging and therapeutic agents. One critical aspect of in vivo nanoparticle use is the characterization of biodistribution profiles. Such studies improve our understanding of particle uptake, specificity, and clearance mechanisms. Currently, the most prevalent nanoparticle biodistribution methods provide either aspatial quantification of whole-organ particle accumulation or nanometer-resolution images of uptake in single cells. Few existing techniques are well-suited to study particle uptake on the micron to millimeter scales relevant to sub-tissue physiology. Here we demonstrate a new method called Hyperspectral Microscopy with Adaptive Detection (HSM-AD) that uses machine learning classification of hyperspectral dark-field images to study interactions between tissues and administered nanoparticles. This label-free, non-destructive method enables quantitative particle identification in histological sections and detailed observations of sub-organ accumulation patterns consistent with organ-specific clearance mechanisms, particle size, and the molecular specificity of the nanoparticle surface. Unlike studies with electron microscopy, HSM-AD is readily applied for large fields of view. HSM-AD achieves excellent detection sensitivity (99.4%) and specificity (99.7%) and can identify single nanoparticles. To demonstrate HSM-AD's potential for novel nanoparticle uptake studies, we collected the first data on the sub-organ localization of large gold nanorods (LGNRs) in mice. We also observed differences in particle accumulation and localization patterns in tumors as a function of conjugated molecular targeting moieties. Thus, HSM-AD affords new degrees of detail for the study of nanoparticle uptake at physiological scales. HSM-AD may offer an auxiliary or alternative approach to study the biodistribution profiles of existing and novel nanoparticles.

Optical coherence tomography (OCT) is a powerful biomedical imaging technology that relies on the coherent detection of backscattered light to image tissue morphology in vivo. As a consequence, OCT is susceptible to coherent noise (speckle noise), which imposes significant limitations on its diagnostic capabilities. Here we show speckle-modulating OCT (SM-OCT), a method based purely on light manipulation that virtually eliminates speckle noise originating from a sample. SM-OCT accomplishes this by creating and averaging an unlimited number of scans with uncorrelated speckle patterns without compromising spatial resolution. Using SM-OCT, we reveal small structures in the tissues of living animals, such as the inner stromal structure of a live mouse cornea, the fine structures inside the mouse pinna, and sweat ducts and Meissner's corpuscle in the human fingertip skin-features that are otherwise obscured by speckle noise when using conventional OCT or OCT with current state of the art speckle reduction methods.

Yecies, Derek W MD; Liba, Orly; SoRelle, Elliot; Dutta, Rebecca; Wilson, Christy; Grant, Gerald A MD; de la Zerda, Adam

Leukocyte populations, especially tumor-associated macrophages (TAMs), are capable of mediating both anti- and pro-tumor processes and play significant roles in the tumor microenvironment. Moreover, TAMs have been shown to exert substantial influence on the efficacy of various cancer immunotherapy treatment strategies. Laboratory investigation into the behavior of TAMs has been limited by a lack of methods capable of resolving the in vivo distribution and dynamics of this cell population across wide fields of view. Recent studies have employed magnetic resonance imaging and intravital microscopy in conjunction with nanoparticle labeling methods to detect TAMs and observe their responses to therapeutic agents. Here we describe a novel method to enable high-resolution, wide-field, longitudinal imaging of leukocytes based on contrast-enhanced Speckle-Modulating Optical Coherence Tomography (SM-OCT), which substantially reduces imaging noise. We were able to specifically label TAMs and activated microglia in vivo with large gold nanorod contrast agents (LGNRs) in an orthotopic murine glioblastoma model. After labeling, we demonstrated near real-time tracking of leukocyte migration and distribution within the tumors. The intrinsic resolution, imaging depth, and sensitivity of this method may facilitate detailed studies of the fundamental behaviors of TAMs in vivo , including their intratumoral distribution heterogeneity and the roles they play in modulating cancer proliferation. In future studies, the method described herein may also provide the necessary means to characterize TAM responses to immunotherapeutic regimens in a range of solid tumors.