Wearable vibrotactile devices have many potential applications, including sensory substitution for accessibility and notifications. Currently, vibrotactile experimentation is done using large lab setups. However, most practical applications require standalone on-body devices and integration into small form factors. Such integration is time-consuming and requires expertise. With a goal to democratize wearable haptics we introduce VHP, a vibrotactile haptics platform. It includes a low-power miniature electronics board that can drive up to 12 independent channels of haptic signals with arbitrary waveforms at a 2 kHz sampling rate. The platform can drive vibrotactile actuators including linear resonant actuators and voice coils. The control hardware is battery-powered and programmable, and has multiple input options, including serial and Bluetooth, as well as the ability to synthesize haptic signals internally. We developed current-based loading sensing, thus allowing for unique features such as actuator auto-classification, and skin-contact quality sensing. Our technical evaluations showed that the system met all our initial design criteria and is an improvement over prior methods as it allows all-day wear, has low latency, has battery life between 3 and 25 hours, and can run 12 actuators simultaneously. We demonstrate unique applications that would be time-consuming to develop without the VHP platform. We show that VHP can be used as bracelet, sleeve and phone-case form factors. The bracelet was programmed with an audio-to-tactile interface and was successfully worn for multiple days over months by developers. To facilitate more use of this platform, we open-source our design and plan to make the hardware widely available. We hope this work will motivate the use and study of vibrotactile all-day wearable devices.

Partial differential equations (PDEs) are typically used as models of physical processes but are also of great interest in PDE-based image processing. However, when it comes to their use in imaging, conventional numerical methods for solving PDEs tend to require very fine grid resolution for stability, and as a result have impractically high computational cost. This work applies BLADE (Best Linear Adaptive Enhancement), a shallow learnable filtering framework, to PDE solving, and shows that the resulting approach is efficient and accurate, operating more reliably at coarse grid resolutions than classical methods. As such, the model can be flexibly used for a wide variety of problems in imaging.

We present a framework for interactive design of new image stylizations using a wide range of predefined filter blocks. Both novel and off-the-shelf image filtering and rendering techniques are extended and combined to allow the user to unleash their creativity to intuitively invent, modify, and tune new styles from a given set of filters. In parallel to this manual design, we propose a novel procedural approach that automatically assembles sequences of filters, leading to unique and novel styles. An important aim of our framework is to allow for interactive exploration and design, as well as to enable videos and camera streams to be stylized on the fly. In order to achieve this real-time performance, we use the \textit{Best Linear Adaptive Enhancement} (BLADE) framework -- an interpretable shallow machine learning method that simulates complex filter blocks in real time. Our representative results include over a dozen styles designed using our interactive tool, a set of styles created procedurally, and new filters trained with our BLADE approach.

In this paper, we broadly connect kernel-based filtering (e.g., approaches such as the bilateral filter and non-local means, but also many more) with general variational formulations of Bayesian regularized least squares and the related concept of proximal operators. Variational/Bayesian/proximal formulations often result in optimization problems that do not have closed-form solutions and therefore typically require global iterative solutions. Our main contribution here is to establish how one can approximate the solution of the resulting global optimization problems using locally adaptive filters with specific kernels. Our results are valid for small regularization strength (i.e., weak noise), but the approach is powerful enough to be useful for a wide range of applications because we expose how to derive a “kernelized” solution to these problems that approximates the global solution in one shot, using only local operations. As another side benefit in the reverse direction, given a local data-adaptive filter constructed with a particular choice of kernel, we enable the interpretation of such filters in the variational/Bayesian/proximal framework.

Denoising is a fundamental imaging problem. Versatile but fast filtering has been demanded for mobile camera systems. We present an approach to multiscale filtering which allows real-time applications on low-powered devices. The key idea is to learn a set of kernels that upscales, filters, and blends patches of different scales guided by local structure analysis. This approach is trainable so that learned filters are capable of treating diverse noise patterns and artifacts. Experimental results show that the presented approach produces comparable results to state-of-the-art algorithms while processing time is orders of magnitude faster.

The Rapid and Accurate Image Super Resolution (RAISR) method of Romano, Isidoro, and Milanfar is a computationally efficient image upscaling method using a trained set of filters. We describe a generalization of RAISR, which we name Best Linear Adaptive Enhancement (BLADE). This approach is a trainable edge-adaptive filtering framework that is general, simple, computationally efficient, and useful for a wide range of problems in computational photography. We show applications to operations which may appear in a camera pipeline including denoising, demosaicking, and stylization.

We have implemented a near-ultrasonic communication protocol in the 18.5-20 kHz band, which is inaudible to most humans, using commodity smartphone speakers and microphones to transmit and receive signals. The protocol described in this paper is a component of Google's Nearby platform, where near-ultrasound signals are used to establish copresence between nearby devices by transmitting a short token. High-frequency sound does not pass through walls (most energy is reflected), so identified devices are constrained to approximately the same room, "within earshot" of one another. Our protocol has a raw data rate of 94.5 b/s, and we find in real indoor environments that transmission between mobile devices is reliable at 2 m distance and often works at 10 m. We use direct-sequence spread spectrum modulation, which makes it highly robust to multipath, motion, and narrowband noise. We use a 127-chip pseudorandom code, repeating once per data symbol, and modulate its amplitude with orthogonal sine waveforms encoding 4-bit symbol values. We add the orthogonal sines to a constant "pedestal," which is inefficient in an information-theoretic sense, but makes synchronization easier. We describe a robust and computationally efficient transmitter and receiver implementations and show experiments on real and simulated data.

We present a system to convert any set of images (e.g., a video clip or a photo album) into a storyboard. We aim to create multiple pleasing graphic representations of the content at interactive rates, so the user can explore and find the storyboard (images, layout, and stylization) that best suits their needs and taste. The main challenges of this work are: selecting the content images, placing them into panels, and applying a stylization. For the latter, we propose an interactive design tool to create new stylizations using a wide range of filter blocks. This approach unleashes the creativity by allowing the user to tune, modify, and intuitively design new sequences of filters. In parallel to this manual design, we propose a novel procedural approach that automatically assembles sequences of filters for innovative results. We aim to keep the algorithm complexity as low as possible such that it can run interactively on a mobile device. Our results include examples of styles designed using both our interactive and procedural tools, as well as their final composition into interesting and appealing storyboards.

Robust and far-field speech recognition is critical to enable true hands-free communication. In far-field conditions, signals are attenuated due to distance. To improve robustness to loudness variation, we introduce a novel frontend called per-channel energy normalization (PCEN). The key ingredient of PCEN is the use of an automatic gain control based dynamic compression to replace the widely used static (such as log or root) compression. We evaluate PCEN on the keyword spotting task. On our large rerecorded noisy and far-field eval sets, we show that PCEN significantly improves recognition performance. Furthermore, we model PCEN as neural network layers and optimize high-dimensional PCEN parameters jointly with the keyword spotting acoustic model. The trained PCEN frontend demonstrates significant further improvements without increasing model complexity or inference-time cost.

The Rapid and Accurate Image Super Resolution (RAISR) method of Romano, Isidoro, and Milanfar is a computationally efficient image upscaling method using a trained set of filters. We describe a generalization of RAISR, which we name Best Linear Adaptive Enhancement (BLADE). This approach is a trainable edge-adaptive filtering framework that is general, simple, computationally efficient, and useful for a wide range of image processing problems. We show applications to denoising, compression artifact removal, demosaicing, and approximation of anisotropic diffusion equations.

Total variation regularization is usually applied with L 2 data fidelity assuming an additive white Gaussian noise model [24, 25]. However, better results are possible when the noise model accurately describes the noise distribution in the given image. Total variation denoising has already been developed with the Laplace noise model [2, 12] ( L 1 data fidelity) and the Poisson noise model [19, 26]. In this paper, we develop two variational methods for total variation regularization using the Rician noise model. Our first model uses the Rician probability distribution function directly in the fidelity term, leading to a slightly nonconvex objective function. We give existence and comparison results for this model and investigate the validity of the model on three dimensional synthetic magnetic resonance imaging (MRI) data corrupted with Rician noise and Gaussian blur, both with known standard deviations. In this case, the numerical solutions are obtained using the L 2 and Sobolev H 1 gradient descent methods. In our second model, we develop a variational method for total variation regularization that closely approximates the Rician noise model. A key challenge is to find a convex approximation of the slightly nonconvex objective function. The variational problem is efficiently solved using the split Bregman method [16], and numerical examples are performed on the MRI data described above. Comparison of both models is presented.

We present a method of characterising tree networks based on a structural triangulation of those networks. Each component triangle is assigned a generation number which reflects the distance of that component from the origin of the network. By interpreting the generation number as an energy level, we can associate a partition function with a tree network which, in terms of the usual statistical thermodynamic interpretation, enables the determination of the internal energy and entropy of the triangulation. These thermodynamic functions depend on a parameter analogous to an inverse temperature that assigns weights to different parts of the network based on the generation numbers of the triangular elements. The systematic variation of these weights permits an examination of the development of the network, from the initial stages at low temperatures, where lower generation numbers have the greatest weight, to the complete network at high temperature, where all generation numbers have similar weights. After working through several examples to illustrate our methodology, we analyze the arterial and venous vasculature of the chorionic plate of 13 human placentas. We attempt to examine the extent to which the entropy function is correlated to the infant birthweight with the sample set. A correlation is postulated as a key factor in determining lifelong health.

Gaussian convolution is a common operation and building block for algorithms in signal and image processing. Consequently, its efficient computation is important, and many fast approximations have been proposed. In this survey, we discuss approximate Gaussian convolution based on finite impulse response filters, DFT and DCT based convolution, box filters, and several recursive filters. Since boundary handling is sometimes overlooked in the original works, we pay particular attention to develop it here. We perform numerical experiments to compare the speed and quality of the algorithms.

Given an image where a specified region is unknown, image inpainting or image completion is the problem of inferring the image content in this region.Traditional retouching or inpainting is the practice of restoring aged artwork, where damaged or missing portions are repainted based on the surrounding content to approximate the original appearance.In the context of digital images, inpainting is used to restore regions of an image that are corrupted by noise or where the data is missing.Inpainting is also used to solve disocclusion, to estimate the scene behind an obscuring foreground object.A popular use of digital inpainting is object removal, for example, to remove a trashcan that disrupts a scene of otherwise natural beauty.Inpainting is an interpolation problem, filling the unknown region with a condition to agree with the known image on the boundary.A classical solution for such an interpolation is to solve Laplace's equation.However, Laplace's equation is usually unsatisfactory for images since it is overly smooth.It cannot recover a step edge passing through the region.Total variation (TV) regularization is an effective inpainting technique which is capable of recovering sharp edges under some conditions (these conditions will be explained).The use of TV regularization was originally developed for image denoising by Rudin, Osher, and Fatemi [3] and then applied to inpainting by Chan and Shen [13].TV-regularized inpainting does not create texture, the method is limited to inpainting the geometric structure.

While many segmentation methods rely heavily in some way on edge detection, the "Active Contours Without Edges" method by Chan and Vese [7, 9] ignores edges completely. Instead, the method optimally fits a two-phase piecewise constant model to the given image. The segmentation boundary is represented implicitly with a level set function, which allows the segmentation to handle topological changes more easily than explicit snake methods. This article describes the level set formulation of the Chan-Vese model and its numerical solution using a semi-implicit gradient descent. We also discuss the Chan-Sandberg-Vese method [8], a straightforward extension of Chan-Vese for vector-valued images.

Deblurring is the inverse problem of restoring an image that has been blurred and possibly corrupted with noise. Deconvolution refers to the case where the blur to be removed is linear and shift-invariant so it may be expressed as a convolution of the image with a point spread function. Convolution corresponds in the Fourier domain to multiplication, and deconvolution is essentially Fourier division. The challenge is that since the multipliers are often small for high frequencies, direct division is unstable and plagued by noise present in the input image. Effective deconvolution requires a balance between frequency recovery and noise suppression. Total variation (TV) regularization is a successful technique for achieving this balance in deblurring problems. It was originally developed for image denoising by Rudin, Osher, and Fatemi and then applied to deconvolution by Rudin and Osher. In this article, we discuss TV-regularized deconvolution with Gaussian noise and its efficient solution using the split Bregman algorithm of Goldstein and Osher. We show a straightforward extension for Laplace or Poisson noise and develop empirical estimates for the optimal value of the regularization parameter λ.

Automatic Color Enhancement "ACE" is an effective method for color image enhancement introduced by Gatta, Rizzi, and Marini based on modeling several low level mechanisms of the human visual system. The direct computation of ACE on an N x N image costs O(N-4) operations. This article describes two fast approximations of ACE. First, the algorithm of Bertalmio, Caselles, Provenzi, and Rizzi uses a polynomial approximation of the slope function to decomposes the main computation into convolutions, reducing the cost to O(N-2 log N). Second, an algorithm based on interpolating intensity levels also reduces the main computation to convolutions. The use of ACE for image enhancement and color correction is demonstrated.