In this work we address the problem of estimating 3D human pose from a single RGB image by blending a feed-forward CNN with a graphical model that couples the 3D positions of parts. The CNN populates a volumetric output space that represents the possible positions of 3D human joints, and also regresses the estimated displacements between pairs of parts. These constitute the ‘unary’ and ‘pairwise’ terms of the energy of a graphical model that resides in a 3D label space and delivers an optimal 3D pose configuration at its output. The CNN is trained on the 3D human pose dataset 3.6M, the graphical model is trained jointly with the CNN in an end-to-end manner, allowing us to exploit both the discriminative power of CNNs and the top-down information pertaining to human pose. We introduce (a) memory efficient methods for getting accurate voxel estimates for parts by blending quantization with regression (b) employ efficient structured prediction algorithms for 3D pose estimation using branch-and-bound and (c) develop a framework for qualitative and quantitative comparison of competing graphical models. We evaluate our work on the Human3.6M dataset, demonstrating that exploiting the structure of the human pose in 3D yields systematic gains.
In this paper, we propose a novel method for tracing object boundaries automatically based on a method called “PointFlow” in image induced vector fields. The PointFlow method comprises two steps: edge detection and edge integration. Basically, it uses an ordinary differential equation for describing the movement of points under the action of an image-induced vector field and generates induced trajectories. The trajectories of the flows allow to find and integrate edges and determine object boundaries. We also extend the original PointFlow method to make it adaptable to images with complicated scenes. In addition, the PointFlow method can be applied to infer certain illusory contours. We test our method on real image dataset. Compared with the other classical edge detection and integration models, our PointFlow method is better at providing precise and continuous curves. The experimental results clearly exhibit the robustness and effectiveness of the proposed method.
Current convolutional neural networks algorithms for video object tracking spend the same amount of computation for each object and video frame [3]. However, it is harder to track an object in some frames than others, due to the varying amount of clutter, scene complexity, amount of motion, and object’s distinctiveness against its background. We propose a depth-adaptive convolutional siamese network that performs video tracking adaptively at multiple neural network depths. Parametric gating functions are trained to control the depth of the convolutional feature extractor by minimizing a joint loss of computational cost and tracking error. Our network achieves accuracy comparable to the state-of-the-art on the VOT2016 benchmark. Furthermore, our adaptive depth computation achieves higher accuracy for a given computational cost than traditional fixed-structure neural networks. The presented framework extends to other tasks that use convolutional neural networks and enables trading speed for accuracy at runtime.
Optimization in the wavelet domain has been a very prominent research topic both for denoising, as well as compression, reflected in its use in the JPEG-2000 standard. Its performance depends to a great extent on the wavelet ψ itself, represented in the form of a filter in the case of the discrete wavelet transform. While other works solely optimize the coefficients in the wavelet domain, we will use a combined approach, optimizing the wavelet ψ and the coefficients simultaneously in order to adapt both to a given image, resulting in a better reconstruction of an image from less coefficients. We will use several orthonormal wavelet bases as a starting point, but we will also demonstrate that we can create wavelets from white Gaussian noise with our approach, which are in some cases even better in terms of performance. Experiments will be conducted on several images, demonstrating how the optimization algorithm adapts to textured, as well as more homogeneous images.
The optical flow within a scene can be an arbitrarily complex composition of motion patterns that typically differ regarding their scale. Hence, using a single algorithm with a single set of parameters is often not sufficient to capture the variety of these motion patterns. In particular, the estimation of large displacements of small objects poses a problem. In order to cope with this problem, many recent methods estimate the optical flow by a fusion of flow candidates obtained either from different algorithms or from the same algorithm using different parameters. This, however, typically results in a pipeline of methods for estimating and fusing the candidate flows, each requiring an individual model with a dedicated solution strategy. In this paper, we investigate what results can be achieved with a pure variational approach based on a standard coarse-to-fine optimization. To this end, we propose a novel variational method for the simultaneous estimation and fusion of flow candidates. By jointly using multiple smoothness weights within a single energy functional, we are able to capture different motion patterns and hence to estimate large displacements even without additional feature matches. In the same functional, an intrinsic model-based fusion allows to integrate all these candidates into a single flow field, combining sufficiently smooth overall motion with locally large displacements. Experiments on large displacement sequences and the Sintel benchmark demonstrate the feasibility of our approach and show improved results compared to a single-smoothness baseline method.
We propose an inverse rendering model for light fields to recover surface normals, depth, reflectance and natural illumination. Our setting is fully uncalibrated, with the reflectance modeled with a spatially-constant Blinn-Phong model and illumination as an environment map. While previous work makes strong assumptions in this difficult scenario, focusing solely on specific types of objects like faces or imposing very strong priors, our approach leverages only the light field structure, where a solution consistent across all subaperture views is sought. The optimization is based primarily on shading, which is sensitive to fine geometric details which are propagated to the initial coarse depth map. Despite the problem being inherently ill-posed, we achieve encouraging results on synthetic as well as real-world data.
Compressed sensing is a powerful mathematical modelling tool to recover sparse signals from undersampled measurements in many applications, including medical imaging. A large body of work investigates the case with linear measurements, while compressed sensing with nonlinear measurements has been considered more recently. We continue this line of investigation by considering a novel type of nonlinearity with special structure that occurs in data acquired by multi-emitter X-ray tomosynthesis systems with spatio-temporal overlap. In [15] we proposed a nonlinear optimization model to deconvolve the overlapping measurements. In this paper we propose a model that exploits the structure of the nonlinearity and a nonlinear tomosynthesis algorithm that has a practical running time of solving only two linear subproblems at the equivalent resolution. We underpin and justify the algorithm by deriving RIP bounds for the linear subproblems and conclude with numerical experiments that validate the approach.
This paper proposes using both spatial and spectral regularizers/priors for hyperspectral image sharpening. Leveraging the recent plug-and-play framework, we plug two Gaussian-mixture-based denoisers into the iterations of an alternating direction method of multipliers (ADMM): a spatial regularizer learned from the observed multispectral image, and a spectral regularizer trained using the hyperspectral data. The proposed approach achieves very competitive results, improving the performance over using a single regularizer. Furthermore, the spectral regularizer can be used to classify the image pixels, opening the door to class-adapted models.
In this work, we focus on the problem of intrinsic scene decomposition in light fields. Our main contribution is a novel prior to cope with cast shadows and inter-reflections. In contrast to other approaches which model inter-reflection based only on geometry, we model indirect shading by combining geometric and color information. We compute a shadow confidence measure for the light field and use it in the regularization constraints. Another contribution is an improved specularity estimation by using color information from sub-aperture views. The new priors are embedded in a recent framework to decompose the input light field into albedo, shading, and specularity. We arrive at a variational model where we regularize albedo and the two shading components on epipolar plane images, encouraging them to be consistent across all sub-aperture views. Our method is evaluated on ground truth synthetic datasets and real world light fields. We outperform both state-of-the art approaches for RGB+D images and recent methods proposed for light fields.
We consider the task of learning a classifier for semantic segmentation using weak supervision in the form of image labels which specify the object classes present in the image. Our method uses deep convolutional neural networks (CNNs) and adopts an Expectation-Maximization (EM) based approach. We focus on the following three aspects of EM: (i) initialization; (ii) latent posterior estimation (E-step) and (iii) the parameter update (M-step). We show that saliency and attention maps, our bottom-up and top-down cues respectively, of simple images provide very good cues to learn an initialization for the EM-based algorithm. Intuitively, we show that before trying to learn to segment complex images, it is much easier and highly effective to first learn to segment a set of simple images and then move towards the complex ones. Next, in order to update the parameters, we propose minimizing the combination of the standard softmax loss and the KL divergence between the true latent posterior and the likelihood given by the CNN. We argue that this combination is more robust to wrong predictions made by the expectation step of the EM method. We support this argument with empirical and visual results. Extensive experiments and discussions show that: (i) our method is very simple and intuitive; (ii) requires only image-level labels; and (iii) consistently outperforms other weakly-supervised state-of-the-art methods with a very high margin on the PASCAL VOC 2012 dataset.
Segmenting moving objects in a video sequence has been a challenging problem and critical to outdoor robotic navigation. While recent literature has laid focus on regularizing object labels over a sequence of frames, exploiting the spatio-temporal features for motion segmentation has been scarce. Particularly in real world dynamic scenes, existing approaches fail to exploit temporal consistency in segmenting moving objects with large camera motion.In this paper, we present an approach for exploiting semantic information and temporal constraints in a joint framework for motion segmentation in a video. We propose a formulation for inferring per-frame joint semantic and motion labels using semantic potentials from dilated CNN framework and motion potentials from depth and geometric constraints. We integrate the potentials obtained into a 3D (space-time) fully connected CRF framework with overlapping/connected blocks. We solve for a feature space embedding in the spatio-temporal space by enforcing temporal constraints using optical flow and long term tracks as a least-squares problem. We evaluate our approach on outdoor driving benchmarks - KITTI and Cityscapes dataset.
Existing adiabatic quantum computers are tailored towards minimizing the energies of Ising models. The quest for implementations of pattern recognition or machine learning algorithms on such devices can thus be seen as the quest for Ising model (re-)formulations of their objective functions. In this paper, we present Ising models for the tasks of binary clustering of numerical and relational data and discuss how to set up corresponding quantum registers and Hamiltonian operators. In simulation experiments, we numerically solve the respective Schrödinger equations and observe our approaches to yield convincing results.
Convexity is known as an important cue in human vision and has been recently proposed as a shape prior for segmenting a single foreground object. We propose a mutli-object convexity shape prior for multilabel image segmentation. We formulate a novel multilabel discrete energy function. To optimize our energy, we extend the trust region optimization framework recently proposed in the context of binary optimization. To that end we develop a novel graph construction. In addition to convexity constraints, our model includes $$L^1$$ color separation term between the background and the foreground objects. It can also incorporate any other multilabel submodular energy term. Our formulation can be used to segment multiple convex objects sharing the same appearance model, or objects consisting of multiple convex parts. Our experiments demonstrate general usefulness of the proposed convexity constraint on real image segmentation examples.
In this paper we establish a link between diffraction theory and graph characterization through the Schrödinger operator. This provides a natural way of characterizing wave propagation on a graph. In order to do so, we compute the spatio-temporal Fourier transform of the operator and then pack its spherical representation in a point of a Stiefel manifold. We show that when the temporal interval of analysis is set according to quantum efficiency principles the proposed approach outperforms the alternatives in graph discrimination.
The Mumford-Shah model for image formation is an important, but also difficult energy functional. In this work we focus on several approaches based on convex relaxation operating on a discretized image domain. Existing methods typically use discretized intensity labels, but in this work we propose to retain the continuous label structure. To this end we employ a recently proposed framework for a new convex relaxation of the Mumford-Shah functional. Numerical results illustrate the performance of the various approaches.
A numerical solution to shape-from-shading under natural illumination is presented. It builds upon an augmented Lagrangian approach for solving a generic PDE-based shape-from-shading model which handles directional or spherical harmonic lighting, orthographic or perspective projection, and greylevel or multi-channel images. Real-world applications to shading-aware depth map denoising, refinement and completion are presented.
In this paper, we introduce a generalized asymmetric fronts propagation model based on the geodesic distance maps and the Eikonal partial differential equations. One of the key ingredients for the computation of the geodesic distance map is the geodesic metric, which can govern the action of the geodesic distance level set propagation. We consider a Finsler metric with the Randers form, through which the asymmetry and anisotropy enhancements can be taken into account to prevent the fronts leaking problem during the fronts propagation. These enhancements can be derived from the image edge-dependent vector field such as the gradient vector flow. The numerical implementations are carried out by the Finsler variant of the fast marching method, leading to very efficient interactive segmentation schemes.
The inverse problem of backward diffusion is known to be ill-posed and highly unstable. Backward diffusion processes appear naturally in image enhancement and deblurring applications. It is therefore greatly desirable to establish a backward diffusion model which implements a smart stabilisation approach that can be used in combination with an easy-to-handle numerical scheme. So far, existing stabilisation strategies in the literature require sophisticated numerics to solve the underlying initial value problem. We derive a class of space-discrete one-dimensional backward diffusion as gradient descent of energies where we gain stability by imposing range constraints. Interestingly, these energies are even convex. Furthermore, we establish a comprehensive theory for the time-continuous evolution and we show that stability carries over to a simple explicit time discretisation of our model. Finally, we confirm the stability and usefulness of our technique in experiments in which we enhance the contrast of digital greyscale and colour images.
In this work we express resource-efficient MAP inference as joint optimization problem w.r.t. (i) messages (i.e. reparametrizations) and (ii) surrogate potentials that are upper bounds for the problem of interest and allow efficient inference. We show that resulting nested optimization task can be solved on trees by a convergent and efficient algorithm, and that its loopy extension also returns convincing MAP solutions in practice. We demonstrate the utility of the method on dense correspondence and image completion problems.
Resolving ambiguities is a fundamental problem in shape from shading (SFS). The classic SFS approach allows to reconstruct the surface locally around singular points up to an ambiguity of convex, concave or saddle point type. In this paper we follow a recent approach that seeks to resolve the local ambiguities in a global graph-based setting so that the complete surface reconstruction is consistent. To this end, we introduce a novel graph theoretic formulation for the underlying problem that allows to prove for the first time in the literature that the underlying surface orientation problem is 𝒩𝒫 -complete. Moreover, we show that our novel framework allows to define an algorithmic framework that solves the disambiguation problem. It makes use of cycle bases for dealing with the graph construction and enables an easy embedding into an optimization method that amounts here to a linear program.
