Traditional narrow collision pipelines for rigid body dynamics typically consist of calculation of collisions between primitives with a dedicated deepest point function for each primitive pair and a much slower general collision calculation between a pair of convex shapes. The mixture of different collision functions makes these methods not CPU-cache friendly and unsuitable for modern GPU hardware. For many reinforcement learning applications, data transfers become the bottle-neck of the whole pipeline. To get a stable behavior of rigid bodies even on a flat ground without jittering, it is necessary to use multiple contact points for each pair of objects. This is achieved by either keeping track of old collision points, or by adding noise to the collision pair. Using the first approach we get a full set of collision points only after multiple frames, using the second, the computational cost largely increases by a factor equal to the number of samples. In this project, we propose to tackle these issues by introducing: a) the quadric decomposition - a data representation for both primitives and general shapes, describing each object as a union of the intersection of planar and quadric inequalities, b) calculation of the deepest point as a maximization of depth over a set of quadric inequalities, c) prediction of the desired number of contact points as a maximization of the surface of the intersection polygon, d) Recurrent neural network (RNN) based solver to speed up the optimization process.

Predictive variability due to data ambiguities has typically been addressed via construction dedicated models with built-in probabilistic capabilities that are trained to predict uncertainty estimates as variables of interest. These approaches require distinct architectural components and training mechanisms, may include restrictive assumptions and exhibit overconfidence, i.e., high confidence in imprecise predictions. In this work, we propose a post-hoc sampling strategy for estimating predictive uncertainty accounting for data ambiguity. The method can generate different plausible outputs for a given input and does not assume parametric forms of predictive distributions. It is architecture agnostic and can be applied to any feed-forward deterministic network without changes to the architecture nor training procedure. Experiments on regression tasks on imaging and non-imaging input data show the method’s ability to generate diverse and multi-modal predictive distributions and how estimated uncertainty correlates with prediction error.

Recent progress in consumer hardware allowed for the collection of a large amount of animated point cloud data, which is on the one hand highly redundant and on the other hand incomplete. Our goal is to bridge this gap and find a low dimensional representation capable of approximation to a desired precision and completion of missing data. Model-less non-rigid 3D reconstruction algorithms, formulated as a linear factorization of observed point tracks into static shape component and dynamic pose, have been found insufficient to create suitable generative models, capable of generating new unobserved poses. This is due to the non-locality of the linear models, over-fitting to the non-causal correlations present in the data, which manifests in the reconstruction containing rigidly behaving not directly connected parts. In this paper, we propose a new method that can distinguish body parts and factorize the data into shape and pose purely using topological properties of the manifold-local deformations and neighborhoods. To obtain localized factorization, we formulate the deformation distance between two point tracks as the smallest deformation along the path between them. After embedding such distance in low dimensional space, a clustering of embedded data leads to close to rigid components, suitable as initialization for fitting a model-a skinned rigged mesh, used extensively in computer graphics. As both local deformations and neighborhoods of a point are local and can be estimated only from the part of the animation, the method can be used to recover unobserved data in each frame.

Decomposing scenes into reflectance and lighting is an important task for applications such as relighting, image matching or content creation. Advanced light transport effects like occlusion and indirect lighting are often ignored, leading to subpar decompositions in which the albedo needs to compensate for insufficiencies in the estimated shading. We show how to account for these advanced lighting effects by utilizing precomputed radiance transfer to estimate reflectance and lighting. Given the geometry of an object and one or multiple images, our method reconstructs the object's surface reflectance properties-such as its albedo and glossiness-as well as a colored lighting environment map. Evaluation on synthetic and real data shows that incorporation of indirect light leads to qualitatively and quantitatively improved results.

Many geometric quantities can be computed efficiently for convex meshes. For general meshes, methods for approximate convex decomposition have been developed that decompose a static, non-convex object into a small set of approximately convex parts. The convex hulls of those parts can then be used as a piecewise convex approximation to the original mesh. While previous work was only concerned with static meshes, we present a method for decomposing animated 3D meshes into temporally coherent approximately convex parts. Given a mesh and several training frames---that is, different spatial configurations of its vertices---we precompute an approximate convex decomposition that is independent of any specific frame. Such a decomposition can be transferred in real-time to novel, unseen frames. We apply our method to a variety of pre-animated meshes as well as a 3D character interactively controlled by a user's body pose. We further demonstrate that our method enables real-time physics simulations to interact with animated meshes.

In this paper we review current state-of-the-art in 3D point cloud classification, present a new 3D point cloud classification benchmark data set of single scans with over four billion manually labeled points, and discuss first available results on the benchmark. Much of the stunning recent progress in 2D image interpretation can be attributed to the availability of large amounts of training data, which have enabled the (supervised) learning of deep neural networks. With the data set presented in this paper, we aim to boost the performance of CNNs also for 3D point cloud labeling. Our hope is that this will lead to a breakthrough of deep learning also for 3D (geo-) data. The semantic3D.net data set consists of dense point clouds acquired with static terrestrial laser scanners. It contains eight semantic classes and covers a wide range of urban outdoor scenes, including churches, streets, railroad tracks, squares, villages, soccer fields, and castles. We describe our labeling interface and show that, compared to those already available to the research community, our data set provides denser and more complete point clouds, with a much higher overall number of labeled points. We further provide descriptions of baseline methods and of the first independent submissions, which are indeed based on CNNs, and already show remarkable improvements over prior art. We hope that semantic3D.net will pave the way for deep learning in 3D point cloud analysis, and for 3D representation learning in general.

PhysicsForests is a fluid-simulation framework that can simulate large scenes with several million particles in real time, including surface generation, foam, rigid-body coupling, and rendering. Instead of solving underlying Navier-Stokes equations, it predicts particle behavior approximately using machine-learning-based regression-forest method that is trained on a large set of randomly generated scenes obtained by a traditional fluid solver. The result is faster and more stable than standard methods. The simulation does not diverge even if the incompressibility constraint cannot be satisfied, for example in case of an extremely aggressive impact caused by user interaction. The parametric-input feature vector allows for simulation of a wide range of materials, differing in density, viscosity, static friction, or surface tension, and coupling of fluids with rigid bodies or with each other.

Submodular function minimization is a key problem in a wide variety of applications in machine learning, economics, game theory, computer vision, and many others. The general solver has a complexity of $O(n^3 \log^2 n . E +n^4 {\log}^{O(1)} n)$ where $E$ is the time required to evaluate the function and $n$ is the number of variables \cite{Lee2015}. On the other hand, many computer vision and machine learning problems are defined over special subclasses of submodular functions that can be written as the sum of many submodular cost functions defined over cliques containing few variables. In such functions, the pseudo-Boolean (or polynomial) representation \cite{BorosH02} of these subclasses are of degree (or order, or clique size) $k$ where $k \ll n$. In this work, we develop efficient algorithms for the minimization of this useful subclass of submodular functions. To do this, we define novel mapping that transform submodular functions of order $k$ into quadratic ones. The underlying idea is to use auxiliary variables to model the higher order terms and the transformation is found using a carefully constructed linear program. In particular, we model the auxiliary variables as monotonic Boolean functions, allowing us to obtain a compact transformation using as few auxiliary variables as possible.

Many machine learning tasks require finding per-part correspondences between objects. In this work we focus on low-level correspondences - a highly ambiguous matching problem. We propose to use a hierarchical semantic representation of the objects, coming from a convolutional neural network, to solve this ambiguity. Training it for low-level correspondence prediction directly might not be an option in some domains where the ground-truth correspondences are hard to obtain. We show how transfer from recognition can be used to avoid such training. Our idea is to mark parts as "matching" if their features are close to each other at all the levels of convolutional feature hierarchy (neural paths). Although the overall number of such paths is exponential in the number of layers, we propose a polynomial algorithm for aggregating all of them in a single backward pass. The empirical validation is done on the task of stereo correspondence and demonstrates that we achieve competitive results among the methods which do not use labeled target domain data.

Surface reconstruction from a point cloud is a standard subproblem in many algorithms for dense 3D reconstruction from RGB images or depth maps. Methods, performing only local operations in the vicinity of individual points, are very fast, but reconstructed models typically contain lots of holes. On the other hand, regularized volumetric approaches, formulated as a global optimization, are typically too slow for real-time interactive applications. We propose to use a regression forest based method, which predicts the projection of a grid point to the surface, depending on the spatial configuration of point density in the grid point neighborhood. We designed a suitable feature vector and efficient oct-tree based GPU evaluation, capable of predicting surface of high resolution 3D models in milliseconds. Our method learns and predicts surfaces from an observed point cloud sparser than the evaluation grid, and therefore effectively acts as a regularizer.

This paper presents a new 3D point cloud classification benchmark data set with over four billion manually labelled points, meant as input for data-hungry (deep) learning methods. We also discuss first submissions to the benchmark that use deep convolutional neural networks (CNNs) as a work horse, which already show remarkable performance improvements over state-of-the-art. CNNs have become the de-facto standard for many tasks in computer vision and machine learning like semantic segmentation or object detection in images, but have no yet led to a true breakthrough for 3D point cloud labelling tasks due to lack of training data. With the massive data set presented in this paper, we aim at closing this data gap to help unleash the full potential of deep learning methods for 3D labelling tasks. Our semantic3D.net data set consists of dense point clouds acquired with static terrestrial laser scanners. It contains 8 semantic classes and covers a wide range of urban outdoor scenes: churches, streets, railroad tracks, squares, villages, soccer fields and castles. We describe our labelling interface and show that our data set provides more dense and complete point clouds with much higher overall number of labelled points compared to those already available to the research community. We further provide baseline method descriptions and comparison between methods submitted to our online system. We hope semantic3D.net will pave the way for deep learning methods in 3D point cloud labelling to learn richer, more general 3D representations, and first submissions after only a few months indicate that this might indeed be the case.

Several machine learning tasks require to represent the data using only a sparse set of interest points. An ideal detector is able to find the corresponding interest points even if the data undergo a transformation typical for a given domain. Since the task is of high practical interest in computer vision, many hand-crafted solutions were proposed. In this paper, we ask a fundamental question: can we learn such detectors from scratch? Since it is often unclear what points are "interesting", human labelling cannot be used to find a truly unbiased solution. Therefore, the task requires an unsupervised formulation. We are the first to propose such a formulation: training a neural network to rank points in a transformation-invariant manner. Interest points are then extracted from the top/bottom quantiles of this ranking. We validate our approach on two tasks: standard RGB image interest point detection and challenging cross-modal interest point detection between RGB and depth images. We quantitatively show that our unsupervised method performs better or on-par with baselines.

Given a picture taken somewhere in the world, automatic geo-localization of such an image is an extremely useful task especially for historical and forensic sciences, documentation purposes, organization of the world’s photographs and intelligence applications. While tremendous progress has been made over the last years in visual location recognition within a single city, localization in natural environments is much more difficult, since vegetation, illumination, seasonal changes make appearance-only approaches impractical. In this work, we target mountainous terrain and use digital elevation models to extract representations for fast visual database lookup. We propose an automated approach for very large scale visual localization that can efficiently exploit visual information (contours) and geometric constraints (consistent orientation) at the same time. We validate the system at the scale of Switzerland (40,000 \(\hbox {km}^2\)) using over 1000 landscape query images with ground truth GPS position.

Given a picture taken somewhere in the world, automatic geo-localization of such an image is an extremely useful task especially for historical and forensic sciences, documentation purposes, organization of the world's photographs and intelligence applications. While tremendous progress has been made over the last years in visual location recognition within a single city, localization in natural environments is much more difficult, since vegetation, illumination, seasonal changes make appearance-only approaches impractical. In this chapter, we target mountainous terrain and use digital elevation models to extract representations for fast visual database lookup. We propose an automated approach for very large-scale visual localization that can efficiently exploit visual information (contours) and geometric constraints (consistent orientation) at the same time. We validate the system at the scale of Switzerland (40 000 km(2)) using over 1000 landscape query images with ground truth GPS position.

We propose an approach for dense semantic 3D reconstruction which uses a data term that is defined as potentials over viewing rays, combined with continuous surface area penalization. Our formulation is a convex relaxation which we augment with a crucial non-convex constraint that ensures exact handling of visibility. To tackle the non-convex minimization problem, we propose a majorizeminimize type strategy which converges to a critical point. We demonstrate the benefits of using the non-convex constraint experimentally. For the geometry-only case, we set a new state of the art on two datasets of the commonly used Middlebury multi-view stereo benchmark. Moreover, our general-purpose formulation directly reconstructs thin objects, which are usually treated with specialized algorithms. A qualitative evaluation on the dense semantic 3D reconstruction task shows that we improve significantly over previous methods.

In this work we make use of recent advances in data driven classification to improve standard approaches for binocular stereo matching and single view depth estimation. Surface normal direction estimation has become feasible and shown to work reliably on state of the art benchmark datasets. Information about the surface orientation contributes crucial information about the scene geometry in cases where standard approaches struggle. We describe, how the responses of such a classifier can be included in global stereo matching approaches. One of the strengths of our approach is, that we can use the classifier responses for a whole set of directions and let the final optimization decide about the surface orientation. This is important in cases where based on the classifier, multiple different surface orientations seem likely. We evaluate our method on two challenging real-world datasets for the two proposed applications. For the binocular stereo matching we use road scene imagery taken from a car and for the single view depth estimation we use images taken in indoor environments.

The automatic extraction of road networks is an interesting and challenging task. In spite of significant research efforts this problem remains largely open. In our work we attempt to leverage context at two different levels to extract accurate and topologically correct road networks. Local context, in the form of powerful features extracted from large neighborhoods, exploits the layout of road pixels and their co-occurrence with visual patterns along the roads. Global context enforces the connectivity of roads in a network, by grouping individual pixels into longer road segments, modeled as large higher-order cliques in a Conditional Random Field. Here, we evaluate different ways of defining these cliques. It turns out that, with modern probabilistic inference techniques, using a smaller number of very large cliques is more efficient than splitting them into a larger number of shorter segments.

Dense semantic 3D reconstruction is typically formulated as a discrete or continuous problem over label assignments in a voxel grid, combining semantic and depth likelihoods in a Markov Random Field framework. The depth and semantic information is incorporated as a unary potential, smoothed by a pairwise regularizer. However, modelling likelihoods as a unary potential does not model the problem correctly leading to various undesirable visibility artifacts. We propose to formulate an optimization problem that directly optimizes the reprojection error of the 3D model with respect to the image estimates, which corresponds to the optimization over rays, where the cost function depends on the semantic class and depth of the first occupied voxel along the ray. The 2-label formulation is made feasible by transforming it into a graph-representable form under QPBO relaxation, solvable using graph cut. The multi-label problem is solved by applying alpha-expansion using the same relaxation in each expansion move. Our method was indeed shown to be feasible in practice, running comparably fast to the competing methods, while not suffering from ray potential approximation artifacts.

In this paper we propose an approach to multi-class semantic segmentation of urban areas in high-resolution aerial images with classspecific object priors for buildings and roads. What makes model design challenging are highly heterogeneous object appearances and shapes that call for priors beyond standard smoothness or co-occurrence assumptions. The data term of our energy function consists of a pixel-wise classifier that learns local co-occurrence patterns in urban environments. To specifically model the structure of roads and buildings, we add high-level shape representations for both classes by sampling large sets of putative object candidates. Buildings are represented by sets of compact polygons, while roads are modeled as a collection of long, narrow segments. To obtain the final pixel-wise labeling, we use a CRF with higher-order potentials that balances the data term with the object candidates. We achieve overall labeling accuracies of > 80%.

The matching function for the problem of stereo reconstruction or optical flow has been traditionally designed as a function of the distance between the features describing matched pixels. This approach works under assumption, that the appearance of pixels in two stereo cameras or in two consecutive video frames does not change dramatically. However, this might not be the case, if we try to match pixels over a large interval of time. In this paper we propose a method, which learns the matching function, that automatically finds the space of allowed changes in visual appearance, such as due to the motion blur, chromatic distortions, different colour calibration or seasonal changes. Furthermore, it automatically learns the importance of matching scores of contextual features at different relative locations and scales. Proposed classifier gives reliable estimations of pixel disparities already without any form of regularization. We evaluated our method on two standard problems - stereo matching on KITTI outdoor dataset, optical flow on Sintel data set, and on newly introduced TimeLapse change detection dataset. Our algorithm obtained very promising results comparable to the state-of-the-art.