technologies and 3D computer-vision algorithms are allowing efficient and real-time acquisition, reconstruction, and understanding of the 3D environment, which in turn are enabling an array of new applications based on life-like machine vision. These applications include immersive and interactive gaming, video conferencing with custom backgrounds and synthetic environments, education and training, virtual home and office decoration, virtual clothes fitting, autonomous machines, and numerous other entertainment and productivity usages that incorporate natural and intuitive interactions. This article discusses the advances in key technologies, including sensors, algorithms, and system integration, that are fueling the recent proliferation of 3D-sensing applications. Specific topics include depth-sensing and tracking technologies, 3D imaging and reconstruction, 3D gestural interactions with hand skeleton tracking, background segmentation, and collaborative augmented reality. Depth Sensing and Tracking Technologies The recent availability of small form factor, low power, and real-time depth-image-capture technologies is a key enabler for mainstream devices and systems with 3D-sensing capabilities and interactive applications. Conventionally, vision-based applications have used 2D image sensors, utilizing the cameras that are already part of mobile devices such as smartphones, tablets, and laptops. These conventional image-acquisition devices are able to convert the visual information in a 3D scene into a projection of 2D arrays of discrete numbers. As a result of this transformation process, the 3D information cannot be accurately recovered from the captured 2D images because the pixels in the images preserve only partial information about the original 3D space. Reconstruction of 3D surfaces from singleintensity images is a widely researched subject. However, recovering 3D spatial information from 2D projections possesses inherent ambiguities. Moreover, this type of approach is generally computing intensive and often requires manual user inputs; hence, it is not suitable for applications that require real-time and unaided understanding of the 3D environment and interactive usages. In contrast, the human visual system incorporates a binocular imaging scheme and is capable of depth perception. These capabilities allow us to understand, navigate, and interact in our 3D environment with ease. Similarly, natural and interactive experiences with vision-based schemes are better accomplished using 3D image-sensing devices, which can capture depth or range information in addition to the color values for a pixel, thereby allowing fast and accurate reconstruction and understanding of the 3D world. Interactive devices and applications utilizing real-time 3D sensing are rapidly gaining adoption and popularity.1 Examples include personal computers and mobile devices using Intel’s RealSense cameras, and gaming console systems such as Microsoft Kinect in living rooms. Figure 1 shows a pair of depth and color images captured with the RealSense 3D Camera. Besides the advances in image sensing, vision-based tracking and mapping techniques have progressed significantly with the advent of visual simultaneous localization and mapping approaches or visual SLAM,2–4 which allow real-time construction and updating of a spatial map while simultaneously tracking features within the environment. This is often built on top of standard 2D imaging devices to allow real-time camera pose estimation and a sparse reconstruction of the environment. In Advances in 3D-Sensing Technologies and Applications

3D sensing enabled by depth‐imaging technologies allows immersive user interfaces and lifelike interactive experiences via real‐time understanding of the 3D environment, people, and objects in real‐world scenes. Advances in this area are being made possible through several key technologies, including sensors, algorithms, and system integration.

The recent developments in depth‐imaging and 3D computer vision techniques allow efficient and real‐time acquisition, reconstruction, and understanding of the 3D environment, which enable an array of life‐like and immersive applications. We show here an excerpt of some of the key technologies spanning sensors, algorithms, and system integration based on depth imaging and 3D sensing technology. The small form factor, the low power and the real‐time depth image capture are the key enablers. We talk here about emerging trends in this burgeoning field and new enabled applications that include immersive and interactive gaming, virtual home and office decoration, virtual cloth fitting, video conferencing with custom background and synthetic environment, education and training, and numerous other new entertainment and productivity usages with natural and intuitive interactions.

We present “It's a Pirate's Life” demonstration, an Augmented Reality (AR) game which makes use of real-time 3D reconstruction and tracking using an Intel® RealSense™ camera system embedded in a tablet to build a dynamic game world. Players can play as a pirate ship captain searching for gold on a virtual sea overlaid on the real-world. Real-world objects become part of the play space; islands in the tropical seas which you have to navigate your ships around while avoiding cannon balls to find the treasure. Players control the wind, and hence, direction of sail by moving the tablet around the play space to guide the virtual ship in the real and virtual environment to the pirate gold.

This article is a revised version of an earlier work on Gravity-Aware Handheld Augmented Reality (AR) (Kurz and Benhimane, 2011 [1]), which investigates how different stages in handheld AR applications can benefit from knowing the direction of the gravity measured with inertial sensors. It presents approaches to improve the description and matching of feature points, detection and tracking of planar templates, and the visual quality of the rendering of virtual 3D objects by incorporating the gravity vector. In handheld AR, both the camera and the display are located in the user's hand and therefore can be freely moved. The pose of the camera is generally determined with respect to piecewise planar objects that have a static and known orientation with respect to gravity.In the presence of (close to) vertical surfaces, we show how Gravity-Aligned Feature Descriptors (GAFDs) improve the initialization of tracking algorithms relying on feature point descriptor-based approaches in terms of quality and performance. For (close to) horizontal surfaces, we propose to use the gravity vector to rectify the camera image and detect and describe features in the rectified image. The resulting Gravity-Rectified Feature Descriptors (GREFDs) provide an improved precision-recall characteristic and enable faster initialization, in particular under steep viewing angles. Gravity-rectified camera images also allow for real-time 6 DoF pose estimation using an edge-based object detection algorithm handling only 4 DoF similarity transforms. Finally, the rendering of virtual 3D objects can be made more realistic and plausible by taking into account the orientation of the gravitational force in addition to the relative pose between the handheld device and a real object.In comparison to the original paper, this work provides a more elaborate evaluation of the presented algorithms. We propose a method enabling the evaluation of inertial-sensor aided visual tracking methods without real inertial sensor data. By synthesizing gravity measurements from ground truth camera poses, we benchmark our algorithms on a large existing dataset. Based on this approach, we also develop and evaluate a gravity-adaptive approach that performs image-rectification only when beneficial. (C) 2012 Elsevier Ltd. All rights reserved.

We present a method to automatically determine a set of feature descriptors that describes an object such that it can be localized under a variety of viewpoints. Based on a set of synthetically generated views, local image features are detected, described and aggregated in a database. Our proposed method evaluates matches between these database features to eventually find a set of the most representative descriptors from the database. Using this scalable offline process, the localization success rate is significantly increased without adding computational load to the runtime method. Moreover, if camera localization is performed with respect to objects at a known gravity orientation, we propose to create multiple reference descriptor sets for different angles between the camera's principal axis and the gravity vector. This approach is particularly suited for handheld devices with built-in inertial sensors and enables matching against a reference dataset only containing the information relevant for camera poses that are consistent with the measured gravity. Comprehensive evaluations of the proposed methods using a large quantity of real camera images, a variety of objects, different cameras and different kinds of feature descriptors confirm that our approaches outperform standard feature descriptor-based methods.

In this paper, we address the problem of simultaneous tracking and reconstruction of non-planar templates in real-time. Classical approaches to template tracking assume planarity and do not attempt to recover the shape of an object. Structure from motion approaches use feature points to recover camera pose and reconstruct the scene from those features, but do not produce dense 3D surface models. Finally, deformable surface tracking approaches assume a static camera and impose strong deformation priors to recover dense 3D shapes. The proposed method simultaneously recovers the camera motion and deforms the template such that an approximation of the underlying 3D structure is recovered. Spatial smoothing is not explicitly imposed, thus templates of smooth and non-smooth objects can be equally handled. The problem is formalized as an energy minimization based on image intensity differences. Quantitative and qualitative evaluation on both real and synthetic data is presented, we compare the proposed approach to related methods and demonstrate that the recovered camera pose is close to the ground truth even in presence of strong blur and low texture.

In the research fields of Augmented Reality (AR) and Mixed Reality (MR), tracking and registration methods are still one of the most important topics. The tracking research field is highly active, and numerous methods appear on a regular basis. The TrakMark working group (WG) was established 2009 to create a benchmark test that permits objective and accurate evaluation of the tracking methods. This year, the workshop will cover a wide range of topics concerning AR/MR registration, tracking and benchmarking. Key areas include, but are not limited to: — Vision-based registration, camera localization — Visual SLAM, structure from motion, camera calibration, sensor fusion — Natural feature tracking, object tracking, feature detection, feature description — Comparison of methods, evaluation of methods, suggestion of new benchmarking scheme — Survey of tracking papers.

For natural interaction with augmented reality (AR) applications, good tracking technology is key. But unlike dense stereo, optical flow or multi-view stereo, template-based tracking which is most commonly used for AR applications lacks benchmark datasets allowing a fair comparison between state-of-the-art algorithms. Until now, in order to evaluate objectively and quantitatively the performance and the robustness of template-based tracking algorithms, mainly synthetically generated image sequences were used. The evaluation is therefore often intrinsically biased. In this paper, we describe the process we carried out to perform the acquisition of real-scene image sequences with very precise and accurate ground truth poses using an industrial camera rigidly mounted on the end effector of a high-precision robotic measurement arm. For the acquisition, we considered most of the critical parameters that influence the tracking results such as: the texture richness and the texture repeatability of the objects to be tracked, the camera motion and speed, and the changes of the object scale in the images and variations of the lighting conditions over time. We designed an evaluation scheme for object detection and interframe tracking algorithms suited for AR and other computer vision applications and used the image sequences to apply this scheme to several state-of-the-art algorithms. The image sequences are freely available for testing, submitting and evaluating new template-based tracking algorithms, i.e. algorithms that detect or track a planar object in an image sequence given only one image of the object (called the template).

This paper investigates means to benchmark methods for camera pose localization and tracking that in addition to a camera image make use of inertial sensor measurements. In particular the direction of the gravity has recently shown to provide useful information to aid vision-based approaches making them outperform visiononly methods. Obviously, it is desirable to benchmark the performance of such methods and to compare them with state-of-the-art approaches, but to the best of our knowledge, all publicly available benchmarking datasets unfortunately lack gravity information. We present different simple means to generate one’s own benchmarks for inertial sensor-aided localization and tracking methods and most considerably show how existing datasets, that do not have inertial sensor data, can be exploited. We demonstrate how to evaluate Gravity-Aligned Feature Descriptors (GAFD) and GravityRectified Feature Descriptors (GREFD) on an existing benchmark dataset with ground truth poses. By synthesizing gravity measurements from these poses we achieve similar results to using real sensor measurements at significantly less effort. Most importantly, the proposed procedure enables the comparison with existing evaluation results on the same data. The paper concludes with a requirements analysis and suggestions for the design of future benchmarking datasets for localization and tracking methods.