Projection-based augmented reality (AR) is a promising medium for realizing pervasive computing environment, and yet the problem of determining projection-suitable regions and where to project remains. To tackle this problem, we introduce FRISP, a projection-based augmented reality (AR) Framework designed for Registering Interactive Spatial Projection. The FRISP framework utilizes a pan-tilt projection-camera (pro-cam) system for capturing geometry and projection mapping. The framework scans and analyzes the geometric properties of a room, in order to determine projection-suitable regions and generate multi-window layouts. Once the multi-windows are registered to the real world, they can be interacted with by the users. The users can assign various widgets or applications to the multi-windows, which are then finally augmented onto the real world and can serve as a base units for realizing the pervasive AR environment.

A fast and accurate algorithm is presented for registering scans from an RGB-D camera on a pan-tilt platform. The pan-tilt RGB-D camera rotates and scans the entire scene in an automated fashion. The proposed algorithm exploits the movement of the camera that is bound by the two rotation axes of the servo motors so as to realize fast and accurate registration of acquired point clouds. The rotation parameters, including the rotation axes, pan-tilt transformations and the servo control mechanism, are calibrated beforehand. Subsequently, fast global registration can be performed during online operation with transformation matrices formed by the calibrated rotation axes and angles. In local registration, features are extracted and matched between two scenes. False-positive correspondences, whose distances to the rotation trajectories exceed a threshold, are rejected. Then, a more accurate registration can be achieved by minimizing the residual distances between corresponding points, while transformations are bound to the rotation axes. Finally, the preliminary alignment result is input to the iterative closed point algorithm to compute the final transformation. Results of comparative experiments validate that the proposed method outperforms state-of-the-art algorithms of various approaches based on camera calibration, global registration, and simultaneous-localization-and-mapping in terms of root-mean-square error and computation time.

Projection Augmented Reality (AR), which uses projectors to augment virtual information and objects in physical space, is one of the AR fields that have been actively researched. There have been many attempts to use mobile devices to implement the interactions available in these AR environments; however, due to mobile devices' limited computational power, they were somewhat difficult to use in real applications. We have solved this problem with the latest off-the-shelf devices and support development kit and have proven, through experimentation, that our solution can be used in actual projection AR applications.

In this paper, we propose three display methods for projection-based augmented reality. In spatial augmented reality (SAR), determining where information, objects, or contents are to be displayed is a difficult and important issue. We use deep learning models to estimate user pose and suggest ways to solve the issue based on this data. Finally, each method can be appropriately applied according to various the applications and scenarios.

This study describes an Adaptive Projection Augmented Reality (AR) system that can provide information in real-time using object recognition. This approach is based on deep-learning through the construction of a 3D space from the real world. Through single system, a projector-camera unit with pan-tilt mechanism,, the 360-degree space of user surroundings can be built into a 3-dimensional space and accomplish object and user recognition. a projection AR environment can be generated instantaneously. Information relevant to real-world objects can be provided through real-time interactions between the user and objects. Using spatial interaction, it also allows for achieving intuitive interactions with projection information, user interface (UI), and contents without touch sensors. Several scenarios for the use of this system are described below.

Current rendering processors are organized mainly to process a triangle as fast as possible and recently parallel 3D rendering processors, which can process multiple triangles in parallel with multiple rasterizers, begin to appear. For high performance in processing triangles, it is desirable for each rasterizer have its own local pixel cache. However, the consistency problem may occur in accessing the data at the same address simultaneously by more than one rasterizer. In this paper, we propose a parallel rendering processor architecture, called DAVID II, resolving such consistency problem effectively. Moreover, the proposed architecture reduces the latency due to a pixel cache miss significantly. The experimental results show that DAVID II achieves almost linear speedup at best case even in sixteen rasterizers.

Since projection augmented reality (AR) robot can provide a lot of information through projector, it can be useful in museums and art galleries that need to provide information to the crowd. Therefore, it is necessary to continue to interact with people, and human-aware path planning is also needed. We prototyped projection AR mobile robot implemented human-aware path planning and wrote about future research direction.

In this paper, we propose PPAP, an augmented reality platform with an actuated projector for dynamic user-perspective projection. In PPAP, a stationary camera is used jointly with a pan-tilt motorized projector-camera unit. With the servo control of the steerable pan-tilt system, the system is able to continuously orient itself to match the user's view center of the projection-mapped surface. This provides users with greatly widened viewing angles in the augmented scene, when compared to the stationary projection. Through user studies, in which users judged the size and distance of a projected virtual object, we verified that the perspective projection with the actuated projector helps users better understand the spatial relationship of the virtual object in the augmented scene in terms of depth perception.

We propose a projection-based augmented reality (AR) robot system that provides pervasive support for the education and safety of preschoolers via a deep learning framework. This system can utilize real-world objects as metaphors for educational tools by performing object detection based on deep learning in real-time, and it can help recognize the dangers of real-world objects that may pose risks to children. We designed the system in a simple and intuitive way to provide user-friendly interfaces and interactions for children. Children's experiences through the proposed system can improve their physical, cognitive, emotional, and thinking abilities.

In this paper, we propose AR Pointer, a new augmented reality (AR) interface that allows users to manipulate three-dimensional (3D) virtual objects in AR environment. AR Pointer uses a built-in 6-degrees of freedom (DoF) inertial measurement unit (IMU) sensor in an off-the-shelf mobile device to cast a virtual ray that is used to accurately select objects. It is also implemented using simple touch gestures commonly used in smartphones for 3D object manipulation, so users can easily manipulate 3D virtual objects using the AR Pointer, without a long training period. To demonstrate the usefulness of AR Pointer, we introduce two use-cases, constructing an AR furniture layout and AR education. Then, we conducted two experiments, performance tests and usability tests, to represent the excellence of the designed interaction methods using AR Pointer. We found that AR Pointer is more efficient than other interfaces, achieving 39.4% faster task completion time in the object manipulation. In addition, the participants gave an average of 8.61 points (13.4%) on the AR Pointer in the usability test conducted through the system usability scale (SUS) questionnaires and 8.51 points (15.1%) on the AR Pointer in the fatigue test conducted through the NASA task load index (NASA-TLX) questionnaire. Previous AR applications have been implemented in a passive AR environment where users simply check and pop up the AR objects those are prepared in advance. However, if AR Pointer is used for AR object manipulation, it is possible to provide an immersive AR environment for the user who want/wish to actively interact with the AR objects.

As aging societies grow, researchers are actively studying care systems concerning the life and diseases of the elderly. Among these diseases, dementia makes it difficult to maintain daily life due to the degradation of cognitive functioning, memory, and reasoning, as well as the ability to perform actions. Moreover, dementia does not have a perfect cure, though therapy and care can slow its onset and provide patients with physical and mental support. In this paper, we developed a projection-based augmented reality system robot that can cover 360 degrees of space. We also propose an application that supports continuous monitoring of dementia patients to address the difficulties they face in daily life. The system is also designed to provide therapy applications, such as entertainment and spatial art, to provide mental care aids for the patients.

Developing innovative and pervasive smart technologies that provide medical support and improve the welfare of the elderly has become increasingly important as populations age. Elderly people frequently experience incidents of discomfort in their daily lives, including the deterioration of cognitive and memory abilities. To provide auxiliary functions and ensure the safety of the elderly in daily living situations, we propose a projection-based augmented reality (PAR) system equipped with a deep-learning module. In this study, we propose three-dimensional space reconstruction of a pervasive PAR space for the elderly. In addition, we propose the application of a deep-learning module to lay the foundation for contextual awareness. Performance experiments were conducted for grafting the deep-learning framework (pose estimation, face recognition, and object detection) onto the PAR technology through the proposed hardware for verification of execution possibility, real-time execution, and applicability. The precision of the face pose is particularly high by pose estimation; it is used to determine an abnormal user state. For face recognition results of whole class, the average detection rate (DR) was 74.84% and the precision was 78.72%. However, for face occlusions, the average DR was 46.83%. It was confirmed that the face recognition can be performed properly if the face occlusion situation is not frequent. By object detection experiment results, the DR increased as the distance from the system decreased for a small object. For a large object, the miss rate increased when the distance between the object and the system decreased. Scenarios for supporting the elderly, who experience degradation in movement and cognitive functions, were designed and realized, constructed using the proposed platform. In addition, several user interfaces (UI) were implemented according to the scenarios regardless of distance between users and the proposed system. In this study, we developed a bidirectional PAR system that provides the relevant information by understanding the user environment and action intentions instead of a unidirectional PAR system for simple information provision. We present a discussion of the possibility of care systems for the elderly through the fusion of PAR and deep-learning frameworks.

A pan-tilt camera system has been adopted by a variety of fields since it can cover a wide range of region compared to a single fixated camera setup. Yet many studies rely on factory-assembled and calibrated platforms and assume an ideal rotation where rotation axes are perfectly aligned with the optical axis of the local camera. However, in a user-created setup where a pan-tilting mechanism is arbitrarily assembled, the kinematic configurations may be inaccurate or unknown, violating ideal rotation. These discrepancies in the model with the real physics result in erroneous servo manipulation of the pan-tilting system. In this paper, we propose an accurate control mechanism for arbitrarily-assembled pan-tilt camera systems. The proposed method formulates pan-tilt rotations as motion along great circle trajectories and calibrates its model parameters, such as positions and vectors of rotation axes, in 3D space. Then, one can accurately servo pan-tilt rotations with pose estimation from inverse kinematics of their transformation. The comparative experiment demonstrates out-performance of the proposed method, in terms of accurately localizing target points in world coordinates, after being rotated from their captured camera frames.

Here we present a projection augmented reality (AR) based assistive robot, which we call the Pervasive Assistive Robot System (PARS). The PARS aims to improve the quality of life by of the elderly and less able-bodied. In particular, the proposed system will support dynamic display and monitoring systems, which will be helpful for older adults who have difficulty moving their limbs and who have a weak memory.We attempted to verify the usefulness of the PARS using various scenarios. We expected that PARSs will be used as assistive robots for people who experience physical discomfort in their daily lives.

Projection-based augmented reality (AR) has much potential, but is limited in that it requires burdensome installations and prone to geometric distortions on display surface. To overcome these limitations, we propose AIR. It can be carried and placed anywhere to project AR using pan/tilting motors, while providing the user with distortion-free projection of a correct 3D view.

Many kinds of preparations are needed when meeting. For example, projector, laptop, cables and ETC. As such, this video have constructed Meet AR-bot, which helps users to keep meeting going smoothly, based on the projection of Augmented Reality(AR). Our system can easily provide meeting room environment through the movable setting via wheel-based stand. Users do not need to carry a personal laptop and connect them to the projector. Robot reconstructs the 3D geometry information through pan-tilt system and compute projection areas to project information in the space. Users can also control through mobile devices. We offer presentation, table interaction, file sharing and virtual object registration by mobile device.

Video conferencing, which helps gather opinions and make decisions quickly among employees who are not in the same location, is now a very important communication tool in the workplace. Our research is one of these video conferencing solutions, specifically proposed to address the difficulties of analog materials sharing and feedback, and has added some useful features for the smooth use of conference participants. We conducted a comparative experiment on our proposed method of file sharing and the method that we had previously used in video conferencing. As a result, the proposed system yielded better results in terms of time and usability during a full-scale collaborative situation in which feedback was provided.

Children encounter a variety of experiences through play, which can improve their ability to form ideas and undergo multi-faceted development. Using Augmented Reality (AR) technology to integrate various digital learning elements with real environments can lead to increased learning ability. This study proposes a 360° rotatable and portable system specialized for education and development through projection-based AR play. This system allows existing projection-based AR technology, which once could only be experienced at large-scale exhibitions and experience centers, to be used in individual and small-scale spaces. It also promotes the development of multi-sensory abilities through a multi-modality which provides various intuitive and sensory interactions. By experiencing the various educational play applications provided by the proposed system, children can increase their physical, perceptive, and emotional abilities and thinking skills.

An estimated 20 percent of the world»s population experience difficulties with physical, cognitive and mental health. Senior citizens experience many inconveniences due to less memory and mobility as a function of the aging process. To solve this, the cARe-bot system was designed to dynamically provide optimal information in the actual environment at any time or place. It is also designed to increase an elder person»s ability to live in a house or in senior homes with daily support, thus establishing the convenience and comfort they knew throughout their lives. The hardware of this system is a 360-degree controllable projection system. It is made with a pan-tilt system, projector, and a depth-recognition camera. Arduino is attached in order to control two sets of servo-motors, and is connected to a PC, which controls the entire process.

We present a method to assist pre-touch on multi touch surfaces using features extracted from the fingertips of the user's hands. Pixel area a feature obtained from the contour of a fingertip is computed and tracked using commodity cameras to obtain the velocity of approaching fingers. We target sound volume control for digital multi-touch instruments an area where pre-touch is demonstrated with our approach to approximating velocity of finger taps. We explore other areas of application with velocity and how it can help anticipate user actions in mobile touch displays. In order to evaluate the effectiveness of the proposed system, a system prototype was presented to users. We confirmed the satisfactory results by performing the recognition rate experiment for measuring the blob size according to the touch speed.