Semantic Mapping: A Semantics-based Approach to Virtual Content Placement for Immersive Environments

Semantic Mapping is a semantics-based interactive system that enables intuitive virtual content placement for projection mapping in intelligent environments. Our semantic mapping system embeds semantic information of the environment to provide a user with an easy way to control and place projected virtual items in the physical world. In contrast to traditional projection mapping that involves manual adjustments, this semantic mapping system enables efficient manipulation of virtual content through inputs from users via speech or text. To build the system, we first use a commercial depth camera for scene reconstruction and an end-to-end deep learning framework for semantic segmentation at the instance level. We illustrate the system by developing a prototype for a set of proof-of-concept, room-scale applications. The accuracy study and user study results show that the system can provide users with accurate semantic information for effective virtual content placement.