Nothing replaces the importance of the sweat, blood and tears of a live simulated training experience in which all your senses (visual, audio, haptic, olfactory, gastronomy, etc.) play into a physical, mental and emotional life-and- death scenario in a fully three dimensional, real-time world. When Virtual Reality is provided as an alternative, it can pale in comparison, as it is a disembodied experience no matter how much artistry has been applied to the aesthetic display and emotional thrill. This statement is applicable even to military simulations that drive complex and intricate training, and yet rarely cause trainees to break a sweat. There is a need for systems that integrate training scenarios into physically responsive live environments, enhanced by compelling entertainment techniques. Such systems must support the delivery of a wide range of simulation applications including vehicular, dismounted and constructive simulation planning. This paper covers recent developments in integrating multi-modal functionality into Mixed Reality (MR), the blending of real and virtual sight, sound and special effects. More specifically, we present an overview of an MR research project and the multi-modal training engine (versus game engine) produced as a consequence of this research. This engine composes real and synthetic sensory stimulations into an interactive, multi-sensory, non-linear, immersive experience. One application of this MR system is to create a MOUT (Military Operations in Urban Terrain) environment that blends real assets, such as buildings or building facades, with virtual assets including neutrals and combatants, both friends and foes. The entire system (software and hardware) is designed to be deployed into the field to transform any site into a MOUT for use in military training, homeland security, emergency response, informal education and entertainment. It can adapt core content experiences to custom environments providing in-the-field lush, compellingly, interactive and non-linear group experiences.

This paper describes the creative exploration of emerging artistic convention within applied Mixed Reality research and its co-evolution with the creation of new scientific methods. We examine current research projects in entertainment , education and training that are defining new challenges and new capabilities for multi-modal immersion with cross-reality media. The areas of exploration cover novel developments in content creation and content experience.

This paper describes the creative exploration of emerging artistic convention within applied Mixed Reality research and its co-evolution with the creation of new scientific methods. We examine current research projects in entertainment , education and training that are defining new challenges and new capabilities for multi-modal immersion with cross-reality media. The areas of exploration cover novel developments in content creation and content experience.