Intelligent Reality refers to the fusion of integrated technological innovations to develop complex computing systems aiming to blend the real with digital worlds and enable humans and machines to seamlessly work together leveraging each other’s strengths. This paper presents the results of a study evaluating the user experience of visiting an Intelligent Reality virtual museum prototype in terms of presence, technology acceptance, and environment perceptions, as a small-scale Cyber-Physical-Social Eco-System proof of concept. A virtual reality (VR) system was designed, and a real robot was placed in a technology museum to foster cyber-physical synchronisation, for users to visit and navigate in the environment. The study results revealed high degree of presence achieved by users in the VR world, with positive acceptance of technology and environment evaluation. Most importantly, the results highlighted the importance of presence in such systems and how it relates to acceptance and to the overall users’ perception of a VR world.
The advent and the rapid expansion of Industry 4.0 solutions and technologies such as IoT, Artificial Intelligence (AI), cloud computing, and digital twinning, is causing industrial companies, utilities, and other organizations to either retrain their workforce or hire experts from other industries who have the knowledge or hands-on experience working with such technologies. With the emergence of remote working routines, providing such hands-on training programs becomes difficult. As such, BCIT's Smart Microgrid Applied Research Team has developed an advanced Virtualized Experiential Learning Platform (VELP) with the support of the Future Skills Centre and its industry partner to provide trainees from industry and academia with hands-on experience in the fields of smart grids, substation automation systems and Operational Technology (OT) cybersecurity utilizing virtualization technologies such as digital twins, AI, and cloud apps. This paper aims to introduce VELP, its features, and functionalities, and explain how monitoring and control layers can be migrated into cyberspace. Moreover, this paper proposes a reliable communication network and a cybersecure topology that can be utilized in similar virtualization platforms. The results of this paper show that such technologies can be applied to other digital experiential learning programs where reliable and secure remote access to physical assets is essential to support relevant experiential learning pedagogical models.
Most virtual reality (VR) applications rely on auditory and visual stimulation which can be limiting in terms of how well the user experience can be. This systematic review looks at multisensory VR systems that incorporate haptic, olfactory, and/or gustatory cues, in addition to audio-visual. It aims to provide a broad overview of how multisensory stimulation affects VR experiences and to identify which types of sensory stimuli are most commonly used in the multisensory context. The authors found that 90% of the studies they examined showed a positive effect of multisensory VR, with haptics attracting considerable attention (65%). They highlight the need to devote more research and development to smell and taste, as they can add significant value to VR and user experiences.
In the not so distant future, the rise of the metaverse will bring multiple opportunities to people and societies all over the world. Through pioneering technologies integrated, it aims to revolutionise and significantly improve our lives. Unfortunately the metaverse can also be exploited and abused to facilitate the operations of extremists and terrorists as it has already happened in social media and online blogs and forums. Using the metaverse’s virtual reality, A.I., digital twins and avatars to their advantage, the various extremist/terrorist organisations and groups can successfully spread their radical ideologies and propaganda, recruit new members and plan their attacks more effectively and with minimum costs. In the metaverse, such operations will have greater impact since the users will be an actual part of the metaverse and therefore fully immersed in it. In this paper, we aim to provide a better understanding of how extremists and terrorists can exploit the metaverse and its services through the usage of appropriate discussion and relevant examples. Concluding, we give some general direction on what needs to be done to prevent such unfortunate scenarios.
Measuring social constructs such as engagement, rapport, and trust often rely heavily on surveys and behavioral observations. This paper describes a method to use features identified by psychology-based language analysis, combined with machine learning, to predict participant survey responses in a training context based on 120 dyad transcripts. The method analyzed data collected from subjects performing a circuit board training task within the project called SCOTTIE, Systematic Communication Objectives and Telecommunications Technology Investigations and Evaluations. In this study, the collected data showed low utterance count and a lack of correlation between features and survey responses, suggesting that the context in which the interactions occurred may limit opportunities for interlocutors to manifest social behaviors verbally, which in turn affected the ability to use language analysis to predict subject perceptions of the interaction. However, the methodology appears sound.
Digital Twins are often cited as both a compelling example of Digital Reality, and an excellent use case for Augmented, Mixed and Virtual Reality. Superficially, they appear to go to the heart of the concept of intelligent reality, combining real-world sensors, analytics and 3D computer graphics presented as an augmented reality overlay, or a fully virtual experience.Digital Twins appear to promise many things, but what do they actually deliver? Who are Digital Twins designed for? And, if they are so incredibly useful, why are they still so rare in the real world?In this brief article, the author will explore some of the key challenges in real world deployments of Digital Twins for commercial buildings, and discuss a path to implementation that could make them more broadly accessible.
This paper presents initial findings from an empirical study utilising an intelligent virtual museum with the aim to explore its learning and engagement capability. The study investigates the participants’ experiences with the environment, their perceptions on its learning and engagement ability, and how they envision future learning scenarios with intelligent virtual environments. Participants found the system interesting, useful, and engaging. Suggestions for improvement highlighted technological, pedagogical, and social aspects which can guide further exploitation of such systems in education.
The applicability of augmented reality (AR) is stunted by the current limitations of localization systems. In various forms, simultaneous localization and mapping (SLAM) has become a common framework for providing device localization in AR systems; however, outside of camera localization data, SLAM systems typically fail to provide additional information about the environment to consumer applications. This limits the domain of potential AR applications, as many applications will require some degree of interaction between the real and virtual worlds. One such application is object labeling for moving objects. In this work, we implement an AR moving object labeling system by utilizing LUMO-SLAM, a SLAM system that registers and localizes unknown moving objects in the environment. Test runs of the system show that moving object information provided by LUMO-SLAM is sufficient for implementing a useful moving object labeling system and potentially other real-world applications of AR.
Digital Twins (DTs) and Virtual Reality (VR) have recently gained significant popularity. The term digital twin refers to a virtual representation of a real-world environment. DTs are known for providing intuitive, accurate, and real-time visualizations of complex physical systems. In this paper, we present a novel, cloud-based framework for creating digital twin environments using VR within a cloud environment. We demonstrate our framework with two real-world case studies inspired by clinical insights obtained from experts in post-stroke rehabilitation. In the case of rehabilitation, sessions within virtual scenarios mimicking patients’ everyday life may significantly improve their rehabilitation quality. Inspired by biofeedback, we used a digital twin in VR to allow patients to enhance hand stability using real-time data streaming. Our second case study demonstrated the utility of our approach for allowing patients to more comfortably navigate within their homes by generating a virtual twin of an apartment. Our work is designed to transcend the boundary between reality and virtuality by letting users interact directly with the real and virtual worlds simultaneously. To achieve real-time synchronization between the real and the virtual environments, we established an infrastructure blueprint that features efficient sensor data collection, streaming, processing, and storage in a secure cloud environment. Data may then be accessed from any site, directly from the cloud, to update the virtual environment. In the end, we show that our infrastructure is generalizable across clinically-relevant tasks. We hope to extend our work to other areas that would benefit from a digital twin-based VR space.
Augmented reality (AR) applications require constant device localization, which is often fulfilled by visual simultaneous localization and mapping (SLAM). SLAM provides realtime camera localization by also dynamically building a 3D map of the environment, but the functionality of SLAM systems generally stops here. Useful applications of AR could make great use of additional information about the environment, such as the structure and location of moving objects in the scene (including objects that were not previously known to be separate from the static points of the map). We present an approach for solving the visual SLAM problem while also registering and localizing moving objects without prior knowledge of the objects’ structure, appearance, or existence. This is accomplished via analysis of reprojection errors and iterative use of the ePnP algorithm in a RANSAC scheme. The approach is demonstrated with the accompanying prototype system, LUMO-SLAM. The initial results achieved by this system indicate that the approach is both sound and potentially viable for some practical applications of AR and visual SLAM.
Simulations are a powerful tools particularly in the case of safety critical scenarios. However, simulating complex temporal events in multi-agent scenarios with vehicles and pedestrians, such as those that exist in urban environments, is challenging. We present CityLifeSim, a simulation for the research community that focuses on rich pedestrian behavior, such as the one that arises when different personalities, environmental events, and group goals are simulated. In our simulations we can see cases of people jay walking a red light, sitting on a bench, waiting for the bus, or calling on the phone, but also more complex creation and management of crowds that might even line up or just keep moving while observing interpersonal distances. CityLifeSim is configurable and can create unlimited scenarios with detailed logging capabilities. As a demonstration we have run CityLifeSim to create a demo dataset for training setups that includes 17 different cameras, views from a moving vehicle in the street under different weather conditions (rain, snow, sun), and from a drone with frontal and downward views. All content is released with the corresponding original configuration files, ground truth pedestrian segmentation, and RGB-D frames. We evaluate our dataset on a pedestrian detection and identification task with state of the art Multi-Object Tracker (MOT), showing the limitations and opportunities for synthetic data in this use case.
In Virtual Environments (VE), audio technologies play a significant role in immersive and interactive experiences. Virtual Reality (VR) simulations must be ecologically enacted by a participatory exploration of sense-making in a network of human and non-human agents, called actors. The guardian of such locus of agency is the digital twin (DT) that fosters intra-actions between humans and technology, dynamically and fluidly redefining all those configurations that are crucial for meaningful sonic experiences. The idea of human-machine entanglement is here mainly declined in an egocentric-spatial perspective related to emerging knowledge of the listener’s subjectivity. Such a systemic view can be interpreted as a working definition of intelligent reality: a perceptual and cognitive co-constitution of physical and virtual worlds through adaptive and reflective behaviors of VR technologies. The main theoretical results reported in this paper reside in the definition of sonic experiences as a multilayer interconnected network of actors lying in two main layers, i.e., immersion and coherence, which are entangled by a DT able to perform transformative actions for the listener.
Symbolic buildings and cities have long been targeted and still are being targeted raising the importance of safeguarding the world’s architectural heritage record, and our cultural memory. This article looks at a series of actions aimed at devastating historic cities and cultural heritage, and examines 3D visualization with Intelligent Reality as a new way to "post-trauma reconstruction". The method of modeling and reconstructing the lost heritage is demonstrated with an application to Palmyra.
Avatars are common virtual representations used in Extended Reality (XR) to support interaction and communication between remote collaborators. Recent advancements in wearable displays provide features such as eye and face-tracking, to enable avatars to express non-verbal cues in XR. The research in this paper investigates the impact of avatar visualization on Social Presence and user's preference by simulating face tracking in an asymmetric XR remote collaboration between a desktop user and a Virtual Reality (VR) user. Our study was conducted between pairs of participants, one on a laptop computer supporting face tracking and the other being immersed in VR, experiencing different visualization conditions. They worked together to complete an island survival task. We found that the users preferred 3D avatars with facial expressions placed in the scene, compared to 2D screen attached avatars without facial expressions. Participants felt that the presence of the collaborator's avatar improved overall communication, yet Social Presence was not significantly different between conditions as they mainly relied on audio for communication.
In this paper we describe the xR4DRAMA system, a solution that makes use of XR capabilities to support professionals who deal with disasters, man-made crises or media productions. The key contribution of this work in progress is the increase of situation awareness, which is achieved by the innovative combination of data collection, multimedia and sensor analysis, linking data, GIS and interactive XR technologies. The proposed platform is designed to facilitate the creation of immersive environments using semantically enriched content and comprises a powerful tool that is applicable to multiple real use case scenarios.
Cultural heritage is a lot more than monuments and collections of objects. In addition, it includes traditions and living expressions inherited from our ancestors and passed on to our descendants, such as songs, performing arts, social practices, rituals, festive events, knowledge and practices concerning nature, or the knowledge and skills to produce traditional crafts. These elements constitute the Intangible Cultural Heritage (ICH). Safeguarding of ICH is an important factor in maintaining social coherence along with cultural diversity, but it also is a challenge in modern societies. The digitization of ICH, with the transcription of its artifacts, is already becoming a springboard for its preservation and re-use, but intangible heritage has more to offer than a library of content. Lately, the advancements in extended reality interactive technologies and in artificial intelligence break new ground in understanding and sensing intangible heritage. In this paper, we present the CHROMATA platform which aims at facilitating the creation of novel immersive experiences to sustain ICH. To achieve this, we are developing a web platform that provides the content retrieval from online sources and cultural institutions; the AI services for multimedia analysis, e.g., 3D pose estimation, folklore dance recognition, and textual analysis; the authoring interface where the non-developers designers can create their virtual experience. The various data (images, video, text, audio, and 3D models) are analysed using state-of-the-art methods for dance recognition and visualization, Laban Movement Analysis and Labanotation generation, textual sentiment analysis and text generation. Moreover, 3D reconstruction techniques are employed to accurately capture places of interest, as well as human activities to create the related virtual spaces. The CHROMATA platform aims to boost the revival of intangible cultural heritage via easing the creation of more complete and enlightening immersive experiences. The platform development is driven and validated by real use case scenarios in which designers, creators and historians are building virtual experiences of customs and dances of Greek culture.
The orbital floor is a thin boney plate that supports the eye and its muscles. When sufficiently large, a fracture of the orbital floor leads to malposition or entrapment of the eye, necessitating surgical reconstruction. To reconstruct the orbital floor, the surgeon must retract the eyeball and dissect deeply through a small incision in order to safely place a synthetic plate beneath the eye, thus replacing the fractured bone. Conventionally, the accuracy of implant placement relies on the surgeon's expertise. Intraoperative imaging and navigation are rarely used due to their cost and setup times, so erroneous implant positioning is often unrecognized until postoperative imaging. This confers risk to the patient's eyeball, orbital vasculature, optic nerves, and stereotactic vision. In this work, we develop the workflow and user interface of an Augmented Reality (AR) system to aid surgeons with intraoperative placement of an orbital floor implant and ultimately reduce rates of implant malposition. The preliminary evaluation of workflow and user interface shows good potential of this platform. With improvements in accuracy through advancements in hardware and sensing method, the proposed method can become a successful AR solution to improve clinical performance.
The IEEE 2888 standard is in the process of development to provide common platform for digtial twin space or metaverse plus. There are four parts of the standard designed to provide standardized interface for synchronization between cyber and physcial world. This paper provides a brief introduction to this on-going standard activity.
Square fiducial markers are commonly used in Augmented Reality (AR) applications to affix AR content to a particular location in the real world. Unoccluded, these markers are quickly and easily identified, and AR content is realistically displayed in real time. However, because most square fiducial marker libraries use a thresholding-based method of detection, small edge occlusions often prevent markers from being found, or cause inaccurate estimations of marker pose. Both of these scenarios result in visual disturbances in the AR content. This is particularly problematic for hand-manipulated AR objects, where markers will suffer frequent edge occlusions by fingers. In this paper, we propose an alternative method of detecting single square fiducial markers where only two diagonal corners of the marker must be visible for detection. Our proposed method finds and classifies corners in the image, pairs candidate diagonal corners based on their gradient directions, and then attempts to find a homography between a standard template and corners in the image that may belong to a marker. Identification of potential markers is done using a commonly-used square fiducial marker identification algorithm. This method detects markers under partial edge occlusions at a rate of up to 2.48x that of a popular square fiducial library, detects and localizes square fiducial markers in isolated frames, and is fast enough to be used for real-time applications.
The COVID-19 pandemic has motivated a shift from physical interaction, approaches, or procedures due to social distancing. More people are at home using digital displays for real-time communication and engagement. With recent innovations in computational hardware for spatial applications, such as extended reality technologies, entry barriers for hosting intimate, interpersonal, virtual events continue to fall. The barrier falls at such a rate that the production or manufacturing of an extended reality system for different and simultaneous, practical scenarios may be built to solve communication issues resulting from COVID-19. This paper aims to describe a low-cost networked holographic system that can be used for various purposes such as communication, education, and gaming. We created three different applications to show the cross-compatibility, effectiveness, and usability of our system.
