Immersive technologies, such as augmented reality (AR) and virtual reality (VR), have received widespread attention in recent years. With the rapid advancement of head-mounted displays (HMDs) technology, AR and VR offer users the capacity to interact with virtual objects in immersive environments. Although AR and VR are both based on HMDs, their own unique features differentiate them from one another. Besides, virtual object manipulation is an essential element in AR and VR due to the immersive features, and the existing research has concentrated on enhancing the user experience for virtual object manipulation in AR and VR applications. However, there is little literature on the influence of different immersive environments on users’ feeling of control for virtual object manipulation. Therefore, this study aims to compare users’ feeling of control for virtual object manipulation between AR and VR environments via self-report and performance measures. According to the results, the study concludes that the VR environment results in a better feeling of control than the AR environment when users manipulate virtual objects with mid-air gestures.
Most existing Human-Object Interaction (HOI) detection methods focus on supervised learning, but labeling all interactions is costly because of the enormous possible combinations of objects and verbs. Zero-shot HOI detection emerges as a promising approach to address this problem but encounters challenges when facing unseen interactions. To this end, we propose a novel two-stage Visual-Text modeling HOI detection (VT-HOI) method which can effectively recognize both seen and unseen interactions. In the first stage, the features of the humans and the objects are extracted by DETR and concatenated as the query sequences. In the second stage, local and global memory features from the Visual Encoder are fused into the corresponding query sequences by our proposed Semantic Representation Decoder with the cross-attention mechanism. Then we perform cosine similarity computation between visual features and text features, which are extracted or label-generated by Visual Representation Head (VRH) and Text Feature Memory (TFM) module respectively. Finally, the similarity matrix is fused with the results of the classification head for training or inference. The comprehensive experiments conducted on HICO-DET datasets demonstrate that the proposed VT-HOI significantly outperforms the state-of-the-art methods.
Virtual reality (VR) demonstration is safe, convenient, low-cost, and has easy access to learning data. Keyframes have been extensively used in learning from demonstration (LfD) to teach robots skills, especially when the skills need to be further updated and adapted. However, few works use keyframe-based methods in VR demonstrations to combine the advantages of both. Besides, human annotations and multiple demonstrations are always required to generate keyframe models, which causes problems of large demonstration workload, poor interpretability, and low generalization performance. To tackle the issues mentioned above, we propose VRCK: a VR teaching system based on generation and editing of constrained keyframes. In this system, only one demonstration is needed to generate the skill keyframes, path keyframes and additional keyframes. Constraints can be generated automatically using object detection and binocular localization algorithms. During the VR teaching process, users can visualize keyframes and constraints, and use UI to bind and edit them at any time. Finally, three task cases and a physical experiment are provided to validate our system and illustrate how it enables intratask generalization.
To deliver ultra-high resolution 360-degree video (such as 8K, 12K, or even higher) across the internet, viewport-dependent streaming becomes necessary to save bandwidth. During viewport switches, clients and servers will instantly exchange coordination info and contents for the given viewports. However, those viewport switches pose a serious challenge for video encoding because the temporal dependency between contents within changing viewports is unpredictable. In existing practices, it is commonly noted that GOP (Group of Pictures) size in a bitstream intrinsically prohibits the reduction of the viewport switch latency, such as Motion-to-photon (MTP) latency, or motion-to-high-quality (MTHQ) latency. In this paper, we presented a Scalable Video Coding (SVC) based bitstream schema, which can structurally remove the impacts of GOP in viewport-dependent streaming and provide instant viewport switches within one-frame time (the best possible). In addition, combined with tiling, this new coding schema allows an efficient packing of the non-adjacent regions within a viewport of 360-degree video. Our experiments also show that the overall encoding with this SVC-based approach is faster than with multi-stream approaches. Compared with current 360-degree video streaming solutions based on MPEG-I OMAF, our approach is superior in terms of viewport switch latency, simplicity of viewport packing, and encoding performance.
The new framework of infrastructure standards for education informatization had been constructed with four levels of digital base, application scenario, system specification and goal guidance, which was combined with the concept of Metaverse in order to improve the quality of ideological and political courses in colleges and universities. The modern history review simulation system was designed by Unity3D game engine and taken as an example, in which C# as the game logic programming language, Node.js as the server programming language, and MySQL as the game database. The path to improve the quality of ideological and political courses had been explored with six key features of new network, new platform, new resources, new campus, new application and new security. The red education resources were introduced into the lessons, and then the transformation of red resources can be realized by the digital technology. It can be seen from the experimental test result, the excellent teaching effect was achieved.
