Most existing Human-Object Interaction (HOI) detection methods focus on supervised learning, but labeling all interactions is costly because of the enormous possible combinations of objects and verbs. Zero-shot HOI detection emerges as a promising approach to address this problem but encounters challenges when facing unseen interactions. To this end, we propose a novel two-stage Visual-Text modeling HOI detection (VT-HOI) method which can effectively recognize both seen and unseen interactions. In the first stage, the features of the humans and the objects are extracted by DETR and concatenated as the query sequences. In the second stage, local and global memory features from the Visual Encoder are fused into the corresponding query sequences by our proposed Semantic Representation Decoder with the cross-attention mechanism. Then we perform cosine similarity computation between visual features and text features, which are extracted or label-generated by Visual Representation Head (VRH) and Text Feature Memory (TFM) module respectively. Finally, the similarity matrix is fused with the results of the classification head for training or inference. The comprehensive experiments conducted on HICO-DET datasets demonstrate that the proposed VT-HOI significantly outperforms the state-of-the-art methods.
Virtual reality (VR) demonstration is safe, convenient, low-cost, and has easy access to learning data. Keyframes have been extensively used in learning from demonstration (LfD) to teach robots skills, especially when the skills need to be further updated and adapted. However, few works use keyframe-based methods in VR demonstrations to combine the advantages of both. Besides, human annotations and multiple demonstrations are always required to generate keyframe models, which causes problems of large demonstration workload, poor interpretability, and low generalization performance. To tackle the issues mentioned above, we propose VRCK: a VR teaching system based on generation and editing of constrained keyframes. In this system, only one demonstration is needed to generate the skill keyframes, path keyframes and additional keyframes. Constraints can be generated automatically using object detection and binocular localization algorithms. During the VR teaching process, users can visualize keyframes and constraints, and use UI to bind and edit them at any time. Finally, three task cases and a physical experiment are provided to validate our system and illustrate how it enables intratask generalization.
To deliver ultra-high resolution 360-degree video (such as 8K, 12K, or even higher) across the internet, viewport-dependent streaming becomes necessary to save bandwidth. During viewport switches, clients and servers will instantly exchange coordination info and contents for the given viewports. However, those viewport switches pose a serious challenge for video encoding because the temporal dependency between contents within changing viewports is unpredictable. In existing practices, it is commonly noted that GOP (Group of Pictures) size in a bitstream intrinsically prohibits the reduction of the viewport switch latency, such as Motion-to-photon (MTP) latency, or motion-to-high-quality (MTHQ) latency. In this paper, we presented a Scalable Video Coding (SVC) based bitstream schema, which can structurally remove the impacts of GOP in viewport-dependent streaming and provide instant viewport switches within one-frame time (the best possible). In addition, combined with tiling, this new coding schema allows an efficient packing of the non-adjacent regions within a viewport of 360-degree video. Our experiments also show that the overall encoding with this SVC-based approach is faster than with multi-stream approaches. Compared with current 360-degree video streaming solutions based on MPEG-I OMAF, our approach is superior in terms of viewport switch latency, simplicity of viewport packing, and encoding performance.
The new framework of infrastructure standards for education informatization had been constructed with four levels of digital base, application scenario, system specification and goal guidance, which was combined with the concept of Metaverse in order to improve the quality of ideological and political courses in colleges and universities. The modern history review simulation system was designed by Unity3D game engine and taken as an example, in which C# as the game logic programming language, Node.js as the server programming language, and MySQL as the game database. The path to improve the quality of ideological and political courses had been explored with six key features of new network, new platform, new resources, new campus, new application and new security. The red education resources were introduced into the lessons, and then the transformation of red resources can be realized by the digital technology. It can be seen from the experimental test result, the excellent teaching effect was achieved.
Gaze estimation is a vital indicator of human behavior and has diverse applications, such as virtual reality, human-machine interaction, and medical analysis. However, it remains a challenging task to accurately predict the direction of a person’s gaze due to factors like unique eye appearance, varying lighting conditions, and head poses. Recent advancements in deep learning have improved appearance-based gaze estimation, but there is still room for improvement. In this paper, we propose a multi-loss convolutional neural network that combines coarse and fine classification to achieve higher accuracy. Our model uses an EfficientNet-B3 backbone and a fine-grained classifier trained with the aid of other coarse-angle units. The final prediction result uses integrated regression. We evaluate our model on four datasets collected under unconstrained settings and demonstrate its state-of-the-art accuracy of 3.89°, 6.93°, and 10.78° on MPIIFaceGaze, RT-Gene, and Gaze360 datasets, respectively. The code will be released on https://github.com/hhqiang/HG-Net/ after the paper’s acceptance.
Virtua1 reality (VR) technology has become one of the research hotspots of technology to promote teaching because of its high immersion and good interactivity. Putting VR games into education can make learning more participatory and incentive. Nevertheless, the majority of current technologies and efforts to accomplish human-computer interaction necessitate additional hardware, which has certain limits. To solve these problems, realize real-time operation to provide direct feedback, and enhance the user’s sense of immersion and experience, we have developed a real-time hand capture algorithm using a monocular camera to perform more accurate user gesture recognition. In the algorithm, we use the backbone architecture of deep residual network ResNet-50 as a feature extractor. Through the combined training of 2D and 3D annotation data, we are able to effectively predict 2D posture and 3D spatial information and implement virtual content interaction. This method achieves real-time performance (90fps) and accuracy (95.6%) on existing datasets and outperforms existing methods in hand mesh/pose accuracy and hand image alignment. We built and implemented a virtual reality game based on the method proposed in this research, and then transplanted it onto the VR platform with an ecological setting. While providing users with an immersive experience, we also want to use virtual reality technology to teach, play, and promote traditional Chinese culture.
Virtual museums have gained popularity due to their interactive and immersive media experiences and have been used to train technical staff and develop design strategies. They present unique challenges for traditional museums, especially when attracting younger audiences. To engage younger visitors, museums must innovate and improve their offerings by providing opportunities for active engagement and immersive cultural experiences. Research has shown that teaching virtual museum design to art students positively impacts their perceptions of museums and increases their interest. This thesis aims to design an effective virtual museum design course using the ADDIE model for undergraduate art students, focusing on the creation of virtual museums using the Unity3D engine, and exploring the potential of virtual museums to enhance young people's attitudes towards museum visits.
In the process of ship painting, the distribution of coating thickness and its calculation is difficult due to the complexity of the coating formed by manual spraying. Therefore, a modeling and simulation method of manual spraying based on virtual reality (VR) is proposed. The manual spraying model (MSM) is established by the relationship between gun parameters, position, direction, and coating thickness. Then the MSM is verified by designing a spraying test. The results show that the simulation and test error is less than 8%, which proves the accuracy of MSM in this paper. Based on the MSM, a VR-based ship spraying simulation scene was developed using C# and Unity3D, and the error was less than 5% by comparing it with the coating thickness of the actual test. This further illustrates the reliability of the MSM in this paper.
This paper proposes SurfChessVR, a novel gaming interface that deploys chess games on parametric surfaces in Virtual Reality (VR). Unlike traditional chess in VR where all chessmen are placed on a plane chessboard, our approach automatically places chessmen on arbitrary parametric surfaces such as spheres, torus, cones, etc. More specifically, given a mathematical equation that describes a parametric surface as input, our approach automatically generates the chessboard in the shape of that parametric surface and places the chessmen on that surface at different squares with corresponding orientations that align with the surface’s normal directions. In order to deliver players with immersive gaming experiences of playing chess on a parametric surface, we implemented a game AI algorithm called the Min-Max algorithm that enables the computer to game against the player in VR. Experiment results and user studies validate the effectiveness and correctness of our approach.
The current mainstream way of cultural relic display is static display, that is, the cultural relic entity is presented to the user in a single way, the information output efficiency is low, and the user experience is ordinary. Our system to improve the display effect of cultural relics is mainly reflected in the hardware design and front-end display part: In the hardware part, we designed the cultural relics virtual-real combination display cabinet, which is mainly composed of an LCD transparent display screen that can be opened and closed and cabinet body. Based on augmented reality technology, the system designs a display mode that combines virtual and real cultural relics, namely the combination of cultural relics entity and digital 3D model of cultural relics. The transparent LCD screen is used to map the 3D model of cultural relics to the front of the cultural relics entity, and at the same time displays the relevant background information of cultural relics, so that the cultural relics information can be displayed to users in an all-round and multi-angle way. Equipped with Leap Motion gesture sensor, gesture recognition is used to realize contact-free interactive operation, which improves users’ browsing experience on the premise of ensuring the safety of cultural relics.
This paper aims to study the official virtual characters “ Huai Xiaobu” and “ Huai Xiaomei” in Huai’an City, how to rely on virtual reality technology to enable canal culture transmission. Under the background of metauniverse, Huai’an Canal culture transmission as the entry point, through the analysis and induction of virtual reality technology, first of all, on the basis of traditional IP, the use of 3D design software to further digital virtual people and VR design creation. Based on the image of “Huai Xiaobu” and “Huai Xiaomei”, a series of derivative design, virtual image and reality combined video design, the virtual character “Huai Xiaobu” communication form is divided into virtual reality, augmented reality, mixed reality three case forms to achieve, further enhance the communication power and influence of Huai’an new media. This article introduces the basic principles of various methods, overcome the specific technology of the case, compare and summarize the advantages and disadvantages of various forms, and finally achieve the promotion of the canal cultural information transmission rate, to meet the emerging needs of the audience, it is a highly comprehensive city history and humanity, not only can directly express the cultural and economic characteristics of Huai’an, broaden the depth and breadth of communication, Cooperatively build multi-level and multi-dimensional communication ecological pattern, but also can become the name card of Huai’an, combined with the characteristics of Huai’an, improve the internal cohesion of Huai’an, external reputation.
The traditional movable-type printing craft is confronted with the problem of difficult learning and inheritance. In order to enable the public to experience or learn the movable-type printing craft with the help of virtual platform and make the movable-type printing craft have more possibilities for future development, the VR experience project of virtual movable-type printing museum is designed and developed through data collection and modeling, 2D interface design, virtual interactive design and systematic development of visual platform. It can retain the integrity of the procedure of movable-type printing craft to the greatest extent, provide a freedom, efficient, safe and green simulation learning platform, rapidly promote the traditional movable-type printing culture in a large range, innovate the presentation of movable-type printing craft and promote the development of printing technology.
Depression is one of the most significant mental or psychiatric disorders that impact millions of people worldwide. Commonly used solutions to treat depression are a combination of nondrug therapies such as psychotherapy and physical therapy, along with antidepressant medication prescriptions. However, therapy sessions are often terminated by patients losing interest in continuing due to the cost and time-consuming nature of the treatments, leading to unsatisfactory recovery progress. To assist in diagnosing and treating depression using advanced technologies, numerous researchers have explored and developed physiological computing techniques. One of the popular trends is the use of neurofeedback based on Virtual Reality (VR), which has grown in popularity. This current survey examines the hierarchy from medicine and therapy to software models, with a particular emphasis on neurofeedback and its significant use in conjunction with virtual reality, which is the spotlight for all researchers. While the search for more universally satisfactory approaches is still ongoing, the main goal of this review is to help researchers by providing a thorough overview of currently used techniques and identifying potential changes that could be made to improve the model’s performance.
Numerical calculation methods with high precision and fast speed are crucial for solving heat conduction problems, a method for solving and simulating the two-dimensional heat conduction process was proposed. This method discretized the two-dimensional heat equation in time and space; then combined the modified Crank-Nicolson method with the alternating direction implicit method to obtain the discrete equations of each node; then the algebraic equations was solved and combined with the Richardson extrapolation method to improve the convergence speed of the calculation; Finally, the two-dimensional heat conduction process was simulated by computer graphics technology, and the dynamic visualization of the temperature field in the two-dimensional heat conduction region was realized. The method is second-order convergent both in time and space, and its order is increased from the second order to the fourth order when combined with the Richardson extrapolation method. Experiments showed that this method can stably realize the simulation and dynamic visualization the of two-dimensional heat conduction process.
Learning accurate 3D shapes from sparse and incomplete point clouds is challenging and meaningful, on account that the point clouds with low resolution always lack representative and informative details. This paper presents a novel deep auto-encoder called TGNet, which is formulated based on a tree-based generative adversarial network (GAN), to address self-supervised learning tasks on the point cloud with low sparsity. On the encoder side, we employ a PointNet-based framework to intensively capture the global representations. To better infer the spatial information in latent space, we propose a spectral graph learning module in with due consideration to graph topology. Further, we present a new loss that combines Wasserstein metric and multi-resolution Chamfer distance to better estimate global 3D geometry and structural details. The proposed TGNet achieves state-of-the-art performance for various point cloud learning tasks. Qualitative and quantitative evaluations demonstrate the novelty of the proposed model.
Manual processing of tomographic data volumes, such as interactive image segmentation in medicine or paleontology, is considered a time-consuming and cumbersome endeavor. Immersive volume sculpting stands as a potential solution to improve its efficiency and intuitiveness. However, current open-source software solutions do not yield the required performance and functionalities. We address this issue by contributing a novel open-source game engine voxel library that supports real-time immersive volume sculpting. Our design leverages GPU instancing, parallel computing, and a chunk-based data structure to optimize collision detection and rendering. We have implemented features that enable fast voxel interaction and improve precision. Our benchmark evaluation indicates that our implementation offers a significant improvement over the state-of-the-art and can render and modify millions of visible voxels while maintaining stable performance for real-time interaction in virtual reality.
There is a rapidly increasing need for learning resources in the field of Artificial Intelligence aimed at newcomers. To this end, we have attempted to make an intuitive visualization of Artificial Neural Networks in Virtual Reality, while trying to make it as general and extensible as possible to allow for future work to build upon it to create a series of highly intuitive and interactive learning resources. Our implementation results in a framework built using careful design considerations that we think results in a solid building block for work that aims to interactively educate individuals in fields such as Artificial Intelligence and other such algorithms.
In the traditional building HVAC system equipment operation and maintenance process, some of the HVAC system equipment installation locations are hidden in the building space, so many HVAC equipment repair and maintenance operations have limited maintenance space and low visualization, and the building air conditioning system fault diagnosis process is often plagued by on-site work collaboration factors resulting in low fault repair efficiency. Based on the above problems, this paper proposes to superimpose Mixed Reality (MR) technology on top of the Building Information Model (BIM) to develop a BIM+MR-based building HVAC system equipment fault diagnosis system to improve the immersive and remote visualization interaction capability in the HVAC system equipment maintenance process through the digital twin technology. The BIM+MR fault diagnosis system is developed to improve the immersive and remote visualization interaction capability in the HVAC system equipment maintenance process using digital twin technology. The technical verification shows that the efficiency of HVAC’s air conditioning equipment fault diagnosis in the pilot project has increased by 1.88 times after applying MR technology.
Spinal cord injury (SCI) can impair hand function needed for activities of daily living. Physical therapy is crucial to rehabilitating function but is time-and effort-intensive. Therefore, training approaches that facilitate greater engagement for faster motor learning are needed. In prior work, we showed how training with augmented feedback from an instrumented glove with computational intelligence to detect secure grasp could improve functional performance immediately. During training, feedback about secure grasp was provided with concurrent visual and audio (i.e., multimodal) cues. Furthermore, the cues were provided at progressively reduced time delays to induce a sense of agency (perception of control) as a cognitive basis to accelerate motor learning. In this study, we newly incorporated virtual reality (VR) with this glove system and training paradigm to enhance the augmented feedback and to assess its effects on persons with cervical-level incomplete SCI. Adding VR to the training regime significantly improved the performance of a grasp-and-place task by persons with SCI (n=6). Moreover, neural activity, as measured from scalp-surface electroencephalography (EEG), was significantly more pronounced after training with VR feedback. Performance improvement and increased neural activity suggest the potency for neuroplasticity and motor learning when adding VR enhancements to our training approach for clinical populations having severe neurological trauma such as SCI.
The sustainable teaching of quality methods in the sense of Lean Management and Six Sigma with the help of assistance systems, such as virtual reality glasses, represents a new and growing aspect of continuing education programs. The development and use of virtual learning environments offers the chance to deepen the theoretical prior knowledge through interactive learning opportunities. In this way, existing learning concepts are supplemented with virtual teaching content. Complex or difficult to present learning settings can be mapped virtually without high material consumption or operating costs. In this paper, the reference to quality methods for teaching with virtual reality is established and a classification in the Assisted Reality Implementation Model is made. Subsequently, the requirements for the programming of a virtual learning environment are shown. Using the example of the quality method Failure Mode and Effects Analysis, the procedure and implementation will be explained and the research question will be examined whether a reasonable implementation in virtual reality is possible. Special attention is paid to the technical implementation and the requirements for the design of the virtual environment. In order to be able to answer the research question, the question of acceptance is put in the foreground and this is carried out with the help of a questionnaire. In the following, first results of the explorative tests from used questionnaires are presented to get an overview of the explorative acceptance.
