In aerospace manufacturing, aircraft skin panels typically require manual secondary trimming prior to assembly to ensure appropriate assembly clearance, thereby meeting requirements for airtightness and structural safety. At present, the trimming process primarily relies on manual measurements and repeated trial-and-error fitting, involving frequent adjustments and multiple trimming operations. This process is inefficient, error-prone, and exhibits a low degree of digitalization. To address this issue, this paper proposes a digital trimming allowance computation and distribution optimization method for aircraft panels based on measuring complex assembly seam features, intending to advance digital trimming technologies for aerospace panel assembly, an area still lacking effective solutions. To tackle the challenge of measuring spatially closed seams under complex curvature conditions, a structure tensor and coordinate optimization-based optimal projection plane search strategy (STCPS) is introduced, along with a flexible seam measurement model (FSMM) to enable high-precision measurement of gap and flush features. In addition, a trimming allowance distribution optimization algorithm based on the trimming-aware coordinated strategy (TACS) is developed to address the low efficiency and quality issues caused by dispersed trimming allowances. Experimental results demonstrate that the proposed method significantly outperforms existing algorithms in terms of both accuracy and efficiency. Specifically, in the simulation environment, the mean measurement errors for gap and flush on eight types of model data are 4.9 mu m and 5.8 mu m, respectively; while on eight types of instance data the corresponding errors are 55.4 mu m and 74.7 mu m. The method achieves a mean trimming optimization rate of 96.84 % across 16 trimming scenarios, with a mean trim region reduction rate of 68.58 %. After intelligent planning of trimming segments, the trimming task volume is reduced by a mean of 40.55 %, and operational consistency is improved by an mean of 17.49 %. This study provides an effective technical solution and practical guidance for advancing the aerospace industry's digital trimming and assembly processes.

In Industry 5.0, high-precision human-robot collaborative assembly requires intuitive interfaces that minimize cognitive load. However, traditional 6-DoF control interfaces are hindered by kinematic coupling, as the control coordinate system (CCS) is rigidly fixed to the robot's end-effector (EEF), forcing operators to perform complex compensatory movements. To address this, we propose the Multi-Point Mapping Interface (MPMI), a Mixed Reality (MR) strategy that dynamically decouples the CCS from the EEF, allowing operators to align control with task-relevant geometric features. A single-factor within-subjects user study (N = 18) was conducted to validate the system against a traditional Single-Point Mapping Interface (SPMI). Experimental results demonstrate that MPMI significantly reduced task completion time by 14.6% (p = 0.022) and improved input efficiency, as indicated by a 15.0% reduction in cumulative operator input pose change (p < 0.001). Furthermore, subjective assessments confirmed a significant decrease in NASA-TLX cognitive load (p = 0.011) and superior usability of the system. These findings empirically validate the principle of dynamic control origin decoupling as a critical methodology for enhancing efficiency in complex robotic assembly, providing a foundation for future cognitive-optimized HRC systems.

Blind area manual assembly arises from the narrow and inter-obscuring structures of products which makes it difficult for workers to see the assembly site. Mixed Reality (MR) visualization systems can provide intuitive interfaces to improve efficiency and reduce errors in blind assembly, but the neural mechanisms in MR visualization of blind area assembly are not well understood. We conducted a user study (N = 24) that compared the difference in cognitive load between using a future advanced visualization system simulated using MR (Low latency, high accuracy and easy detection of current movements) and blind area assembly with the intention of explaining its core dynamics. Significant differences in electroencephalographic (EEG) features including power spectral density (PSD), coherence (Coh) and event-related potentials (ERPs) were analyzed between the two conditions. And the overall EEG was divided into three segments based on three events (Pickup, Localization and Insertion). The results showed that MR visualization exhibits PSD features such as decreased theta power, increased gamma power, and increased beta power, Coh features such as weakened theta, alpha connectivity, and increased beta, gamma connectivity, and shorter latencies for ERP components such as N2 and P300 at specific phases. We discuss these insights and directions for future work. We hope that research this may provide suggestions for the construction of current blind area assembly visualization systems.

Whether aviation angle pieces in aviation equipment are correctly assembled has a significant impact on structural stability and flight safety. In actual production, the assembly process of aviation angle pieces suffers from issues such as prone to picking similar wrong pieces, non-standard manual operations, and low assembly efficiency, which severely restrict the production efficiency and service life of aviation equipment. Therefore, this paper proposes a multidimensional compliance monitoring and AR-guided method for error-prone assembly process of aviation angle pieces based on TCN-improved cross-modal action recognition and intelligent process perception, providing a digital and intelligent solution for process monitoring and guidance at assembly site of small parts in the aviation equipment manufacturing industry. In this paper, a Progressive Group-interactive Length-aware Spatiotemporal Graph Convolutional Network (PG-LSTGCN) is proposed. At the temporal convolution (TCN) level, it incorporates a multi-scale group-interactive convolution to capture features of both single joint and inter-joint coordinated movements and a new spatiotemporal decoupled attention mechanism to adaptively extract features from action samples of varying durations. This design addresses the limitations of conventional temporal convolutions in recognizing complex joint motion patterns and sensitivity of existing models to action samples’ lengths. Moreover, by fusing assembly action features from image and skeletal modalities, a cross-modal recognition method is proposed for similar assembly actions of angle pieces, which effectively avoids the recognition ambiguity caused by relying solely on the skeletal modality when distinguishing similar assembly actions. Furthermore, a new intelligent assembly process perception algorithm based on process graph retrieval and dynamic matching is proposed, which can automatically perceive the current assembly progress and perform compliance monitoring of assembly operations, thereby ensuring an accurate, standardized, and efficient assembly workflow for angle pieces. Finally, a human-region-object multi-dimensional assembly monitoring and AR-based intelligent guidance system is developed, enabling real-time monitoring of angle piece types, assembly regions, and assembly workflows, as well as intelligent recognition, automatic process progression, and real-time tracking of the assembly process. Experimental results show that the proposed PG-LSTGCN network effectively improves recognition accuracy on both public and private action datasets. The cross-modal similar assembly action recognition method achieves an accuracy of 95.84% on the similar angle piece assembly action dataset, outperforming the single skeletal modality by 3.96%. The AR-assisted assembly region monitoring and intelligent process perception method reduces the frequency of incorrect angle piece selection and omitted assembly steps by over 90%, decreases the occurrence of non-standard actions by up to 48.39%, and improves assembly efficiency by up to 10.49%, while simultaneously alleviating the cognitive load and mental demands of assembly operators.

Freehand curve drawing on virtual surfaces constitutes a fundamental interaction in virtual reality (VR) design systems and is widely used for tasks such as 3D model decoration, annotation, and shaping. However, existing projection-based techniques frequently encounter trade-offs among intuitive control, interaction efficiency, and both trajectory continuity and stability. This paper presents FlexiStick, a novel interaction technique for drawing on virtual surfaces in VR. The method leverages historical projection points and incorporates frame-to-frame translational and rotational increments of the controller or stylus to improve directional control while maintaining trajectory continuity. In addition, an optimized closest-point projection strategy based on offset surfaces is employed to further enhance the projection stability in geometrically complex regions. A user study with 12 participants was conducted to compare FlexiStick with two representative projection methods. The results indicate that FlexiStick demonstrated higher speed controllability while maintaining high drawing accuracy and trajectory continuity. In addition, FlexiStick received the highest ratings in both user preference and system usability evaluations.

Aircraft manufacturing involves manually drilling numerous high-precision holes by experienced workers using drilling templates. Most of the previous AR-assisted drilling research is related to drilling in surgery and mining, which involve workflows distinct from those in aircraft manufacturing. Some spatial augmented reality (SAR) assistance systems have been used to project virtual images onto the aircraft panels showing hole positions to assist in manual drilling, as they do not require additional wearable or handheld devices, support hands-free operation, and are more suitable for long-duration work compared to other types of augmented reality assistance systems. However, existing systems only display hole positions, lacking guidance for some other important steps in template-based manual drilling. This work presents a SAR-assisted system for template-based manual drilling of aircraft panels that guides a worker through the entire workflow of template-based manual drilling. It consists of a smart shelf to assist in retrieving and returning drilling templates and checking for errors, a binocular vision system to automatically determine drilling template installation positions by detecting and matching location holes, and a projector to display SAR drilling instructions. A user study with 16 participants was conducted to evaluate the system usability. The results showed that compared to traditional documentation guidance, the developed SAR system can speed up manual drilling and provide better perceived ease-of-use and lower perceived workload. The limitations of this research are also discussed, and some directions for future work are provided.

Augmented Reality (AR) assembly instructions can be used to guide precise manual assembly, providing detailed real-time information to assist in executing operations. Most AR instructions for precise manual assembly are based on initial assembly tolerance allocation schemes (ATASs), which provide a precision constrained zone and require assembly according to the initial zone. Dynamic ATASs can adjust the subsequent tolerances according to the completed assembly errors, thereby reducing the assembly difficulty. However, ATASs do not affect the AR cues for assembly guidance, and their effects on the usability of AR instructions are subtle and unclear. This study focuses on evaluating the effects of initial and dynamic ATASs on the usability of AR instructions. An AR instruction based on an initial ATAS and an AR instruction based on a dynamic ATAS were developed fora precise manual assembly task. A user study was conducted with 16 participants to compare the two AR instructions. The results showed significant differences in assembly speed, assembly quality, perceived ease-of-use, and perceived workload. We found that the AR instruction based on the dynamic ATAS can expedite assembly with better perceived ease-of-use and lower perceived workload, but may reduce assembly quality. Based on the results, several recommendations were provided to help researchers and developers design better AR instructions for precise manual assembly.

Mixed reality remote collaboration assembly is a type of computer-supported collaborative assembly work that uses mixed reality technology to enable spatial information and collaboration status sharing among geographically distributed collaborators, including remote experts and local users. However, due to the abundance of mixed virtual and real-world information in the MR space and the limitations imposed by narrow field-of-view augmented reality (AR) glasses, users face challenges in effectively focusing on relevant and valuable visual information. Our research aims to enhance users' visual attention to critical guidance information in MR collaborative assembly tasks, thereby improving the clear expression of instructions and facilitating the transmission of collaborative intention. We developed the Information Recommendation and Visual Enhancement System (IRVES) through an assembly process information hierarchy division mechanism, a content-based information recommendation system, and a gesture interaction-based information visual enhancement method. IRVES can leverage the guidance expertise and preferences of remote experts to recommend information to filter out irrelevant information and present the key information that the remote expert conveys to the local user in an intuitive way through visual enhancement. We conducted a user study experiment of a collaborative assembly task of a small engine in a laboratory environment. The experimental results indicate that IRVES outperforms traditional MR remote collaborative assembly methods (VG3DV) in terms of time performance, operational errors, cognitive performance and user experience. Our research contributes a human-centered information visualization approach for remote experts and local users, providing a novel method and idea for designing visual information interfaces in MR remote collaboration assembly tasks.

When working together in asynchronous collaboration, one key aspect for collaborators is understanding co-workers' actions. While prior actions can often be reviewed, the recording and sharing of co-workers' bio-signal cues remain largely underexplored. In this paper, we present a Mixed Reality (MR) asynchronous collaboration system that enables workers to check their co-workers' previous actions and physiological state. We designed several different viewing modes for workers to playback and observe their co-workers' operations and heart rate. We conducted a user study to investigate how sharing bio-signal data between asynchronous collaborators could affect the overall user experience. We found that by showing additional physiological cues (e.g., heart rate), asynchronous collaboration could achieve a significantly stronger sense of co-presence and greater experience for the participants than sharing actions only.

It is vital to enable and establish both comprehension and simulation of environment-induced damage processes for predicting the remaining service life of engineering structures and components, conducting reliability analysis, and designing to enhance the material’s overall resistance to such damage. A fully coupled mechano-chemical peridynamic (PD) model for environment-induced degradation, including corrosion, was developed based on both peridynamic corrosion theory and the mechano-chemical effect theory. When the conditions for phase transition are satisfied, the movement of boundaries occurs autonomously, without requiring any supplementary boundary conditions to be specified within the model. This model effectively simulates degradation arising from the combined and interactive influences of mechano-chemical phenomena. To validate the model, in-situ electrochemical tests and stress corrosion cracking tests were conducted, with the results used to explore the effects of stress and/or load on the kinetics of environment-induced damage in an aluminum alloy. The experimental electrochemical parameter values closely match theoretical predictions, validating the mechano-chemical effects. As stress levels increase, the corrosion potential of aluminum alloy 7050 shifts negatively, corrosion current density increases, and the severity of corrosion worsens. The explicit finite difference method was employed to simulate the damage evolution of a typical stress corrosion crack in the aluminum alloy. This numerical model easily simulates the morphological evolution of corrosion pits with arbitrary shapes under different stress conditions during growth. The numerical predictions closely match the experimental findings. This innovative study demonstrates that the fully coupled mechano-chemical peridynamic corrosion model can accurately capture environment-induced damage and is a valuable tool for investigating the propagation and growth of such damage in aggressive environments.OPEN ACCESS Received: 24/07/2024 Accepted: 15/11/2024 Published: 20/04/2025

Virtual content in augmented reality (AR) applications can be tailored to a designer’s specifications. However, real-world environments are challenging to control precisely or replicate fully. Consequently, prototyping AR applications for specific environments is often difficult. One potential solution is employing mixed reality (MR) to simulate an AR system, enabling controlled experiments. Nevertheless, the effectiveness of using MR to simulate AR office work remains underexplored. In this paper, we report the results of a user study (N = 40) that investigated the impact of an MR simulation of an AR office on participants’ task performance and cognitive workload (CWL). Participants completed several office tasks in both an AR scene featuring a virtual monitor and an MR-simulated AR scene. During these tasks, CWL was measured using electroencephalography (EEG) and a subjective questionnaire. The results show that the performance of the pass-through window is a major constraint on the effectiveness of the MR simulation office. Finally, we discuss the study’s limitations and directions for future research.

Current Virtual Reality (VR) sketching systems lack natural bimanual interactions for drawing curves on 3D models. This article introduces HoldTabSketch, a VR sketching system that uses a tracked graphics tablet as a touch-sensitive physical proxy to enable realistic bimanual interactions for 3D curve drawing. The system features a Reoriented Bounding Boxes (ROBB) haptic redirection technique that automatically aligns virtual pen and hand movements with various virtual objects while maintaining tactile feedback. The system supports two additional bimanual interaction modes: 1) bare-hand object manipulation with pen-based sketching using visual feedback, and 2) tablet-assisted tangible interactions combining grasp support and surface drawing. A 12-user comparative study showed the tablet-based physical proxy significantly improved the realism of 3D object manipulation and surface drawing, yielding higher user preference and usability compared to vision-only or basic tangible approaches. The system demonstrates how hybrid physical-digital proxies can enhance 3D creative workflows in VR.

Manual precise manipulation of objects is an essential skill in everyday life, and Augmented Reality (AR) is increasingly being used to support such operations. In this study, we investigate whether detailed visualizations of position and orientation deviations are helpful for AR-assisted manual precise manipulation of objects. We developed three AR instructions with different visualizations of deviations: the logical deviation baseline instruction, the precise numerical deviations-based instruction, and the intuitive color-mapped deviations-based instruction. All three instructions visualized the required directions for manipulation and the logical values of whether the object met the accuracy requirements. Additionally, the latter two instructions provided detailed visualizations of deviations through numerical text and color-mapping respectively. A user study was conducted with 18 participants to compare the three AR instructions. The results showed that there were no significant differences found in speed, accuracy, perceived ease-of-use, and perceived workload between the three AR instructions. We found that the visualizations of the required directions for manipulation and the logical values of whether the object met the accuracy requirements were sufficient to guide manual precise manipulation. The detailed visualizations of the real-time deviations could not improve the speed and accuracy of manual precise manipulation, and although they could improve the perceived ease-of-use and user experience, the effects were not significant. Based on the results, several recommendations were provided for designing AR instructions to support precise manual manipulation.

Interaction with virtual objects is one of the essential features of Augmented Reality (AR) systems. One of its main issues is how to provide precise manipulation of distant virtual objects in AR. In this work, we explore bare-hand manipulation of distant objects in AR with DOF (degree-of-freedom) separation, motion scaling, and near-field metaphors. We developed two manipulation techniques: the distant widget-based metaphor (DWBM), and the near-field widget-based metaphor (NFWBM). We conducted a user study with 20 participants to compare the two techniques in terms of performance and user experience. We found that NFWBM has faster speed, lower mental effort, better ease-of-use, and a friendlier user experience. However, there is no significant difference in terms of precision; both techniques could manipulate distant objects precisely, with an average position error of less than 0.7 cm and an average orientational error of less than 1 degrees. We also discussed the limitations of this research and directions for future work.

Mixed Reality (MR) remote collaboration supports a shared workspace and various non-verbal communication cues that enable remote experts to express ideas in an immersive Virtual Reality (VR) space to guide local workers in industrial physical tasks by creating Augmented Reality (AR) instructions, such as remote assembly/disassembly guidance, emergency maintenance, and training. However, due to the information non-symmetrical caused by geographical distribution, it can be challenging and high-workload for remote experts to create clear and detailed AR instructions for local workers to match spatial relationships with the local workspace, while comprehensively considering the situation information (e.g., task-related objects and partner’s environment) in complex industrial physical tasks. Advances in ambient intelligence bring ideas for building a new remote collaboration framework that supports the adaptive generation of instructions. In the study, we developed a novel MR remote collaboration prototype system, which can automatically and adaptively generate clear, detailed, and standardized AR instructions based on remote experts’ simple and intuitive interactions and local contextual information to assist remote experts in guiding local workers to complete industrial physical tasks, especially industrial assembly tasks. The effect of two interfaces on industrial assembly tasks involving relatively complex process information and environments was explored in a user study: one was a baseline solution that uses a typical MR remote collaboration interface to support remote experts to create instructions (RECI) using popular non-verbal communication cues in the shared local 3D stereoscopic scene, and the other was our novel interface that additionally supported the adaptive generation of instructions via context awareness (AGICA) on top of the typical interface. The result of the user study showed that AGICA significantly improves collaboration performance, reduces errors and workload, and enhances usability and user experience compared to RECI. This demonstrates that supporting the adaptive generation of instructions is a feasible way to enhance MR remote collaboration in industrial assembly tasks with relatively complex process information and environments. Our research findings have a certain guiding significance for the design of MR remote collaboration systems in such tasks.

The visualization of complex spatial data is typically achieved using the Windows, Icons, Menus, and Pointer (WIMP) method. However, this approach has limitations, including a steep learning curve, non-intuitive interfaces, and limited visualization capabilities. To overcome these challenges, this study proposes an augmented reality-based (AR-based) visualization and interaction method for complex spatial data. By customizing the rendering pipeline, various visualization effects, such as clipping and lighting adjustments, can be achieved. Additionally, a prototype system called ARSpatialDX is developed. Furthermore, a user study is conducted to compare the AR and WIMP methods. The results indicate that AR outperforms WIMP in terms of virtual-reality matching, perception of physical properties, attractiveness, stimulation, perspicuity, and novelty aspects. Users generally find the direct interaction with data using their hands in AR to be highly interesting and express a strong willingness to continue using it.

Virtual content in Augmented Reality (AR) applications can be constructed according to the designer's requirements, but real environments, are difficult to be accurate control or completely reproduce. This makes it difficult to prototype AR applications for certain real environments. One way to address this issue is to use Virtual Reality (VR) to simulate an AR system, enabling the design of controlled experiments and conducting usability evaluations. However, the effectiveness of using VR to simulate AR has not been well studied. In this paper, we report on a user study (N=20) conducted to investigate the impact of using an VR simulation of AR on participants' task performance and cognitive workload (CWL). Participants performed several office tasks in an AR scene with virtual monitors and then again in the VR-simulated AR scene. While using the interfaces CWL was measured with Electroencephalography (EEG) data and a subjective questionnaire. Results showed that frequent visual checks on the keyboard resulted in decreased task performance and increased cognitive workload. This study found that using AR centered on virtual monitor can be effectively simulated using VR. However, there is more research that can be done, so we also report on the study limitations and directions for future work.

In this research, a blind area information perception and guidance approach for dynamic context is proposed as a solution to the issue of difficult and time-consuming assembly in blind areas. The proposed approach involves the utilization of real-time RGBD data to perceive both blind area context and operator hand information. The resulting data is then used to visualize the blind area scene and provide assembly guidance through the application of augmented reality technology. Unlike conventional methods, the proposed solutions are based on dynamic RGBD data rather than static predefined CAD models, making it simpler to configure and adapt to more scenarios. A user study was designed and conducted to confirm the feasibility of the suggested approach. The results indicate that the suggested approach can decrease assembly time by 49.5%, greatly lower the percentage of assembly errors, reduce the mental load on the workers, and significantly enhance their operational experience.

Fine operations exist in the form of teleoperation, human-machine cooperative work, and fine manual assembly. However, there is a lack of research that explores the characteristics of different assistive guidance methods and their use for fine operation tasks with different levels of difficulty. For this issue, this study first designed a prototype of a magnetic-based kinesthetic interaction device, and then using this device, a user study (n = 27) was designed and conducted. The experiment explored the performance of 2D and 3D visual cues as well as haptic cues when facing operation tasks with different levels of difficulty and analyzed the characteristics of each guidance modality and its reasons. The experimental results suggest that (1) visual cues are better suited than haptic cues to convey directional information in fine operations, (2) haptic cues can introduce an additional physical load, and (3) 2D visual cues' missing depth information can affect operations in the view direction.

Manual precise manipulation of objects is a critical skill in daily life, and Augmented Reality (AR) is increasingly used to support such tasks. In this study, we propose a system utilizing pseudo-haptic feedback to support precise manipulation for six degrees of freedom (6DOF). Two types of AR instruction interfaces were developed: Visual Deviation Instruction Interface (VDI) and Pseudo-Haptic Instruction Interface (PHI). A user study with 18 participants compared the two instruction interfaces in terms of performance and user experience. The objective measures of performance (task completion time, deviation), and the subjective measures (system usability scale, NASA Task Load Index) were collected. Results show that both instruction interfaces effectively support manual precise manipulation, achieving position deviations under 2 mm and orientation deviations under 1 degrees. PHI outperformed VDI in speed, mental effort, physical demand, performance, perceived workload, custom user experience elements and reduction deviations of manual precise manipulation. Finally, we discuss research limitations and future directions.