As manufacturing scales up and demand for customization grows, computer-aided inspection technologies face increasing pressure to enhance efficiency and capability. Inspection path planning (IPP) for touch-trigger probes on coordinate measuring machines (CMMs) still suffers from heavy reliance on manual rules and poor coordination among planning stages. To address these limitations, this paper proposes a novel approach for IPP based on a hybrid planning framework, called Inspection Path Optimization via Deep Reinforcement Learning & Heuristic Search (InPO-DRLHS). InPO-DRLHS leverages hierarchical reinforcement learning (HRL) to generate collision-free paths (CFPs) while accounting for probe orientation changes, and integrates this architecture with heuristic-based optimization of measurement sequence to improve global planning efficiency. Specifically, within the HRL module: the lower-level agent employs an improved deep Q-learning approach to generate optimal CFPs under fixed probe orientations; the higher-level agent predicts optimal probe rotation positions from a voxelized scene representation, guiding orientation adjustments for the lower-level planning. Furthermore, the value function learned by the HRL module serves as a surrogate heuristic estimate within the heuristic algorithm during sequence planning, enabling path quality assessment without explicit CFP generation. Experimental results show that the proposed method achieves over 90% success rate in 3D CFP planning and reduces total measurement time by 21.7% compared to conventional approaches, indicating substantial gains in inspection efficiency.

Sequence modeling provides a self-attention mechanism-guided full-scale communication for multi-agent decision-making, showing potential in cooperative tasks. However, message sharing among all agents involves redundant observations and computational overhead. When dealing with complex multi-agent systems, it becomes difficult to distinguish valuable information for collaboration under limited exploration, leading to poor action prediction. In this paper, we propose a novel framework named group-based Multi-Agent Transformer (g-MAT). Guided by the role-discovery mechanisms, it constructs a compressed communication topology and achieves an efficient sequential policy. Specifically, the agents are divided into several role-labeled groups according to the behavioral effects. Then, the hybrid attention layer is designed to model the high-level interrelationships among agents based on the grouping scheme, where the local information from teammates and global information from other groups helps to promote collaboration. Extensive experiments on the challenging benchmarks demonstrate the superiority of g-MAT across a variety of cooperative tasks, where agents are able to aggregate into explainable groups with less manual assistance and task priors.

Pre-trained transformers have significantly advanced point cloud learning, but the conventional “pre-training + full fine-tuning” paradigm for these transformers requires separate fine-tuned models for each downstream task, incurring substantial storage costs. While prompt tuning offers a parameter-efficient alternative, existing methods struggle to effectively utilize local spatial information at a granular level and exhibit an over-reliance on task-specific data for prompt generation, hindering generalization. This paper introduces Prior-Aware Fine-Grained Prompt Tuning (PFPT) to address these limitations. PFPT introduces a fine-grained prompt generator that leverages local spatial features to produce per-token prompts, effectively capturing the local geometric structure within the point cloud. Furthermore, a prior-aware prompt enhancement module is proposed to mitigate the limitations of relying solely on task-specific data. This module utilizes prototypes derived from the pre-training data and establishes geometric correlations with the input point cloud to generate prior-enhanced features that enrich the prompts. Moreover, a Geometrically Consistent Fusion Module is designed to ensure geometric consistency between the final prompts and the point cloud tokens. Extensive experiments demonstrate that PFPT achieves performance comparable to or surpassing full fine-tuning while tuning less than 6.3% of the model parameters. Notably, our method surpasses SOTA Parameter-Efficient Fine-Tuning methods on challenging datasets such as ScanObjectNN PB 50 RS.

Optimizing the geometry deformation characteristics in contact problems with random rough surfaces is an important component of improving product performance, such as assembly accuracy, sealing percolation, contact thermal resistance, and electrical resistance. Traditionally, the deformation is computed by numerically solving the partial differential equations that govern the contact problems. In the optimization process, the deformations under a variety of random rough surfaces need to be solved. It is computationally intensive and necessitates a surrogate model to approximate the numerical solutions. This study employs non-uniform rational B-splines (NURBS) to represent the geometries involved in the contact problem and proposes treating the NURBS control points as image pixels, treating the deformations of these points as image pixel values. Furthermore, an image generator-enhanced deep operator network (IGE-DeepONet) that leverages an image generator as a trunk net is proposed to predict the deformations and a concatenation-based information fusion mechanism between the trunk net and branch net of the DeepONet was developed to improve the prediction accuracy. Based on the contact problem between a smooth elastomer cube and a rigid cuboid with a random rough surface, it was demonstrated that the proposed IGE-DeepONet has smaller test error and reduced training time compared to the standalone image generator and the traditional DeepONet which uses a fully connected neural network as trunk net.

Estimating the 6D poses of industrial parts is a fundamental task in automated industries. However, the scarcity of industrial part datasets and the effort involved to retrain networks present challenges when estimating poses for unseen parts. Although a few pre-trained networks demonstrate effectiveness on unseen objects, they often struggle to encode correct viewpoint for unseen industrial parts, which have significant geometrical differences compared to the pre-trained objects. Additionally, they overlook the viewpoint non-uniformity that frequently occurs in industrial settings, resulting in significant 3D rotation errors. To address these issues, a novel 6D pose estimator for unseen industrial parts is proposed. First, a Self-to-Inter (S2I) viewpoint encoder is introduced to efficiently generate discriminative descriptors that capture the viewpoint information of the observed image. The S2I viewpoint encoder utilizes an Inter-viewpoint attention module to facilitate prior viewpoint communication and leverages a saliency descriptor selection strategy to boost inference speed. Second, a viewpoint alignment module (VAM) is established and integrated with the ICP refiner. The VAM aligns non-uniform viewpoints in an analytical paradigm, leading to enhanced efficiency of the refinement process and more accurate final predictions. Experimental results on the LINEMOD dataset demonstrate competitive performance compared to state-of-the-art methods. Furthermore, the experiments conducted on eight unseen industrial parts validate the exceptional generalizability of our method, highlighting its potential in industrial applications.

Accurate depth perception is fundamental for numerous computer vision applications, yet depth maps acquired from commodity sensors often suffer from artifacts and inaccuracies, necessitating effective enhancement techniques. Polarization imaging, capturing rich geometric cues robust to illumination variations, offers a promising modality to guide this process. However, effectively integrating these cues within learning-based depth enhancement frameworks remains challenging. Existing methods often overlook the inherent representational gap between depth and polarization features and employ context-agnostic fusion mechanisms, incapable of generating prompts adaptive to cross-modal relationships and local context. To address these limitations, we propose a novel Rectified and Context-Aware Polarization Prompting (ReCAP2) framework for depth enhancement models. The ReCAP2 first performs initial feature rectification across both channel and spatial dimensions to bridge the modality gap. Subsequently, it generates fine-grained polarization prompts by leveraging dual-level context: utilizing cross-modal context ensures the prompts encode pertinent inter-modality relationships, while processing spatial neighborhood context yields prompts spatially tailored to regional content. Consequently, these dual-context aware prompts provide precise, adaptive guidance for the foundation model, facilitating more robust depth enhancement. Extensive experiments demonstrate the effectiveness of our method. On the multi-modal HAMMER dataset, our method shows superior accuracy and robustness across diverse sensor types in indoor scenes under both full fine-tuning and prompt tuning settings. Furthermore, cross-domain evaluations on the challenging CroMo dataset validate its strong generalization to outdoor environments.

Learning-based 6-DOF (6D) pose tracking, serving as a basis for most real-time applications such as augmented reality and robot manipulation, receives attention transiting from 2D to 3D vision, with the popularity of depth sensors. However, the irregular nature of 3D point clouds challenges this task, especially since the lack of explicit alignments hinders the interaction and fusion between the observed point clouds. Therefore, this paper proposes a novel approach named PA-Pose to achieve 6D pose tracking in point clouds. It takes the forward-predicted dense correspondences within an overlap as reliable alignments, to guide the feature fusion of the partial-to-partial point clouds. Then, the relative transformation pose of adjacent observations is continuously regressed from the point-wisely fused features by confidence scoring, avoiding non-differentiable pose fitting. In addition, a shifted point convolution (SPConv) operation is introduced in the fusion process, to further promote the local context interaction of the observed point cloud pair in the expanded alignment field. Extensive experiments on two benchmark datasets (YCB-Video and YCBInEOAT) demonstrate that our method achieves state-of-the-art performance. Even though only 3D point clouds are taken as input, our PA-Pose is still competitive with those methods fully utilizing RGB-D information in the single view. Finally, experiments in the real scene for tracking industrial objects also validates the effectiveness of the proposed method.

PurposeAssembly action recognition plays an important role in assembly process monitoring and human-robot collaborative assembly. Previous works overlook the interaction relationship between hands and operated objects and lack the modeling of subtle hand motions, which leads to a decline in accuracy for fine-grained action recognition. This paper aims to model the hand-object interactions and hand movements to realize high-accuracy assembly action recognition.Design/methodology/approachIn this paper, a novel multi-stream hand-object interaction network (MHOINet) is proposed for assembly action recognition. To learn the hand-object interaction relationship in assembly sequence, an interaction modeling network (IMN) comprising both geometric and visual modeling is exploited in the interaction stream. The former captures the spatial location relation of hand and interacted parts/tools according to their detected bounding boxes, and the latter focuses on mining the visual context of hand and object at pixel level through a position attention model. To model the hand movements, a temporal enhancement module (TEM) with multiple convolution kernels is developed in the hand stream, which captures the temporal dependences of hand sequences in short and long ranges. Finally, assembly action prediction is accomplished by merging the outputs of different streams through a weighted score-level fusion. A robotic arm component assembly dataset is created to evaluate the effectiveness of the proposed method.FindingsThe method can achieve the recognition accuracy of 97.31% and 95.32% for coarse and fine assembly actions, which outperforms other comparative methods. Experiments on human-robot collaboration prove that our method can be applied to industrial production.Originality/valueThe author proposes a novel framework for assembly action recognition, which simultaneously leverages the features of hands, objects and hand-object interactions. The TEM enhances the representation of dynamics of hands and facilitates the recognition of assembly actions with various time spans. The IMN learns the semantic information from hand-object interactions, which is significant for distinguishing fine assembly actions.

Human–robot collaborative assembly (HRCA) is one of the hot trends in intelligent manufacturing and has gained the attention of many researchers. In HRCA, errors in assembly sequence may reduce working efficiency and damage workpieces. To detect the assembly sequence errors rapidly and not influence the assembly process, the assembly sequence monitoring system demands high real-time performance and non-contact requirements. Therefore, an assembly sequence monitoring method based on workflow modeling is proposed for HRCA. The framework of the monitoring method is divided into an assembly state recognition block, an action library, and a monitoring block. The assembly state recognition block characterizes the complicated and unstructured assembly environment as a state vector in real time. The action library is constructed to model the workflow of the assembly sequences. The monitoring block matches the state-change vector calculated from the state vectors with defined actions in the action library and outputs the monitoring result. The assembly state recognition block is tested in simulation, and the assembly sequence monitoring method is validated in simulation and real-world experiments to exhibit the effectiveness. The comparison with other similar methods illustrates the superiority in monitoring range and recognition accuracy.

The height probability distribution (HPD) of random rough surface topography has significant effect on its contact behaviors. In this paper, an optimization model to calculate the optimal HPD of random rough surface topography to make its contact force-deformation characteristic satisfy the given target force-deformation characteristic was established. In the solution to calculate the optimal HPD, using Bezier interpolation curve to represent the curve of HPD function and using the positions of the control points of the Bezier interpolation curve as optimization variables were proposed. The solution was validated by numerical simulations implemented using matlab. The effect of the number of control points on the minimum objective function value was investigated. Through analyzing the obtained result, we found the most appropriate number of the control points is 35. Comparison with the method using height parameters as optimization variables in literature was implemented. It was found that the proposed method is applicable to broader types of contact force-deformation characteristic requirements than the method using height parameters as optimization variables.

Human-robot collaborative assembly (HRCA) is one of the current trends of intelligent manufacturing, and assembly action recognition is the basis of and the key to HRCA. A multi-scale and multi-stream graph convolutional network (2MSGCN) for assembly action recognition is proposed in this paper. 2MSGCN takes the temporal skeleton sample as input and outputs the class of the assembly action to which the sample belongs. RGBD images of the operator performing the assembly actions are captured by three RGBD cameras mounted at different viewpoints and pre-processed to generate the complete human skeleton. A multi-scale and multi-stream (2MS) mechanism and a feature fusion mechanism are proposed to improve the recognition accuracy of 2MSGCN. The 2MS mechanism is designed to input the skeleton data to 2MSGCN in the form of a joint stream, a bone stream and a motion stream, while the joint stream further generates two sets of input with rough scales to represent features in higher dimensional human skeleton, which obtains information of different scales and streams in temporal skeleton samples. And the feature fusion mechanism enables the fused feature to retain the information of the sub-feature while incorporating union information between the sub-features. Also, the improved convolution operation based on Ghost module is introduced to the 2MSGCN to reduce the number of the parameters and floating-point operations (FLOPs) and improve the real-time performance. Considering that there will be transitional actions when the operator switches between assembly actions in the continuous assembly process, a transitional action classification (TAC) method is proposed to distinguish the transitional actions from the assembly actions. Experiments on the public dataset NTU RGB+D +D 60 (NTU 60) and a self-built assembly action dataset indicate that the proposed 2MSGCN outperforms the mainstream models in recognition accuracy and real-time performance.

To grasp the randomly moving objects in unstructured environment, a novel robotic grasping method based on multi-agent TD3 with high-quality memory (MA-TD3H) is proposed. During the grasping process, the MA-TD3H algorithm obtains the object's motion state from the vision detection module and outputs the velocity of the gripper. The quality of the sampled memory plays a crucial role in reinforcement learning models. In MA-TD3H, transitions are saved in the memory buffer and high-quality memory (H-memory) buffer respectively. When updating the actor network, transitions are adaptively sampled from the two buffers by a set ratio according to the current grasping success rate of the algorithm. Also, the multi-agent mechanism enables the MA-TD3H algorithm to control multiple agents for simultaneous training and experience sharing. In the simulation, MATD3H improves the success rate of grasping the moving object by around 25 percent, compared with TD3, DDPG and SAC. While in most cases, MA-TD3H spends 80 percent of the time of the other algorithms. In realworld experiments on grasping objects in different shapes and trajectories, the average grasping prediction success rate (GPSR) and grasping reaching success rate (GRSR) of MA-TD3H are above 90 percent and 80 percent respectively, and the average GRSR is improved by 20-30 percent compared with the other algorithms. In summary, simulated and real-world experiments validate that the MA-TD3H algorithm outperforms the other algorithms in robotic grasping for moving objects.

To enjoy a wonderful cooking experience in a smart kitchen, users and designers need a visual interaction platform. This paper proposes a DT framework incorporating data processing, flow field online simulation, equipment monitoring, interaction and visualization. Specifically, the DT online simulation and visualization of the kitchen fume flow field serve as the foundation for appliance design and control. Additionally, users can gain deeper insight into the state and change trends of the kitchen. To address the online simulation of the flow field, a RFNO online simulation method is proposed. In addition, this paper proposes an Echarts-based 2D and UE5-based 3D flow field visualization method to enable dynamic visualization and interaction of the flow field. The proposed DT framework was successfully verified in the case of the smart kitchen, demonstrating its efficiency and effectiveness.

Assembly positioning by visual servoing (VS) is a basis for autonomous robotic assembly. In practice, VS control suffers potential stability and convergence problems due to image and physical constraints, e.g., field of view constraints, image local minima, obstacle collisions, and occlusion. Therefore, this article proposes a novel deep reinforcement learning-based hybrid visual servoing (DRL-HVS) controller for motion planning of VS tasks. DRL-HVS controller takes current observed image features and camera pose as inputs, and the core parameters of hybrid VS are dynamically optimized using a deep deterministic policy gradient (DDPG) algorithm to obtain an optimal motion scheme, considering image/physical constraints and robot motion performance. In addition, an adaptive exploration strategy is proposed to further improve the training efficiency by adaptively tuning the exploration noise parameters. In this way, the offline pretrained DRL-HVS controller in the virtual environment, where the DDPG actor–critic network is continuously optimized, can be quickly deployed to a real robot system for real-time control. Experiments based on an eye-in-hand VS system are conducted with a calibrated HIKVISION RGB camera mounted on the end-effector of a GSK-RB03A1 six degree-of-freedom (6-DoF) robot. Basic VS task experiments show that the proposed controller achieves better performance than the existing methods: the servoing time is 24% smaller than that of the five-dimensional VS method, a 100% success rate with the perturbed ranges of the initial position within 25 mm for translation and 20° for rotation, and a 48% efficiency improvement. Moreover, a planetary gear component assembly process case study, where the robot aims to automatically put the gears on the gear shafts, is conducted to demonstrate the applicability of the proposed method in practice.

For the large redundant manipulator, due to its long working distance and large mass, the number of links (i.e., manipulator's arms) that can be driven to move simultaneously is limited. Otherwise, the control accuracy and motion stability of the manipulator will deteriorate. Focusing on that, a weighted Newton iteration (WNI) algorithm for trajectory planning of the manipulator is firstly proposed, where the motion of the manipulator joints is controlled by a weight matrix, which is constant and related to each link's energy consumption. To dynamically adjust the weight matrix according to kinematic constraints and acquire better energy efficiency, an adaptive WNI (AWNI) algorithm is further proposed. In AWNI, the weight matrix is adjusted in real-time during the planning process, with considerations of the kinematic constraints and the energy consumption of the manipulator. The switch of the links between the working state and the non-working state is made through the weight matrix to achieve flexible control of the manipulator motion. Two evaluation functions are established to validate the effectiveness of AWNI in energy saving and motion stability control. Taking a 6 degrees of freedom (DOF) manipulator as an example, simulation experiments on trajectory planning are carried out and the results show the effectiveness of the proposed AWNI algorithm.

介绍了一种通用的利用三维CAD模型进行三坐标检测程序离线编制的系统开发方法,该系统基于三维CAD建模平台,集成了测头与坐标系管理模块、测量规划模块以及编程与仿真模块等功能,实现了工件在虚拟端的采样规划、路径规划、测量程序生成以及测量过程的虚拟验证,有效提升了测量程序编制的效率和质量,有助于实现批量零件的自动化测量.

Accurate and robust 6-DOF (6D) pose estimation from a single RGB image and depth map (RGB-D) image is an essential task of intelligent manufacturing, such as robot assembly and digital twin. However, incomplete and noisy 3-D data acquired from depth sensors make the task challenging, especially for various industrial parts without sufficient textures, where the occlusion further exacerbates the problem. To tackle this issue, this article proposes a geometry-enhanced network with incomplete shape recovery (GER-Net) to estimate the 6D pose of industrial parts. First, an incomplete 3-D shape recovery (ISR) module with a learnable shape protection (SP) layer is introduced to recover the complete 3-D geometry shapes of raw point clouds obtained from depth measurements. Subsequently, the multimodal features extracted from raw RGB-D data are enhanced with the geometry information from the recovered point cloud via multiscale concatenation and recurrent forward fusion in the point cloud space. In this way, the enhanced RGB-D representations contribute to the regression of accurate 6D pose. Experiments on two popular benchmark datasets (LineMOD and Occlusion-LineMOD) show that the proposed approach achieves state-of-the-art performance. Furthermore, a real-world low-texture industrial part dataset industrial texture-less machined and 3-D-printed parts (ITM3D) is presented to fully validate the effectiveness of our method, where it also achieves the best performance with remarkable accuracy and robustness.

Accurate 3D object segmentation in point clouds is a basis for industrial robot applications, such as robot manipulation and digital twin, which require an understanding of the 3D environment. However, the unstructured and disordered nature of point clouds makes it challenging, especially for the incomplete 3D data under a single view in the real-world scenario. To this end, this paper proposes a novel 3D object segmentation framework (3DT-Seg) based on Cross-Window Point Transformer (CP-Former). CP-Former captures the long-range dependencies between local windows and latent semantic boundaries to enhance the point-wise features extracted from irregular point clouds via a bidirectional cross-attention mechanism. In addition, a contrastive learning loss and an adaptive dual aggregation strategy are introduced on semantic transition regions during the semantic supervising and instance clustering process, respectively. In this way, the latent boundary information is further utilized to improve the overall segmentation performance. Experiments on the popular benchmark (SI3DS) dataset show the state-of-the-art performance of the proposed approach in terms of semantic and instance segmentation. Furthermore, a real-world point cloud dataset (IP-Cloud) for the robotic grasping task is presented to fully validate the effectiveness of our method in practice, where it also achieves remarkable performance.

针对传统智能优化算法在机械臂关节空间进行时间最优轨迹规划应用中存在的寻优效率低、优化结果全局性和稳定性差的问题,提出新的机械臂时间最优轨迹规划方法.在建立机械臂关节空间内的时间最优轨迹规划模型时考虑位置约束,根据输入的关节点列,使用S形曲线估算时间的取值区间,对生成算法的所有个体进行多种群竞争迭代,得出机械臂关节空间轨迹规划的时间最优解.与不同算法的仿真对比试验结果表明,所提方法较传统的优化算法具有更高的寻优效率和更好的优化全局性;所提方法的稳定性好,其多次优化结果的方差相较单种群算法低3个数量级.