Instance-level 6D object pose tracking involves tracking a known object in 3D space and estimating its six degrees of freedom (6DoF) pose across consecutive images, starting from the initial pose in the first frame. This technology has wide-ranging applications in various fields, including robotics, augmented reality, and human–machine interaction. Over the years, significant progress has been made in this field. Many methods tackle the problem of instance-level 6D pose tracking from RGB images. These techniques can be classified based on their use of keypoints, edges, region information or direct optimization. Additionally, the availability of affordable RGB-D sensors has prompted the utilization of depth data for 6D pose tracking. Another notable advancement is the adoption of deep neural networks, which have shown promising results. Despite these developments, survey studies on the latest advancements in this field are lacking. Therefore, this work aims to fill this gap by providing a comprehensive review of recent progress in instance-level 6D object tracking, covering the aforementioned advancements. This paper provides a detailed examination of metrics, datasets, and methodology employed in this field. Based on the problem modeling approach, methods reviewed in this paper are categorized into optimization-based, learning-based, filtering-based approaches and hybrid approaches that combine various techniques. Furthermore, quantitative results on several publicly available datasets are presented and analyzed, along with applications and open challenges for future research directions.

The prosthesis grasping force feedback can be conveyed to users via sensory substitution (vibrotactile encoding) or modality-matched stimulation (physical force exertion). To date, the two approaches are rarely compared and, typically, they are designed to convey only the normal grasping force component. This letter developed a lab based closed-loop myoelectric prosthetic control system using a collaborative robotic arm to render high-fidelity 3D grasping force from the prosthetic end-effector on the user. Experiments with 11 subjects (10 able-bodied, 1 amputee) demonstrated that: (1) Tangential force feedback facilitated accurate perception of slippage tendencies under visual deprivation. (2) In stiffness identification, visual + force feedback significantly outperformed both visual only and visual + vibrotactile feedback in accuracy (87.1% vs. 43.1% and 80.2%) and decision time (9.8 s vs. 11.6 s and 10.7 s). (3) In force tracking, exclusive force feedback (rather than exclusive vibrotactile feedback) significantly minimized tracking errors compared to visual feedback, with visual + force feedback achieving the lowest RMSE (3.70 N) compared to the visual feedback (4.89 N) and no feedback (5.34 N). The results indicate that visual feedback alone is inadequate for force control, and the addition of a haptic channel significantly improves performance. Overall, the proposed 3D force feedback system offers broad applicability, implying that its socket-integrated miniaturization could facilitate dynamic grasping and manipulation.

In response to the current convolutional neural network-based robotic arm grasp pose detection networks being susceptible to interference from redundant information, and the tendency to misjudge object contour areas as grasping execution poses , leading to low grasping success rates of robotic arms, this paper introduces SA-Grasp, a lightweight convolutional neural network for robotic arm grasping that integrates self-attention mechanism. It reduces interference from redundant information and improves grasping accuracy. SA-Grasp achieved high detection accuracies of 98.37

Bridge infrastructure maintenance is crucial for public safety and efficient transportation networks. Deep-learning-based object detection models offer promising solutions for automated bridge crack detection, reducing the reliance on traditional visual inspection methods which are often time-consuming, subjective, and potentially hazardous. This study conducts a comprehensive evaluation of two state-of-the-art object detection models, You Only Look Once (YOLO) v8 and YOLOv5, in the context of bridge crack detection. Our experimental evaluation involves training and testing both models on a dataset comprising images of bridge surfaces with varying degrees of crack severity. We compare their performance in detection accuracy, inference speed, and model size. The results demonstrate that, while both YOLOv8 and YOLOv5 exhibit strong performance in detecting bridge cracks, there are discernible differences in their detection accuracy and computational efficiency. YOLOv8, particularly the nano model, demonstrates superior accuracy in identifying crack patterns. It achieved an impressive mean average precision of 0.602 at 0.5 intersection over union while maintaining a remarkably low processing time of just 1.1 ms per image and a compact model size of 5.96 MB. These achievements make it better suited to meet the lightweight and accuracy requirements of object detection using smart device applications.

To solve the problems of low trajectory tracking accuracy, large moment amplitude and slow response of manipulator arm, a radial basis function neural network and fuzzy sliding mode control (RBF-FSMC) algorithm was proposed. Firstly, an improvement is made to the quasi-sliding mode in traditional sliding mode variable structure control by replacing the saturation function with an enhanced hyperbolic tangent function as the sliding mode smooth switching function. Sub-sequently, the RBF neural network model is utilized for its superior approximation capability and generalization ability to dynamically compensate for sudden unknown disturbances, thereby mitigating the adverse effects of external disturbances and modelling errors on the control system. The fuzzy adaptive logic algorithm approximates the switching gain in the sliding mode controller, adaptively adjusting the switching gain based on the system's varying states to reduce the substantial chattering associated with sliding mode control. Finally, MATLAB/Simulink simulation results demonstrate that the proposed RBF-FSMC algorithm significantly improves trajectory tracking accuracy, effectively reduces the amplitude of torque oscillations, and confirms the efficiency and advanced nature of the RBF-FSMC algorithm.

The task of utilizing machine vision for the detection of casting surface defects is characterized by small targets, real-time performance, and ease of mobility. The direct application of current mainstream object detection networks for defect detection presents issues of low accuracy and efficiency. Consequently, in this paper, we introduce Casting-DETR, an end-to-end network designed for casting surface defect detection. To assess and validate the model’s performance, 554 images of casting samples with surface defects were employed. Casting-DETR achieved an impressive detection rate of 98.97% on the test set, with a single image detection time of 91.5ms. Furthermore, a real-time detection system, built using PyQT6, was tested in four different environments. Casting-DETR exhibited exceptional performance, maintaining a single-frame detection time of approximately 90 ms, demonstrating the model’s high robustness and suitability for real-time detection. The Casting-DETR network proposed in this paper is an end-to-end solution with rapid convergence, superior detection accuracy, and swift detection speeds, offering a fresh perspective for similar detection tasks within the industry.

Human hands possess the dexterity to interact with diverse objects such as grasping specific parts of the objects and/or approaching them from desired directions. More importantly, humans can grasp objects of any shape without object-specific skills. Recent works synthesize grasping motions following single objectives such as a desired approach heading direction or a grasping area. Moreover, they usually rely on expensive 3D hand-object data during training and inference, which limits their capability to synthesize grasping motions for unseen objects at scale. In this paper, we unify the generation of hand-object grasping motions across multiple motion objectives, diverse object shapes and dexterous hand morphologies in a policy learning framework GraspXL. The objectives are composed of the graspable area, heading direction during approach, wrist rotation, and hand position. Without requiring any 3D hand-object interaction data, our policy trained with 58 objects can robustly synthesize diverse grasping motions for more than 500k unseen objects with a success rate of 82.2%. At the same time, the policy adheres to objectives, which enables the generation of diverse grasps per object. Moreover, we show that our framework can be deployed to different dexterous hands and work with reconstructed or generated objects. We quantitatively and qualitatively evaluate our method to show the efficacy of our approach. Our model and code will be available.

In this letter, a grasping neural network for a bio-inspired gripper (BIG-Net) trained on a synthetic dataset is proposed for the picking of novel objects. The grasp feasibility is evaluated by tracking the deformation of the soft gripping pad and three types of gripping forces during simulation. Over 420 K grasp scenes with 4.3 B grasps have been synthesized with stacked objects to train the neural network, instead of isolated objects in many existing methods. The BIG-Net takes in a depth image and provides pixel-wise grasp parameters for a grasp scene. Various experiments in both simulation and real world indicate that the BIG-Net grasping method outperforms the traditional and state-of-the-art methods. It achieves the average grasp success rates of 94% for the random picking of household items in clutter and 86% for adversarial items at real-time speeds (25 ms).

This article studies the detection of discontinuous false data-injection (FDI) attacks on cyber-physical systems (CPSs). Considering the unknown stochastic properties of the process noise and measurement noise, deep reinforcement learning is applied to designing an FDI attack detector. First, the discontinuous attack detection problem is modeled as a partially observable Markov decision process (POMDP) and a neural network is used to explore the POMDP. In the network, sliding observation windows which are composed of the offline fragment historical data are used as the input. An approach to designing the reward in POMDP is provided to ensure the precision of the detection when there are even some state recognition errors. Second, sufficient conditions on attack frequency and duration to guarantee the applicability of the detector and the expected estimation performance are further given. Finally, simulation examples illustrate the effectiveness of the attack detector.

Spiking Neural Network (SNN) has been recognized as the third generation of neural networks. Conventionally, a SNN can be converted from a pre-trained Artificial Neural Network (ANN) with less computation and memory than training from scratch. But, these converted SNNs are vulnerable to adversarial attacks. Numerical experiments demonstrate that the SNN trained by optimizing the loss function will be more adversarial robust, but the theoretical analysis for the mechanism of robustness is lacking. In this paper, we provide a theoretical explanation by analyzing the expected risk function. Starting by modeling the stochastic process introduced by the Poisson encoder, we prove that there is a positive semidefinite regularizer. Perhaps surprisingly, this regularizer can make the gradients of the output with respect to input closer to zero, thus resulting in inherent robustness against adversarial attacks. Extensive experiments on the CIFAR10 and CIFAR100 datasets support our point of view. For example, we find that the sum of squares of the gradients of the converted SNNs is 13∼160 times that of the trained SNNs. And, the smaller the sum of the squares of the gradients, the smaller the degradation of accuracy under adversarial attack.

In this article, a six-step approach is proposed to simulate the grasp and evaluate the grasp quality for a versatile vacuum gripper by tracking the deformation and force-torque wrench of the gripping pad. Over 100 K synthetic grasps are generated for neural network training. Furthermore, a gripping attention convolutional neural network (GA-CNN) is developed to predict the grasp quality for real-world grasp, running by 15 Hz closed-loop control with the real-time robotic observation and force-torque feedback. Various experiments in both the simulation and physical grasps indicate that our GA-CNN can focus on the crucial region of the soft gripping pad to predict grasp qualities and perform a lower average error compared with a same-scale traditional CNN. In addition, the complexity of grasping clutters is defined from Level 1 to Level 9. The proposed grasping method achieves an average success rate of 90.2% for static clutters at Level 1 to Level 8 and an average success rate of >80.0% for dynamic grasping at Level 1 to Level 7, which outperforms state-of-the-art grasping methods.

The skill of grasping is one of fundamental and primary skills for a robot. In this work, we present a lightweight one-stage algorithm for generating robot grasping pose estimates, which outputs the grasping pose and grasping quality predictions on each pixel directly end-to-end through a backbone network. To address the problems of long detection time and large computational effort of the two-target detection algorithm, pixel-level learning is performed by combining an improved Dense-Attention module for deep feature extraction to achieve grasping pose estimation. Compared to other complex architectures of similar grasp networks our network has fewer parameters and detects quickly thus better meeting the requirements of real-time detection. Furthermore, we investigate the different input data impact in terms of performance of the algorithm. We find that input depth data and RGB data to the method can get the best performance and achieves the highest detection accuracy, despite reducing the computational speed to some extent.

To achieve multiple obstacle avoidance tasks consisting of zero-space obstacle avoidance and terminal obstacle avoidance with self-avoidance permission, we propose an algorithm for multiple obstacle avoidance tasks for dual-arm robots. First, we design a self-obstacle avoidance gradient optimization function to implement the self-obstacle avoidance task between the two arms of dual-arm robots by using the pseudo-distance as the basis for the collision judgment. Second, the terminal obstacle avoidance task for dual-arm robots is implemented by applying an adaptive terminal obstacle avoidance velocity. Third, a zero-space obstacle avoidance task for dual-arm robots is realized using an adaptive virtual repulsion force. Finally, a dual-arm robot consisting of two 6-DOF Jaco2 manipulators is simulated in a Matlab environment to verify the obstacle avoidance algorithm of dual-arm robots. The simulation results show that the algorithm proposed in this paper can complete the multi-obstacle avoidance tasks of terminal obstacle avoidance and zero-space obstacle avoidance simultaneously with the premise of realizing the self-obstacle avoidance of dual-arm robots, and has a minor terminal trajectory error, which proves that the algorithm is in the feasibility of completing terminal obstacle avoidance and zero-space obstacle avoidance with the premise of realizing dual-arm robots self-obstacle avoidance.

Since the current grasping success rate of robots is low when performing grasping tasks in complex environments, in order to improve this problem, this paper proposes a robot grasping detection network SA-U2GNet combining U 2 -Net and Shuffle Attention networks. The network can not only achieve information communication between different sub-features through the attention mechanism, but also capture more contextual information from RGB-D images through the two-level nested U-shaped structure. Training and testing were performed on the Cornell and Jacquard grasp datasets, the accuracy rates reached 97.9% and 94.7% respectively, and the time required to process RGB-D images was 30ms. Compared with other methods, this method improves the accuracy and time efficiency, and the experiment verifies the feasibility and effectiveness of this method.

This article proposes a novel method for surface defect recognition of die-casting parts based on deep learning YOLOv5 network model. Previous methods, such as based on machine learning and based on template matching, can only classify defect type, and the accuracy and generalization of them are limited. The novel surface defects recognition method based on YOLOv5 algorithm can classify surface defects of die castings and accurately locate their positions which is import in powder metallurgy. To train the casting surface defect detection method based on the YOLOv5 algorithm, the transfer learning is initialized and trained on the Microsoft COCO dataset, we expanded the dataset based on the cyclegan algorithm, and used the kmeans++ algorithm to initialize the anchor-box size. We set up many groups of experiments, and experimental results show that our proposed method performed better than the previous method in joint identification of surface defects, and it can achieve very high mean of average precision (mAP@.5 and mAP@.5:.95) with more than 95%.

Chemical vapor deposition (CVD) has been widely applied to create thin films in semiconductor manufacturing. Virtual metrology (VM) can assist the quality prediction in CVD based on control variables and preceding metrology results. However, the multitask learning problem and limited labeled data from available real metrology are challenges for VM modeling. Accordingly, this paper presents the improved method to combine multitask Gaussian process (MTGP) and adaptive active learning (AAL) for the VM modeling in CVD systems. Initially, a multitask Gaussian processes-based virtual metrology (MTGPVM) model is built based on the intrinsic coregionalization model (ICM). Subsequently, active learning methods based on different sampling criteria are improved to address the limited training data issue. Furthermore, an adaptive algorithm is promoted to update the VM model according to the temporary performance of active learning. Finally, the evaluation of the proposed methods was carried out using the practical dataset in a factory. The proposed MTGPVM achieved prediction performance with 1.44–1.79% mean-absolute-percentage error (MAPE) in thickness and 0.39–0.49% MAPE in refractive index. The proposed AAL algorithm can enhance the learning accuracy of the MTGPVM model with a small sample size.

This study presents a LiDAR-Visual-Inertial Odometry (LVIO) based on optimized visual point-line features, which can effectively compensate for the limitations of a single sensor in real-time localization and mapping. Firstly, an improved line feature extraction in scale space and constraint matching strategy, using the least square method, is proposed to provide a richer visual feature for the front-end of LVIO. Secondly, multi-frame LiDAR point clouds were projected into the visual frame for feature depth correlation. Thirdly, the initial estimation results of Visual-Inertial Odometry (VIO) were carried out to optimize the scanning matching accuracy of LiDAR. Finally, a factor graph based on Bayesian network is proposed to build the LVIO fusion system, in which GNSS factor and loop factor are introduced to constrain LVIO globally. The evaluations on indoor and outdoor datasets show that the proposed algorithm is superior to other state-of-the-art algorithms in real-time efficiency, positioning accuracy, and mapping effect. Specifically, the average RMSE of absolute trajectory in the indoor environment is 0.075 m and that in the outdoor environment is 3.77 m. These experimental results can prove that the proposed algorithm can effectively solve the problem of line feature mismatching and the accumulated error of local sensors in mobile carrier positioning.

In practical applications, an appropriate path planning algorithm needs to be selected according to the actual situation. Using the Cartographer algorithm based on graph optimization to build the map, and using AMCL positioning at the same time, the problem of low accuracy caused by its own defects has been successfully solved. A* algorithm original path trajectory inflection point, node redundancy phenomenon is obvious, longer running time, and even difficult to plan the optimal path An improved A* algorithm is proposed, which uses adding weight coefficient w, adaptively adjusting the step size algorithm and cubic Bezier curve make up for the lack of more turning points, larger turning angles and longer running time in the algorithm search. The hybrid path planning algorithm combining the improved A* algorithm and the dynamic window method solves the problem that the A* algorithm cannot avoid dynamic obstacles in complex environments, and prevents the mobile robot from falling into local optimum. Experiments show that the improved algorithm can effectively solve the problems of more turning points, larger turning angles and longer running time encountered by the mobile robot in the search.

Recycling of Waste Electrical and Electronic Equipment (WEEE) is challenging due to the high variety in both the design and composition of the products in this waste stream. Therefore, targeted demanufacturing and appropriate material sorting processes are crucial, but remain often a manual and thus labor-intensive and costly activity. Despite first steps towards the automation of these sorting processes can be noticed, many of them are in material sorting, but very few in the sorting at product level. In this perspective, one of the key tasks is the automated classification, picking and manipulation of WEEE products.

To address the problem of insufficient image feature information and the accumulated error of visual-inertial odometry (VIO) in an outdoor complex environment, this paper proposes a VIO and global navigation satellite system (GNSS) algorithm based on a combination of point and line features. First, the paper designs a minimization strategy using geometric constraints, which performs constraint matching between the front and back frames on the extracted line features. In addition, an improved VIO-GNSS loose coupling model is built, in which the accumulated error in VIO pose estimation is suppressed using the global observation value of the GNSS. The experimental results in the KIITI data set and measured data set show that the proposed algorithm can maintain good accuracy and robustness compared with several similar algorithms in large outdoor complex environments with good/rejected/interrupted GNSS signals and meet the real-time requirements of driving positioning.