Ensuring the safety of human workers in a collaborative environment with robots is of utmost importance. Although accurate pose prediction models can help prevent collisions between human workers and robots, they are still susceptible to critical errors. In this study, we propose a novel approach called deep ensembles of temporal graph neural networks (DE-TGN) that not only accurately forecast human motion but also provide a measure of prediction uncertainty. By leveraging deep ensembles and employing stochastic Monte-Carlo dropout sampling, we construct a volumetric field representing a range of potential future human poses based on covariance ellipsoids. To validate our framework, we conducted experiments using three motion capture datasets including Human3.6M, and two human-robot interaction scenarios, achieving state-of-the-art prediction error. Moreover, we discovered that deep ensembles not only enable us to quantify uncertainty but also improve the accuracy of our predictions.

This paper presents a novel knowledge-informed graph neural planner (KG-Planner) to address the challenge of efficiently planning collision-free motions for robots in high-dimensional spaces, considering both static and dynamic environments involving humans. Unlike traditional motion planners that struggle with finding a balance between efficiency and optimality, the KG-Planner takes a different approach. Instead of relying solely on a neural network or imitating the motions of an oracle planner, our KG-Planner integrates explicit physical knowledge from the workspace. The integration of knowledge has two key aspects: 1) We present an approach to design a graph that can comprehensively model the workspace's compositional structure. The designed graph explicitly incorporates critical elements such as robot joints, obstacles, and their interconnections. This representation allows us to capture the intricate relationships between these elements; 2) We train a Graph Neural Network (GNN) that excels at generating nearly optimal robot motions. In particular, the GNN employs a layer-wise propagation rule to facilitate the exchange and update of information among workspace elements based on their connections. This propagation emphasizes the influence of these elements throughout the planning process. To validate the efficacy and efficiency of our KG-Planner, we conduct extensive experiments in both static and dynamic environments. These experiments include scenarios with and without human workers. The results of our approach are compared against existing methods, showcasing the superior performance of the KG-Planner. A short video introduction of this work is available via this https://zh.engr.tamu.edu/wp-content/uploads/sites/310/2024/03/KGPlanner.mp4link.

Drones have increasingly collaborated with human workers in some workspaces, such as warehouses. The failure of a drone flight may bring potential risks to human beings' life safety during some aerial tasks. One of the most common flight failures is triggered by damaged propellers. To quickly detect physical damage to propellers, recognise risky flights, and provide early warnings to surrounding human workers, a new and comprehensive fault diagnosis framework is presented that uses only the audio caused by propeller rotation without accessing any flight data. The diagnosis framework includes three components: leverage convolutional neural networks, transfer learning, and Bayesian optimisation. Particularly, the audio signal from an actual flight is collected and transferred into time–frequency spectrograms. First, a convolutional neural network‐based diagnosis model that utilises these spectrograms is developed to identify whether there is any broken propeller involved in a specific drone flight. Additionally, the authors employ Monte Carlo dropout sampling to obtain the inconsistency of diagnostic results and compute the mean probability score vector's entropy (uncertainty) as another factor to diagnose the drone flight. Next, to reduce data dependence on different drone types, the convolutional neural network‐based diagnosis model is further augmented by transfer learning. That is, the knowledge of a well‐trained diagnosis model is refined by using a small set of data from a different drone. The modified diagnosis model has the ability to detect the broken propeller of the second drone. Thirdly, to reduce the hyperparameters' tuning efforts and reinforce the robustness of the network, Bayesian optimisation takes advantage of the observed diagnosis model performances to construct a Gaussian process model that allows the acquisition function to choose the optimal network hyperparameters. The proposed diagnosis framework is validated via real experimental flight tests and has a reasonably high diagnosis accuracy.

There has been a growing utilization of industrial robots as complementary collaborators for human workers in remanufacturing sites. Such a human-robot collaboration (HRC) aims to assist human workers in improving the flexibility and efficiency of labor-intensive tasks. In this article, we propose a human-aware motion planning framework for HRC to effectively compute collision-free motions for manipulators when conducting collaborative tasks with humans. We employ a neural human motion prediction model to enable proactive planning for manipulators. Particularly, rather than blindly trusting and utilizing predicted human trajectories in the manipulator planning, we quantify uncertainties of the neural prediction model to further ensure human safety. Moreover, we integrate the uncertainty-aware prediction into a graph that captures key workspace elements and illustrates their interconnections. Then, a graph neural network (GNN) is leveraged to operate on the constructed graph. Consequently, robot motion planning considers both the dependencies among all the elements in the workspace and the potential influence of future movements of human workers. We experimentally validate the proposed planning framework using a 6-degree-of-freedom manipulator in a shared workspace where a human is performing disassembling tasks. The results demonstrate the benefits of our approach in terms of improving the smoothness and safety of HRC. A brief video introduction of this work is available as the supplemental materials.

Industrial manipulators have extensively collaborated with human operators to execute tasks, e.g., disassembly of end-of-use products, in intelligent remanufacturing. A safety task execution requires real-time path planning for the manipulator's end-effector to autonomously avoid human operators. This is even more challenging when the end-effector needs to follow a planned path while avoiding the collision between the manipulator body and human operators, which is usually computationally expensive and limits real-time application. This paper proposes an efficient hybrid motion planning algorithm that consists of an A$^*$ algorithm and an online manipulator reconfiguration mechanism (OMRM) to tackle such challenges in task and configuration spaces respectively. The A$^*$ algorithm is first leveraged to plan the shortest collision-free path of the end-effector in task space. When the manipulator body is risky to the human operator, our OMRM then selects an alternative joint configuration with minimum reconfiguration effort from a database to assist the manipulator to follow the planned path and avoid the human operator simultaneously. The database of manipulator reconfiguration establishes the relationship between the task and configuration space offline using forward kinematics, and is able to provide multiple reconfiguration candidates for a desired end-effector's position. The proposed new hybrid algorithm plans safe manipulator motion during the whole task execution. Extensive numerical and experimental studies, as well as comparison studies between the proposed one and the state-of-the-art ones, have been conducted to validate the proposed motion planning algorithm.

This paper presents a deep learning enhanced adaptive unscented Kalman filter (UKF) for predicting human arm motion in the context of manufacturing. Unlike previous network-based methods that solely rely on captured human motion data, which is represented as bone vectors in this paper, we incorporate a human arm dynamic model into the motion prediction algorithm and use the UKF to iteratively forecast human arm motions. Specifically, a Lagrangian-mechanics-based physical model is employed to correlate arm motions with associated muscle forces. Then a Recurrent Neural Network (RNN) is integrated into the framework to predict future muscle forces, which are transferred back to future arm motions based on the dynamic model. Given the absence of measurement data for future human motions that can be input into the UKF to update the state, we integrate another RNN to directly predict human future motions and treat the prediction as surrogate measurement data fed into the UKF. A noteworthy aspect of this study involves the quantification of uncertainties associated with both the data-driven and physical models in one unified framework. These quantified uncertainties are used to dynamically adapt the measurement and process noises of the UKF over time. This adaption, driven by the uncertainties of the RNN models, addresses inaccuracies stemming from the data-driven model and mitigates discrepancies between the assumed and true physical models, ultimately enhancing the accuracy and robustness of our predictions. Compared to the traditional RNN-based prediction, our method demonstrates improved accuracy and robustness in extensive experimental validations of various types of human motions.

This article presents a disassembly task planning algorithm considering human–robot collaboration (HRC) and human behavior prediction (HBP). Unlike assembly procedures, the disassembly of end-of-life (EOL) products has been a labor-intensive process with uncertainties difficult to cope with. Meanwhile, it is usually challenging to obtain an optimal sequence efficiently without excessive computational cost. Also, the conventional human-centered task planning, in which the robot has to halt frequently due to unsafe interruptions by human motions, may decrease the efficiency of the disassembly process. In this article, a sequence planner is proposed to assign tasks in real time between a human operator and a robot to overcome the aforementioned challenges. The cost function includes the effort of the human and the robot in terms of both movement distance and time spent on the tasks. The constraints include the disassembly rules and the safety of the human operation. The optimal sequence is generated by solving an optimization problem in a receding-horizon way. In particular, at each step, the proposed disassembly sequence planner locates the workers (a human operator and a robot) and the to-be-disassembled components, predicts human movement for the next several steps, and obtains the optimal disassembly sequence for the next several steps following disassembly rules and safety constraints. Experiments have been extensively conducted on the disassembly of a wooden toybox and a used hard disk drive (HDD) to validate the proposed disassembly sequence planner. The planner has successfully generated the disassembly sequence in an HRC setting explicitly considering real-time human motion prediction and assigned the human operator and the robot to collaboratively complete disassembly tasks without violating disassembly rules and safety constraints.

While the disassembly of high-precision electronic devices is a predominantly labor-intensive process, collaborative robots provide a promising solution through human–robot collaboration. To ensure efficient yet safe collaboration, this article presents a new way to generate task-constrained and collision-free motion for a collaborative robot operating in a dynamic environment involving human movement, which is traditionally challenging due to the high degree of freedom of the corobot and the uncertainty nature of human motion. We first establish a neural human motion prediction model with quantified uncertainty, and then optimize the configuration of the robot online by taking the human motion and uncertainties into consideration. While such rationale is straightforward in nature, our method explicitly quantified the uncertainty of the neural human prediction model to further enhance the collaboration safety, and integrated the quantified uncertainty into the task-satisfied motion planning in real time to efficiently conduct tasks. Extensive experimental tests and comparison studies have been conducted to validate the efficiency and effectiveness of the proposed planning method.

This work presents a framework aimed at mitigating adverse effects of high-amplitude drone noise ranging from hearing loss to reduced productivity in human–robot collaborative environments by infusing acoustic awareness in a path planning algorithm without imposing any additional design layers or hardware to an operational drone. Following a detailed outline of the proposed approach, it is shown that a significant reduction of noise levels perceived by human workers at noise-sensitive locations is realized via a path planner which generates optimal paths ranging from quietest to shortest paths. The approach is then augmented with a path-correction mechanism which accounts for noise exposure duration to ensure the aforementioned optimal paths are compliant with a given industrial/environmental standard. The correction mechanism enforces an adjustment of subsets of the planned paths inside quiet zones designated around noise-sensitive locations. The presented concepts were verified using numerical simulations conducted for a 2-dimensional rasterized obstacle field followed by a statistical design of experiments. The proposed framework is highly versatile and integrable with widely used industrial path planners, rendering it a highly valuable tool for noisy collaborative workplaces.

Human motion prediction is the foundation stone of human–robot collaboration in intelligent manufacturing. The nonlinear and stochastic nature of human motion has made it challenging to predict the motion accurately. Many recent deep-learning-based approaches, e.g., convolutional neural networks or recurrent neural networks (RNNs), have been applied to address this challenge. On the other hand, existing works tend to ignore the importance of human dynamics in motion prediction, especially the effect of muscle force on the motion. This article proposes a novel dynamic model informed motion prediction method. It utilizes an unscented Kalman filter (UKF) to predict the state of the future arm dynamic model such that the future motion of the human arm can be obtained. In particular, the arm dynamic model is developed based on Lagrangian mechanics and represented by differential equations. Embracing the future muscle force predicted by RNN into the differential equations, such a dynamic model is capable of explicitly establishing the intrinsic relation between the future muscle force and the corresponding future arm motion. UKF is leveraged to predict the future joint position and velocity of the human arm based on the dynamic model. Experiments on three motion datasets validate that the proposed prediction method, compared with the traditional RNN-based prediction using skeleton vectors, significantly improves the prediction accuracy regarding elbow and wrist positions.

Obstacle avoidance is one of the fundamental problems in human-robot collaboration (HRC) studies. The close proximity between robots and human usually leaves robots a short period of time to re-plan a safe motion, especially when facing non-static obstacles. Therefore, to identify collisions in advance and mitigate the computational efforts regarding robot motion re-planning, this paper proposes a network-based stop-go algorithm that uses only images capturing the states of the robot arm and a non-static obstacle without accessing any robot dynamics. In particular, a deep convolutional long-short-term memory (ConvLSTM) neural network is first developed to learn the spatial features of images, and predict both the robot arm and the non-static obstacle states five steps in advance. Next, the predictions are set back to the robot arm so that the robot arm would halt the current task when a potential future collision is identified. Eventually, the robot arm resumes the task after the non-static obstacle is clear. Extensive numerical studies have been conducted to validate the effectiveness of the proposed trajectory prediction scheme in presence of obstacles.

In recent years, the demands of using robotic hands have been increased dramatically in a variety of applications such as intelligent manufacturing. The current robotic hands usually are functioned relying on the assembly of numerous components, which bring unnecessary dead weight and complex control systems. Such shortcomings limit the further application of robotic hands. To address this issue, this paper presents a lightweight and low-complexity five-fingered robotic hand called the Integrated Finger Robotic Hand (IFRH). Each finger of the robotic hand is built as an integrated object based on a compliant joint called the Elastic Knuckle Connection (EKC). The 3D-printed finger design minimizes the number of components as well as the weight of the system. The IFRH can be assembled quickly and maintained easily due to its low number of parts compared to traditional separate-finger robotic hands. The IFRH is driven by servo motors which transmit motion to a monofilament fishing string. This fishing string is connected to each finger, which mirrors the grasping motion of a real human hand when contracted. With feedback from force sensors, the IFRH accomplishes the goal of high precision grasping of daily objects. The IFRH can be controlled by a potentiometer powered by Arduino, further increasing the user-friendly nature of the robotic hand. A number of experimental tests are conducted to verify the accuracy of the force transmission system, the grasping capabilities, and the grasping precision.

The safety of human workers has been the main concern in human-robot close collaboration. Along with rapidly developed artificial intelligence techniques, deep learning models using two-dimensional images have become feasible solutions for human motion detection. These models serve as “sensors” in the closed-loop system that involve humans and robots. Most existing methods that detect human motion using images do not consider the uncertainty from the deep learning model itself. The mappings established by deep learning models should not be taken blindly, and thus uncertainty should be a natural part of this type of sensor. In particular, model uncertainty should be explicitly quantified and incorporated into robot motion control to guarantee safety. With this motivation, to rigorously quantify the uncertainty of these “sensors”, this letter proposes a probabilistic interpretation method and automatically provides a framework to benefit from a deep model's uncertainty. Experimental data from human-robot collaboration has been collected and used to validate the proposed method. A training strategy is proposed to efficiently train surrogate models that learn to refine the prediction of the main Bayesian models. The proposed framework is also compared with Ego hands benchmark showing a 4.7% increase in mIoU.

3D imaging sensors are important for intelligent robots to figure out complicated and unknown assembly tasks. To perform a smooth robotic assembly with high precision, it is crucial to incorporate 3D imaging sensors that can scan, recognize, and characterize fine-scale features (e.g., small screws). We developed an integrated robotic 3D vision system based on high-resolution structured light 3D vision based on fringe projection. Our preliminary testing has demonstrated the promise of our integrated system in precision assembly operations supported by the capability of extracting fine-scale key features through high-resolution surface mapping.

Currently, studying the vehicle-human interactive behavior in the emergency needs a large amount of datasets in the actual emergent situations that are almost unavailable. Existing public data sources on autonomous vehicles (AVs) mainly focus either on the normal driving scenarios or on emergency situations without human involvement. To fill this gap and facilitate related research, this paper provides a new yet convenient way to extract the interactive behavior data (i.e., the trajectories of vehicles and humans) from actual accident videos that were captured by both the surveillance cameras and driving recorders. The main challenge for data extraction from real-time accident video lies in the fact that the recording cameras are un-calibrated and the angles of surveillance are unknown. The approach proposed in this paper employs image processing to obtain a new perspective which is different from the original video's perspective. Meanwhile, we manually detect and mark object feature points in each image frame. In order to acquire a gradient of reference ratios, a geometric model is implemented in the analysis of reference pixel value, and the feature points are then scaled to the object trajectory based on the gradient of ratios. The generated trajectories not only restore the object movements completely but also reflect changes in vehicle velocity and rotation based on the feature points distributions.

Quadrotor unmanned aerial vehicles (UAVs) have been developed and applied into several types of workplaces, such as warehouses, which usually involve human workers. The co-existence of human and UAVs brings new challenges to UAVs: potential failure of UAVs may cause risk and danger to surrounding human. Effective and efficient detection of such failure may provide early warning to the surrounding human workers and reduce such risk to human beings as much as possible. One of the most common reasons that cause the failure of the UAV's flight is the physical damage to the propellers. This paper presents a method to detect the propellers damage only based on the audio noise caused by the UAV's flight. The diagnostic model is developed based on convolutional neural network (CNN) and transfer learning techniques. The audio data is collected from the UAVs in real time, transformed into the time-frequency spectrogram, and used to train the CNN-based diagnostic model. The developed model is able to detect the abnormal features of the spectrogram and thus the physical damage of the propellers. To reduce the data dependence on the UAV's dynamic models and enable the utilization of the training data from UAVs with different dynamic models, the CNN-based diagnostic model is further augmented by transfer learning. As such, the refinement of the well-trained diagnostic model ground on other UAVs only requires a small amount of UAV's training data. Experimental tests are conducted to validate the diagnostic model with an accuracy of higher than 90%.