Multimodal Learning-Based Proactive Human Handover Intention Prediction Using Wearable Data Gloves and Augmented Reality

Efficient object handover between humans and robots holds significant importance within collaborative manufacturing environments. Enhancing the efficacy of human-robot handovers involves enabling robots to comprehend and foresee human handover intentions. This article introduces human-teaching-robot-learning-prediction framework, allowing robots to learn from diverse human demonstrations and anticipate human handover intentions. The framework facilitates human programming of robots through demonstrations utilizing augmented reality and a wearable dataglove, aligned with task requirements and human working preferences. Subsequently, robots enhance their cognitive capabilities by assimilating insights from human handover demonstrations, utilizing deep neural network algorithms. Furthermore, robots can proactively seek clarification from humans via an augmented reality system when confronted with ambiguity in human intentions, mirroring how humans seek clarity from their counterparts. This proactive approach empowers robots to anticipate human intentions and assist human partners during handovers. Empirical results underscore the benefits of the proposed approach, demonstrating highly accurate prediction of human intentions in human-robot handover tasks. A novel human-teaching-robot-learning-prediction framework is presented in this paper. Using the proposed framework, the human can teach the robot through multimodal demonstrations. The robot can learn from human demonstrations and predict human handover intentions using the learned prediction model. Experimental results have demonstrated the advantages of the proposed approach in predicting human handover intentions during human-robot handover.image (c) 2024 WILEY-VCH GmbH