Learning Task Constraints and Robot Motion from Unreliable Human Skeleton Data Using Human Action Recognition Algorithm

Utilizing a single vision sensor for learning from human demonstration (LfD) planning offers numerous benefits. However, the accurate identification of key task constraints for motion planning is a prevailing challenge, stemming from data unreliability. This research introduces a novel LfD planning framework that employs a human action recognition algorithm to extract task constraints from unreliable human skeleton data derived from single camera images, thereby enhancing motion planning. Initially, the method aims to obtain task constraints from the skeleton data using action labels associated with specific constraints, for instance, “pick upward while maintaining a consistent pose.” Subsequently, a robot motion is crafted to comply with the extracted constraints and is catalogued in the motion database as a path experience. For novel planning challenges, this path experience is tailored to new environments, serving as a reference for discerning a valid motion via the random modification of any inconsistent segments. Simulation results, focused on pick-and-place tasks, indicate that the introduced method surpasses the state-of-the-art approach by elevating the success rate and diminishing the computation time and path length by 8%, 20%, and 15%, respectively. Furthermore, superior performance was also observed in two real-world scenarios, including a task that requires more complex human actions, such as rotating the wrist. Although the execution time was slightly increased, the proposed method was shown to increase the success rate by 11% and 28% and reduce the average computation time by 14% and 29% for the two real-world scenarios.