Enabling collaborative assembly between humans and robots using a digital twin system

Human-robot collaboration (HRC) systems are intelligent systems that guide robots to collaborate with humans based on a cognitive understanding of human intention, ensuring safe, flexible, and efficient collaboration between humans and robots in shared workspaces. In industrial settings, the current methods for constructing a human digital twin model rely on motion capture devices that require personnel to wear cumbersome equipment, which goes against the principle of flexible interaction advocated for HRC. Furthermore, the current methods do not model humans and robots in a unified space, which is both unintuitive and inconvenient for perceiving and understanding the overall environment. To address these limitations, this paper proposes a digital twin system for HRC. This system facilitates the construction of a digital twin scene, the mapping from the real space to the virtual space, and the planning and execution of collaborative strategies from the virtual to the real space. Designed explicitly for common workstation settings, a robust human mesh recovery algorithm is introduced to address the challenge of reconstructing occluded human bodies. Additionally, uncertainty estimation is employed to enhance the action recognition algorithm, ensuring a controllable level of risk in the recognition process. Experimental results demonstrate the superiority of the proposed methods over baseline methods. Finally, the feasibility and effectiveness of the HRC system are validated through a case study involving component assembly.