Soft Optoelectronic Sensors with Deep Learning for Gesture Recognition

With the rapid development of deep learning and computing power, human-computer interactions, and interfaces are attracting attentions in industrial and academic research. Flexible human-computer interaction can greatly improve productivity and enable robots to work in extreme environments that humans cannot tolerate. The research of gesture recognition is emerging and provides a new way of studying the human-computer interactions. However, compared with the entire human body, human hands are dexterous organs with more complex and flexible joints, which makes hand gesture recognition a challenging problem. Here, a robust and cost-effective gesture recognition system is reported through the soft optoelectronic sensors. An array of polymer-encapsulated U-shaped microfiber (UMF) attached to a glove is fabricated for sensitive finger motion detection. The anisotropic strain response of UMF is measured with a sensitivity of 15.98 (2.20) in the x-direction (y-direction). A deep learning network (VGGNet) is developed to process the optical signals for analyzing and classifying hand gestures. The experiments show that VGGNet has high recognition accuracy of 99.2% for the test datasets with ten classified gestures. This work provides a potential optical interface in studying gesture recognition and biomechanical signatures, which can also be applied in virtual reality systems and interactive game platforms.