Sign language recognition based on skeleton and SK3D-Residual network

Most of the existing dynamic sign language recognition methods based on deep learning directly use the video sequence or the whole sequence based on RGB information, not just the video sequence representing the change of gesture. These make it difficult for sign language recognition to achieve good accuracy. In order to solve these problems, this paper proposes a method of sign language recognition based on skeleton and SK3D-Residual network. In SK3D-Residual network, a key frame optimization algorithm for skeleton sequence based on mutual information is designed. The 3D-LSTM module extracts spatiotemporal features from the skeleton key frame sequences, analyzes the features of each action in the sequence, and then recognizes sign language. The experimental accuracy is 88.6%. In addition, the accuracy of the combination of RGB and skeleton information is 93.2%. Our experiment has achieved a good recognition accuracy.