Real-Time Dynamic Sign Language Recognition Using LSTM Based on MediaPipe Hand Data

Sign Language is one of the important communication methods for the hearing-impaired person. As it is hard to learn, it has been researched several times throughout various motion capture technologies. In this paper, MediaPipe Hand, Google’s AI framework, and LSTM, a type of RNN, were used for training our dynamic Sign Language recognition system. Unlike in previous researches, our trained model translates the dynamic Sign Language in real-time with displaying the translated sentences. Sign language differs from existing English grammar in terms of language structure. To solve this problem, there is a way to use Neural Machine Translation, but due to the limitation of the number of words used in this research, we used a simple rule-based structure converting the recognized word sequences into sentences with English grammar.