In this work, we present an ensemble based sign video recognition method. Our proposed method uses different input representations – such as RGB video and body key-points or pose data – to model sign videos in a multi-modal manner. We represent an input sign video in two ways: the dense frame and the sparse frame inputs. The dense input uses 3D Convolutional Neural Network (CNN) on a 64 frame input window and Long Short Term Memory (LSTM) Network on 32 frame pose input. The sparse input picks 5 representative frames from a sign video, and utilizes CNN and Graph Convolutional Network (GCN) based modeling. These representative frames for a video are selected using pose confidences that are obtained from an off-the-shelf pose estimation model. Our experimental results show that, while the dense 3D CNN model achieves best performance as a single classifier, the GCN based sparse model provides extra recognition capacity. More specifically, the sparse modeling source, when added with the dense modeling in an ensemble manner, can disambiguate similar looking sign classes. Our proposed multi-source ensemble method outperforms several state-of-the-art methods on AUTSL Turkish sign language benchmark dataset.

Gestures in American Sign Language (ASL) are characterized by fast, highly articulate motion of upper body, including arm movements with complex hand shapes and facial expressions. In this work, we propose a new method for word-level sign recognition from American Sign Language (ASL) using video. Our method uses both motion and hand shape cues while being robust to variations of execution. We exploit the knowledge of the body pose, estimated from an off-the-shelf pose estimator. Using the pose as a guide, we pool spatio-temporal feature maps from different layers of a 3D convolutional neural network. We train separate classifiers using pose guided pooled features from different resolutions and fuse their prediction scores during test time. This leads to a significant improvement in performance on the WLASL benchmark dataset [25]. The proposed approach achieves 10%, 12%, 9.5% and 6.5% performance gain on WLASL100, WLASL300, WLASL1000, WLASL2000 subsets respectively. To demonstrate the robustness of the pose guided pooling and proposed fusion mechanism, we also evaluate our method by fine tuning the model on another dataset. This yields 10% performance improvement for the proposed method using only 0.4% training data during fine tuning stage.

American Sign Language recognition is a difficult gesture recognition problem, characterized by fast, highly articulate gestures. These are comprised of arm movements with different hand shapes, facial expression and head movements. Among these components, hand shape is the vital, often the most discriminative part of a gesture. In this work, we present an approach for effective learning of hand shape embeddings, which are discriminative for ASL gestures. For hand shape recognition our method uses a mix of manually labelled hand shapes and high confidence predictions to train deep convolutional neural network (CNN). The sequential gesture component is captured by recursive neural network (RNN) trained on the embeddings learned in the first stage. We will demonstrate that higher quality hand shape models can significantly improve the accuracy of final video gesture classification in challenging conditions with variety of speakers, different illumination and significant motion blurr. We compare our model to alternative approaches exploiting different modalities and representations of the data and show improved video gesture recognition accuracy on GMU-ASL51 benchmark dataset.

This work presents an approach for American Sign Language (ASL) gesture recognition from videos. Gestures are comprised of various upper body motions involving hand shapes, motion of both hands with facial expression and head movements. Previous approaches tackled this problem by directly learning 3D convolutional spatio-temporal models from video in a simplified settings with uniform backgrounds. To handle more complex variation in appearance and backgrounds we propose to exploit recent advances in estimation of 2D body pose using Deep Convolutional Neural Networks trained on large corpus of human pose annotations. We use the trajectories of 2D skeletal data estimated from video to train a baseline recursive neural network gesture recognition model. The basic model is further extended using embeddings of hand images obtained from another hand shape recognition model [15] with dynamics modeled by another recursive neural network. The final model learns how to fuse two Long Short Term Model (LSTM) recursive neural network models for skeletal and hand image data. We train and evaluate this model on the GMU-ASL51 dataset of 12 users and 51 ASL gestures [8] demonstrating its superior performance compared to several baseline models.

Home assistant devices such as Amazon Echo and Google Home have become tremendously popular in the last couple of years. However, due to their voice-controlled functionality, these devices are not accessible to Deaf and Hard-of-Hearing (DHH) people. Given that over half a million people in the United States communicate using American Sign Language (ASL), there is a need of a home assistant system that can recognize ASL. The objective of this work is to design a home assistant system for DHH users (referred to as mmASL) that can perform ASL recognition using 60 GHz millimeter-wave wireless signals. mmASL has two important components. First, it can perform reliable wake-word detection using spatial spectrograms. Second, using a scalable and extensible multi-task deep learning model, mmASL can learn the phonological properties of ASL signs and use them to accurately recognize the ASL signs. We implement mmASL on 60 GHz software radio platform with phased array, and evaluate it using a large-scale data collection from 15 signers, 50 ASL signs and over 12K sign instances. We show that mmASL is tolerant to the presence of other interfering users and their activities, change of environment and different user positions. We compare mmASL with a well-studied Kinect and RGB camera based ASL recognition systems, and find that it can achieve a comparable performance (87% average accuracy of sign recognition), validating the feasibility of using 60 GHz mmWave system for ASL sign recognition.

Over half a million people in the United States use American Sign Language (ASL) as their primary mode of communication. Automatic ASL recognition would enable Deaf and Hard of Hearing (DHH) users to interact with others who are not familiar with ASL as well as voice-controlled digital assistants (e.g., Alexa, Siri, etc.). While ASL recognition has been extensively studied, there is a little attention given to recognition of ASL non-manual body markers. The non-manual markers are typically expressed through head, torso and shoulder movements, and add essential meaning and context to the signed sentences. In this work, we present ExASL, a sentence-level ASL recognition system using millimeter-wave radars. ExASL can recognize manual markers (hand gestures) and non-manual markers (head and torso movements). It utilizes multi-distance clustering to recognize body parts and cluster mmWave point clouds. We then present a multi-view deep learning algorithm that can learn from clustered body part representation for an expressive sentence-level recognition. Our evaluation shows that ExASL can recognize ASL sentences with a word error rate of 0.79%, sentence error rate of 1.25%, and non-manual markers with an accuracy of 83.5%.

Voice-controlled personal and home assistants (such as the Amazon Echo and Apple Siri) are becoming increasingly popular for a variety of applications. However, the benefits of these technologies are not readily accessible to Deaf or Hard-ofHearing (DHH) users. The objective of this study is to develop and evaluate a sign recognition system using multiple modalities that can be used by DHH signers to interact with voice-controlled devices. With the advancement of depth sensors, skeletal data is used for applications like video analysis and activity recognition. Despite having similarity with the well-studied human activity recognition, the use of 3D skeleton data in sign language recognition is rare. This is because unlike activity recognition, sign language is mostly dependent on hand shape pattern. In this work, we investigate the feasibility of using skeletal and RGB video data for sign language recognition using a combination of different deep learning architectures. We validate our results on a large-scale American Sign Language (ASL) dataset of 12 users and 13107 samples across 51 signs. It is named as GMUASL51. We collected the dataset over 6 months and it will be publicly released in the hope of spurring further machine learning research towards providing improved accessibility for digital assistants.

Human motion tracking is a common requirement for many real world applications such as video surveillance, games, cultural and medical applications. In this paper an appearance based tracking system is proposed which tracks human motion from a video scene. The system is based on appearance based color histogram and color correlogram model. Multiple people can be tracked consistently by this proposed system. This system works well in indoor and semi-indoor environment. The proposed system comprises of five major steps. Firstly all motion blobs in the video scene are extracted. Morphological image closing are applied then on extracted motion blobs for removing anomalies associated with blobs. After that all the motion blobs are counted and labeled by connected component finding. Correlogram and histogram model for each blob are built from the color information of moving blobs in the next step. Similarity measures between the blobs of the current image frame and previous image frame are calculated. Finally system performs tracking based on the similarity measures from frame to frame. The system is consistent during partial occlusion. During occlusion occluding entities are tracked as one entity and tracked separately when they split.