Modeling Signs Using Functional Data Analysis

We present a functional data analysis (FDA) based method to statistically model continuous signs of the American Sign Language (ASL) for use in the recognition of signs in continuous sentences. We build models in the Space of Probability Functions (SoPF) that captures the evolution of the relationships among the low-level features (e.g. edge pixels) in each frame. The distribution (histogram) of the horizontal and vertical displacements between all pairs of edge pixels in an image frame forms the relational distributions. We represent these sequence of relational distributions, corresponding to the sequence of image frames in a sign, as a sequence of points in a multi-dimensional space, capturing the salient variations in these relational distributions over time; we call this space the SoPF. Each sign model consists of a mean sign function and covariance functions, capturing the variability of each sign in the training set. We use functional data analysis to arrive at this model. Recognition and sign localization is performed by correlating this statistical model with any given sentence. We also present a method to infer and learn sign models, in an unsupervised manner, from sentence samples containing the sign; there is no need for manual intervention.