On the simultaneous recognition of identity and expression from BU-3DFE datasets

We propose a new linear model, based on resampling 3D face meshes to convert them to 3D matrices, to recognize identity and expression simultaneously. This contrasts with bilinear models currently used with 3D meshes. The matrices are amenable to algebraic operations for facial data analysis and synthesis. Facial emotion is represented as a linear combination of its identity and expression using principal components extracted from training data as neutral-to-emotion deformations. The linear model is applicable to other mesh data with pose variations after correction using recently available techniques. The proposed approach avoids the problem of correspondence between pairs of person's neutral and emotion meshes for estimating facial deformations used as features. Identity and expression recognition accuracies, obtained by representing resampled matrices as linear combinations of composite depth-color (gray) PCs, are better than the results in the literature on both simultaneous identity-expression using bilinear models and expression-only recognition using deformable models, facial action codes, distances between pairs of annotated facial points as features and others. The proposed framework can also be used to generate synthetic matrices displaying a wide array of natural and mixed emotions for any chosen identity. A byproduct is the result that second-order deformations as features do not seem to perform as effectively as first-order deformations for identity and expression recognition.