In this paper, we propose a novel algorithm for general 2D image matching, which is known to be an NP-complete optimization problem. With our algorithm, the complexity is handled by sequentially optimizing the image columns from left to right in a two-level dynamic programming procedure. On a local level, a set of hypotheses is computed for each column, while on a global level the best sequence of these hypotheses is selected. The optimization on the local level is guided by a lookahead that gives an estimate about the not yet optimized part of the image. We evaluate the algorithm on the task of pose-invariant face recognition in an automatic setup and show that the suggested method is competitive and achieves very good recognition accuracies on the popular face recognition databases CMU-PIE and CMU-MultiPIE.

A survey of video databases that can be used within a continuous sign language recognition scenario to measure the performance of head and hand tracking algorithms either w.r.t. a tracking error rate or w.r.t. a word error rate criterion is presented in this work.Robust tracking algorithms are required as the signing hand frequently moves in front of the face, may temporarily disappear, or cross the other hand.Only few studies consider the recognition of continuous sign language, and usually special devices such as colored gloves or blue-boxing environments are used to accurately track the regions-of-interest in sign language processing.Ground-truth labels for hand and head positions have been annotated for more than 30k frames in several publicly available video databases of different degrees of difficulty, and preliminary tracking results are presented.

We present a novel large vocabulary OCR system, which implements a 5 confidenceand margin-based discriminative training approach for model adap6 tation of an HMM based recognition system to handle multiple fonts, different 7 handwriting styles, and their variations. Most current HMM approaches are HTK 8 based systems which are maximum-likelihood (ML) trained and which try to adapt 9 their models to different writing styles using writer adaptive training, unsupervised 10 clustering, or additional writer specific data. Here, discriminative training based 11 on the Maximum Mutual Information (MMI) and Minimum Phone Error (MPE) 12 criteria are used instead. For model adaptation during decoding, an unsupervised 13 confidence-based discriminative training within a two-pass decoding process is pro14 posed. Additionally, we use neural network based features extracted by a hierar15 chical multi-layer-perceptron (MLP) network either in a hybrid MLP/HMM ap16 proach or to discriminatively retrain a Gaussian HMM system in a tandem approach. 17 The proposed framework and methods are evaluated for closed-vocabulary isolated 18 handwritten word recognition on the IfN/ENIT Arabic handwriting database, where 19 the word-error-rate is decreased by more than 50% relative compared to a ML 20 trained baseline system. Preliminary results for large-vocabulary Arabic machine 21 printed text recognition tasks are presented on a novel publicly available newspaper 22 database. 23 RWTH Aachen University Human Language Technology and Pattern Recognition Ahornstr 55, D-52056 Aachen, Germany Tel.: +49-241-80-21613 Fax: +49-241-80-22219 e-mail: @cs.rwth-aachen.de

This paper systematically analyzes the strengths and weaknesses of existing image warping algorithms on the tasks of face recognition. Image warping is used to cope with local and global image variability and in general is an NP-complete problem. Although many approximations have recently been proposed, neither thorough comparison, nor systematic analysis of methods in a common scheme has been done so far. We follow the bottom-up approach and analyze the methods with increasing degree of image structure preserved during optimization. We evaluate the presented warping approaches on four challenging face recognition tasks in highly variable domains. Our findings indicate that preserving maximum dependencies between neighboring pixels by imposing strong geometrical constraints leads to the best recognition results while making optimization efficient.

We present a novel confidence- and margin-based discriminative training approach for model adaptation of a hidden Markov model (HMM)-based handwriting recognition system to handle different handwriting styles and their variations. Most current approaches are maximum-likelihood (ML) trained HMM systems and try to adapt their models to different writing styles using writer adaptive training, unsupervised clustering, or additional writer-specific data. Here, discriminative training based on the maximum mutual information (MMI) and minimum phone error (MPE) criteria are used to train writer-independent handwriting models. For model adaptation during decoding, an unsupervised confidence-based discriminative training on a word and frame level within a two-pass decoding process is proposed. The proposed methods are evaluated for closed-vocabulary isolated handwritten word recognition on the IFN/ENIT Arabic handwriting database, where the word error rate is decreased by 33% relative compared to a ML trained baseline system. On the large-vocabulary line recognition task of the IAM English handwriting database, the word error rate is decreased by 25% relative.

We use neural network based features extracted by a hierarchical multilayer-perceptron (MLP) network either in a hybrid MLP/HMM approach or to discriminatively retrain a Gaussian hidden Markov model (GHMM) system in a tandem approach. MLP networks have been successfully used to model long-term and non-linear features dependencies in automatic speech and optical character recognition. In offline hand writing recognition, MLPs have been mostly used for isolated character and word recognition in hybrid approaches. Here we analyze MLPs within an LVCSR framework for continuous handwriting recognition using discriminative MMI/MPE training. Especially hybrid MLP/HMM and discriminatively retrained MLP-GHMM tandem approaches are evaluated. Significant improvements and competitive results are re ported for a closed-vocabulary task on the IfN/ENIT Arabic handwriting database and for a large-vocabulary task using the IAM English handwriting database.

The task of image recognition is very challenging due to many intra-class variations. Especially the example of face recognition offers many challenges such as illumination, facial expressions, occlusions or different poses. Many different approaches have been proposed such as analyzing the sub-space spanned by face images, feature matching or three-dimensional models. However the approach of two-dimensional warping is particularly suited for this task. It defines a similarity measure between images that is very tolerant to local deformations and can be used for nearest-neighbor classification, which allows mug-shot recognition. In this thesis novel two-dimensional warping algorithms for image recognition within a nearest-neighbor classification framework are proposed. The new algorithms maintain full dependencies in both dimensions defined by a two-dimensional grid. This approach allows the enforcement of geometric constraints such as Sakoe constraints. This is implemented without sacrificing efficiency by relaxing the general two-dimensional warping criterion. During the alignment of columns a lookahead inspired by Tree-Serial Dynamic Programming (TSDP) is used. The developed algorithms are evaluated and compared to state-of-the-art warping methods on the task of face recognition. For this purpose the AR-Face and the CMU-PIE database are considered. To raise the level of difficulty, the former is altered by introducing artificial rotations. Experiments on both databases show the competitiveness of the novel algorithms and the effectiveness of the lookahead.

In this work, we present novel warping algorithms for full 2D pixel-grid deformations for face recognition. Due to high variation in face appearance, face recognition is considered a very difficult task, especially if only a single reference image, for example a mug-shot, per face is available. Usually model-based approaches with additional training data are used to cope with several types of variation occurring in facial imaging. Image warping contrarily yields a distance measure which is invariant with regard to several types of variation. This allows for precise recognition even using only very few reference observations. Due to the computationally complex problem of optimal 2D warping, pseudo-2D warping-based approaches in the past represented strong approximations of the original problem, and were mainly successful on data with low variability or rectified images. We propose a novel 2D warping method which is globally optimal and makes no prior assumtions on the data variability besides two-dimensional smootheness constraints which both avoid local mirroring and gaps and significantly speed up the optimization. Furthermore, we show that occlusion handling is imperative to obtain smooth warpings in a variety of domains. We evaluate our novel algorithm on various well known databases, such as the AR-Face and CMU-PIE database, and provide a detailed comparison to existing warping approaches. We show that by using simple relative 2D constraints, strong local features and a kernel, which is robust w.r.t. occlusions, our computationally complex approaches outperform state-of-the-art results for recognizing faces under varying expressions, occlusions and poses. Most interestingly, we achieve higher accuracy using fewer training instances per class compared to methods learning a model of the 3D shape.

In this work, we propose a novel extension of pseudo 2D image warping (P2DW) which allows for joint alignment and recognition of non-rectified face images. P2DW allows for optimal displacement inference in a simplified setting, but cannot cope with stronger deformations since it is restricted to column-to-column mapping. We propose to implement additional flexibility in P2DW by allowing deviations from column centers while preserving vertical structural dependencies between neighboring pixel coordinates. In order to speed up the recognition we employ hard spacial constraints on candidate alignment positions. Experiments on two well-known face datasets show that our algorithm significantly improves the recognition quality under difficult variability such as 3D rotation (poses), expressions and illuminations, and can reliably classify even automatically detected faces. We also show an improvement over state-of-the-art results while keeping computational complexity low.

The SignSpeak project will be the first step to approach sign language recognition and translation at a scientific level already reached in similar research fields such as automatic speech recognition or statistical machine translation of spoken languages. Deaf communities revolve around sign languages as they are their natural means of communication. Although deaf, hard of hearing and hearing signers can communicate without problems amongst themselves, there is a serious challenge for the deaf community in trying to integrate into educational, social and work environments. The overall goal of SignSpeak is to develop a new vision-based technology for recognizing and translating continuous sign language to text. New knowledge about the nature of sign language structure from the perspective of machine recognition of continuous sign language will allow a subsequent breakthrough in the development of a new vision-based technology for continuous sign language recognition and translation. Existing and new publicly available corpora will be used to evaluate the research progress throughout the whole project.

We analyze the usage of matching algorithms for image recognition. We focus on the approaches which aim at finding nonlinear deformations of an entire image. ZeroOrder Warping (ZOW), Pseudo 2D Hidden Markov Model (P2DHMM) and Tree-Serial Dynamic Programming (TSDP) are studied. The effects of different constraints and parameter settings are analyzed. Furthermore, a new version of the TSDP and extensions for the P2DHMM are proposed. The proposed approaches allow to compensate for large disparities and additionally intend to preserve the monotonicity and continuity of the warping. The problem of local and global image variability occurs in many image recognition tasks and is a typical issue in the domain of face recognition. Many local deformations caused by changes in facial expression and pose make conventional distance functions fail. Additionally, face registration errors worsen the performance of most holistic methods. The P2DHMM is limited to a column-to-column mapping and is sensitive to registration errors, while the previously known version of the TSDP restricts the absolute displacement of each pixel. We propose to extend the P2DHMM by allowing deviations from a column which preserve the first-order dependencies between the pixels. Furthermore, we propose to relax some of the constraints imposed on the warping to cope with registration errors. A new version of the TSDP algorithm is proposed which relaxes the absolute constraints and intends to preserve the monotonicity and continuity of the warping. The proposed extensions are compared to the already known methods. Experimental results on the AR Face and the Labeled Faces in the Wild dataset show that the proposed approaches can outperform state-of-the-art methods.

Typical training criteria for string recognition like for example minimum phone error (MPE) and maximum mutual information (MMI) in speech recognition are based on a (regularized) loss function. In contrast, large-margin classifiers-the de-facto standard in machine learning-maximize the separation margin. An additional loss term penalizes misclassified samples. This paper shows how typical training criteria like for example MPE or MMI can be extended to incorporate the margin concept, and that such modified training criteria are smooth approximations to support vector machines with the respective loss function. The proposed approach takes advantage of the generalization bounds of large-margin classifiers while keeping the efficient framework for conventional discriminative training. This allows us to directly evaluate the utility of the margin term for string recognition. Experimental results are presented using the proposed modified training criteria for different tasks from speech recognition (including large-vocabulary continuous speech recognition tasks trained on up to 1500-h audio data), part-of-speech tagging, and handwriting recognition.

In this work, the recent additions to the RWTH-Phoenix corpus, a data collection of interpreted news announcement, are analysed. The corpus features videos, gloss annotation of German Sign Language and transcriptions of spoken German. The annotation procedure is reported, and the corpus statistics are discussed. We present automatic machine translation results for both directions, and discuss syntactically motivated enhancements.

We propose to use energy minimization in MRFs for matching-based image recognition tasks. To this end, the Tree-Reweighted Message Passing algorithm is modified by geometric constraints and efficiently used by exploiting the guaranteed monotonicity of the lower bound within a nearest-neighbor based classification framework. The constraints allow for a speedup linear to the dimensionality of the reference image, and the lower bound allows to optimally prune the nearest-neighbor search without loosing accuracy, effectively allowing to increase the number of optimization iterations without an effect on runtime. We evaluate our approach on well-known OCR and face recognition tasks and on the latter outperform current state-of-the-art.

Communicative actions are specific movements or gestures which are accomplished by vocal tract articulators (lips, tongue, velum etc.) for speech, by facial articulators (eye brows, eye lids etc.) for co-verbal facial expression and by other bodily articulators (hands, arms etc.) for co-verbal gesturing. While action-based approaches already exist for spoken language processing it is the aim of this paper to adopt action theory for signed language processing (i.e. production and perception of sign language). Method: An action-based method for the phonetic annotation of sign language has been developed and a 100 sentence American Sign Language corpus has been analyzed using this method. Results: Five basic types of sign actions were identified, all indicating the importance of movement phases even if the goal of a gesture is to reach a specific target (e.g. specific hand shape, orientation, location, and/or direction). Conclusion: This study is a starting point for investigating sign language production quantitatively in terms of a unified action theory.

The SignSpeak project will be the first step to approach sign language recognition and translation at a scientific level already reached in similar research fields such as automatic speech recognition or statistical machine translation of spoken languages. Deaf communities revolve around sign languages as they are their natural means of communication. Although deaf, hard of hearing and hearing signers can communicate without problems amongst themselves, there is a serious challenge for the deaf community in trying to integrate into educational, social and work environments. The overall goal of SignSpeak is to develop a new vision-based technology for recognizing and translating continuous sign language to text. New knowledge about the nature of sign language structure from the perspective of machine recognition of continuous sign language will allow a subsequent breakthrough in the development of a new vision-based technology for continuous sign language recognition and translation. Existing and new publicly available corpora will be used to evaluate the research progress throughout the whole project.

We analyze the usage of Speeded Up Robust Features (SURF) as local descriptors for face recognition. The effect of different feature extraction and viewpoint consistency constrained matching approaches are analyzed. Furthermore, a RANSAC based outlier removal for system combination is proposed. The proposed approach allows to match faces under partial occlusions, and even if they are not perfectly aligned or illuminated. Current approaches are sensitive to registration errors and usually rely on a very good initial alignment and illumination of the faces to be recognized. A grid-based and dense extraction of local features in combination with a block-based matching accounting for different viewpoint constraints is proposed, as interest-point based feature extraction approaches for face recognition often fail. The proposed SURF descriptors are compared to SIFT descriptors. Experimental results on the AR-Face and CMU-PIE database using manually aligned faces, unaligned faces, and partially occluded faces show that the proposed approach is robust and can outperform current generic approaches.

We present a writer adaptive training and writer clustering approach for an HMM based Arabic handwriting recognition system to handle different handwriting styles and their variations. Additionally, a writing variant model refinement for specific writing variants is proposed. Current approaches try to compensate the impact of different writing styles during preprocessing and normalization steps. Writer adaptive training with a CMLLR based feature adaptation is used to train writer dependent models. An unsupervised writer clustering with Bayesian information criterion based stopping condition for a CMLLR based feature adaptation during a two-pass decoding process is used to cluster different handwriting styles of unknown test writers. The proposed methods are evaluated on the IFN/ENIT Arabic handwriting database.

For the recognition of continuous sign language we analyse whether we can improve the results by explicitly incorporating depth information. Accurate hand tracking for sign language recognition is made difficult by abrupt and fast changes in hand position and configuration, overlapping hands, or a hand signing in front of the face. In our system depth information is extracted using a stereo-vision method that considers the time axis by using pre- and succeeding frames. We demonstrate that depth information helps to disambiguate overlapping hands and thus to improve the tracking of the hands. However, the improved tracking has little influence on the final recognition results.