In this paper, we propose a dual-encoder ASR architecture for joint modeling of close-talk (CT) and far-talk (FT) speech, in order to combine the advantages of CT and FT devices for better accuracy. The key idea is to add an encoder selection network to choose the optimal input source (CT or FT) and the corresponding encoder. We use a single-channel encoder for CT speech and a multi-channel encoder with Spatial Filtering neural beamforming for FT speech, which are jointly trained with the encoder selection. We validate our approach on both attention-based and RNN Transducer end-to-end ASR systems. The experiments are done with conversational speech from a medical use case, which is recorded simultaneously with a CT device and a microphone array. Our results show that the proposed dual-encoder architecture obtains up to 9% relative WER reduction when using both CT and FT input, compared to the best single-encoder system trained and tested in matched condition.

In this paper, we apply Semi-Supervised Learning (SSL) along with Data Augmentation (DA) for improving the accuracy of End-to-End ASR. We focus on the consistency regularization principle, which has been successfully applied to image classification tasks, and present sequence-to-sequence (seq2seq) versions of the FixMatch and Noisy Student algorithms. Specifically, we generate the pseudo labels for the unlabeled data on-the-fly with a seq2seq model after perturbing the input features with DA. We also propose soft label variants of both algorithms to cope with pseudo label errors, showing further performance improvements. We conduct SSL experiments on a conversational speech data set with 1.9kh manually transcribed training data, using only 25% of the original labels (475h labeled data). In the result, the Noisy Student algorithm with soft labels and consistency regularization achieves 10.4% word error rate (WER) reduction when adding 475h of unlabeled data, corresponding to a recovery rate of 92%. Furthermore, when iteratively adding 950h more unlabeled data, our best SSL performance is within 5% WER increase compared to using the full labeled training set (recovery rate: 78%).

Deep Neural Network (DNN) acoustic models are sensitive to the mismatch between training and testing environments. When a trained model is tested on unseen speakers, domain, or environment, recognition accuracy can degrade substantially. In such a case, offline adaptation with a fair amount of field data can improve recognition accuracy significantly, and is commonly applied to ASR systems in practice. Ideally, such kind of adaptation should be done online as well in order to catch any unexpected dynamic changes in the environments during the inference process. However, online adaptation is subject to strict constraints on computational cost. On the other hand, the small amount of available data and the nature of unsupervised adaptation make online adaptation a very challenging task, especially for DNN acoustic models which normally contain millions of parameters. In this paper, we introduce a simple and effective online adaptation technique to compensate training and testing mismatch for DNN acoustic models. It is done via online adaptation of the parameters associated with the batch normalization applied to the model training process. Our results show that this technique can improve accuracy significantly in a domain mismatched scenario for different DNN architectures.

In this paper, we propose a word confidence measure based on phone durations depending on large contexts. The measure is based on the expected duration of each recognized phone in a word. In the approach here proposed the duration of each phone is in principle context-dependent, and the measure is a function of the distance between the observed and expected phone duration distributions within a word. Our experiments show that, since the “duration confidence” does not make use of any acoustic information, its Equal Error Rate (EER) in terms of False Accept and False Rejection rates is not as good as the one obtained by using the more informed acoustic confidence measure. However, combining the two measures by a simple linear interpolation, the system EER improves by 6% to 10% relative on an isolated word recognition task in several languages.

Neural networks are systems based on simple computational elements (units) massively connected by weighted links. The unit performs a measure of similarity between its input vector X=(Xl,…,Xn) and its weight vector Wi=(Wi1,…. Win). In literature there exist two main ways to compute this similarity: the former is the inner product (IP) of the vector x and the vector Wi(related to the projection of x on Wi) while the latter is based on some distance measure (DM) between the x and Wi vectors. This work continues the activity reported in [Gemello,91] and [Gemello,92] devoted to introduce hypersphere units. a kind of units based on DM similarity. and to describe their capability to improve the rejection of extraneous patterns. The focus of this paper is to extend the previous DM unit in order to create an hyperelliptical shaped decision region and to perform feature selection. The architecture we propose. the Closed Decision Region MLP net (CDRMLP). integrates both IP and DM unit. that in the following we call hyperellipse unit (HE). into the same architecture. We show how a gradient descent approach can be successfully used to train this network. The kind of unit we propose is not new: there already exist in the literature some connessionist approaches based on distance metric such as Learning Vector Quantization [Kohonen.92]. Restriced Coulomb Energy [Cooper.87]. Probabilistic Neural Networks [Specht.92]. Radial Basic Functions [Musavi.92]. The differences rely on the learning strategy (gradient descent). the kind of computation. and the feature selection capability.

This paper describes the conceptual and algorithmic evolutions of Memory Based Parametric Equalization (MPEQ) needed to exploit the potentialities of the method within the state-of-the-art Loquendo ASR. MPEQ is the memory-based evolution of Parametric Non-Linear Equalization (PEQ) introduced to overcome the problem of unreliable statistics estimation in presence of very limited acoustic information in the test utterance to be normalized. The main limitations of the method that prevented its practical application were the lack of online implementation, the unrealistic unimodal assumption about the training statistics, the unconditioned application of equalization, and the need for retraining the acoustic models.The paper describes how these limitations have been overcome and reports a large experimentation on many corpora that shows improvements in a variety of mismatched conditions, while preserving performances in matched conditions.

This paper investigates the use of a Neural Network classifier for topic identification from conversational telephone speech, which exploits rich recognition results coming from an automatic speech recognizer. The baseline features used to feed the neural classifier are produced using the words extracted from the 1-best sequence. Rich recognition results include the word union of the first n-best sequences, the consensus hypothesis and the full or pruned Word Confusion Network generated from the n-best sequences. Different probabilistic information attached to the words, including confidence and word posterior probabilities, is investigated together with classical and probabilistic feature weighting schemes. A large experimentation on conversational telephone speech of Fisher corpus is reported, showing significant improvements when compared to the state of the art.

In this paper we describe the implementation of a complete ANN training procedure for speech recognition using the block mode back-propagation learning algorithm. We exploit the high performance SIMD architecture of GPU using CUDA and its C-like language interface. We also compare the speed-up obtained implementing the training procedure only taking advantage of the multi-thread capabilities of multi-core processors. Our approach has been tested by training acoustic models for large vocabulary speech recognition tasks, showing a 6 times reduction of the time required to train real-world large size networks with respect to an already optimized implementation using the Intel MKL libraries.

This paper analyzes the benefits and drawbacks of PEQ (Parametric Non-linear Equalization), a features normalization technique based on the parametric equalization of the MFCC parameters to match a reference probability distribution. Two limitations have been outlined: the distortion intrinsic to the normalization process and the lack of accuracy in estimating normalization statistics on short sentences. Two evolutions of PEQ are presented as solutions to the limitations encountered. The effects of the proposed evolutions are evaluated on three speech corpora, namely WSJ0, AURORA-3 and HIWIRE cockpit databases, with different mismatch conditions given by convolutional and/or additive noise and non-native speakers. The obtained results show that the encountered limitations can be overcome by the newly introduced techniques.

In this paper, we propose a word confidence measure based on phone durations depending on large contexts. The measure is based on the expected duration of each recognized phone in a word. In the approach here proposed the duration of each phone is in principle context-dependent, and the measure is a function of the distance between the observed and expected phone duration distributions within a word. Our experiments show that, since the "duration confidence" does not make use of any acoustic information, its Equal Error Rate (EER) in terms of False Accept and False Rejection rates is not as good as the one obtained by using the more informed acoustic confidence measure. However, combining the two measures by a simple linear interpolation, the system EER improves by 6% to 10% relative on an isolated word recognition task in several languages.

This paper presents a front-end consisting of an Artificial Neural Network (ANN) architecture trained with multilingual corpora. The idea is to train an ANN front-end able to integrate the acoustic variations included in databases collected for different languages, through different channels, or even for specific tasks. This ANN front-end produces discriminant features that can be used as observation vectors for language or task dependent recognizers. The approach has been evaluated on three difficult tasks: recognition of non-native speaker sentences, training of a new language with a limited amount of speech data, and training of a model for car environment using a clean microphone corpus of the target language and data collected in car environment in another language.

Training Artificial Neural Networks (ANNs) with large amounts of speech data is a time intensive task due to the intrinsically sequential nature of the back-propagation algorithm.This paper presents an approach for training ANNs using sentence and frame selection. The goal is to speed-up the training process, and to balance the phonetic coverage of the selected frames, trying to mitigate the classification problems related to the prior probabilities of the individual phonetic classes.These techniques, together with a three-step training approach and software optimizations, reduced by an order of magnitude the training time of our models.

This paper presents the results of a large number of experiments performed on the Hiwire cockpit database with a hybrid HMM-ANN speech recognition model(1). The Hiwire database is a noisy and non-native English speech corpus for cockpit communication. The noisy component of the database has been used to test two noise reduction methods recently introduced, while the adaptation component is exploited to perform supervised and unsupervised adaptation of the HMM-ANN model with an innovative technology, both in multi-speaker and speaker dependent way. Baseline results are presented, and the improvements obtained with noise reduction and adaptations are reported, showing an error reduction of about 60%.

This paper focuses on the adaptation of Automatic Speech Recognition systems using Hybrid models combining Artificial Neural Networks (ANN) with Hidden Markov Models (HMM). Most adaptation techniques for ANNs reported in the literature consist in adding a linear transformation network connected to the input of the ANN. This paper describes the application of linear transformations not only to the input features, but also to the outputs of the internal layers. The motivation is that the outputs of an internal layer represent discriminative features of the input pattern suitable for the classification performed at the output of the ANN. In order to reduce the effect due to the lack of adaptation samples for some phonetic units we propose a new solution, called Conservative Training. Supervised adaptation experiments with different corpora and for different types of adaptation are described. The results show that the proposed approach always outperforms the use of transformations in the feature space and yields even better results when combined with linear input transformations.

A method is described for predicting acoustic feature variability by analyzing the consensus and relative entropy of phoneme posterior probability distributions obtained with different acoustic models having the same type of observations. Variability prediction is used for diagnosis of automatic speech recognition (ASR) systems. When errors are likely to occur, different feature sets are considered for correcting recognition results. Experimental results are provided on the CH1 Italian portion of AURORA3