The presence of a corresponding talking face has been shown to significantly improve speech intelligibility in noisy conditions and for hearing impaired population. In this paper, we present a system that can generate landmark points of a talking face from an acoustic speech in real time. The system uses a long short-term memory (LSTM) network and is trained on frontal videos of 27 different speakers with automatically extracted face landmarks. After training, it can produce talking face landmarks from the acoustic speech of unseen speakers and utterances. The training phase contains three key steps. We first transform landmarks of the first video frame to pin the two eye points into two predefined locations and apply the same transformation on all of the following video frames. We then remove the identity information by transforming the landmarks into a mean face shape across the entire training dataset. Finally, we train an LSTM network that takes the first- and second-order temporal differences of the log-mel spectrogram as input to predict face landmarks in each frame. We evaluate our system using the mean-squared error (MSE) loss of landmarks of lips between predicted and ground-truth landmarks as well as their first- and second-order temporal differences. We further evaluate our system by conducting subjective tests, where the subjects try to distinguish the real and fake videos of talking face landmarks. Both tests show promising results.
Object-based audio has the potential to enable multimedia content to be tailored to individual listeners and their reproduction equipment. In general, object-based production assumes that the objects—the assets comprising the scene—are free of noise and interference. However, there are many applications in which signal separation could be useful to an object-based audio workflow, e.g., extracting individual objects from channel-based recordings or legacy content, or recording a sound scene with a single microphone array. This paper describes the application and evaluation of blind source separation (BSS) for sound recording in a hybrid channel-based and object-based workflow, in which BSS-estimated objects are mixed with the original stereo recording. A subjective experiment was conducted using simultaneously spoken speech recorded with omnidirectional microphones in a reverberant room. Listeners mixed a BSS-extracted speech object into the scene to make the quieter talker clearer, while retaining acceptable audio quality, compared to the raw stereo recording. Objective evaluations show that the relative short-term objective intelligibility and speech quality scores increase using BSS. Further objective evaluations are used to discuss the influence of the BSS method on the remixing scenario; the scenario shown by human listeners to be useful in object-based audio is the worst-case scenario among those tested.
We assume a second-order source separation model where the observed multivariate time series is a linear mixture of latent, temporally uncorrelated time series with some components pure white noise. To avoid the modelling of noise, we extract the non-noise latent components using some standard method, allowing the modelling of the extracted univariate time series individually. An important question is the determination of which of the latent components are of interest in modelling and which can be considered as noise. Bootstrap-based methods have recently been used in determining the latent dimension in various methods of unsupervised and supervised dimension reduction and we propose a set of similar estimation strategies for second-order stationary time series. Simulation studies and a sound wave example are used to show the method's effectiveness.
In recent years, there has been a renaissance of research on the role of the spectral phase in single-channel speech enhancement. One of the recent proposals is to not only estimate the clean speech phase but also use this phase estimate as an additional source of information to facilitate the estimation of the clean speech magnitude. To assess the potential benefit of such approaches, in this paper we systematically explore in which situations additional information about the clean speech phase is most valuable. For this, we compare the performance of phase-aware and phase-blind clean speech estimators in different noise scenarios, i.e. at different signal to noise ratios (SNRs) and for noise sources with different degrees of stationarity. Interestingly, the results indicate that the greatest benefits can be achieved in situations where conventional magnitude-only speech enhancement is most challenging, namely in highly non-stationary noises at low SNRs.
We develop an extension to Poisson factorization, to model Multinomial data using a moment parametrization. Our construction is an alternative to the canonical construction of generalized linear models. This is achieved by defining K component Poisson Factorization models and constraining the sum of observation tensors across components. A family of fully conjugate tensor decomposition models for binary, ordinal or multinomial data is devised as a result, which can be used as a generic building block in hierarchical models for arrays of such data. We give parameter estimation and approximate inference procedures based on Expectation Maximization and variational inference. The flexibility of the resulting model on binary and ordinal matrix factorizations is illustrated. Empirical evaluation is performed for movie recommendation on ordinal ratings matrix, and for knowledge graph completion on binary tensors. The model is tested for both prediction and producing ranked lists.
Far-field speech capture systems rely on microphone arrays to spatially filter sound, attenuating unwanted interference and noise and enhancing a speech signal of interest. To design effective spatial filters, we must first estimate the acoustic transfer functions between the source and the microphones. It is difficult to estimate these transfer functions if the source signals are unknown. However, in systems that are activated by a particular speech phrase, we can use that phrase as a pilot signal to estimate the relative transfer functions. Here, we propose a method to estimate relative transfer functions from known speech phrases in the presence of background noise and interference using template matching and time-frequency masking. We find that the proposed method can outperform conventional estimation techniques, but its performance depends on the characteristics of the speech phrase.
We propose an extension of the canonical polyadic (CP) tensor model where one of the latent factors is allowed to vary through data slices in a constrained way. The components of the latent factors, which we want to retrieve from data, can vary from one slice to another up to a diffeomorphism. We suppose that the diffeomorphisms are also unknown, thus merging curve registration and tensor decomposition in one model, which we call registered CP. We present an algorithm to retrieve both the latent factors and the diffeomorphism, which is assumed to be in a parametrized form. At the end of the paper, we show simulation results comparing registered CP with other models from the literature.
The Deep Brain Stimulation (DBS) is a surgical procedure efficient to relieve symptoms of some neurodegenerative disease like the Parkinson’s disease (PD). However, apply permanently the deep brain stimulation due to the lack of possible control lead to several side effects. Recent studies shown the detection of High-Voltage Spindles (HVS) in local field potentials is an interesting way to predict the arrival of symptoms in PD people. The complexity of signals and the short time lag between the apparition of HVS and the arrival of symptoms make it necessary to have a fast and robust model to classify the presence of HVS ( Y=1 ) or not ( Y=-1 ) and to apply the DBS only when needed. In this paper, we focus on a Gaussian process model. It consists to estimate the latent variable f of the probit model: Pr(Y=1|input) = (f(input)) with the distribution function of the standard normal distribution.
Nonnegative matrix factorization is a well-known unsupervised learning method for part-based feature extraction and dimensionality reduction of a nonnegative matrix with a variety of applications. One of them is a matrix completion problem in which missing entries in an observed matrix is recovered on the basis of partially known entries. In this study, we present a geometric approach to the low-rank image completion problem with separable nonnegative matrix factorization of an incomplete data. The proposed method recursively selects extreme rays of a simplicial cone spanned by an observed image and updates the latent factors with the hierarchical alternating least-squares algorithm. The numerical experiments performed on several images with missing entries demonstrate that the proposed method outperforms other algorithms in terms of computational time and accuracy.
This paper introduces a framework for robust speaker localization in reverberant environments based on a causal analysis of the temporal relationship between direct sound and corresponding reflections. It extends previously proposed localization approaches for spherical microphone arrays based on a direct-path dominance test. So far, these methods are applied in the time-frequency domain without considering the temporal context of direction-of-arrival measurements. In this work, a causal analysis of the temporal structure of subsequent directions-of-arrival estimates based on the Granger causality test is proposed. The cause-effect relationship between estimated directions is modeled via a causal graph, which is used to distinguish the direction of the direct sound from corresponding reflections. An experimental evaluation in simulated acoustic environments shows that the proposed approach yields an improvement in localization performance especially in highly reverberant conditions.
In this study we address models with latent variable in the context of neural networks. We analyze a neural network architecture, mixture of deep experts (MoDE), that models latent variables using the mixture of expert paradigm. Learning the parameters of latent variable models is usually done by the expectation-maximization (EM) algorithm. However, it is well known that back-propagation gradient-based algorithms are the preferred strategy for training neural networks. We show that in the case of neural networks with latent variables, the back-propagation algorithm is actually a recursive variant of the EM that is more suitable for training neural networks. To demonstrate the viability of the proposed MoDE network it is applied to the task of speech presence probability estimation, widely applicable to many speech processing problem, e.g. speaker diarization and separation, speech enhancement and noise reduction. Experimental results show the benefits of the proposed architecture over standard fully-connected networks with the same number of parameters.
Structured low-rank approximation is used in model reduction, system identification, and signal processing to find low-complexity models from data. The rank constraint imposes the condition that the approximation has bounded complexity and the optimization criterion aims to find the best match between the data—a trajectory of the system—and the approximation. In some applications, however, the data is sub-sampled from a trajectory, which poses the problem of sparse approximation using the low-rank prior. This paper considers a modified Hankel structured low-rank approximation problem where the observed data is a linear transformation of a system’s trajectory with reduced dimension. We reformulate this problem as a Hankel structured low-rank approximation with missing data and propose a solution methods based on the variable projections principle. We compare the Hankel structured low-rank approximation approach with the classical sparsity inducing method of ℓ _1 -norm regularization. The ℓ _1 -norm regularization method is effective for sum-of-exponentials modeling with a large number of samples, however, it is not suitable for damped system identification.
An approach is proposed for underdetermined blind separation of nonnegative dependent (overlapped) sources from their nonlinear mixtures. The method performs empirical kernel maps based mappings of original data matrix onto reproducible kernel Hilbert spaces (RKHSs). Provided that sources comply with probabilistic model that is sparse in support and amplitude nonlinear underdetermined mixture model in the input space becomes overdetermined linear mixture model in RKHS comprised of original sources and their mostly second-order monomials. It is assumed that linear mixture models in different RKHSs share the same representation, i.e. the matrix of sources. Thus, we propose novel sparseness regularized joint nonnegative matrix factorization method to separate sources shared across different RKHSs. The method is validated comparatively on numerical problem related to extraction of eight overlapped sources from three nonlinear mixtures.
Knowing the signal-to-noise ratio of a noisy speech signal is important since it can help improve speech applications. This paper presents a two-stage approach for estimating the long-term signal-to-noise ratio (SNR) of speech signals that are corrupted by background noise. The first stage produces noise residuals from a speech separation module. The second stage then uses the residuals and a deep neural network (DNN) to predict long-term SNR. Traditional SNR estimation approaches use signal processing, unsupervised learning, or computational auditory scene analysis (CASA) techniques. We propose a deep-learning based approach, since DNNs have outperformed other techniques in several speech processing tasks. We evaluate our approach across a variety of noise types and input SNR levels, using the TIMIT speech corpus and NOISEX-92 noise database. The results show that our approach generalizes well in unseen noisy environments, and it outperforms several existing methods.
We propose a new algorithm for Independent Component Extraction that extracts one non-Gaussian component and is capable to exploit the non-Gaussianity of background signals without decomposing them into independent components. The algorithm is suitable for situations when the signal to be extracted is determined through initialization; it shows an extra stable convergence when the target component is dominant. In simulations, the proposed method is compared with Natural Gradient and One-unit FastICA, and it yields improved results in terms of the Signal-to-Interference ratio and the number of successful extractions.
Joint Diagonalization (JD) is a well-known method for blind signal separation (BSS) by exploiting the nonstationarity of signals. In this paper, we propose Synchronized Joint Diagonalization (SJD) that solves multiple JD problems simultaneously and tries to synchronize the activity of the same signal along the time axis over the multiple JD problems. SJD attains not only signal separation by the mechanism of JD but also permutation alignment by the synchronization when applied to frequency-domain BSS. Although the formulation of SJD starts from the minimization of multi-channel Itakura-Saito divergences between a covariance matrix and a diagonal matrix, the simplified cost function with the finest time blocks becomes similar to that of Independent Vector/Component Analysis (IVA/ICA). We discuss the relationship among SJD and existing techniques. Experimental results on speech separation are shown to demonstrate the behavior of these methods.
The state of the art in audio declipping has currently been achieved by SPADE (SParse Audio DEclipper) algorithm by Kitić et al. Until now, the synthesis/sparse variant, S-SPADE, has been considered significantly slower than its analysis/cosparse counterpart, A-SPADE. It turns out that the opposite is true: by exploiting a recent projection lemma, individual iterations of both algorithms can be made equally computationally expensive, while S-SPADE tends to require considerably fewer iterations to converge. In this paper, the two algorithms are compared across a range of parameters such as the window length, window overlap and redundancy of the transform. The experiments show that although S-SPADE typically converges faster, the average performance in terms of restoration quality is not superior to A-SPADE.
Most convolutional neural network architectures explored so far for musical audio separation follow an autoencoder structure, where the mixture is considered to be a corrupted version of the original source. On the other hand, many approaches based on deep neural networks make use of several networks with different objectives for estimating the sources. In this paper we propose a discriminative approach based on traditional convolutional neural network architectures for image classification and speech recognition. Our results show that this architecture performs similarly to current state of the art approaches for separating singing voice, and that the addition of convolutional layers allows improving separation results with respect to using only fully-connected layers.
Nonnegative matrix factorization (NMF) has been well-known as a powerful spectral model for audio signals. Existing work, including ours, has investigated the use of generic source spectral models (GSSM) based on NMF for single-channel audio source separation and shown its efficiency in different settings. This paper extends the work to multichannel case where the GSSM is combined with the source spatial covariance model within a unified Gaussian modeling framework. Especially, unlike a conventional combination where the estimated variances of each source are further constrained by NMF separately, we propose to constrain the total variances of all sources altogether and found a better separation performance. We present the expectation-maximization (EM) algorithm for the parameter estimation. We demonstrate the effectiveness of the proposed approach by using a benchmark dataset provided within the 2016 Signal Separation Evaluation Campaign.
Independent Component Analysis (ICA) can be used to model gene expression data as an action of a set of statistically independent hidden factors. The ICA analysis with a downstream component analysis was successfully applied to transcriptomic data previously in order to decompose bulk transcriptomic data into interpretable hidden factors. Some of these factors reflect the presence of an immune infiltrate in the tumor environment. However, no foremost studies focused on reproducibility of the ICA-based immune-related signal in the tumor transcriptome. In this work, we use ICA to detect immune signals in six independent transcriptomic datasets. We observe several strongly reproducible immune-related signals when ICA is applied in sufficiently high-dimensional space (close to one hundred). Interestingly, we can interpret these signals as cell-type specific signals reflecting a presence of T-cells, B-cells and myeloid cells, which are of high interest in the field of oncoimmunology. Further quantification of these signals in tumoral transcriptomes has a therapeutic potential.
